Moving from LLM literacy to model literacy

Sometime a few days ago I started getting really frustrated with the output of some of my most frequent workflows. Answers were less specific, the rigor was gone, and the dreaded glazing of late GPT-4o had returned.

I use the platform enough that I'm pretty used to silent user testing on new models. It was immediately obvious to me that the dry precision of GPT-5 was being pulled out from under me. I picked up my ball and went home. Or rather, I went to Claude Sonnet. It wasn't my preferred model (not quite as dry, but it deals with precision better than Gemini, for example), but I could rely on it over whatever was happening in the background. A few days later OpenAI officially launched GPT-5.1 and switching back and forth between that and GPT-5 in their legacy model picker made it clear exactly what had happened.

Model Whiplash

A lot of people in the genAI space reacted pretty derisively when we started getting news about people falling to pieces because the "personality" of their AI changed when OpenAI launched GPT-5. Many people found it jarring, and it is especially so for people who are still learning how to ask the right questions.

When I talk to people about building AI workflows, I always recommend they stay as simple as possible. Often the simplest approach is a couple of familiar templates and the web user interface. Sure, it might be slightly less efficient, involve slightly more data shepherding if you have a multi-step process, but for most people the creation and upkeep of more robust (even small) tools eats that marginal efficiency up pretty quickly.

Naturally, these tend to be built for a single model, and especially for people new to the space, this can build up muscles that respond to idiosyncrasies of their first model. They don't have the experience to differentiate between their abilities to interact with genAI as a whole vs a specific model.

But success in the long run depends on flexibility. Because a model change can break everything completely. You need to have a wide familiarity with models across generations and companies, because it's not an if, but a when you will need to find a fallback.

Task-Defined Needs

Concerns about psychological reliance on AI companionship aside (and I have a lot of those, despite being an AI optimist), the magnitude and duration of the frustration in the switch from GPT-4o to GPT-5 is an indicator of the problems with viewing genAI through an interaction lens instead of a task lens. Models are not forever, behaviors are not guaranteed—monolithic reliance is a guarantee of failure.

It's helpful to build a mental model of a high-level task categorization, and get a sense of what workflows fall where. Here's a few that I think about (not exhaustive by any means)

• Step by step execution: A --> B --> C
• Exploratory discussion: "I want to have an unstructured stream of consciousness, but need you to keep me focused as I do so"
• Data extraction: To-do's from meeting notes, review sentiment, turn-this-mess-into-a-table
• Thought-partner: I have ideas and I need you to push back

Portfolio Mindset

I built this understanding by a lot of trial and error (and often by running the same task through multiple models in parallel). It might seem like a waste of time , but it helps you make decisions about the value of tools to complete your tasks. This lets you build mental models task mapping like this:

• Step by step execution: Sonnet 4.5, GPT-5.0, GPT-5.1, Gemini 2.5
• Exploratory discussion: GPT-4o, Opus 4.1, Gemini 2.5
• Data extraction: Sonnet 4.5, GPT-5
• Thought-partner: Opus 4.1, Sonnet 4.5

Once you build familiarity, it becomes an incremental learning process (manageable day-to-day). You build a portfolio of tools to keep you on the front edge of what can be done with simple browser-based interaction.

Of course, this goes deeper. Part of what we do to improve our agentic web underpinning Feather is a process of continual evolution of model interactions. As different models interact with each other, behavior emerges and niches become available for a wide range of models, and like evolution, it's all about what works for that time and place, not what is "best". One of our biggest unlocks was returning to a less powerful model for certain elements in the chain because the behavior was easier to control and more reliable than trying to force a different model into behavior outside its norm.

Model Literacy for Operators

Model literacy is about moving from memorizing prompts, favorite models, or accepting the default. It's about understanding use cases, about being willing to open the dropdown and chose another model. It's about having opinions and processes and getting the experience to build mental heuristics.

In a work environment, you go to different people for different things. Christy and Bob might know the answer to your question. But Bob will overexplain and leave you confused, while Christy might shortcut under pressure. If Janet is gone, you might go to Jeff for the next best answer. We naturally learn these things about the human expertise around us, and go to different people for different needs.

If you want success in the new world being built around us, you need to treat your genAI expertise the same.