For the last year, we’ve seen companies blaming AI for layoffs, saying that AI would deliver greater productivity and efficiency gains than humans.

This thinking was echoed by leadership and management of AI labs. For example, some said that software engineering was largely solved and that AI would be able to manage many routine coding tasks on its own.

But, as we have seen, the promise of AI’s ability to manage complex tasks autonomously has not matched up with reality. (This doesn’t mean that AI won’t take on a large part of cognitively difficult work, just that it’s going to take a while, or may even be deemed undesirable over time.)

Instead what’s happened is that people started to recognize that when it comes to working with AI:

• More does not always equal better

• Sometimes slowing down is essential

• Thinking carefully about what AI is going to output is vital

• Simplifying complex tasks into components is a best practice

• Cultivating judgement about what good looks like is essential

it seems that some companies are starting to recognize that human + AI is optimal and are beginning to accelerate hiring, and even re-hire individuals they let go previously. According to the Wall Street Journal:

“Companies ranging from railroad giant CSX to Google parent Alphabet have recently told investors that they plan to hire to meet growth goals or to seize on emerging technologies.”

My belief is that the next big deal when it comes to organizations working with AI

is understanding how to up-skill employees to get the most out of AI. This will require:

• Improving understanding of AI’s mechanics (tokens, context, etc.)

• Focusing on the daily, micro actions people take with AI (such as how they write prompts, manage context, reduce retries) to improve efficiency and increase task success rates

We’re still in the AI’s early innings. The modest reversal in the prioritizing AI over humans trend is heartening. The next trend to look out for will be efforts to understand how best to ensure humans are working with AI at maximum capability.

—
Benefit from These AI Efficiency and Productivity Resources

• LLM Whisperer Method: 3-part system helping people understand how to optimize AI micro actions to get the most out of these systems

• Understanding and Managing the AI Agent Footprint: Practical guidance on what AI agents install, access and change on devices and workflows and how to manage the AI footprint

I started my journey in generative AI when ChatGPT was first launched. It was a time of great promise, but also scarcity.

AI models (I was using GPT 3.5 and 4.0 at the time), were very context limited and had narrow reasoning capabilities.

I was working on early versions of retrieval augmented generation (it wasn’t called that at the time), where I’d provide the model with content from a data store. I also manually set up tools to deliver content from the Web to the model via Python scripts.

This work was extremely frustrating. Not only would the model hallucinate a lot, but you could only put so much information into its context. I had to put content through customized summarization pipelines and carefully construct my prompts to make sure the LLM saw the right things.

This was the minimum required to allow these models to function. There was also a financial incentive because input and output tokens were, on a per-token basis, expensive to generate.

The Scarcity Mindset

These early experiences shape how I use generative AI today. Even though models have larger context windows and can reason much more effectively, I’m still stingy with it. I don’t view inference as an unlimited resource and am careful with how I use these models.

This scarcity mindset influenced how I looked at a recent Anthropic study, focusing on AI’s use for work tasks. Anthropic found that AI is helping people engage in tasks that deliver more economic value over time. This is fantastic, but I thought: how much is this actually costing us?

So I decided to measure it. The result of this work is the Secrets of the LLM Whisperer study, which looked at the habits and behaviors that contribute to inefficient (and more expensive) LLM use.

A key driving factor is that, although we have more AI inference available to us, it is still a scarce resource that is being heavily subsidized. AI labs are feverishly buying data centers to meet the voracious appetite for inference. And, a backlash is growing, with some states imposing moratoriums on data center construction.

The cost of inference is also moving up. Anthropic is putting its most powerful current model Fable, behind stringent usage caps and pay-as-you-go payment schemes. Many are angry about this, but labs are heavily incentivized to stop subsidizing AI inference so heavily and have users pay closer to the market rate for it.

The study results and these market forces inspired me to launch this newsletter. In it, I’ll explore questions like:

• How can we use AI more smartly?

• What are the strategies and tactics being used to conserve resources (such as water, land and energy) consumed by AI?

• How can we make the social and economic costs of AI lower so ROI is higher?

• How can we operate AI more securely (which has its own efficiency benefits)?
  

Thanks for joining me on this journey.