The AI race can sometimes feel like a numbers game.
Earlier this month, Bob Sternfels, the CEO of McKinsey & Company, made a surprising announcement. The firm, he said, had a workforce of over 60,000 people — 40,000 human employees and 25,000 AI agents.
For Sternfels, it was an example of McKinsey’s all-in approach to AI. Others in the industry, however, say the eye-popping number actually says little about the company’s successful adoption of artificial intelligence.
Dan Priest, the chief AI officer at PwC, told Business Insider that evaluating a firm’s AI use by the number of agents it has is not the best metric.
“There was this emerging bragging right around the number of agents I had or I have in production,” he said. “I think that’s probably the wrong measure.”
The value of AI deployment is better measured by the quality — not the quantity — of agents, he said.
He said one way to do that is to look at the number of agents that are authorities on a given task, which will encourage humans to use them, Priest said. The other is to evaluate the number of humans using those agents to execute tasks to achieve a prioritized outcome for a company. An example could be a better customer experience by transforming a call center.
Over the past two years, agents have come to dominate how companies talk about AI adoption. Priest said that focus on agents is the right approach. “Agents are at a place now where they’re the best way to unlock value from AI,” he said.
Humans still drive the workforce, however, and a better way to measure an agent’s value is by how effectively people use them — not just by how much work the agents have the potential to automate.
At PwC, about 82% of its employees were actively using the firm’s AI tools. Priest said that AI agents are embedded across teams at PwC, and the firm tracks how agents interact, how accurately they complete tasks, whether they are making processes faster, higher quality, or higher performing. Humans have a role in reviewing agents’ output and providing feedback.
“The human is still accountable,” he said. “The humans are the ones who get certified. The humans are the ones who get licensed. The humans are the ones who get empowered.”
Priest said that PwC and its clients first took a bottom-up approach to AI adoption. Many business leaders, he said, tried to “crowdsource” approaches to adoption from employees because they themselves didn’t have the answers.
That led to a “fairly disappointing” return on investment, he said.
Priest said a shift to a “top-down” approach has been more effective, allowing them to focus on fewer agents with a deeper mastery of a smaller set of tasks.
“That agent, I’ve given them permission to access certain data sets,” he said. “I’ve given them permission to perform certain tasks. I’ve given them permission to produce certain outcomes. Those permissions are monitored, they expire, they’re managed.”
