How Many LLMs Do You Really Need? A Model Portfolio Strategy for the Enterprise

Fourteen models and no strategy

A CIO told me about the day her architecture team finished an inventory she had asked for months earlier. She expected to find two or three large language models in production. They found fourteen. Marketing used one through a content tool. Engineering used two coding assistants. Customer service ran a fourth through its contact-center vendor. Three business units had signed their own API keys. Nobody could say which model handled which data, or what any of it cost per transaction.

Like the other scenes in this series, that story is a composite of conversations with technology leaders. The question it raises is now one of the most practical in enterprise AI: how many models do we actually need?

What the data says

Multi-model is already the norm. In Andreessen Horowitz's 2025 survey of 100 enterprise CIOs, 37% said they used five or more models, up from 29% a year earlier, and the firm reported that differentiation by use case, not just avoiding lock-in, was driving the shift ^[1]. In its next CIO survey, 81% of companies used three or more model families, up from 68% less than a year earlier ^[2].

No single provider dominates. Menlo Ventures estimated that by late 2025, Anthropic accounted for 40% of enterprise LLM API usage, OpenAI 27% and Google 21% ^[3]. Two years earlier, OpenAI had held 50% ^[3]. Leadership has changed hands within a couple of product cycles.

The mix is also shifting toward smaller models. Gartner predicts that by 2027, organizations will use small, task-specific AI models at least three times more, by usage volume, than general-purpose LLMs ^[4].

Why one is too few, and fourteen is too many

The case against a single model

Tasks differ. Coding, long-document analysis, high-volume classification and customer conversation reward different strengths. a16z found enterprises increasingly pick models by use case ^[1].

Cost differs by orders of magnitude. Sending every request to the most capable model wastes money on easy tasks. In LMSYS's RouteLLM research, a trained router achieved 95% of GPT-4's performance on MT Bench while cutting cost by over 85% compared with sending everything to GPT-4 ^[5].

Leadership changes. The provider with the best model this year may not have it next year, as the shift in enterprise share shows ^[3]. A single-model architecture turns every leapfrog into a migration project.

Concentration is a resilience risk. One provider's outage, price change or model deprecation shouldn't stop your business.

The case against model sprawl

Every model is a data path. Each provider has its own retention, processing-location and training terms. Fourteen models means fourteen sets of terms to review and monitor.

Evaluation doesn't scale linearly. Each model needs its own test runs on your use cases, and every new version needs re-testing.

Costs become invisible. When usage is scattered across business-unit contracts and embedded vendor tools, nobody can see cost per transaction. In McKinsey's 2026 survey, about one in five respondents said AI operating costs, including tokens, were constraining their AI use ^[6].

Skills spread thin. Teams that know one or two model families deeply get better results than teams that dabble in many.

A tiered model portfolio

Think of models the way you think of compute instances: a few standard sizes, each matched to a class of work.

Most enterprises can cover these with three to five models in total, ideally from at least two providers so that no single vendor change stops the business. The small-and-fast tier is where Gartner expects usage to grow fastest ^[4].

The controlled-deployment tier is optional and should be justified by a specific data or regulatory requirement. Open-weight models accounted for about 11% of enterprise LLM usage in late 2025, down from 19% a year earlier, according to Menlo Ventures ^[3], which suggests most enterprises are not choosing self-hosting by default.

Implementation: making a portfolio work

1. Put a gateway in front of every model

Route all model calls from your applications through one internal gateway. It is the control point for authentication, logging, data-loss prevention, rate limits and cost attribution by team and use case. It also makes switching a model a configuration change rather than a code change. Cloud platforms that host models from several providers, such as Amazon Bedrock with Anthropic's Claude models ^[7], can simplify contracting, but a gateway you control keeps the abstraction yours.

2. Choose a routing strategy

The research case for smarter routing is strong. RouteLLM's learned routers matched 95% of GPT-4 quality on MT Bench at over 85% lower cost ^[5]. Stanford's FrugalGPT showed that a cascade of models could match the best individual model at up to 98% lower cost on the benchmarks it tested ^[8]. Results on your own traffic will differ; measure before you commit.

3. Build one evaluation harness for all models

Create a test set per use case from real, anonymized requests, with expected answers or scoring rubrics. Run every candidate model, and every new version, through the same harness. This is what lets you add, replace or retire a model on evidence instead of opinion.

4. Plan for model change as routine

Providers release new versions and retire old ones on their own schedules. Pin model versions in production, track each provider's deprecation notices, and keep a tested fallback for every critical workflow.

5. Make cost visible per transaction

Report cost per resolved ticket, per document processed or per answer served, by model and by team. Total spend tells you how much; cost per transaction tells you whether a different tier would do.

A phased roadmap

1. Inventory (first 30 days). Find every model in use: direct API keys, cloud accounts, and models embedded in SaaS tools. Record the data each one touches and what it costs.

2. Consolidate (months 1 to 3). Stand up the gateway, define the tiers and pick a primary and an alternate model for each. Move direct API usage behind the gateway.

3. Optimize (months 3 to 6). Build the evaluation harness, then test routing on your highest-volume workloads. Shift easy traffic down a tier where quality holds.

4. Govern (ongoing). Review the portfolio quarterly: add a model only when it beats an incumbent on your evaluations, and retire one when it no longer earns its place.

Pitfalls I see most often

Standardizing on one model for simplicity. It feels efficient until that model is deprecated, repriced or overtaken.

Letting every team choose. Sprawl grows quietly through business-unit API keys and tool purchases.

Choosing models on public benchmarks. Leaderboards measure generic tasks; your evaluation set measures your business.

Routing before measuring. A router without an evaluation harness just moves errors around faster.

Forgetting embeddings. Changing your embedding model usually means re-indexing your content, so choose it deliberately.

CIO checklist

• Inventory all models in use, including those inside SaaS tools
• Route every direct model call through a single gateway
• Define tiers and name a primary and an alternate model for each
• Use at least two providers across the portfolio
• Build an evaluation harness with test sets per use case
• Pin model versions and track deprecation schedules
• Report cost per transaction by model, tier and team
• Review the portfolio quarterly against your evaluations

A founder's perspective

We designed Yavi.ai to be model-agnostic, because we learned early that customers' knowledge outlives any single model. The durable assets are the unified knowledge layer, the workflows built on it and the evaluations that prove they work. Models are components that should be easy to change.

So my answer to "how many LLMs do you need?" is: as few as cover your tiers, from at least two providers, behind one gateway, chosen by your own evaluations. For most enterprises that is three to five. The exact number will change every year. The architecture that lets you change it should not.

Go back to that inventory of fourteen. The problem was never the number. It was that nobody had chosen it.