Modeldle

What do AI model parameters mean?

What the B in 70B means, total vs active parameters in mixture of experts models, why bigger is not always better, and why many labs keep model sizes secret.

Last updated Oct 1, 2026

AI model names are full of numbers like 27B, 70B and 405B. The B stands for billion, and the number counts the model's parameters. It is the closest thing a model has to a measure of size, but it is easy to read too much into it.

What a parameter is

A language model is a giant mathematical function. Text goes in as tokens, passes through many layers of arithmetic, and comes out as a prediction of the next token. Parameters are the adjustable numbers inside that function, mostly weights that decide how strongly each part of one layer influences the next.

Before training, they are random. During training, the model reads an enormous amount of text and keeps trying to predict the next token. After each batch of guesses, every parameter is nudged slightly in the direction that would have made those guesses better. Repeat that over trillions of tokens and the numbers end up encoding grammar, facts, writing styles and patterns of reasoning. No single parameter holds a fact. Knowledge is spread across billions of them at once.

A rough analogy is a huge mixing desk. A bigger desk has more knobs, so it can capture finer detail, but only if the knobs are set well.

Size and memory

Parameters have a very practical cost: each one has to sit in memory while the model runs. At 16-bit precision, a common format, every parameter takes 2 bytes. That gives a quick rule of thumb:

ModelTotal parametersWeights at 16-bitWeights at 4-bit
Gemma 4 31B31 billionAbout 62 GBAbout 16 GB
Llama 3.3 70B70 billionAbout 140 GBAbout 35 GB
Llama 3.1 405B405 billionAbout 810 GBAbout 200 GB
DeepSeek V4 Pro1.6 trillionAbout 3.2 TBAbout 800 GB

Storing the numbers with fewer bits, called quantization, cuts memory sharply for a modest loss in quality. It is how people run mid-sized open models on a single graphics card. The model also needs working memory for the conversation itself, so treat these figures as a minimum.

Total vs active parameters

Many of today's largest models use a design called mixture of experts, or MoE. Instead of one giant block that every token passes through, parts of the model are split into many smaller "expert" networks. A small router picks a few experts to handle each token, and the rest sit idle for that token.

That creates two different numbers:

  • Total parameters are everything stored in the model. This decides how much memory you need.
  • Active parameters are the share actually used for each token. This drives how much computing each token takes, and so how fast and cheap the model can be to run.

Some model names spell out both. Qwen3.5 397B-A17B has 397 billion parameters in total, but only 17 billion are active for any one token, about 4 percent. Qwen3 235B-A22B has 235 billion in total and 22 billion active.

Mixtral 8x7B shows how a name can mislead. It has eight experts and uses two of them for each token. You might expect 8 × 7 = 56 billion parameters, but the catalog lists 46.7 billion, because the experts share parts of the network such as the attention layers. Only the expert blocks are repeated.

Mixture of experts is how labs build models with enormous totals while keeping them affordable to serve. Several catalog models pass a trillion: Kimi K3 at 2.8 trillion, DeepSeek V4 Pro at 1.6 trillion and Kimi K2.6 at 1.04 trillion. At that scale, a model that used every parameter for every token would be very slow and expensive to run.

Why bigger is not always better

If parameters were everything, the largest model would always win. The catalog says otherwise. Gemma 4 31B, released in April 2026, scores 1443 on the LMArena text leaderboard. Llama 3.1 405B, released in July 2024 with about 13 times as many parameters, scores 1284. On the arena's scale, that gap means people prefer the smaller model's answers about 7 times out of 10.

Several things matter as much as size, or more:

  • Training data. More, cleaner and better chosen data makes each parameter count for more.
  • The training recipe. Steady improvements in architecture and training methods keep making models more capable at a given size.
  • Post-training. Fine-tuning on good examples, learning from human and automated feedback, and training for step-by-step reasoning can change how useful a model feels.
  • Distillation. Smaller models are often trained to imitate larger ones, inheriting much of their skill in a compact form.

Size has costs, too. Bigger models are slower, need more hardware and cost more to run. For many everyday tasks, a well trained smaller model is the better pick.

Why many labs keep sizes secret

The catalog has no parameter count for any closed model: not Claude, not Gemini, not Grok, and not OpenAI's GPT-5 or GPT-6 lines. Anthropic, OpenAI, Google and xAI do not publish the sizes of their closed models. A few reasons are usually given:

  • Competition. Total and active size hint at how a model was built and how much it costs to serve, which rivals would love to know.
  • It is a weak signal. As the Gemma example shows, the number alone says little about quality, so publishing it can invite misleading comparisons.
  • Customers do not need it. With an API-only model, nobody outside the lab runs the weights, so nobody needs to plan memory around them.

Estimates for closed models circulate online, but they are guesses. Open weights models are different: anyone can download the files and count. That is why every parameter count in Modeldle belongs to an open weights model.

In the quiz

Model size questions ask "Which is the bigger model?" or "Which of these open models is the biggest?", always by total parameters. Since only open weights models publish their sizes, those are the models you will see. Watch the names closely: Qwen3.5 397B-A17B is 397 billion in total, not 17. Play today's quiz and see how your sense of scale holds up.