Inference

What does inference mean in AI?

In everyday English and in logic, an inference is a conclusion reached from evidence. In statistics, inference means drawing conclusions about a population from a sample. In AI, the word has a narrower, practical meaning: the model takes an input and infers an output, using what it learned during training. If a model answers, classifies, transcribes, or generates anything, it is doing inference.

How inference works

The analogy: training is a chef spending years learning to cook; inference is cooking one dish when an order comes in. The learning is already done; each order uses it.

The precise version for a language model: your prompt is split into tokens and fed through the model's fixed, already-trained weights. The model generates the reply one token at a time, each new token depending on everything before it, until it finishes or hits a length limit. Settings like temperature shape how it picks each token. No learning happens during inference; the weights do not change.

Training vs inference

• Training: happens rarely, uses huge amounts of data and compute, and changes the model's weights.
• Inference: happens every time the model is used, uses one prompt at a time, and leaves the weights unchanged.
• Training sets what a model can do. Inference is where you feel its cost and speed.

A real example of inference

A salon's booking assistant receives "Do you have anything Saturday morning for a cut and color?" The model reads the message, calls the calendar tool, and writes a reply. Each of those model calls is an inference request, billed by the tokens read and written. A busy assistant might run thousands of inference requests a day on a model that was trained once.

Why inference matters

For anyone using AI, inference is the running cost. API providers typically price usage per token, with separate rates for input and output, so long prompts and long answers cost more. Inference also drives latency: bigger models and longer outputs take longer. That is why picking the right-sized model for each job, and keeping prompts focused, matters for both budget and speed.

Related terms

• Training: the other half of a model's life.
• Token: the unit inference is measured and billed in.
• Latency: how long inference takes to respond.
• Quantization: a way to make inference cheaper and lighter on memory.

Common misconceptions about inference

• The model learns from each conversation. Standard inference does not update the model; any memory comes from the app storing context.
• Inference is cheap because training was the expensive part. Per request it is small, but at scale it becomes the main ongoing cost.
• Inference means reasoning. In AI it simply means running the model, whether or not the task involves reasoning.

Inference in practice

Inference is what you pay for when an AI tool answers the phone or drafts a reply. See how per-use costs add up for real businesses in our breakdown of AI phone answering costs, and read the latency entry for the speed side.