Back to Study Material Deep Learning

Model inference

Inference is the process of running a trained neural network on new input data to produce a prediction. Unlike training, inference does not compute gradien...

Model inference

What Happens During Inference

Inference is the process of running a trained neural network on new input data to produce a prediction. Unlike training, inference does not compute gradients or update weights. The network simply performs a forward pass: input data flows through each layer in sequence, and the final layer produces an output such as a class probability, a generated token, or a bounding box coordinate.

Consider a sentiment classifier deployed on a product review website. When a user submits a review, the text is tokenized, converted to embeddings, passed through transformer layers, and the output layer produces a probability for each sentiment class. The entire process may complete in under 100 milliseconds on suitable hardware, making the experience feel instantaneous to the user.

Exam tip: Interviewers sometimes ask how inference differs from training in terms of computational requirements. The key points are that inference requires only a forward pass, does not need to store intermediate activations for backpropagation, and typically uses lower batch sizes, which changes the optimal hardware choice.

Latency, Throughput, and Batch Size

Two metrics dominate inference optimization discussions: latency and throughput. Latency measures how long a single request takes from input to output. Throughput measures how many requests the system can process per unit of time. These two goals are often in tension. Batching multiple requests together improves throughput because the GPU can process them in parallel, but it increases latency for any individual request that must wait for the batch to fill.

Real-time applications such as voice assistants or interactive chatbots prioritize low latency and typically process one or a few requests at a time. Batch processing workloads such as offline document classification prioritize throughput and use large batch sizes to maximize GPU utilization.

Choosing the right batch size for your deployment scenario is one of the first decisions in production inference planning. Profiling on representative traffic patterns before deploying to production avoids surprises in latency under load.

Inference Optimization Techniques

Several techniques specifically target inference speed without retraining the model from scratch. Quantization converts weight values from 32-bit or 16-bit floating point to 8-bit integers or even 4-bit integers, reducing memory and enabling faster integer arithmetic on supported hardware. Compiled inference engines such as TensorRT or ONNX Runtime apply graph-level optimizations including operator fusion, which merges multiple sequential operations into a single kernel call, reducing memory bandwidth overhead.

Speculative decoding is a newer technique used for large language models. A small draft model generates candidate tokens quickly, and the large model verifies many tokens in a single forward pass, achieving higher throughput than running the large model alone.

An interview question about inference optimization is an opportunity to demonstrate awareness of the full stack, from model-level changes like quantization to system-level changes like batching strategies and hardware-specific compilers. Showing familiarity with at least one production inference framework strengthens the answer.