Model deployment
Model deployment is the process of making a trained machine learning model available for use in a real production environment where it can generate predict...
Bringing Models from Experiment to Production
Model deployment is the process of making a trained machine learning model available for use in a real production environment where it can generate predictions for actual users or systems. A model that stays in a notebook creates no value; deployment is where research becomes impact — and where new challenges emerge.
Deployment patterns depend on latency and throughput requirements. Batch prediction processes large volumes of examples offline on a schedule — a bank scoring all customers overnight, for example. It is simpler to operate and allows the model to run on powerful hardware without latency constraints. Online prediction (real-time inference) serves predictions on demand, usually via a REST API, and must respond within milliseconds. Streaming prediction sits between these, processing data from event queues with near-real-time latency.
Packaging a model for deployment involves serializing the trained model (using formats like ONNX, pickle, or SavedModel), wrapping it in a service that handles requests, and containerizing the service (typically with Docker) so it runs consistently across environments. Cloud providers offer managed model serving platforms that handle scaling, monitoring, and versioning.
A critical concept is feature serving: the features used at inference time must match exactly those used during training. Discrepancies — different aggregation windows, different default values for missing inputs, different categorical encodings — cause silent prediction errors that are difficult to diagnose.
Model versioning and rollback capabilities are essential. When a new model version is deployed, A/B testing or shadow deployment (running both old and new models and comparing outputs) validates the new version before full rollout. A broken deployment must be reversible quickly.
A concrete example: a recommendation engine is exported as an ONNX model, containerized in a FastAPI service, deployed to Kubernetes with auto-scaling, and exposed behind a load balancer. Predictions are cached for 10 minutes to reduce latency and cost for popular requests.
Exam tip: be ready to describe the difference between batch and online inference, and explain why feature consistency between training and serving is a critical operational concern.