Skip to main content

Deploy & Host a Model

Turn a model version into a live, running inference endpoint. In Foundry, deploying a model is hosting it.

Steps​

  1. Open the model's detail page — From Model HUB, click into the model you want to host.
  2. Find a ready version — Under Model versions, locate the version with status ready.
  3. Click Deploy — This opens the Deploy Model screen, pre-filled with sensible defaults.
  4. Review Deployment Settings — Confirm the Model URL and Model Source (huggingface or s3), and optionally set a Served Model Name and the vLLM Version to run.
  5. Set Resources — Choose Replicas (1–10), Request CPU (1–64), Request GPU (1–2), and Request Memory (e.g. 24Gi).
  6. Click Deploy Model — Foundry provisions the deployment with the resources you specified.
  7. Track it — Go to AI Service → Model Deployments. Your new deployment appears there and its status moves to Healthy once it's up.

Important: When deploying a compressed model from S3, always specify the Served Model Name. After the deployment status becomes Healthy, allow approximately two minutes for the model to become available in the Playground.

Field reference​

FieldWhat it does
Model URLThe deployable model identifier: a Hugging Face repo ID, or an s3:// URI.
Model Sourcehuggingface downloads weights from the HF Hub at startup; s3 syncs from the given S3 URI before the server starts.
Served Model NameName vLLM reports for this model. Required when deploying a compressed model from S3.
vLLM VersionThe vLLM container image tag to run, e.g. v0.19.0, or latest.
ReplicasNumber of model server copies (min 1, max 10). More replicas = more concurrent capacity.
Request CPU / GPU / MemoryCompute reserved per replica (CPU: 1–64, GPU: 1–2, Memory e.g. 24Gi).