Configuring Dedicated Model Inference

SummaryDedicated Model Inference on the Together AI platform consists of three parts: the endpoint (a stable name you or your clients call), deployments (specific model + hardware combinations running replicas behind it), and configs (recipes for how a model runs). A capacity-aware traffic split ties these three entities together. This architecture enables various other features such as rollouts, A/B tests, shadow experiments, zero-downtime changes possible. Below we'll show how capacity-aware routing works against a live endpoint with measured traffic.How the resource model worksA config is a recipe that specifies the engine, the GPU type and count as well as the parallelism and optimization profile (throughput, latency, balanced). Configs are immutable and every config has an ID like cr_... . Your deployment always points at exactly the config that was tested.A deployment binds one model (at a specific revision) to one config, gives it an autoscaling policy, and runs replicas. Deployments are disposable on purpose and creating and destroying should be considered routine.An endpoint is a fixed identity: a qualified name (<project_slug>/<endpoint_name>) that your applications pass as the model parameter in the ordinary inference API. The endpoint has a traffic split which is used to determine request routing over its deployments.One easy way to understand this is by referring to the ID, every ID in the system tells you what it is by prefix, which makes logs and scripts self-documenting: proj_ (project), ml_ (model), cr_ (config revision), endpoint_, dep_ (deployment), rol_ (rollout).Every advanced operation in the platform is just "add a deployment, and assign it a traffic routing weight" An A/B test is deployments with cohort assignments. A shadow experiment is a deployment at weight zero receiving mirrored traffic. Stopping a deployment is bounding min and max replicas to 0/0.Weight based traffic splitThe traffic split is a list of {deployment_id, weight} entries where a weight is per ready replica**.** The router computes each deployment's effective capacity as weight × ready_replicas and routes proportionally to this capacity.Walk through the diagram above: both deployments have weight 1 but deployment A has 1 ready replica (capacity 1), deployment B has 3 (capacity 3), so traffic follows a 25%/75% split. Equal weights mean equal per-replica load, not equal traffic share.We designed it this way because it makes routing and scaling the same conversation:Autoscaling composes for free. When deployment A scales from 1 replica to 3, its capacity triples and it automatically absorbs proportionally more traffic. With naive percentages, a deployment pinned at "25%" would saturate when scaled down and be idle when scaled up.Per-replica load is what you control. Weight 1 vs weight 2 says "each replica of B should work twice as hard as each replica of A"Replicas that aren't ready don't count. A deployment mid-cold-start, or degraded to 0 ready replicas, contributes zero capacity, so traffic flows to what can actually serve.Weights are positive numbers with no sum constraint so 0.7/0.3 and 700/300 describe identical routing. If you want fixed traffic shares regardless of replica count you can setup A/B experiment cohorts which allow for integer percents that sum to 100%.Setting a split is one PATCH with a field mask: tg beta endpoints update $DEPLOYMENT_A --traffic-weight 1 tg beta endpoints update $DEPLOYMENT_B --traffic-weight 1 # take a deployment out of rotation without scaling it down tg beta endpoints update $DEPLOYMENT_B --traffic-weight 0 ⚠️ Important A brand-new deployment will not get any traffic until it appears in the endpoint's traffic split. If you create an endpoint and a deployment and have it `READY`, send a request, and get back a `routing_error` it is probably because the traffic split is not specified and the router hasn’t yet been told that this deployment should receive traffic. The CLI's `