Tag: serving

All the articles with the tag "serving".

• Throughput per dollar, once the traffic is bursty

  26 May, 2026·23 min read

  One roofline number sets the speculative-decoding gate, the batch-size knee and the chunked-prefill floor. Why max-num-seqs is a latency control.