Why can a small model feel slow even when its throughput is high?

The Triton documentation explains that dynamic batching combines multiple requests, which generally helps improve throughput. It also allows a request to wait for others to join the batch. The official recommendation is to test batch sizes and waiting times incrementally within a latency budget, rather than pursuing the largest possible batch. This mechanism applies to inference services supported by Triton; it should not be assumed that every Jev service offers the same configuration. A starting point for discussion For interactive decisions, network round trips, queueing, inference, and result processing together determine the time the user experiences. The editors suggest recording end-to-end latency percentiles under idle conditions, steady concurrency, and bursts of traffic, alongside timeout rates and throughput of successful requests. Even when the average looks good, a small number of requests waiting a long time can still damage the experience. An offline batch-classification job and a voice interaction may not belong in the same queue, either. Let's discuss 1. Would you reserve a separate queue for real-time interactions? 2. How would you avoid measuring only the cache for repeated requests during load testing? 3. If throughput doubled but p95 latency worsened, how much degradation would you accept? Source: NVIDIA Triton: Official Dynamic Batcher documentation https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/batcher.html Editorial note: The official S1 Hub editorial account used AI assistance to compile public sources. This was not submitted by the original author, and this site has not conducted independent hands-on testing. Additional evidence and differing views are welcome.