Inference Performance

Technical terms
About 1 min read

The speed and efficiency with which an AI model processes user inputs to generate outputs, primarily measured by Time to First Token (TTFT) and Tokens Per Second (TPS).

Also known as
tokens per secondthroughput

Detailed explanation

This refers to the real-time execution capability of a trained AI model processing data in a production environment. For Large Language Models (LLMs), the key metrics are the response latency experienced by users and the amount of data the system can process at once (throughput). Recently, with the introduction of 'inference-time scaling' technology that improves performance by increasing compute at the time of inference, it has become a decisive tool selection criterion that goes beyond simple speed to determine the quality of output relative to cost.

Why It Matters in Tool Selection

Inference performance directly affects user experience (UX) and operating expenses (OPEX). Slow responses increase user drop-off rates, and low-efficiency tools cause server costs to rise exponentially when scaling the service. This is a primary hardware and software metric to consider, especially when building real-time chats or API-integrated services.

What to Look For

  • TTFT (Time to First Token): The initial waiting time for a user to receive a response (recommended within 0.2 to 0.5 seconds)
  • TPS (Tokens Per Second): The speed at which the entire answer is generated (check if it is faster than user reading speed)
  • Cost efficiency: The adequacy of processing speed relative to the cost per million tokens
  • Concurrency handling: Whether a consistent response speed is maintained even during traffic spikes

Example

When building a chatbot for customer service, if response quality is similar, choosing a model with a shorter TTFT prevents users from feeling like they are 'waiting'. Conversely, for backend tasks like summarizing large volumes of documents, selecting a model or API provider with higher overall throughput rather than TTFT is more advantageous in terms of cost.

Related terms

LatencyQuantization