Latency

Technical terms
About 1 min read

The time it takes from the moment a user sends a request to the AI until the first response appears on the screen or the full result is completed.

Also known as
response timeinference speed

Detailed explanation

A key metric determining AI tool performance. Especially in Large Language Models (LLMs), it is divided into 'TTFT (Time To First Token)', the output time of the first character, and 'TPS (Tokens Per Second)', the generation speed per second. As of 2026, LPU-based hardware like Groq records speeds of over 500–800 TPS, and real-time voice AI aims for a latency of 300ms or less for natural conversation. It is determined by the network environment, the model's parameter size, and the level of server hardware optimization.

Why It Matters in AI Tool Selection

Latency directly impacts user experience (UX). For real-time consultation chatbots or voice assistants, a latency exceeding 1 second disrupts the flow of conversation, causing credibility to plummet. On the other hand, for bulk report summarization or code generation tools, overall processing speed (TPS) is more important. You should prioritize between 'response speed' and 'processing capacity' based on your business objectives.

Checklist for Selection

  • Real-time chat apps: Check if TTFT is under 500ms.
  • Voice AI services: Check if end-to-end (E2E) latency is between 300–600ms.
  • Bulk data processing: Choose tools with higher overall completion time (throughput) rather than individual response speed.
  • When offline environments are required: Consider on-device AI models that do not have internet connection delays.

Performance Examples

As of May 2026, GPT-4o shows an average TTFT of around 460ms, making it suitable for general chat. In contrast, Groq's Llama 3-based API, an ultra-fast specialized inference engine, exceeds 800 TPS, providing lag-free real-time feedback even in complex agent workflows.

Related terms

Inference PerformanceOn-device AI