Streaming

Most LLM APIs return full responses only after the entire output is generated, which can slow down UX and delay response time. This is especially noticeable for long outputs like summaries, explanations, or answers in real-time applications.

LitAI enables streaming generation out of the box so tokens are returned as they're generated, improving responsiveness and allowing users to read as the model thinks.

Background

What is Streaming?

Streaming allows a web server to continuously transmit data from the server to the client. For example, this could be chat messages generated word by word from a model or video frames from a video. This differs from traditional web servers, which have a more discrete communication style, typically providing one response per request.

Applications

Streaming lets us access partial outputs token by token, reducing latency and enhancing the interactivity of the application. This is ideal for:

• Chatbots and assistants
• Developer tools
• Long-form content generation
• Voice-based or CLI apps

Streaming with LitAI

To stream the generated text, pass stream=True to the chat(...) method.

The following code will print the text output as soon as they are generated.

from litai import LLM

llm = LLM(model="openai/gpt-4", teamspace="owner-name/teamspace-name")

# Receive tokens as they're generated instead of waiting for the full response.
for chunk in llm.chat("Explain LLM streaming in simple terms", stream=True):
    print(chunk, end="", flush=True)

Output format

Each chunk is a string representing the next generated token or phrase. LitAI abstracts away provider-specific token formats so we can treat each chunk as plain text.

Full OpenAI response format

Many developer tools and IDEs like Cursor, Continue, and VS Code extensions expect streaming responses in the official OpenAI format. LitAI supports this format out of the box.

To enable it, set full_response=True when calling .chat(...). This returns a stream of OpenAI-compatible response objects, including fields like choices[0].delta.content.

from litai import LLM

llm = LLM(model="openai/gpt-4", teamspace="owner-name/teamspace-name")

result = llm.chat("Write an essay on AGI?", max_tokens=10, full_response=True, stream=True)

for chunk in result:
    print(chunk.choices[0].delta.content, end="")

Use the OpenAI response format when:

• Integrating with tools that rely on OpenAI's API schema
• Inspect or log metadata (e.g., role, finish_reason, etc.)
• Building tools on top of the OpenAI agent protocol