Async LLM generation

In applications that handle multiple users, API calls, or agents, blocking LLM requests can become a bottleneck. Python’s standard synchronous execution forces each LLM call to wait, reducing throughput and responsiveness. LitAI solves this by supporting asynchronous LLM calls, allowing us to run multiple requests concurrently. This improves performance and scalability in high-concurrency environments like web servers, chatbots, and agent runtimes.

Background

What is async?

Asynchronous programming lets our application perform other tasks while waiting for operations like API calls or file I/O. In Python, this is done using async and await.

Async with LLMs

Calling an LLM API is a network-bound operation. Without async, each request blocks the main thread, limiting scalability. With async enabled, we can handle many concurrent requests making our application faster and more efficient under load.

Use LitAI asynchronously

LitAI provides a built-in option to enable async via enable_async=True. We can then use await llm.chat(...) inside any async function.

import asyncio
from litai import LLM

async def main():
    llm = LLM(model="openai/gpt-4", enable_async=True)
    print(await llm.chat("Who are you?"))

if __name__ == "__main__":
    asyncio.run(main())

High-concurrency example

When calling LLMs in a loop using synchronous code, each request must finish before the next one starts. This is inefficient and slow, especially when working with multiple users, tools, or tasks in parallel.

With enable_async=True, LitAI lets us issue multiple LLM requests concurrently using asyncio.gather. This means our program doesn't sit idle while waiting for one response. It makes better use of system resources and dramatically reduces total latency.

import asyncio
from litai import LLM

llm = LLM(model="openai/gpt-4", enable_async=True)

async def ask(question):
    return await llm.chat(question)

async def main():
    questions = ["What is AGI?", "Who built Lightning AI?", "Tell me a joke"]
    results = await asyncio.gather(*(ask(q) for q in questions))

    for res in results:
        print(res)

asyncio.run(main())

Conclusion

Async support in LitAI helps scale applications efficiently by running multiple LLM calls in parallel. With a single argument enable_async=True , we can drastically improve the overall throughput and total latency.