LitAI - Docs

LLM router and minimal agent framework in one

Lightning fast ⚡️

open source

pip install litai

Switch between LLM providers without rewriting your prompts — LitAI gives you a thin, composable router across OpenAI, Anthropic, Gemini, local models, and your own self-hosted endpoints, plus a minimal agent framework on top.

Provider-agnostic prompts
Streaming and tool use
Minimal agent loop
Self-host or call APIs

Get started

LitAI

Examples

Toy Example

Walk through a toy example to learn LitAI

Clone free

Chat with the modelUse the chat method to send a message to the model.Stream the responseUse the stream parameter to stream the response from the model.Fallback & retriesFallback to alternate LLMs in case of downtime.

1/3

2/3

3/3

from litai import LLM

llm = LLM(model="openai/gpt-4")
print(llm.chat("who are you?"))
# I'm an AI by Open AI

llm = LLM(model="openai/gpt-4")
for chunk in llm.chat("hello", stream=True):
    print(chunk, end="")

llm = LLM(
    model="openai/gpt-4",
    fallback_models=["mistral/mixtral", "anthropic/claude-3"],
    max_retries=4,
)