Serve LLMs

LitServe is a general-purpose framework designed to serve any AI model, including but not limited to LLMs - (AI is bigger than LLMs 😉).

For an LLM-specific solution, check out LitGPT, built on LitServe:

# install litgpt
pip install 'litgpt[all]'

# deploy an out-of-the-box LLM
litgpt serve microsoft/phi-2

# deploy your own trained model
litgpt serve path/to/microsoft/phi-2/checkpoint

Why use a general purpose framework?

LitServe isn’t just an alternative to vLLM or Ollama; it’s more versatile and offers experts the control to build their own high-performance server like vLLM or Ollama.

• ✅ Add your own KV-caching.
• ✅ Add your own batching logic.
• ✅ Add your own kernels.
• ✅ Add your own architectures.
• ✅ etc...

Full control

A general purpose framework gives you full control of the server setup, and the inference process. Most servers need more than just a model... for example database connections, vector databases and the ability to connect your own business logic.

With an "easy" llm serving framework you trade off "easy to use" for the control over what happens inside the framework.

On the left you see an "easy-to-use" LLM framework. It's great for getting started quickly and launching a prototype. But if you ever need to change the model (maybe add a non LLM), connect to a special vector DB, load a dataset or do something bespoke, suddenly the "easy" became a blocker.

On the right you see the equivalent in LitServe. Yes there is more code, but when you need to go beyond a simple prototype, LitServe gives you access to connect your own business logic like vector DBs, other models, etc...

So, if all you want is a quick prototype and internal control is not important for you, then use a one-line LLM server like LitGPT, Ollama or vLLM. But if you suspect you'll need to go beyond a POC, then you need something more flexible like LitServe.

AI is more than LLMs

What if you need to build a breast cancer detector? satellite imagery segmentator? dna embedder? etc... a huge portion of AI use cases today go beyond LLMs. Here are some examples built with LitServe:

LLM deployment examples

Below we show a few ways you can deploy LLMs with LitServe.

Easy one-liner

For an easy one-liner, use LitGPT which is built on top of LitServe.

# install litgpt
pip install 'litgpt[all]'

# option 1: deploy an out-of-the-box LLM
litgpt serve microsoft/phi-2

# option 2: deploy your own trained model
litgpt serve path/to/microsoft/phi-2/checkpoint

Use a LitGPT LLM

In this example we bring in an LLM from LitGPT.

import litgpt
import litserve as ls

class SimpleLitAPI(ls.LitAPI):
    def setup(self, device):
        self.llm = litgpt.LLM.load("meta-llama/Meta-Llama-3-8B-Instruct")

    def decode_request(self, request):
        return request["prompt"]

    def predict(self, prompt):
        return self.llm.generate(prompt, max_new_tokens=200)

    def encode_response(self, output):
        return {"output": output}

if __name__ == "__main__":
    api = SimpleLitAPI()
    server = ls.LitServer(api)
    server.run(port=8000)

When you use the LitGPT serve command, you are using something like this under the hood, but much more optimized with specialized KV-caching, dynamic batching, etc...

Use a Hugging Face LLM

In this example we bring in an LLM from Hugging Face.

from transformers import pipeline
import litserve as ls

class HFAPI(ls.LitAPI):
    def setup(self, device):
        self.pipe = pipeline("text-generation", model="meta-llama/Meta-Llama-3-8B-Instruct", trust_remote_code=True)

    def decode_request(self, request: dict):
        return request["prompt"]

    def predict(self, prompt):
        messages = [{"role": "user", "content": prompt}]
        return self.pipe(messages)

    def encode_response(self, output):
        return {"output": output[0]["generated_text"][-1]["content"]}

if __name__ == "__main__":
    api = HFAPI()
    server = ls.LitServer(api)
    server.run(port=8000)

Use a vLLM model

In this example we bring in an LLM from vLLM.

from vllm import LLM, SamplingParams
import litserve as ls

class VLLMLlamaAPI(ls.LitAPI):
    def setup(self, device):
        self.llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct", quantization="fp8")

    def decode_request(self, request):
        return request["prompt"]

    def predict(self, prompt):
        sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=100)
        outputs =  self.llm.generate(prompt, sampling_params)
        return outputs[0].outputs[0].text

    def encode_response(self, output):
        return {"output": output}

if __name__ == "__main__":
    api = VLLMLlamaAPI()
    server = ls.LitServer(api)
    server.run(port=8000)

LLM serving speed

We don't benchmark against libraries like vLLM or Ollama because LitServe is not a direct alternative to those. Our special-purpose LLM serving library (LitGPT) is the correct comparison for those.

To achieve the speeds in those frameworks, you need specialized batching, KV-caching, etc... Those advanced techniques require specialized knowledge by experts. If you have the need for that level of control, then LitServe is the perfect foundation to build your own vLLM-like server.