Multiple models one server

Deploying multiple models or data sources often requires separate services, creating complexity and extra maintenance. LitServe removes this friction by letting us run several models—text, vision, or multi-modal—together on one server. This unified setup streamlines deployment, reduces overhead, and enables richer AI workflows without extra infrastructure.

Examples of multiple models in a system.

Background

Compound AI systems

A compound AI system integrates multiple components, such as different models, data sources, or databases, to perform more complex tasks than a single model could handle. For instance, a system could combine a text-processing model with an image-recognition model to provide richer outputs by merging both modalities.

This example shows a compound system that uses a text model to annotate an image model.

# pip install torch clip
import torch, clip, multiprocessing
from PIL import Image
from io import BytesIO
import litserve as ls


class CLIPAPI(ls.LitAPI):
    def setup(self, device):
        self.device = device
        self.model, self.preprocess = clip.load("ViT-B/32", device=device)

        # Load candidate captions
        with open('captions.txt', 'r') as file:
            self.candidate_captions = [line.strip() for line in file.readlines()]

        # Pre-tokenize and pre-encode text captions
        with torch.no_grad():
            self.text_tokens = clip.tokenize(self.candidate_captions).to(self.device)
            self.text_embeddings = self.model.encode_text(self.text_tokens)

    def decode_request(self, request):
        # Convert request to input tensor
        image_bytes = bytes.fromhex(request["image_bytes"])
        image = Image.open(BytesIO(image_bytes))
        return self.preprocess(image).unsqueeze(0).to(self.device)

    def predict(self, image_tensor):
        # Compare the image against the list of captions
        with torch.no_grad():
            image_embedding = self.model.encode_image(image_tensor)

            # Calculating similarities with pre-encoded text features
            similarities = (100.0 * image_embedding @ self.text_embeddings.T).softmax(dim=-1)
            max_indices = similarities.argmax(dim=-1)

            return [self.candidate_captions[idx] for idx in max_indices.cpu().numpy()]

    def encode_response(self, output):
        return {"description": output[0]}

if __name__ == "__main__":
    api = CLIPAPI()
    server = ls.LitServer(api, accelerator="auto")
    server.run(port=8000)

Multimodal systems

Multimodal models are perhaps the best known example of a compound system.

A multimodal architecture can process input data of one type and generate data of another type, such as text-to-image or text-to-speech.

A multimodal server can be built either with a single model that has the ability to process multiple domains or with multiple models where the first model outputs a data type that is supported as an input by the second model.

Image-to-text & speech-to-text are two examples of multi-modal models

Applications of multimodal

Multimodal models have diverse applications, including:

Image Captioning: Generating descriptive text for images.
Speech-to-Text: Converting audio inputs into textual outputs.
Visual Question Answering: Answering questions based on image inputs.

Enable multiple models

To enable a multimodal model with LitServe, follow these steps to configure the server and handle multimodal inputs.

Initialize the models inside setup and decode the prompts or any other input in decode_request
Run inference in predict and return the encoded response.

Server

Here is a toy example that serves image and text embedding models to calculate the similarity between a given image and text prompt.

import numpy as np
import litserve as ls


class SimpleImageTextAPI(ls.LitAPI):
    def setup(self, device):
        # Embedding models
        self.vision_model = lambda x: [1, 2, 3, 4]
        self.text_model = lambda x: [1, 2, 3, 5]

    def decode_request(self, request):
        image_data = request.get("image_bytes")
        text_data = request.get("prompt")
        return image_data, text_data

    def predict(self, inputs):
        image, text = inputs

        image_embedding = self.vision_model(image)
        text_embedding = self.text_model(text)

        cos_sim = np.dot(image_embedding, text_embedding) / (
                np.linalg.norm(image_embedding) * np.linalg.norm(text_embedding))

        return cos_sim

    def encode_response(self, output):
        return {"similarity": output}


if __name__ == "__main__":
    api = SimpleImageTextAPI()
    server = ls.LitServer(api, accelerator="auto")
    server.run(port=8000)

Client

import requests
import base64

with open("image.jpg", "rb") as image_file:
    image_bytes = base64.b64encode(image_file.read()).decode("utf-8")

data = {
    "image_bytes": image_bytes,
    "prompt": "A quick brown fox jumps over the lazy dog",
}
response = requests.post("http://127.0.0.1:8000/predict", json=data)
print(response.json())

Examples

Image captioning with CLIP

CLIP is multimodal model from OpenAI which understands and establishes meaningful connections between text and images.

This is a more complex real-world example that serves as a multimodal model. In this example, CLIP takes an image and a list of sentences as candidate captions to predict an appropriate caption for the given image.

import clip, PIL, io
import litserve as ls

class CLIPAPI(ls.LitAPI):

    def setup(self, device):
        self.model, self.preprocess = clip.load("ViT-B/32", device=device)

    def decode_request(self, request):
        image_bytes = bytes.fromhex(request["image_bytes"])
        image = PIL.Image.open(io.BytesIO(image_bytes))
        candidate_captions = request["candidate_captions"]
        return self.preprocess(image).unsqueeze(0).to(self.device), candidate_captions

    def predict(self, inputs):
        image_tensor, candidate_captions = inputs
        text_tokens = clip.tokenize(candidate_captions).to(self.device)
        text_embeddings = self.model.encode_text(text_tokens)
        image_embedding = self.model.encode_image(image_tensor)
        similarities = (image_embedding @ text_embeddings.T).softmax(dim=-1)
        max_indices = similarities.argmax(dim=-1)
        return [candidate_captions[idx] for idx in max_indices.cpu().numpy()]

    def encode_response(self, output):
        return {"description": output[0]}

if __name__ == "__main__":
    api = CLIPAPI()
    server = ls.LitServer(api)
    server.run(port=8000)

You can make requests to the server using the following client (replace the image path):

import requests, io
from PIL import Image

if __name__ == "__main__":
    image = Image.open("image.jpg").convert("RGB")
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")
    image_bytes = buffered.getvalue()

    data = {
        "image_bytes": image_bytes.hex(),
        "candidate_captions": ["A cat sitting on a sofa", "A dog playing in the park"],
    }
    response = requests.post("http://127.0.0.1:8000/predict", json=data)
    print(response.json())

Was this page helpful?