Get started

To transform datasets at scale or optimize data for fast AI model training.

LitData scales data processing tasks (data scraping, image resizing, distributed inference, embedding creation) on local or cloud machines. It also enables optimizing datasets to accelerate AI model training and work with large remote datasets without local loading.

Choose your workflow:

1. Speed up model training
2. Transform dataset

Speed up model training

Accelerate model training (20x faster) by optimizing datasets for streaming directly from cloud storage. Work with remote data without local downloads with features like loading data subsets, accessing individual samples, and resumable streaming.

Step 1: Optimize the data

This step will format the dataset for fast loading. The data will be written in a chunked binary format.

import numpy as np
from PIL import Image
import litdata as ld

def random_images(index):
    fake_images = Image.fromarray(np.random.randint(0, 256, (32, 32, 3), dtype=np.uint8))
    fake_labels = np.random.randint(10)

    # You can use any key:value pairs. Note that their types must not change between samples, and Python lists must
    # always contain the same number of elements with the same types.
    data = {"index": index, "image": fake_images, "class": fake_labels}

    return data

if __name__ == "__main__":
    # The optimize function writes data in an optimized format.
    ld.optimize(
        fn=random_images,                   # the function applied to each input
        inputs=list(range(1000)),           # the inputs to the function (here it's a list of numbers)
        output_dir="fast_data",             # optimized data is stored here
        num_workers=4,                      # The number of workers on the same machine
        chunk_bytes="64MB"                  # size of each chunk
    )

Step 2: Put the data on the cloud

Upload the data to a Lightning Studio (backed by S3) or your own S3 bucket:

aws s3 cp --recursive fast_data s3://my-bucket/fast_data

Step 3: Stream the data during training

Load the data by replacing the PyTorch DataSet and DataLoader with the StreamingDataset and StreamingDataloader

import litdata as ld

train_dataset = ld.StreamingDataset('s3://my-bucket/fast_data', shuffle=True, drop_last=True)
train_dataloader = ld.StreamingDataLoader(train_dataset)

for sample in train_dataloader:
    img, cls = sample['image'], sample['class']

Key benefits

✅ Accelerate training: Optimized datasets load 20x faster.

✅ Stream cloud datasets: Work with cloud data without downloading it.

✅ Pytorch-first: Works with PyTorch libraries like PyTorch Lightning, Lightning Fabric, Hugging Face.

✅ Easy collaboration: Share and access datasets in the cloud, streamlining team projects.

✅ Scale across GPUs: Streamed data automatically scales to all GPUs.

✅ Flexible storage: Use S3, GCS, Azure, or your own cloud account for data storage.

✅ Compression: Reduce your data footprint by using advanced compression algorithms.

✅ Run local or cloud: Run on your own machines or auto-scale to 1000s of cloud GPUs with Lightning Studios.

✅ Enterprise security: Self host or process data on your cloud account with Lightning Studios.

Optimization features:

• ✅ Filter illegal data
• ✅ Compress data
• ✅ Encrypt data

Streaming features:

• ✅ Any clouds
• ✅ Multi GPU, Multi Node
• ✅ LLM support
• ✅ Combine datasets
• ✅ Merge datasets
• ✅ Splits datasets
• ✅ Up-sample
• ✅ Down-sample

Transform datasets

Accelerate data processing tasks (data scraping, image resizing, embedding creation, distributed inference) by parallelizing (map) the work across many machines at once.

Here's an example that resizes and crops a large image dataset:

from PIL import Image
import litdata as ld

# use a local or S3 folder
input_dir = "my_large_images"     # or "s3://my-bucket/my_large_images"
output_dir = "my_resized_images"  # or "s3://my-bucket/my_resized_images"

inputs = [os.path.join(input_dir, f) for f in os.listdir(input_dir)]

# resize the input image
def resize_image(image_path, output_dir):
  output_image_path = os.path.join(output_dir, os.path.basename(image_path))
  Image.open(image_path).resize((224, 224)).save(output_image_path)

ld.map(
    fn=resize_image,
    inputs=inputs,
    output_dir="output_dir",
)

Key benefits:

✅ Parallelize processing: Reduce processing time by transforming data across multiple machines simultaneously.

✅ Scale to large data: Increase the size of datasets you can efficiently handle.

✅ Flexible usecases: Resize images, create embeddings, scrape the internet, etc...

✅ Run local or cloud: Run on your own machines or auto-scale to 1000s of cloud GPUs with Lightning Studios.

✅ Enterprise security: Self host or process data on your cloud account with Lightning Studios.

Features:

• ✅ Distributed transform