Hugging Face Compatible

To use your favorite Hugging Face dataset with LitData, simply pass its URL to StreamingDataset.

import litdata as ld

hf_uri = "hf://datasets/leonardPKU/clevr_cogen_a_train/data"

dataset = ld.StreamingDataset(hf_uri)

for sample in ds:
    print(f"{sample[1]}; {sample[2]}")

You don’t need to worry about indexing the dataset or any other setup. LitData will handle all the necessary steps automatically and cache the index.json file, so you won't have to index it again.

This ensures that the next time you stream the dataset, the indexing step is skipped..

Find the Hugging Face URL

You can simply follow those steps, copy the URL and paste it within the example above.

Indexing the HF dataset

If the Hugging Face dataset hasn't been indexed yet, you can index it first using the index_hf_dataset method, and then stream it using the code above.

import litdata as ld

hf_uri = "hf://datasets/leonardPKU/clevr_cogen_a_train/data"

ld.index_hf_dataset(hf_uri)

Benchmarks

Below is the benchmark for the Imagenet dataset (155 GB), demonstrating that optimizing the dataset using LitData is faster and results in smaller output size compared to raw Parquet files