Shuflr
softwareA Rust data-streaming tool for deterministic shuffling of large JSONL corpora without loading the dataset into memory
period:2026-present
tech:
RustMachine LearningJSONLZstandardData Streaming
Shuflr converts record-oriented corpora to seekable Zstandard once, then streams them in seed-deterministic shuffled order through a CLI, HTTP, or a compact binary protocol.
It supports buffer, chunk, permutation, and reservoir shuffle modes; deterministic distributed rank partitioning; compressed local and remote inputs; and optional Parquet and Hugging Face sources. In a documented production run, one process served 62 dataset shards and streamed roughly 2.4 TB over more than four days with no observed errors. The repository includes the Rust server and a Python training client.