// preprocessing
High-throughput fine-tuning data prep
Before a single epoch runs, a large corpus has to be read, decompressed, and deduplicated. That preprocessing pass is often re-run every experiment.
Who it's for: ML teams iterating on large corpora who re-run preprocessing across many experiments.
the problem
Repeated read-and-dedupe passes over a growing corpus add up. Single-threaded Python preprocessing becomes a recurring tax on iteration speed.
how fast-axolotl helps
Use the Rust streaming reader plus parallel deduplication to cut the preprocessing pass. deduplicate_indices can filter against a previously-seen hash set so incremental corpus updates only process new rows.
What that looks like in practice
- →Multi-threaded SHA256 (1.9x) over rows for deduplication across cores.
- →existing_hashes lets you dedupe incrementally against prior runs.
- →Streaming decompression (ZSTD/Gzip) avoids a separate unpack step.
- →num_threads=0 auto-selects the core count for the hashing pass.
Try it on this workflow
Follow a guide, read the architecture, or see more use cases.