// data-bound throughput
Speed up the Axolotl data pipeline
On many fine-tuning runs the GPU sits waiting on the data pipeline. The read and dedupe stages, not the kernels, dominate wall-clock.
Who it's for: Engineers whose fine-tuning wall-clock is dominated by data loading rather than GPU compute.
the problem
Pure-Python streaming and single-threaded hashing cannot keep a modern GPU fed. The bottleneck moves upstream of training, and adding GPUs does not help.
how fast-axolotl helps
fast-axolotl swaps the two clean-win hot paths for Rust at import time: streaming readers (77x on Parquet, 50k rows) and multi-threaded SHA256 deduplication (1.9x over a hashlib loop, 100k rows, 16 cores).
What that looks like in practice
- →Streaming reader carries the README's 77x Parquet number at 50,000 rows.
- →parallel_hash_rows fans SHA256 across cores; deduplicate_indices returns unique indices + hashes.
- →Existing axolotl.utils.* import paths get the Rust implementations — no code change.
- →Wins land on read and dedupe, the stages that dominate when data is the bottleneck.
Try it on this workflow
Follow a guide, read the architecture, or see more use cases.