The full acceleration surface

streaming_dataset_reader iterates a file in fixed-size batches, never materializing the whole dataset. 77x on Parquet at 50,000 rows (16-core Linux). Format-detected by extension.

parallel_hash_rows fans rows across cores; deduplicate_indices returns unique indices plus new hashes. 1.9x over a hashlib loop at 100,000 rows.

pack_sequences composes input_ids, labels, and attention_mask in a single Rust pass. Honest caveat: ~0.4x at 10,000 sequences — FFI overhead dominates until larger batches.

pad_sequences supports left/right padding and pad_to_multiple_of for hardware alignment. Same FFI-overhead caveat (~0.5x) at small sizes.

import fast_axolotl before importing axolotl. The shim installs itself into sys.modules — no source edits to Axolotl.

Existing axolotl.utils.* import paths route to Rust automatically. Opt into streaming with dataset_use_rust_streaming: true when you want it.

Python 3.10 through 3.13 on Linux, macOS, and Windows. Nothing to compile — uv add fast-axolotl or pip install.

uninstall() and install() switch the shim off and on within a process; is_available() confirms the extension linked.

fast-axolotl targets the data pipeline — reads, dedup, packing, padding. It does not touch training kernels; for that, reach for a tool like Unsloth.

Packing and padding are exposed but only expected to pay off at larger training-scale batches. The 0.4x/0.5x small-batch numbers are published in BENCHMARK.md.

If your wall-clock is GPU-bound rather than data-bound, fast-axolotl will not move it. The wins land when data loading is the bottleneck.

All four operation numbers come from BENCHMARK.md on a documented system (Linux x86_64, 16 cores). No invented benchmarks.