Verify the speedup on your dataset
The README publishes 77x streaming and 1.9x hashing on 16-core Linux, and is honest that packing and padding show FFI overhead at small batches. This guide measures the real numbers on your own dataset.
- 1
Baseline with the shim off
Disable the Rust paths and time the read/dedupe stage in pure Python to establish a baseline.
import fast_axolotl, time fast_axolotl.uninstall() # ... time your read/dedupe here - 2
Measure with the shim on
Re-enable the Rust implementations and time the same stage. Compare against the baseline you just captured.
fast_axolotl.install() # ... time the same read/dedupe here - 3
Tune thread count for hashing
The hashing path scales with cores. Pass num_threads=0 to auto-select, or set it explicitly to match the machine.
from fast_axolotl import deduplicate_indices deduplicate_indices(rows, num_threads=0) - 4
Interpret the packing/padding numbers
At small batch sizes, pack_sequences and pad_sequences can be slower than Python due to FFI overhead — the README shows ~0.4x/0.5x at 10k sequences. Expect wins at larger training-scale batches, not tiny ones.
Read the honest matrix
- →Streaming (77x) and parallel hashing (1.9x) are the clean wins today.
- →pack_sequences (~0.4x) and pad_sequences (~0.5x) show overhead at 10,000-sequence batches.
- →Full system details are in BENCHMARK.md in the repo.
More guides in the index, or read the architecture to see what the shim is doing.