// glossary
The vocabulary of fast-axolotl
fast-axolotl borrows terms from Rust, Python packaging, and the Axolotl data pipeline. Here is what each one means in context.
- shim
- A thin substitution layer. fast-axolotl imports itself into sys.modules so calls to Axolotl's data utilities resolve to Rust implementations instead of the Python originals — with no source edits.
- FFI
- Foreign Function Interface — the boundary where Python calls into Rust. Each crossing has a fixed cost, which is why large-work calls (streaming, hashing) win but tiny-work calls (small-batch packing) can lose.
- streaming reader
- streaming_dataset_reader reads a file in fixed-size batches without ever materializing the whole dataset in memory. Carries the README's 77x Parquet number and bounds peak RAM by batch size.
- token packing
- Combining multiple sequences into fixed-length training rows (input_ids, labels, attention_mask) to minimize padding waste. pack_sequences does this in Rust; a win only at larger batch sizes.
- batch padding
- Extending sequences to a common length (or a hardware-friendly multiple) so they form a rectangular tensor. pad_sequences supports left/right padding and pad_to_multiple_of.
- Parquet streaming
- Reading columnar Parquet files incrementally. fast-axolotl's Rust reader streams Parquet (and Arrow, JSON, JSONL, CSV, text) with transparent ZSTD/Gzip decompression.
- SHA256 dedup
- Detecting duplicate rows by hashing them. parallel_hash_rows fans SHA256 across cores (1.9x over a hashlib loop); deduplicate_indices returns the unique indices and their hashes.
- wheel
- A prebuilt Python package binary. fast-axolotl ships wheels for Python 3.10–3.13 on Linux, macOS, and Windows, so installing needs no local Rust toolchain.
- hot path
- A code path that dominates wall-clock. fast-axolotl targets the four data-pipeline hot paths — read, dedup, pack, pad — rather than the training kernels.
- maturin
- The build tool that compiles the Rust extension into a Python wheel. It is how fast-axolotl produces its cross-platform binaries.
- sys.modules shim
- The mechanism by which import fast_axolotl replaces entries in Python's module cache so subsequent Axolotl imports transparently get the accelerated implementations.
- is_available()
- A runtime check that returns True when the compiled Rust extension is linked into the process — the way to confirm the shim actually took effect.