// continuous integration
Fast dataset validation in CI
Data-quality checks in CI have to read potentially large files on a memory-limited runner, and they run on every change.
Who it's for: Platform teams running data-quality and dedup checks on large datasets inside CI.
the problem
Loading a large dataset into RAM on a CI runner is slow and can OOM the job. Single-threaded hashing makes dedup checks the long pole of the pipeline.
how fast-axolotl helps
Stream the file in fixed-size batches so the runner never materializes it, and use parallel_hash_rows to compute row hashes across the runner's cores for duplicate detection.
What that looks like in practice
- →Bounded memory keeps CI jobs from OOM-killing on large data.
- →Parallel hashing shortens the dedup/validation step on multi-core runners.
- →Prebuilt wheels for Python 3.10–3.13 on Linux, macOS, and Windows install cleanly in CI.
- →is_available() lets the job assert the Rust extension actually linked.
Try it on this workflow
Follow a guide, read the architecture, or see more use cases.