tuna is a fast, streaming k-mer counter for FASTA/FASTQ input. It partitions minimizer superkmers, then counts them with a partition-local canonical rolling-hash table, keeping memory usage low and throughput high.
It uses unordered_dense as its streaming k-mer hash table. Phase 1 parsing uses a C++ port of helicase (FASTX parser), and minimizer computing uses a two-stack sliding-window minimum adapted from simd-minimizers, with a regular random minimizer scheme.
tuna runs a two-phase pipeline:
-
Partition (Phase 1) — overlaps gzip decompression, FASTX parsing, and minimizer iteration. Whenever the minimizer changes, the current packed superkmer is flushed to a binary partition in memory or on disk. tuna estimates the superkmer size up front and keeps phase 1 in memory when it fits the RAM budget; if the budget is reached while running, the partitions held in memory are spilled to disk and counting continues from there, so a run never exceeds its budget. Running with
-dbgreports when a spill happens. The number of partitions is auto-tuned from input size or set explicitly with-n. -
Count (Phase 2) — replays each partition and inserts its k-mers into a hash table, incrementing the count of each. Identical superkmers are collapsed first, so a repeated one is counted once and its multiplicity added in a single step. Each worker holds one partition table at a time.
-
Output (Phase 2, cont.) — iterates the table, applies
-ci/-cxcount filters, and writes results to the output file in TSV or KFF format.
Partitions are processed in parallel across threads (up to -n partitions at a time), keeping peak memory proportional to a single partition's k-mer set.
- Platform: Linux or macOS (x86_64 and AArch64/Apple Silicon)
- C++20 compiler: GCC >= 9.1 or Clang >= 9.0
- CMake >= 3.17
All other dependencies (zlib-ng, kff-cpp-api) are fetched and built automatically by CMake — no manual installation needed.
Debian/Ubuntu:
sudo apt-get install build-essential cmakeFedora/RHEL:
sudo dnf install gcc-c++ cmakemacOS:
brew install llvm cmakegit clone https://github.com/vicLeva/tuna.git
cd tuna/
mkdir build && cd build/
cmake ..
make -j$(nproc)The tuna binary will be at build/tuna.
You do not. Left out, -m is derived from k: 13 or less below k=31, 17 at or
above.
A build therefore instantiates one m per k. If for some reason (we do not recommend it) you would want to choose -m freely, add the following to the cmake command:
cmake .. -DTUNA_ALL_M=ON # every odd m for every kExpect the following make to be slow and memory-hungry: it will be 270 instantiations against 28.
The default build accepts odd k in [11, 63], plus k=127, each carrying
one minimizer length, auto_m(k). That is 28 template instantiations.
Most users want exactly this and should stop here. Leave -m alone: the
value tuna picks is within a few percent of the best one, and overriding it
means building every m, which is ten times the instantiations and by far the
slowest build.
If the build is still too slow and you know your k in advance, pin it:
cmake -S . -B build_k31 -DFIXED_K=31 && cmake --build build_k31 --target tuna -j$(nproc)One instantiation instead of 28, a smaller binary, and the fastest build.
The binary then accepts only k=31. Use a separate directory per k, which is
also how you keep several around at once: build_k31/tuna, build_k63/tuna.
| cmake flags | k accepted | m accepted | pairs |
|---|---|---|---|
| (none) | odd 11–63, 127 | auto_m(k) |
28 |
-DFIXED_K=K |
K |
auto_m(K) |
1 |
-DTUNA_ALL_M=ON |
odd 11–63, 127 | any odd m | 270 |
-DFIXED_K=K -DTUNA_ALL_M=ON |
K |
any odd m | ~12 |
-DFIXED_K=K -DFIXED_M=M |
any k ≤ 256 | M |
1 |
Three notes on that table. The two switches do one thing each and are
independent: FIXED_K narrows which k, TUNA_ALL_M widens which m, so
sweeping m at a single k wants both.
The last row is the exception. The first four pick from a hardcoded list of k,
and FIXED_K can only remove entries from it, never add one — so
-DFIXED_K=256 alone builds an empty table and fails at runtime. Giving
FIXED_M too switches to a branch with no table at all, which instantiates
just the pair you named and so reaches any k <= 256, even k included:
cmake .. -DFIXED_K=256 -DFIXED_M=31FIXED_M does nothing without FIXED_K, and when both are set they win over
TUNA_ALL_M. Whatever the build, -k and -m at runtime must match what it
carries, or tuna exits saying which value it has.
Other compile-time options
Debug build — disables optimisations, enables debug symbols for gdb/valgrind:
cmake .. -DCMAKE_BUILD_TYPE=Debugtuna [options] <input1.fa [input2.fa ...]> <output_file>
tuna [options] @<input_list_file> <output_file>
Input files can be FASTA or FASTQ, plain or gzipped.
Instead of listing files directly, you can pass @list.txt where list.txt is a newline-separated file of paths.
| Flag | Argument | Default | Description |
|---|---|---|---|
-k |
<int> |
31 |
k-mer length. Odd values in [11, 63] plus 127 in the default build. Any value in [2, 256], even ones included, needs a build with both -DFIXED_K=k and -DFIXED_M=m; -k must then match it. See k range. |
-t |
<int> |
1 |
Number of threads. Compressed phase 1 pipelines decompression, parsing, and partitioning; phase 2 parallelizes over partitions. |
-ci |
<int> |
1 |
Minimum count to report |
-cx |
<int> |
max |
Maximum count to report |
-ram |
<int> |
auto | RAM budget in GB. Phase 1 starts in memory when the estimated superkmer size fits the budget and goes to disk otherwise; if the budget is reached mid-run, the partitions held in memory are spilled to disk and the run finishes on the disk path. Also sizes write buffers. Set lower than physical RAM to leave headroom for other processes, or higher to keep phase 1 in memory |
-w |
<dir> |
next to output | Working directory for temporary partition files. |
-kff |
— | off | Write output in KFF binary format instead of TSV. Auto-detected from a .kff output extension. |
-b |
— | off | Disable canonical k-mers: count forward and reverse-complement strands independently. By default, a k-mer and its reverse complement are merged into a single count (the canonical, lexicographically smaller form is reported). Use -b when strand orientation matters or to match tools that count each strand separately. |
-h / --help |
— | — | Print usage |
Advanced / benchmarking flags
| Flag | Argument | Default | Description |
|---|---|---|---|
-m |
<int> |
from k |
Partition minimizer length, normally left alone: tuna derives it from k (13 below k=31, 17 at or above, clamped to k-2), within a few percent of the best value. A build carries only that one m per k, so another value needs -DTUNA_ALL_M=ON or -DFIXED_M — see Choosing m. Odd values only. |
-n |
<int> |
auto | Number of partitions. Auto-tuned to ~2 MB input/partition when omitted |
-hp |
— | off | Hide progress messages (phase timings are always emitted to stderr) |
-kt |
— | off | Keep temporary partition files after the run |
-co |
— | off | Count only: skip k-mer serialization while still reporting total and distinct k-mer counts |
-tp |
— | off | Stop after partitioning — Phase 1 only |
-dbg |
— | off | Per-partition table summary + minimizer coverage CSV written to <work_dir>/debug_min_coverage.csv |
Count k-mers in a reference genome, k=31, 4 threads:
tuna -k 31 -t 4 genome.fa counts.tsvCount only k-mers seen at least twice:
tuna -k 31 -t 4 -ci 2 genome.fa counts.tsvCount from a list of files:
tuna -k 31 -t 8 @genomes.list counts.tsvWrite KFF binary output (auto-detected from extension):
tuna -k 31 -t 8 @genomes.list counts.kffBenchmark counting without serializing k-mers:
tuna -k 31 -t 8 -co @genomes.list /dev/nullLarge genomes — counting one human assembly (3.0 Gbp) yields about 2.5 billion distinct 31-mers. In TSV that is roughly 85 GB, so prefer KFF, or
/dev/nullwith-cowhen benchmarking; at k=31, KFF uses 8 sequence bytes plus 1–4 count bytes per k-mer.
Plain text, tab-separated, one k-mer per line:
ACGTACGTACGTACGTACGTACGTACGTACG 42
TGCATGCATGCATGCATGCATGCATGCATGC 7
...
K-mer File Format binary output. Each k-mer is stored as a 2-bit packed sequence (A=0, C=1, G=2, T=3) with its smallest lossless 1–4 byte big-endian count. Workers batch records by count width before writing KFF raw sections, avoiding per-record section changes. The file is marked canonical=true and unique=true (or canonical=false when -b is used). Roughly 3–4× smaller than TSV for k=31.
KFF files can be read with kff-cpp-api or any other KFF-compatible tool.
Only k-mers with counts in [ci, cx] are written. By default, the canonical (lexicographically smaller of forward/reverse-complement) form of each k-mer is reported. With -b, the observed strand form is reported instead.
tuna can be embedded directly in a C++ project
#include <tuna/tuna.hpp>
// Collect all k-mers into a map (simple)
auto kmers = tuna::count_to<31>({"genome.fa"}); // std::unordered_map<std::string, uint32_t>
// Stream k-mers through a callback (memory-efficient)
tuna::count<31>({"genome.fa"}, [](std::string_view kmer, uint32_t count) {
// called for every canonical k-mer; may run from multiple threads
});
// Large k: any value in [2, 256], both k and m are template parameters
tuna::count<127, 21>({"genome.fa"}, [](std::string_view kmer, uint32_t count) { ... });
// Options mirror the CLI flags; m defaults to 17, as on the command line
tuna::Options opts;
opts.threads = 8; // -t
opts.ram_gb = 64; // -ram, 0 = auto-detect
opts.canonical = false; // -b: count each strand separately
tuna::count<31>({"reads.fq.gz"}, [](std::string_view kmer, uint32_t count) { ... }, opts);The callback runs the same counting path as the CLI and differs only in where
counted k-mers go, so an API run and an equivalent tuna invocation produce
identical counts.
CMake integration:
add_subdirectory(tuna) # or use FetchContent
target_link_libraries(my_target PRIVATE tuna::tuna)For a full walkthrough: CMake setup, FetchContent, container customisation, thread safety, see the wiki: Using tuna as a library.
All numbers below were measured at k=31, m=21 (the default at the time; the
default is now m=17), 8 threads, with a 256 GB memory budget, on a GenOuest
node (4x8 Xeon E5-2660 at 2.20 GHz, 1.5 TB RAM, CentOS 7), against
KMC 3.2.4 and
FastK. Every tool was asked to report
all k-mers from a count of one upwards, so the three produce the same set of
counts.
| dataset | organism | type | files | where to get it |
|---|---|---|---|---|
| Ecoli | E. coli | assemblies (plain FASTA) | 3682 | Zenodo 6577997 |
| Salmonella | S. enterica | assemblies (gz) | 10000 | ENA2018-bacteria-661k |
| Gut | gut MAGs | assemblies (gz) | 10000 | HumGut |
| Human | H. sapiens | assemblies (gz) | 60 | HPP Year 1 Assemblies |
| Tara | Tara Oceans (sea water) | reads (gz) | 10 | 6 ENA runs from PRJEB4352, listed below |
| Gallus | G. gallus | reads (gz) | 12 | ENA SRR105788, SRR105789, SRR105792, SRR105794, SRR197985, SRR197986 (paired) |
| Human3 | H. sapiens | reads (gz) | 36 | ENA ERR174324-ERR174341 (paired) |
| HumanR | H. sapiens | reads (gz) | 1 | ENA SRR622461, forward reads only |
The per-file experiments use the first 100 files of Ecoli, Salmonella and Gut, and the first 10 of Human and Tara. Gallus and Human3 are counted whole, in a single run over the entire collection. HumanR drives the coverage sweep.
The Tara files are surface-water samples, size fraction QQSS, from stations 7, 23, 30 and 11 of the Tara Oceans expedition. They come from six sequencing runs of four samples, of which the benchmark uses ten files:
| ENA run | sample | station | files used |
|---|---|---|---|
ERR315827 |
ERS327852 | 7 SUR1 | AHX_AAGOSU_6_1_814P9ABXX, _6_2_814P9ABXX |
ERR318594 |
ERS329507 | 23 SUR2 | AHX_AASOSU_1_2_62FGDAAXX |
ERR318583 |
ERS329507 | 23 SUR2 | AHX_AASOSU_2_1_62FGDAAXX, _2_2_62FGDAAXX |
ERR318616 |
ERS329505 | 30 SUR2 | AHX_ABEOSU_1_1_62J5HAAXX, _1_2_62J5HAAXX |
ERR318585 |
ERS329505 | 30 SUR2 | AHX_ABEOSU_2_1_62J5HAAXX, _2_2_62J5HAAXX |
ERR1726642 |
ERS488262 | 11 | AHX_ACXIOSF_6_1_C2FGHACXX.IND4_clean |
The submitted files can be fetched directly:
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR315/ERR315827/AHX_AAGOSU_6_1_814P9ABXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR315/ERR315827/AHX_AAGOSU_6_2_814P9ABXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR318/ERR318594/AHX_AASOSU_1_2_62FGDAAXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR318/ERR318583/AHX_AASOSU_2_1_62FGDAAXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR318/ERR318583/AHX_AASOSU_2_2_62FGDAAXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR318/ERR318616/AHX_ABEOSU_1_1_62J5HAAXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR318/ERR318616/AHX_ABEOSU_1_2_62J5HAAXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR318/ERR318585/AHX_ABEOSU_2_1_62J5HAAXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR318/ERR318585/AHX_ABEOSU_2_2_62J5HAAXX.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR172/ERR1726642/AHX_ACXIOSF_6_1_C2FGHACXX.IND4_clean.fastq.gz
Every script that produced these numbers is in scripts/, one per
experiment, documented in scripts/README.md. They share
bench_common.sh and are resumable: anything already in the output CSV is
skipped, so an interrupted run continues where it stopped.
Median wall time per file, each file counted in its own run. binary is each tool writing its own compact format (KFF for tuna, native databases for the others), ASCII is the whole cost of obtaining a plain-text table, including the separate conversion pass KMC and FastK each require.
| dataset | tuna (bin) | KMC3 (bin) | FastK (bin) | tuna (ASCII) | KMC3 (ASCII) | FastK (ASCII) |
|---|---|---|---|---|---|---|
| Ecoli | 0.29 s | 0.57 s | 0.44 s | 0.41 s | 1.22 s | 1.11 s |
| Salmonella | 0.23 s | 0.59 s | 0.42 s | 0.34 s | 1.22 s | 1.08 s |
| Gut | 0.14 s | 0.44 s | 0.26 s | 0.20 s | 0.74 s | 0.56 s |
| Human | 31.2 s | 39.3 s | 105.6 s | 77.9 s | 229.8 s | failed |
| Tara | 35.1 s | 40.6 s | 128.6 s | 76.7 s | 192.8 s | 289.0 s |
FastK's Tabex conversion step segfaults on every Human file, so it has no
ASCII result there.
Whole collections counted in a single run, binary output:
| dataset | input | tuna | KMC3 | FastK | tuna RSS | KMC3 RSS | FastK RSS |
|---|---|---|---|---|---|---|---|
| Gallus | 22.3 GB | 2.7 min | 3.5 min | timeout | 43 GB | 238 GB | - |
| Human3 | 572 GB | 60.0 min | 66.4 min | 71.2 min | 3 GB | 239 GB | 192 GB |
Both FastK runs on Gallus exceeded the six-hour limit. On Human3 the input no longer fits the memory budget, so tuna switches to its disk pipeline and counts 20.9 billion distinct k-mers in 3 GB of RAM, while still finishing first.