Make deduplication opt-in and prepare 2.2.0beta3 - #683
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request makes duplicate-buffer bypassing opt-in via --deduplicate (disabled by default) to restore run-to-run reproducibility, while still generating a deterministic duplicates.txt report for all scans. It also updates documentation and bumps the project version to 2.2.0beta3.
Changes:
- Add
--deduplicateCLI flag and plumb it throughscanner_configto control duplicate-buffer bypassing. - Always create
duplicates.txtand emit deterministicpath1<TAB>path2<TAB>SHA-1rows at shutdown. - Add/adjust end-to-end and API tests, update docs/manpage/release notes, and bump version/tag references to
2.2.0beta3.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/test_be3.cpp | Adds an end-to-end regression test validating default vs --deduplicate behavior and duplicates.txt contents. |
| src/bulk_extractor.cpp | Adds --deduplicate option and wires it into scanner_config. |
| src/be20_api/test_be20_api.cpp | Adds/updates unit tests for default dedup behavior and duplicates.txt creation. |
| src/be20_api/scanner_set.h | Reworks duplicate tracking storage and adds duplicates recorder name constant. |
| src/be20_api/scanner_set.cpp | Implements opt-in bypassing while always recording duplicates and writing duplicates.txt at shutdown. |
| src/be20_api/scanner_params.cpp | Adjusts scanner-facing duplicate check behavior (used by scan_zip). |
| src/be20_api/scanner_config.h | Adds scanner_config::deduplicate flag. |
| man/bulk_extractor.1 | Documents --deduplicate and duplicates.txt. |
| doc/scanner_api.md | Documents deduplication default/opt-in behavior and duplicates reporting. |
| doc/RELEASE_NOTES.md | Updates 2.2.0 draft notes for beta3 and deduplication change; references #682. |
| doc/latex_manuals/BECurrentGuide.tex | Updates user guide text and CLI reference entry for --deduplicate and duplicates.txt. |
| configure.ac | Bumps project version to 2.2.0beta3. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| bool scanner_params::check_previously_processed(const sbuf_t &s) const | ||
| { | ||
| assert(ss!=nullptr); | ||
| return ss->previously_processed_count(s)==0; | ||
| const bool first_seen = ss->previously_processed_count(s)==0; | ||
| return sc.deduplicate && first_seen; | ||
| } |
There was a problem hiding this comment.
Fixed in 572af19. check_previously_processed() now records every observation, returns false for the first observation, and returns true only for repeats when --deduplicate is enabled. Added focused tests for both opt-in suppression and default-mode recording. Validation: make -C src check TESTS=test_be20_api -j1, make check -j1, and make distcheck -j1 all pass. On the issue #682 2.1 GB corpus, the default run remains at 542,141 feature rows; three deduplicated runs each produced 515,566 rows and identical 1,997-row duplicates.txt reports.
Summary
--deduplicate, with deduplication disabled by defaultduplicates.txtwith deterministicpath1<TAB>path2<TAB>SHA-1rows2.2.0beta3Diagnosis
The report in #682 is caused by the existing concurrent content-deduplication path, independently of the thread-pool shutdown fix in #679. A content hash is claimed by whichever equal recursive buffer reaches the shared seen set first. The later buffer is bypassed, so the feature count is conserved while the retained nested-GZIP forensic path depends on worker scheduling.
With this change, duplicate detection and reporting remain enabled, but bypassing is disabled unless
--deduplicateis supplied.Validation
make -C src check TESTS=test_be20_api -j1make -C src check TESTS=test_be -j1make check -j1make distcheck -j1c0172d79ec23b2fce54e725b00062a38fc3988dfc036b4aa99bbaf243628b3fb458a6663684626d076bf6153b958329e5d7ecee3d58f6056759e92508dd5b4c7--deduplicateruns: 514,470 normalized feature rows each (the reporter's count), with differing content hashes and differences confined to competing nested-GZIP pathsduplicates.txtwas identical across all three runs in each modeAddresses #682 without closing it.