From e1e6e557e5847b1fd01e179d9325ab5034627a1f Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 15:32:55 -0400 Subject: [PATCH 1/9] Add release procedure and automation gates --- .dockerignore | 20 ++++ .github/CODEOWNERS | 6 ++ .github/ISSUE_TEMPLATE/release.yml | 97 +++++++++++++++++ .github/workflows/ci-cd.yml | 2 + .github/workflows/codecov.yml | 2 + .github/workflows/latex-docs.yml | 2 + .github/workflows/mingw.yml | 2 + AGENTS.md | 17 +++ Makefile.am | 70 ++++++++++++ doc/RELEASE_CHECKLIST.txt | 13 --- doc/RELEASE_NOTES.md | 47 ++++++-- doc/RELEASE_PROCEDURE.md | 167 +++++++++++++++++++++++++++++ etc/Dockerfile.fedora-distcheck | 24 +++++ etc/Dockerfile.ubuntu-distcheck | 24 +++++ 14 files changed, 472 insertions(+), 21 deletions(-) create mode 100644 .dockerignore create mode 100644 .github/CODEOWNERS create mode 100644 .github/ISSUE_TEMPLATE/release.yml delete mode 100644 doc/RELEASE_CHECKLIST.txt create mode 100644 doc/RELEASE_PROCEDURE.md create mode 100644 etc/Dockerfile.fedora-distcheck create mode 100644 etc/Dockerfile.ubuntu-distcheck diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 000000000..a4f688944 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,20 @@ +.git +autom4te.cache +build-aux +config.h +config.status +configure +Makefile +doc/Makefile +doc/latex_manuals/Makefile +man/Makefile +specfiles/Makefile +specfiles/bulk_extractor.fedora.spec +specfiles/bulk_extractor.opensuse.spec +specfiles/bulk_extractor.spec.m4 +src/.libs +src/Makefile +src/tests/Makefile +tests/Makefile +bulk_extractor-* +release-artifacts diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 000000000..0ee25b93e --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,6 @@ +# All changes require approval from the human repository owner. +* @simsong + +# Documentation-only changes do not require code-owner approval. +*.md +*.txt diff --git a/.github/ISSUE_TEMPLATE/release.yml b/.github/ISSUE_TEMPLATE/release.yml new file mode 100644 index 000000000..8e0d4255e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/release.yml @@ -0,0 +1,97 @@ +name: Release +description: Plan, validate, and publish a versioned bulk_extractor release. +title: "release: " +labels: [release] +body: + - type: markdown + attributes: + value: | + Use this issue as the release record. Do not attach forensic images, unredacted scan output, credentials, presigned URLs, or other sensitive material. Record only approved image identifiers/checksums and redacted validation evidence. + - type: input + id: version + attributes: + label: Target version + description: The intended release version, for example `2.2.1`. + placeholder: "2.2.1" + validations: + required: true + - type: input + id: tag + attributes: + label: Release tag + description: Immutable signed annotated tag to create after release-PR merge. + placeholder: "v2.2.1" + validations: + required: true + - type: input + id: commit + attributes: + label: Reviewed commit SHA + description: Full SHA at which the tag will be created. Do not use a mutable branch name. + placeholder: "0123456789abcdef..." + validations: + required: true + - type: textarea + id: people + attributes: + label: Release manager and package maintainers + description: Name the release manager and the maintainers authorized to submit Debian and RPM packages. + placeholder: | + Release manager: @... + Debian repository maintainer: @... + RPM repository maintainer: @... + validations: + required: true + - type: textarea + id: large_images + attributes: + label: Approved large-image validation inputs + description: Record only a safe image identifier, SHA-256 checksum, approval source, and storage/access policy. Never attach or link case data. + placeholder: | + Identifier: ... + SHA-256: ... + Approval: ... + Access policy: ... + validations: + required: true + - type: textarea + id: build_evidence + attributes: + label: Build and validation evidence + description: Add URLs for macOS, Ubuntu, Fedora/RPM, Windows, and AWS validation jobs as they complete. Include redacted result summaries and checksums only. + placeholder: | + - [ ] macOS `make distcheck`: + - [ ] macOS large-image scan: + - [ ] Ubuntu `make distcheck`: + - [ ] Fedora/RPM `make distcheck`: + - [ ] Windows executable and Unicode test: + - [ ] AWS large-image validation and cleanup: + validations: + required: true + - type: textarea + id: artifacts + attributes: + label: Release artifacts and provenance + description: Add each artifact filename, SHA-256 checksum, producing job URL, and eventual publication URL. + placeholder: | + - [ ] Source archive: + - [ ] Windows executable: + - [ ] Debian package: + - [ ] RPM package: + - [ ] AWS validation summary: + - [ ] SHA256SUMS: + validations: + required: true + - type: checkboxes + id: gates + attributes: + label: Release gates + options: + - label: The release PR updates `configure.ac`, release notes, and applicable package metadata. + required: true + - label: The tag will be signed, annotated, immutable, and created at the reviewed SHA only after all required checks pass. + required: true + - label: I will use a draft GitHub Release and publish it only after the evidence and artifacts are reviewed. + required: true + - label: I will not include confidential forensic data or credentials in this issue. + required: true diff --git a/.github/workflows/ci-cd.yml b/.github/workflows/ci-cd.yml index 5b81fd8ec..e18343db2 100644 --- a/.github/workflows/ci-cd.yml +++ b/.github/workflows/ci-cd.yml @@ -4,6 +4,7 @@ on: push: branches: [main] pull_request: + types: [opened, reopened, synchronize, ready_for_review] workflow_dispatch: permissions: @@ -11,6 +12,7 @@ permissions: jobs: build: + if: github.event_name != 'pull_request' || !github.event.pull_request.draft strategy: fail-fast: false matrix: diff --git a/.github/workflows/codecov.yml b/.github/workflows/codecov.yml index e34d8b454..3aa52082a 100644 --- a/.github/workflows/codecov.yml +++ b/.github/workflows/codecov.yml @@ -6,6 +6,7 @@ on: push: branches: [main] pull_request: + types: [opened, reopened, synchronize, ready_for_review] workflow_dispatch: permissions: @@ -13,6 +14,7 @@ permissions: jobs: build: + if: github.event_name != 'pull_request' || !github.event.pull_request.draft runs-on: ubuntu-22.04 env: diff --git a/.github/workflows/latex-docs.yml b/.github/workflows/latex-docs.yml index a447e956a..109d172c8 100644 --- a/.github/workflows/latex-docs.yml +++ b/.github/workflows/latex-docs.yml @@ -2,6 +2,7 @@ name: Build and publish LaTeX documentation on: pull_request: + types: [opened, reopened, synchronize, ready_for_review] paths: - 'doc/**/*.tex' - '.github/workflows/latex-docs.yml' @@ -23,6 +24,7 @@ concurrency: jobs: build: + if: github.event_name != 'pull_request' || !github.event.pull_request.draft runs-on: ubuntu-latest steps: - name: Checkout diff --git a/.github/workflows/mingw.yml b/.github/workflows/mingw.yml index ee1563399..8f366de76 100644 --- a/.github/workflows/mingw.yml +++ b/.github/workflows/mingw.yml @@ -4,6 +4,7 @@ on: push: branches: [main] pull_request: + types: [opened, reopened, synchronize, ready_for_review] workflow_dispatch: permissions: @@ -15,6 +16,7 @@ env: jobs: mingw: + if: github.event_name != 'pull_request' || !github.event.pull_request.draft name: Ubuntu MinGW executable runs-on: ubuntu-24.04 diff --git a/AGENTS.md b/AGENTS.md index 470e4ec35..c2ca6059d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,23 @@ GitHub activity authored by Codex must use `@${USER}-codex` when that account exists. For this workspace, use `@simsong-codex`; do not use the personal `@simsong` identity for GitHub writes, pushes, issues, pull requests, reviews, or comments. +Before creating or amending a Codex-authored commit, configure and verify this +repository's author and committer as `Codex AI Assistant ` +and verify the configured signing key belongs to that identity. SSH remote +authentication (including `github-codex`) controls push access only; it does +not set commit metadata. Use `git log --format='%G? %GS %an <%ae> %cn <%ce>'` +to verify the resulting commit before pushing. When correcting existing +commits, use `git commit --amend --reset-author -S` rather than only amending +the signature. + +For every Codex-authored pull request, request a Copilot review after publishing it, then monitor the pull request until Copilot has responded and all required CI checks have completed. Address actionable feedback before declaring the pull request ready. When pushing a fix in response to Copilot, reply to its thread with the fix and validation evidence, but leave the thread unresolved for `@simsong` to review and resolve. Once it is green and feedback-free, mark it ready for review and assign it to `@simsong`; do not approve or merge it unless explicitly asked. + +Delete a local branch once it has merged into `main`. First verify that it is an ancestor of the current `main`; preserve unmerged branches and local files in linked worktrees. + +Do not close GitHub issues. You may validate an issue, record evidence, and recommend +closure in a comment, but change an issue's open/closed state only when the repository +owner explicitly instructs you to do so. + Before changing a scanner or writing a scanner plug-in, read [doc/scanner_api.md](doc/scanner_api.md). Start new loadable scanners from [doc/scanner_template.cpp](doc/scanner_template.cpp), and preserve its phase diff --git a/Makefile.am b/Makefile.am index 618a7104a..6b843166e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -21,6 +21,9 @@ SUBDIRS = doc doc/latex_manuals man src src/tests specfiles tests include Makefile.auto_defs EXTRA_DIST = $(SRC_WIN_DIST) $(AUTO_DOC_FILES) $(AUTO_ETC_FILES) $(AUTO_LICENSES) \ + $(srcdir)/.dockerignore \ + $(srcdir)/etc/Dockerfile.fedora-distcheck \ + $(srcdir)/etc/Dockerfile.ubuntu-distcheck \ $(srcdir)/AGENTS.md \ $(srcdir)/.gitignore \ $(srcdir)/CODING_STANDARDS.md \ @@ -86,3 +89,70 @@ libinstall: install -D src/libbulkextractor.so $(prefix)/lib/libbulkextractor.so install -D python/module/bulkextractor.py $(prefix)/lib/python2.7/dist-packages/bulkextractor.py if [ -r /sbin/ldconfig ]; /sbin/ldconfig -v ; fi + +# Release assembly is deliberately fail-closed. Artifact producers are +# independently validated release jobs; this target stages their outputs and +# the source distribution only when each required input is present. +RELEASE_ARTIFACT_DIR ?= $(abs_builddir)/release-artifacts + +.PHONY: release release-source release-linux-distcheck release-windows release-deb release-rpm release-aws-result release-checksums +release: + $(MAKE) release-source + $(MAKE) release-linux-distcheck + $(MAKE) release-windows + $(MAKE) release-deb + $(MAKE) release-rpm + $(MAKE) release-aws-result + $(MAKE) release-checksums + +release-source: + $(MAKE) distcheck + mkdir -p $(RELEASE_ARTIFACT_DIR) + cp $(PACKAGE)-$(VERSION).tar.gz $(RELEASE_ARTIFACT_DIR)/ + +# Each Finch build copies a clean source context and runs the real distcheck +# inside it. Finch selects native arm64 and emulated amd64 on Apple Silicon. +CONTAINER_ENGINE ?= finch +CONTAINER_PLATFORMS ?= arm64 amd64 + +.PHONY: distcheck-ubuntu-container distcheck-fedora-container distcheck-containers +distcheck-ubuntu-container: + @set -e; for platform in $(CONTAINER_PLATFORMS); do \ + $(CONTAINER_ENGINE) build --platform "$$platform" --progress=plain \ + --file etc/Dockerfile.ubuntu-distcheck .; \ + done + +distcheck-fedora-container: + @set -e; for platform in $(CONTAINER_PLATFORMS); do \ + $(CONTAINER_ENGINE) build --platform "$$platform" --progress=plain \ + --file etc/Dockerfile.fedora-distcheck .; \ + done + +distcheck-containers: distcheck-ubuntu-container distcheck-fedora-container + +release-linux-distcheck: + $(MAKE) distcheck-containers + +release-windows: + test -n "$(RELEASE_WINDOWS_EXE)" && test -s "$(RELEASE_WINDOWS_EXE)" + mkdir -p $(RELEASE_ARTIFACT_DIR) + cp "$(RELEASE_WINDOWS_EXE)" $(RELEASE_ARTIFACT_DIR)/bulk_extractor64.exe + +release-deb: + test -n "$(RELEASE_DEB)" && test -s "$(RELEASE_DEB)" + mkdir -p $(RELEASE_ARTIFACT_DIR) + cp "$(RELEASE_DEB)" $(RELEASE_ARTIFACT_DIR)/ + +release-rpm: + test -n "$(RELEASE_RPM)" && test -s "$(RELEASE_RPM)" + mkdir -p $(RELEASE_ARTIFACT_DIR) + cp "$(RELEASE_RPM)" $(RELEASE_ARTIFACT_DIR)/ + +release-aws-result: + test -n "$(RELEASE_AWS_RESULT)" && test -s "$(RELEASE_AWS_RESULT)" + mkdir -p $(RELEASE_ARTIFACT_DIR) + cp "$(RELEASE_AWS_RESULT)" $(RELEASE_ARTIFACT_DIR)/ + +release-checksums: + rm -f $(RELEASE_ARTIFACT_DIR)/SHA256SUMS + cd $(RELEASE_ARTIFACT_DIR) && shasum -a 256 * > SHA256SUMS diff --git a/doc/RELEASE_CHECKLIST.txt b/doc/RELEASE_CHECKLIST.txt deleted file mode 100644 index 2253d25a0..000000000 --- a/doc/RELEASE_CHECKLIST.txt +++ /dev/null @@ -1,13 +0,0 @@ -Manual checklist for release. - -This document should be *printed* and hand-checked. - -- [ ] Compiles and executes "make distcheck" on clean install of MacOS ___. - -- [ ] Compiles and executes "make distcheck" on clean install of Ubuntu ___. - -- [ ] Compiles and executes "make distcheck" on clean install of Fedora ____. - -- [ ] On Fedora, compile bulk_extractor.exe and test_be.exe with mingw64-configure, validate test_be.exe under Wine, and then run bulk_extractor.exe on a windows system. - -- [ ] Restart Checked: Running program on ubnist1, type control-C, then up-arrow, and restart works. diff --git a/doc/RELEASE_NOTES.md b/doc/RELEASE_NOTES.md index 64b877aec..e303f31c9 100644 --- a/doc/RELEASE_NOTES.md +++ b/doc/RELEASE_NOTES.md @@ -27,11 +27,11 @@ through the project's normal pull-request and CI process. ### Highlights -- A standalone 64-bit Windows `.exe` is planned to return as a simple download. - It is cross-compiled with MinGW on Ubuntu and is checked to ensure that it - imports no non-system Windows DLLs. This remains a release item until - [PR #543](https://github.com/simsong/bulk_extractor/pull/543) is merged and the - resulting executable is attached to the 2.2.0 release. +- A standalone 64-bit Windows `.exe` is built with MinGW on Ubuntu and is + checked to ensure that it imports no non-system Windows DLLs. GitHub Actions + publishes it as a downloadable artifact; attaching it to the 2.2.0 release + remains a release task + ([PR #543](https://github.com/simsong/bulk_extractor/pull/543)). - The source tree is self-contained: `be20_api`, DFXML, schemas, and UTF support are now versioned in this repository instead of being supplied through fragile Git submodules @@ -76,6 +76,24 @@ through the project's normal pull-request and CI process. - Prevented empty MSXML extraction from causing recursion and changed residual `sbuf` diagnostics from an abort to a DFXML warning ([PR #537](https://github.com/simsong/bulk_extractor/pull/537)). +- Applied `--max_minute_wait` to phase-1 work and shutdown, reporting timeout + failures instead of blocking indefinitely + ([PR #540](https://github.com/simsong/bulk_extractor/pull/540)). +- Made recursive input traversal deterministic and safe around symlinks and + permission-denied paths, and proved the Windows path through the original + Unicode-filename regression case + ([PR #541](https://github.com/simsong/bulk_extractor/pull/541), + [PR #549](https://github.com/simsong/bulk_extractor/pull/549)). +- Restored stop-list and alert-list processing, including the normal and + diverted feature outputs, with CLI-level regression tests + ([PR #552](https://github.com/simsong/bulk_extractor/pull/552), + [PR #553](https://github.com/simsong/bulk_extractor/pull/553)). +- Added carving for validated RawTherapee `Image8` RGB thumbnail records, + writing PPM output without an image-sized intermediate copy + ([PR #556](https://github.com/simsong/bulk_extractor/pull/556)). +- Fixed a SQLite-size arithmetic overflow and reduced the scheduled Coverity + workflow token to read-only repository contents + ([PR #538](https://github.com/simsong/bulk_extractor/pull/538)). ### Build, configuration, and testing @@ -94,11 +112,24 @@ through the project's normal pull-request and CI process. `bulk_extractor64.exe` as a GitHub Actions artifact ([PR #543](https://github.com/simsong/bulk_extractor/pull/543)). +### Documentation and project maintenance + +- Consolidated release history and this 2.2.0 draft in one versioned document + ([PR #544](https://github.com/simsong/bulk_extractor/pull/544)). +- Moved the living technical-debt scoreboard to + [Discussion #545](https://github.com/simsong/bulk_extractor/discussions/545), + with GitHub issues remaining the actionable work records + ([PR #546](https://github.com/simsong/bulk_extractor/pull/546)). +- Added code-of-conduct, contribution, issue, and pull-request guidance, and + documented the required Codex GitHub identity + ([PR #551](https://github.com/simsong/bulk_extractor/pull/551), + [PR #542](https://github.com/simsong/bulk_extractor/pull/542)). + ### Known limitations and release work -- Merge and keep the standalone Windows artifact workflow green, then attach its - `.exe` to the 2.2.0 GitHub release. The current proposed Windows build disables - libewf, so the `.exe` does not read E01 images directly. +- Keep the Windows artifact workflow green and attach its `.exe` to the 2.2.0 + GitHub release. The current build disables libewf, so the `.exe` does not + read E01 images directly. - Lightgrep remains an unsupported, source-broken optional configuration and should not be represented as a working 2.2.0 feature. - BEViewer is not bundled with bulk_extractor 2. diff --git a/doc/RELEASE_PROCEDURE.md b/doc/RELEASE_PROCEDURE.md new file mode 100644 index 000000000..7c0e83bf7 --- /dev/null +++ b/doc/RELEASE_PROCEDURE.md @@ -0,0 +1,167 @@ +# bulk_extractor release procedure + +This procedure defines how to prepare, validate, assemble, and publish a +versioned `bulk_extractor` release. Use it with the release issue and retain +the completed evidence there. Do not record case data, unredacted scan output, +credentials, or presigned URLs in GitHub. + +## Release model + +A normal pull request runs CI and may create short-lived test artifacts. It +must not publish release assets or use package-signing credentials. A release +PR is the sole place to prepare a versioned release: it updates the version in +`configure.ac`, release notes, package metadata, and any procedure changes. + +Draft pull requests are planning/release-preparation records. Required CI jobs +skip them and run when the pull request is marked ready for review. GitHub still +records a skipped workflow run; Actions cannot filter `pull_request` events on +the draft field before a workflow starts. Pushes to `dev-release` do not run the +main-branch push workflows. + +After the release PR is merged and all required checks are green, create a +signed annotated tag named `vX.Y.Z` at the reviewed commit. The tag is the +immutable release identity; never move or reuse it. A protected tag-triggered +workflow should create a *draft* GitHub Release. The release manager reviews +the evidence below and explicitly publishes the draft. + +Use a `release/X.Y` branch only when maintaining an established release line +while `main` continues with new development. Do not create a release branch for +an ordinary single-release cycle. + +## Prerequisites + +Before beginning, open a release issue and record: + +- target version, tag, and reviewed commit SHA; +- the release manager and package-repository maintainers; +- approved large-image identifiers and checksums, without exposing case data; +- URLs for all build and validation jobs; +- every artifact filename, SHA-256 checksum, provenance, and publication URL. + +Confirm that the release PR has a clean required-check status and that its +version matches the intended tag. Do not release from an unreviewed local +checkout or from a mutable branch reference. + +## Build and validation gates + +Complete and record each gate before artifact publication. + +1. On a clean local macOS installation, bootstrap, configure, build, and run + `make distcheck`. Record macOS version, architecture, compiler, commands, + and logs. +2. On that Mac, scan an approved large disk image. Record the immutable image + identifier/checksum, command line, elapsed time, peak resources, exit + status, and output checksum/location. Never upload confidential image or + output content to GitHub. +3. Run `make distcheck-ubuntu-container`. It builds a clean Ubuntu 22.04 + container and runs `make distcheck` for both `linux/arm64` and + `linux/amd64` by default. +4. Run `make distcheck-fedora-container`. It builds a clean Fedora 44 container + and runs `make distcheck` for both `linux/arm64` and `linux/amd64` by + default. Together, `make distcheck-containers` runs both operating-system + gates. +5. Run `.github/workflows/mingw.yml` and verify the Windows Unicode filename + test against the exact `bulk_extractor64.exe` artifact. The current build + uses `--disable-libewf`; it is neither signed nor E01-capable unless the + workflow and its evidence are changed accordingly. +6. Run the AWS large-image validation described below, then preserve only its + redacted result summary, checksums, and pass/fail evidence. + +## AWS large-image validation + +The AWS validation infrastructure is tracked in [#624][aws-issue]. Until that +work exists, this gate is manual and must not be represented as automated. + +The eventual CloudFormation or SAM implementation must: + +- use a fixed instance type, maximum runtime, and automatic instance/volume + cleanup; +- use encrypted S3, scoped IAM roles, and GitHub OIDC or presigned URLs rather + than long-lived repository credentials; +- redact image and output data; post only an attested summary, logs, checksums, + and pass/fail status to GitHub; +- have AWS Budget alerts and a separate runtime/cost guardrail. Budgets are + alerts, not instantaneous hard spending caps, so the runtime guardrail is + required to keep expected CPU costs at or below $10. + +Confirm that the instance, volumes, and temporary S3 objects have been removed +before marking the gate complete. + +## Artifact assembly + +`make release` is deliberately an assembly gate, not a credential-bearing +publisher. It runs `make distcheck`, stages the source archive plus validated +inputs in `release-artifacts/`, and writes `SHA256SUMS`. It refuses to proceed +when an input is absent. + +`make release` also runs `make distcheck-containers`, so Finch is a release +prerequisite on macOS. On Apple Silicon, `arm64` runs natively and `amd64` uses +Finch's Linux emulation. The emulated run is slower and is compatibility +coverage rather than a performance result. Use +`CONTAINER_PLATFORMS=arm64` when diagnosing a native-only failure, or +`CONTAINER_PLATFORMS=amd64` when reproducing the emulated x86_64 result. + +Finch is the default engine because it is available in this development +environment. The targets use the portable `build --platform` interface; a +developer who has started Colima and installed the Docker CLI may instead run +them with `CONTAINER_ENGINE=docker`. Use one active engine per run; do not +assume a stopped Colima profile can serve Finch commands. + +Provide the artifact paths explicitly: + +```sh +make release \ + RELEASE_WINDOWS_EXE=/path/to/bulk_extractor64.exe \ + RELEASE_DEB=/path/to/bulk_extractor_VERSION_ARCH.deb \ + RELEASE_RPM=/path/to/bulk_extractor-VERSION-RELEASE.ARCH.rpm \ + RELEASE_AWS_RESULT=/path/to/aws-large-image-summary.txt +``` + +The staged set must contain: + +| Artifact | Current producer | Required release evidence | +| --- | --- | --- | +| Source archive | `make distcheck` | Version and source-tag match | +| Windows `.exe` | MinGW workflow | Exact artifact passed Windows test | +| Debian `.deb` | [#622][deb-issue] | Clean package build and installed-package smoke test | +| RPM/SRPM | [#623][rpm-issue] | Clean RPM build and installed-package smoke test | +| AWS summary | [#624][aws-issue] | Redacted large-image build/scan result and cleanup evidence | +| `SHA256SUMS` | `make release` | Checksums verified before upload | + +Before upload, verify each staged filename, version, checksum, and provenance +against the signed tag. A source archive or package built from any other commit +is a release failure. + +## Publication + +1. Attach the staged artifacts and `SHA256SUMS` to the draft GitHub Release. +2. Check the release notes, tag, commit, asset names, checksums, and links. +3. Submit signed Debian artifacts to the selected Debian repository and record + the resulting URL. The reproducible build and repository choice are tracked + in [#622][deb-issue]. +4. Submit signed RPM artifacts to the selected RPM repository and record the + resulting URL. The reproducible build and repository choice are tracked in + [#623][rpm-issue]. +5. Publish the GitHub Release only after a release manager has reviewed every + required gate and external package publication is either complete or clearly + disclosed in the release notes. + +## Automation roadmap + +The following release-engineering issues close the remaining automation gaps: + +- [#621][github-release-issue]: tag-driven draft GitHub Release creation, + Windows artifact assembly, and checksums. +- [#622][deb-issue]: reproducible Debian builds, signing, and publication. +- [#623][rpm-issue]: reproducible RPM builds, signing, and publication. +- [#624][aws-issue]: budget-capped AWS large-image validation and secure + reporting. + +Once these are complete, a release manager should be able to direct Codex to +prepare a release PR, validate the tag, trigger the protected workflow, and +review the resulting draft release without handling package or cloud secrets. + +[github-release-issue]: https://github.com/simsong/bulk_extractor/issues/621 +[deb-issue]: https://github.com/simsong/bulk_extractor/issues/622 +[rpm-issue]: https://github.com/simsong/bulk_extractor/issues/623 +[aws-issue]: https://github.com/simsong/bulk_extractor/issues/624 diff --git a/etc/Dockerfile.fedora-distcheck b/etc/Dockerfile.fedora-distcheck new file mode 100644 index 000000000..074468195 --- /dev/null +++ b/etc/Dockerfile.fedora-distcheck @@ -0,0 +1,24 @@ +FROM fedora:44 + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +RUN dnf -y install \ + abseil-cpp-devel autoconf automake ca-certificates exiv2-devel \ + expat-devel flex gcc-c++ libgcrypt-devel libtool re2-devel \ + libxml2-devel make openssl-devel pkgconf-pkg-config sqlite-devel wget \ + zlib-devel \ + && dnf clean all + +ARG LIBEWF_URL=https://github.com/libyal/libewf-legacy/releases/download/20140814/libewf-20140814.tar.gz +RUN wget -q "$LIBEWF_URL" -O /tmp/libewf.tar.gz \ + && tar -C /tmp -xzf /tmp/libewf.tar.gz \ + && cd /tmp/libewf-* \ + && ./configure --quiet --enable-silent-rules --prefix=/usr/local \ + && make -j"$(nproc)" \ + && make install \ + && ldconfig \ + && rm -rf /tmp/libewf /tmp/libewf-* /tmp/libewf.tar.gz + +WORKDIR /src +COPY . /src +RUN bash bootstrap.sh && ./configure --quiet && make distcheck diff --git a/etc/Dockerfile.ubuntu-distcheck b/etc/Dockerfile.ubuntu-distcheck new file mode 100644 index 000000000..ed03da3c7 --- /dev/null +++ b/etc/Dockerfile.ubuntu-distcheck @@ -0,0 +1,24 @@ +FROM ubuntu:22.04 + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + autoconf automake ca-certificates flex g++ libabsl-dev libexiv2-dev \ + libexpat1-dev libgcrypt20-dev libre2-dev libssl-dev libtool \ + libxml2-utils make pkg-config wget zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* + +ARG LIBEWF_URL=https://github.com/libyal/libewf-legacy/releases/download/20140814/libewf-20140814.tar.gz +RUN wget -q "$LIBEWF_URL" -O /tmp/libewf.tar.gz \ + && tar -C /tmp -xzf /tmp/libewf.tar.gz \ + && cd /tmp/libewf-* \ + && ./configure --quiet --enable-silent-rules --prefix=/usr/local \ + && make -j"$(nproc)" \ + && make install \ + && ldconfig \ + && rm -rf /tmp/libewf /tmp/libewf-* /tmp/libewf.tar.gz + +WORKDIR /src +COPY . /src +RUN bash bootstrap.sh && ./configure --quiet && make distcheck From f1cccb54dafbc685f844788707e0e86f30909ce2 Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 15:45:52 -0400 Subject: [PATCH 2/9] Isolate release orchestration in a script --- Makefile.am | 49 +++---------------------- doc/RELEASE_PROCEDURE.md | 14 +++++-- scripts/release.sh | 79 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 46 deletions(-) create mode 100755 scripts/release.sh diff --git a/Makefile.am b/Makefile.am index 6b843166e..662559a1e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -24,6 +24,7 @@ EXTRA_DIST = $(SRC_WIN_DIST) $(AUTO_DOC_FILES) $(AUTO_ETC_FILES) $(AUTO_LICENSES $(srcdir)/.dockerignore \ $(srcdir)/etc/Dockerfile.fedora-distcheck \ $(srcdir)/etc/Dockerfile.ubuntu-distcheck \ + $(srcdir)/scripts/release.sh \ $(srcdir)/AGENTS.md \ $(srcdir)/.gitignore \ $(srcdir)/CODING_STANDARDS.md \ @@ -90,25 +91,14 @@ libinstall: install -D python/module/bulkextractor.py $(prefix)/lib/python2.7/dist-packages/bulkextractor.py if [ -r /sbin/ldconfig ]; /sbin/ldconfig -v ; fi -# Release assembly is deliberately fail-closed. Artifact producers are -# independently validated release jobs; this target stages their outputs and -# the source distribution only when each required input is present. +# Release orchestration is deliberately fail-closed and isolated from the +# active checkout. See scripts/release.sh and doc/RELEASE_PROCEDURE.md. RELEASE_ARTIFACT_DIR ?= $(abs_builddir)/release-artifacts +RELEASE_SCRIPT = $(abs_srcdir)/scripts/release.sh -.PHONY: release release-source release-linux-distcheck release-windows release-deb release-rpm release-aws-result release-checksums +.PHONY: release release: - $(MAKE) release-source - $(MAKE) release-linux-distcheck - $(MAKE) release-windows - $(MAKE) release-deb - $(MAKE) release-rpm - $(MAKE) release-aws-result - $(MAKE) release-checksums - -release-source: - $(MAKE) distcheck - mkdir -p $(RELEASE_ARTIFACT_DIR) - cp $(PACKAGE)-$(VERSION).tar.gz $(RELEASE_ARTIFACT_DIR)/ + RELEASE_ARTIFACT_DIR="$(RELEASE_ARTIFACT_DIR)" RELEASE_SOURCE_DIR="$(abs_srcdir)" "$(RELEASE_SCRIPT)" # Each Finch build copies a clean source context and runs the real distcheck # inside it. Finch selects native arm64 and emulated amd64 on Apple Silicon. @@ -129,30 +119,3 @@ distcheck-fedora-container: done distcheck-containers: distcheck-ubuntu-container distcheck-fedora-container - -release-linux-distcheck: - $(MAKE) distcheck-containers - -release-windows: - test -n "$(RELEASE_WINDOWS_EXE)" && test -s "$(RELEASE_WINDOWS_EXE)" - mkdir -p $(RELEASE_ARTIFACT_DIR) - cp "$(RELEASE_WINDOWS_EXE)" $(RELEASE_ARTIFACT_DIR)/bulk_extractor64.exe - -release-deb: - test -n "$(RELEASE_DEB)" && test -s "$(RELEASE_DEB)" - mkdir -p $(RELEASE_ARTIFACT_DIR) - cp "$(RELEASE_DEB)" $(RELEASE_ARTIFACT_DIR)/ - -release-rpm: - test -n "$(RELEASE_RPM)" && test -s "$(RELEASE_RPM)" - mkdir -p $(RELEASE_ARTIFACT_DIR) - cp "$(RELEASE_RPM)" $(RELEASE_ARTIFACT_DIR)/ - -release-aws-result: - test -n "$(RELEASE_AWS_RESULT)" && test -s "$(RELEASE_AWS_RESULT)" - mkdir -p $(RELEASE_ARTIFACT_DIR) - cp "$(RELEASE_AWS_RESULT)" $(RELEASE_ARTIFACT_DIR)/ - -release-checksums: - rm -f $(RELEASE_ARTIFACT_DIR)/SHA256SUMS - cd $(RELEASE_ARTIFACT_DIR) && shasum -a 256 * > SHA256SUMS diff --git a/doc/RELEASE_PROCEDURE.md b/doc/RELEASE_PROCEDURE.md index 7c0e83bf7..be71108a6 100644 --- a/doc/RELEASE_PROCEDURE.md +++ b/doc/RELEASE_PROCEDURE.md @@ -90,9 +90,14 @@ before marking the gate complete. ## Artifact assembly `make release` is deliberately an assembly gate, not a credential-bearing -publisher. It runs `make distcheck`, stages the source archive plus validated -inputs in `release-artifacts/`, and writes `SHA256SUMS`. It refuses to proceed -when an input is absent. +publisher. It invokes `scripts/release.sh`, which preflights every required +input, creates a detached temporary Git worktree at `HEAD`, and runs bootstrap, +macOS `make distcheck`, and container gates there. It captures logs and source +provenance, stages the source archive and validated inputs in +`release-artifacts/`, writes `SHA256SUMS`, then removes the worktree. It refuses +to run from a checkout with tracked or staged changes, to overwrite an existing +artifact directory, or to proceed when an input is absent. The active checkout +is not built, configured, or cleaned by the release process. `make release` also runs `make distcheck-containers`, so Finch is a release prerequisite on macOS. On Apple Silicon, `arm64` runs natively and `amd64` uses @@ -117,6 +122,9 @@ make release \ RELEASE_AWS_RESULT=/path/to/aws-large-image-summary.txt ``` +To use a different empty staging directory, pass +`RELEASE_ARTIFACT_DIR=/absolute/path` to `make release`. + The staged set must contain: | Artifact | Current producer | Required release evidence | diff --git a/scripts/release.sh b/scripts/release.sh new file mode 100755 index 000000000..7fc5b0610 --- /dev/null +++ b/scripts/release.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# Assemble a release in an isolated worktree. Invoked only by `make release`. +set -euo pipefail + +source_dir=${RELEASE_SOURCE_DIR:?make release must set RELEASE_SOURCE_DIR} +artifact_dir=${RELEASE_ARTIFACT_DIR:?make release must set RELEASE_ARTIFACT_DIR} +required_artifacts=(RELEASE_WINDOWS_EXE RELEASE_DEB RELEASE_RPM RELEASE_AWS_RESULT) + +for variable in "${required_artifacts[@]}"; do + value=${!variable:-} + if [[ -z $value || ! -s $value ]]; then + printf 'ERROR: %s must name a non-empty validated artifact\n' "$variable" >&2 + exit 2 + fi +done + +if [[ -e $artifact_dir ]]; then + printf 'ERROR: release artifact directory already exists: %s\n' "$artifact_dir" >&2 + exit 2 +fi + +release_root=$(mktemp -d "${TMPDIR:-/tmp}/bulk-extractor-release.XXXXXX") +release_src="$release_root/source" +log_dir="$artifact_dir/logs" + +cleanup() { + git -C "$source_dir" worktree remove --force "$release_src" 2>/dev/null || true + rm -rf "$release_root" +} +trap cleanup EXIT + +run_logged() { + local name=$1 + shift + printf '==> %s\n' "$name" + "$@" 2>&1 | tee "$log_dir/$name.log" +} + +git -C "$source_dir" diff --quiet || { + printf 'ERROR: tracked changes in the source checkout; commit them before release\n' >&2 + exit 2 +} +git -C "$source_dir" diff --cached --quiet || { + printf 'ERROR: staged changes in the source checkout; commit them before release\n' >&2 + exit 2 +} + +mkdir -p "$log_dir" +git -C "$source_dir" worktree add --detach "$release_src" HEAD + +run_logged bootstrap bash -c 'cd "$1" && bash bootstrap.sh' _ "$release_src" +run_logged configure bash -c 'cd "$1" && ./configure --quiet' _ "$release_src" +run_logged macos-distcheck make -C "$release_src" distcheck +run_logged linux-container-distcheck make -C "$release_src" distcheck-containers + +archive=$(find "$release_src" -maxdepth 1 -type f -name 'bulk_extractor-*.tar.gz' -print -quit) +if [[ -z $archive ]]; then + printf 'ERROR: distcheck did not produce a source archive\n' >&2 + exit 1 +fi + +cp "$archive" "$artifact_dir/" +cp "$RELEASE_WINDOWS_EXE" "$artifact_dir/bulk_extractor64.exe" +cp "$RELEASE_DEB" "$artifact_dir/" +cp "$RELEASE_RPM" "$artifact_dir/" +cp "$RELEASE_AWS_RESULT" "$artifact_dir/" + +{ + printf 'source_commit=%s\n' "$(git -C "$release_src" rev-parse HEAD)" + printf 'created_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf 'artifact_directory=%s\n' "$artifact_dir" +} > "$artifact_dir/RELEASE_METADATA.txt" + +(cd "$artifact_dir" && shasum -a 256 \ + "$(basename "$archive")" bulk_extractor64.exe \ + "$(basename "$RELEASE_DEB")" "$(basename "$RELEASE_RPM")" \ + "$(basename "$RELEASE_AWS_RESULT")" RELEASE_METADATA.txt > SHA256SUMS) + +printf 'Release artifacts staged in %s\n' "$artifact_dir" From f4b2025280972fabbd9d3db1a5eb93ef4388fea2 Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 16:44:55 -0400 Subject: [PATCH 3/9] Skip Debian CI job for draft pull requests --- .github/workflows/ci-cd.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/ci-cd.yml b/.github/workflows/ci-cd.yml index 38265db91..fbf4865b4 100644 --- a/.github/workflows/ci-cd.yml +++ b/.github/workflows/ci-cd.yml @@ -19,6 +19,7 @@ permissions: jobs: debian: + if: github.event_name != 'pull_request' || !github.event.pull_request.draft name: Debian Bookworm build runs-on: ubuntu-latest From e2735214a92f2d505738f2b1de600bb44291de04 Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 16:57:57 -0400 Subject: [PATCH 4/9] Document distribution source-package submissions --- doc/RELEASE_NOTES.md | 6 ++++++ doc/RELEASE_PROCEDURE.md | 39 ++++++++++++++++++++++++++++----------- 2 files changed, 34 insertions(+), 11 deletions(-) diff --git a/doc/RELEASE_NOTES.md b/doc/RELEASE_NOTES.md index 12cb2b4be..53a8a6aa9 100644 --- a/doc/RELEASE_NOTES.md +++ b/doc/RELEASE_NOTES.md @@ -212,6 +212,12 @@ through the project's normal pull-request and CI process. ([PR #561](https://github.com/simsong/bulk_extractor/pull/561), [PR #558](https://github.com/simsong/bulk_extractor/pull/558), [PR #557](https://github.com/simsong/bulk_extractor/pull/557)). +- Documented a controlled release procedure and release-issue template, with + isolated artifact assembly, macOS and container `distcheck` gates, and + source-level downstream submission paths for Debian/Kali and + Fedora/openSUSE ([#621](https://github.com/simsong/bulk_extractor/issues/621), + [#622](https://github.com/simsong/bulk_extractor/issues/622), + [#623](https://github.com/simsong/bulk_extractor/issues/623)). ### Known limitations and release work diff --git a/doc/RELEASE_PROCEDURE.md b/doc/RELEASE_PROCEDURE.md index be71108a6..b755c2727 100644 --- a/doc/RELEASE_PROCEDURE.md +++ b/doc/RELEASE_PROCEDURE.md @@ -131,8 +131,8 @@ The staged set must contain: | --- | --- | --- | | Source archive | `make distcheck` | Version and source-tag match | | Windows `.exe` | MinGW workflow | Exact artifact passed Windows test | -| Debian `.deb` | [#622][deb-issue] | Clean package build and installed-package smoke test | -| RPM/SRPM | [#623][rpm-issue] | Clean RPM build and installed-package smoke test | +| Debian `.deb` | [#622][deb-issue] | Clean package build and installed-package smoke test; direct-download artifact only | +| RPM/SRPM | [#623][rpm-issue] | Clean RPM build and installed-package smoke test; direct-download artifact only | | AWS summary | [#624][aws-issue] | Redacted large-image build/scan result and cleanup evidence | | `SHA256SUMS` | `make release` | Checksums verified before upload | @@ -144,24 +144,41 @@ is a release failure. 1. Attach the staged artifacts and `SHA256SUMS` to the draft GitHub Release. 2. Check the release notes, tag, commit, asset names, checksums, and links. -3. Submit signed Debian artifacts to the selected Debian repository and record - the resulting URL. The reproducible build and repository choice are tracked - in [#622][deb-issue]. -4. Submit signed RPM artifacts to the selected RPM repository and record the - resulting URL. The reproducible build and repository choice are tracked in - [#623][rpm-issue]. -5. Publish the GitHub Release only after a release manager has reviewed every +3. Publish the signed upstream source archive and its checksums. Distribution + archives build their own binaries: do not submit a prebuilt `.deb` or RPM to + Debian, Kali, Fedora, or openSUSE as an archive update. +4. Submit the Debian *source package* through an authorized Debian maintainer + or sponsor and record the accepted source-package URL. Kali normally imports + from Debian; when it carries packaging changes, submit a version-bump request + or merge request against Kali's packaging repository using the same tagged + upstream source. The reproducible build and downstream tracking are in + [#622][deb-issue]. +5. Submit source-level RPM packaging changes: use the Fedora package-review or + dist-git workflow as applicable, and fork the openSUSE package repository to + submit a Gitea pull request containing the updated spec, source reference, + and changelog. Let the distribution build service build and archive the RPMs; + record the accepted request URL. The reproducible build and downstream + tracking are in [#623][rpm-issue]. +6. Publish the GitHub Release only after a release manager has reviewed every required gate and external package publication is either complete or clearly disclosed in the release notes. +The release may attach locally built `.deb`, RPM, and SRPM files to GitHub for +download and installation testing. Those files are not substitutes for the +source-package submissions above. Distribution archives apply their own signing +and build policies; keep any contributor or maintainer credentials outside this +repository and outside `make release`. + ## Automation roadmap The following release-engineering issues close the remaining automation gaps: - [#621][github-release-issue]: tag-driven draft GitHub Release creation, Windows artifact assembly, and checksums. -- [#622][deb-issue]: reproducible Debian builds, signing, and publication. -- [#623][rpm-issue]: reproducible RPM builds, signing, and publication. +- [#622][deb-issue]: reproducible Debian source/binary builds and Debian/Kali + source-package submissions. +- [#623][rpm-issue]: reproducible RPM builds and Fedora/openSUSE source-level + packaging submissions. - [#624][aws-issue]: budget-capped AWS large-image validation and secure reporting. From d79d91bfee1a601189045a3dfa873b9c3086b493 Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 17:01:47 -0400 Subject: [PATCH 5/9] Harden release preparation workflow --- Makefile.am | 1 - doc/RELEASE_NOTES.md | 3 ++- doc/RELEASE_PROCEDURE.md | 16 ++++++++++++---- scripts/release.sh | 18 ++++++++++++++++-- 4 files changed, 30 insertions(+), 8 deletions(-) diff --git a/Makefile.am b/Makefile.am index ffc7d37a6..f73abe1b6 100644 --- a/Makefile.am +++ b/Makefile.am @@ -21,7 +21,6 @@ SUBDIRS = doc doc/latex_manuals man src src/tests specfiles tests include Makefile.auto_defs EXTRA_DIST = $(SRC_WIN_DIST) $(AUTO_DOC_FILES) $(AUTO_ETC_FILES) $(AUTO_LICENSES) \ - $(srcdir)/.dockerignore \ $(srcdir)/etc/Dockerfile.fedora-distcheck \ $(srcdir)/etc/Dockerfile.ubuntu-distcheck \ $(srcdir)/scripts/release.sh \ diff --git a/doc/RELEASE_NOTES.md b/doc/RELEASE_NOTES.md index 53a8a6aa9..a4e56fa8a 100644 --- a/doc/RELEASE_NOTES.md +++ b/doc/RELEASE_NOTES.md @@ -217,7 +217,8 @@ through the project's normal pull-request and CI process. source-level downstream submission paths for Debian/Kali and Fedora/openSUSE ([#621](https://github.com/simsong/bulk_extractor/issues/621), [#622](https://github.com/simsong/bulk_extractor/issues/622), - [#623](https://github.com/simsong/bulk_extractor/issues/623)). + [#623](https://github.com/simsong/bulk_extractor/issues/623), + [#626](https://github.com/simsong/bulk_extractor/issues/626)). ### Known limitations and release work diff --git a/doc/RELEASE_PROCEDURE.md b/doc/RELEASE_PROCEDURE.md index b755c2727..13070d5a8 100644 --- a/doc/RELEASE_PROCEDURE.md +++ b/doc/RELEASE_PROCEDURE.md @@ -91,10 +91,12 @@ before marking the gate complete. `make release` is deliberately an assembly gate, not a credential-bearing publisher. It invokes `scripts/release.sh`, which preflights every required -input, creates a detached temporary Git worktree at `HEAD`, and runs bootstrap, +input, requires `RELEASE_SOURCE_DIR` to name a Git worktree, creates a detached +temporary worktree at `HEAD`, and runs bootstrap, macOS `make distcheck`, and container gates there. It captures logs and source -provenance, stages the source archive and validated inputs in -`release-artifacts/`, writes `SHA256SUMS`, then removes the worktree. It refuses +provenance, stages the source archive and supplied artifact inputs in +`release-artifacts/`, writes `SHA256SUMS` with `shasum` or `sha256sum`, then +removes the worktree. It refuses to run from a checkout with tracked or staged changes, to overwrite an existing artifact directory, or to proceed when an input is absent. The active checkout is not built, configured, or cleaned by the release process. @@ -151,7 +153,10 @@ is a release failure. or sponsor and record the accepted source-package URL. Kali normally imports from Debian; when it carries packaging changes, submit a version-bump request or merge request against Kali's packaging repository using the same tagged - upstream source. The reproducible build and downstream tracking are in + upstream source. Ubuntu normally imports Debian packages into Universe before + its Debian Import Freeze; after that point, record the required Launchpad + sync request. Do not pursue Ubuntu's default third-party-source list for this + free-software tool. The reproducible build and downstream tracking are in [#622][deb-issue]. 5. Submit source-level RPM packaging changes: use the Fedora package-review or dist-git workflow as applicable, and fork the openSUSE package repository to @@ -181,6 +186,8 @@ The following release-engineering issues close the remaining automation gaps: packaging submissions. - [#624][aws-issue]: budget-capped AWS large-image validation and secure reporting. +- [#626][snap-issue]: optional project-owned Snap Store publication for Ubuntu + users; it does not replace the Debian-to-Ubuntu path. Once these are complete, a release manager should be able to direct Codex to prepare a release PR, validate the tag, trigger the protected workflow, and @@ -190,3 +197,4 @@ review the resulting draft release without handling package or cloud secrets. [deb-issue]: https://github.com/simsong/bulk_extractor/issues/622 [rpm-issue]: https://github.com/simsong/bulk_extractor/issues/623 [aws-issue]: https://github.com/simsong/bulk_extractor/issues/624 +[snap-issue]: https://github.com/simsong/bulk_extractor/issues/626 diff --git a/scripts/release.sh b/scripts/release.sh index 7fc5b0610..a45d0f07e 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -6,10 +6,15 @@ source_dir=${RELEASE_SOURCE_DIR:?make release must set RELEASE_SOURCE_DIR} artifact_dir=${RELEASE_ARTIFACT_DIR:?make release must set RELEASE_ARTIFACT_DIR} required_artifacts=(RELEASE_WINDOWS_EXE RELEASE_DEB RELEASE_RPM RELEASE_AWS_RESULT) +if [[ $(git -C "$source_dir" rev-parse --is-inside-work-tree 2>/dev/null || true) != true ]]; then + printf 'ERROR: RELEASE_SOURCE_DIR must name a Git worktree: %s\n' "$source_dir" >&2 + exit 2 +fi + for variable in "${required_artifacts[@]}"; do value=${!variable:-} if [[ -z $value || ! -s $value ]]; then - printf 'ERROR: %s must name a non-empty validated artifact\n' "$variable" >&2 + printf 'ERROR: %s must name a non-empty artifact\n' "$variable" >&2 exit 2 fi done @@ -71,7 +76,16 @@ cp "$RELEASE_AWS_RESULT" "$artifact_dir/" printf 'artifact_directory=%s\n' "$artifact_dir" } > "$artifact_dir/RELEASE_METADATA.txt" -(cd "$artifact_dir" && shasum -a 256 \ +if command -v shasum >/dev/null 2>&1; then + checksum_command=(shasum -a 256) +elif command -v sha256sum >/dev/null 2>&1; then + checksum_command=(sha256sum) +else + printf 'ERROR: neither shasum nor sha256sum is available\n' >&2 + exit 2 +fi + +(cd "$artifact_dir" && "${checksum_command[@]}" \ "$(basename "$archive")" bulk_extractor64.exe \ "$(basename "$RELEASE_DEB")" "$(basename "$RELEASE_RPM")" \ "$(basename "$RELEASE_AWS_RESULT")" RELEASE_METADATA.txt > SHA256SUMS) From 61773d9efff5d055398106492679f79990143de8 Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 18:21:01 -0400 Subject: [PATCH 6/9] Consolidate historical roadmaps into release notes --- doc/RELEASE_NOTES.md | 28 +++- doc/ROADMAP_1.4.md | 314 ----------------------------------------- doc/ROADMAP_1.5.md | 311 ----------------------------------------- doc/ROADMAP_1.6.md | 261 ---------------------------------- doc/ROADMAP_2.0.md | 324 ------------------------------------------- src/TECH_DEBT.md | 12 +- 6 files changed, 26 insertions(+), 1224 deletions(-) delete mode 100644 doc/ROADMAP_1.4.md delete mode 100644 doc/ROADMAP_1.5.md delete mode 100644 doc/ROADMAP_1.6.md delete mode 100644 doc/ROADMAP_2.0.md diff --git a/doc/RELEASE_NOTES.md b/doc/RELEASE_NOTES.md index a4e56fa8a..969cc6392 100644 --- a/doc/RELEASE_NOTES.md +++ b/doc/RELEASE_NOTES.md @@ -7,6 +7,10 @@ not exhaustive changelogs. Some legacy announcement dates disagree with tag dates; the version history below therefore uses dates only where the repository history is clear. +The historical roadmaps were planning documents, not release records. Their +relevant context is incorporated below; preserved, commit-specific copies are +linked from the [historical source map](#historical-source-map). + ## 2.2.0 (draft) **Status:** Unreleased. The source version is currently @@ -271,9 +275,10 @@ and production-oriented defaults replaced the broader experimental posture of version 1. The rewrite also reorganized scanner and feature-recorder APIs, improved DFXML -reporting and testability, and introduced the initial version-2 E01 path. See -the contemporary [2.0 roadmap](ROADMAP_2.0.md) for the design goals; it is a -planning record rather than a final release announcement. +reporting and testability, and introduced the initial version-2 E01 path. Its +planning record proposed C++14, continuous integration, systematic unit and +end-to-end tests, and a narrower command-line distribution. It also identified +future scanner and carving work that was not a release commitment. ### Maintenance through 2.0.3 @@ -308,6 +313,8 @@ placement for the 64-bit executable was corrected. The surviving [1.6.0 announcement](announce/announce_1.6.0.md) is explicitly a pre-release draft. This summary combines it with the repository `ChangeLog` and tagged source rather than treating that draft as a final historical record. +The contemporary roadmap also recorded longer-term work on scanner development, +carving, testing, and BEViewer; those proposals were not commitments for 1.6.0. ### 1.5.x @@ -324,7 +331,9 @@ Encryption, added RFC 4648 Base64 handling, and introduced the MSXML scanner for Microsoft Office Open XML documents. See the original [1.5 announcement](announce/announce_1.5.md) and -[1.5.2 notes](announce/announce_1.5.2.md). +[1.5.2 notes](announce/announce_1.5.2.md). The associated roadmap carried +forward proposals for future scanners, carvers, testing, and user-interface +work; it does not define the shipped 1.5.x scope. ### 1.4.x @@ -343,7 +352,9 @@ Command-line compatibility changed: block size and word-size controls moved from refactored. See the original [1.4 announcement](announce/announce_1.4.txt) and -[1.4.1 notes](announce/announce_1.4.1.txt). +[1.4.1 notes](announce/announce_1.4.1.txt). The associated roadmap recorded +planned scanner, raw-device, restart, and validation work, including proposals +deferred to later releases. ## Historical source map @@ -351,7 +362,10 @@ See the original [1.4 announcement](announce/announce_1.4.txt) and version-2 development periods. - [`doc/announce`](announce/) contains announcements for 1.2, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.2, 1.6.0, and 2.1.0. -- [`doc/ROADMAP_2.0.md`](ROADMAP_2.0.md) records the goals and migration plan for - the version-2 rewrite. +- The original planning records are preserved at the commit that last contained + them: [1.4 roadmap](https://github.com/simsong/bulk_extractor/blob/d79d91bfee1a601189045a3dfa873b9c3086b493/doc/ROADMAP_1.4.md), + [1.5 roadmap](https://github.com/simsong/bulk_extractor/blob/d79d91bfee1a601189045a3dfa873b9c3086b493/doc/ROADMAP_1.5.md), + [1.6 roadmap](https://github.com/simsong/bulk_extractor/blob/d79d91bfee1a601189045a3dfa873b9c3086b493/doc/ROADMAP_1.6.md), + and [2.0 roadmap](https://github.com/simsong/bulk_extractor/blob/d79d91bfee1a601189045a3dfa873b9c3086b493/doc/ROADMAP_2.0.md). - Git tags and their trees remain the authoritative source for the exact code in each release. diff --git a/doc/ROADMAP_1.4.md b/doc/ROADMAP_1.4.md deleted file mode 100644 index 43064fd6c..000000000 --- a/doc/ROADMAP_1.4.md +++ /dev/null @@ -1,314 +0,0 @@ -============================================================================== -Bulk Extractor 1.4. Feature Freeze: 1 JUN 2013. Release: 1 AUG 2013 -============================================================================== - -OTHER REFERENCES: -- https://github.com/simsong/bulk_extractor/issues - -DOCUMENTATION: -- document how to write a new scanner and add it to the mainstream. - -BUGFIXES: -- scan_net occasionally throws exceptions. Find out why and stop it. -- scan_net does not properly report timestamps -+ scan_pdf should use multiple strategies for extracting text. - -FEATURES: -+ Inverting bytes - -+ Work with windows raw-device (e.g. \\.\physicaldrive0 ) - when run as Administrator. - - http://msdn.microsoft.com/en-us/library/aa363858(v=vs.85).aspx - -+ Track number of bytes processed - -+ Construction of a stop-list from standard installs of OS and Apps - -+ Replaced hacky XML reading in restart with a proper Expat-based parser. - -+ Fixed exception throwing in MyFlexLexer.h so that msg is properly passed as *what(). - -On Hold: - -- scanner for emails and usernames. "Simson L. Garfinkel" - -- improved testing and validation of CMU LIFT software - -- Support for checkpointing using BLCR. - -- slg: scan_net.cpp - replace all buffer arithmetic with sbuf pointer get. - - Figure out why this is causing a assertion failure: - - /Users/simsong/domex/src/bulk_extractor/trunk/src/bulk_extractor -Z -o out4 -j1 -Y 7805599744 /corp/nps/drives/nps-2011-2tb/nps-2011-2tb.E01 -- simplify beregex_vector, word_and_context_list, and regex_list into a single structure. - -- Integrate Digital Assembly video carving - -- Filter mode - reads from stdin and writes from stdout. -- It's not BASE64 unless you have at least X characters from above 16; -- It's not BASE16 unless you have at least X characters from above 10 -- It's not BASE85 unless you have at least X characters from above 64 - -- scan_rar — integrate JHUAPL code - detect the presence of RAR-compressed data, report it, - and recursively re-process it. Handles both RAR and RAR2 - -- represent all files examined in report.xml file (.001,.002, etc.) - -- Windows shortcut files & IE history - -- Improved regression testing for release: - - bulk_diff.py - - identify_files.py - - Benchmark testing for execution against reference disk images - -- Escape processing to search term histogram - -- Improved restarting, so that each page is retried once. - (Retry it if we see a single start in the XML file but not two starts.) - -- Make sure identify_filenames will not process histogram files and it should produce an excel file. - -- Performance optimization - -- Add NIST hacking case to regression testing. - -- UTF-16 email addresses sometimes have the last character removed; figure out why and fix. - -- Add the classification label of media from .E01 files into the Feature file as a comment. -EWF files have a Notes field in which a classification label may be placed. -This field may be filled with classification labels such as UNCLASSIFIED//FOUO. -bulk_extractor may detect this field and forward a corresponding comment -in generated Feature files such as "# CLASSIFICATION: UNCLASSIFIED". -Classification comments may also be inserted into Feature files using the "-b" banner option. - -BEViewer (Requested but not assigned): -- Display the file path, if there is one, of selected Features. -We may use fiwalk and identify_filenames to additionally display the file -associated with the Feature that is currently navigated to. - -- Revise, document and deploy multi-drive correlator - -================================================================ -Bulk Extractor 1.5: Sometime in 2014 -================================================================ - - -- scan_windir: - - Add support for MBR and GPRT decoding (can we just hijack the SleuthKit code?) - - - -============================================================================== -Bulk Extractor 2.0. Sometime in 2016? -============================================================================== - - -- Source code scanner - - Will this be part of scan_lift? - -- Carvers: - - MPEG carving (Integrate results of Digital Assembly work) - - AVI carving - - Carve iCalendar entries - -- 7Zip Scanner (scan_lzma) - -- Timestamp scanner - -- scan_lzma — detect the presence of LZMA-compressed data, report it, - and recursively re-process it. (Model scan_zip). - -- scan_bzip2 — detect the presence of bzip2-compressed data, report - it, and recursively re-process it. (Model scan_zip). - -- scan_msi — detect the presence of MSI-compressed data, report it, - and recursively re-process it. Find the code for MSI compression in - The Unarchiver. (Model scan_zip). - -- scan_cab — detect the presence of CAB-compressed data, report it, - and recursively re-process it. Find the code for CAB compression in - The Unarchiver. (Model scan_zip). - -- scan_ntfs — detect the presence of NTFS-compressed data, report it, - and recursively re-process it. This is especially difficult because - NTFS compression has no magic numbers, so trial compression needs - to be done! (Model scan_hiber). - -- scan_mime — Some way to handle two MIME quoting problems — =\n - should be replaced by “”, and =40 should be replaced by “@”. But - should all “=” escapements be handled? - - This will handle: - - user@loc= - alhost - - user=40localhost - - loc^M - alhost - -- scan_sqlite — Find, identify, and validate sqlite databases. Ideally - does carving of unallocated SQLITE pages. - -- Modify DFXML so that absolute path of disk image is reported. - http://stackoverflow.com/questions/143174/c-c-how-to-obtain-the-full-path-of-current-directory - -- make feature_recroder::get_name raise an exception rather than aborting? - -- Update scan_net to carve PPP packets (alegedly common with 3G and 4G modem cards) - -- Python bridge, so scanners can be written in python - - Requires that each Python interperter be run in its own address space, - as python is not thread-safe - -- C# bridge, so scanners can be written in C# - -- Codepage / CJKV identification - - typically Windows-Codepage 1252 and / or UTF-8 - -- Human Language identification. - - Identify the kind of language that's present. - - http://sourceforge.net/projects/la-strings/ - - http://lucene.apache.org/nutch/apidocs-0.8.x/org/apache/nutch/analysis/lang/LanguageIdentifier.html - - http://github.com/vcl/cue.language - - http://alias-i.com/lingpipe/demos/tutorial/langid/read-me.html - - http://textcat.sourceforge.net/ - -- Explore integration of http://itextpdf.com/itext.php for PDF text extraction. - - rewrite scan_pdf? - -- Allow bulk_extractor to scan just unallocated area. - - Unallocated lists can come from: - 1 - Real-time analysis of disk using sleuthkit - 2 - DFXML file - 3 - list of blocks from sleuthkit blk_find - - Not clear we want to this in bulk_extractor, rather than just having it scan from stdin? - -- More options for suppression: - - Suppress known sectors (hash list of sector hashes?) - -- Improve documentation - - Document the feature file syntax - - The syntax of Feature files will be documented. - - Basically: We have Feature Files and Histogram Files. - - These files have tab-delineated data. - - BOM is ignored. - - Lines starting with "#" are ignored. - - Entries in most Feature files contain three fields: - - 1) Offset in decimal or else a forensic path, - - 2) the Feature (which might be XML) - - 3) the "context." (which might be XML) - - Entries in gps.txt and exif.txt contain three fields: 1) offset, 2) MD5SUM, 3) formatted content. - - Entries in Histogram files contain two fields: - - 1) histogram count prefixed by "n=" and - - 2) the Feature. - - All bytes below space (" ") are converted to Octal and are escaped with "\". - -- scan_winprefetch - - Add ability to extract executable's location from prefetch hash value - http://www.woanware.co.uk/?page_id=173 - - - Ability to detect and analyze SuperFetch files - https://forensics.wiki/superfetch - -- scan_plist - - create. Give it the ability to find and decode Mac plist files (binary and XML) - -- scan_im: - - Skype - - Pidgon - - Google Talk - - Yahoo! Messenger including decryption (XOR of the @yahoo account name) - - QQ Messenger including decryption (Blowfish with the key being the QQ account number?) - - etc. - -- Windows Jump List scanner? - -- VM detection? ie: - - VirtualBox; VMware; QEMU/KVM; Parallels; Virtual PC - -============================================================================== -Possible Projects -============================================================================== -- new scanner for Windows iedownloadhistory index.dat file contents -File /users//appdata/roaming/microsoft/windows/iedownloadhistory/index.dat -contains download history and timestamp information from IE9. - -Here is the data structure, contributed by Fornzix on linux_forensics on 6/26/12: -1. Records show up as gibberish until the computer is restarted for -some reason. Even shutting down IE9 didn't help. After the restart, -the records are readable. -2. Individual download records are sized in multiples of 128 bytes -(896,1024,1152,1280,.....). -3. Individual downloads start with "URL" (bytes 1-3). -4. Byte 4 = unknown. -5. Byte 5-6 = These two bytes make a 16 bit Integer which is the -length of the record in 128 byte chunks (i.e. hex 0B 00 = 11, and 11 x -128 = 1408, which is the total record length from "URL" to #12 below). -6. Bytes 17-24 = 8 byte Windows Date / Time when the download -finished. -7. Bytes 81-84 = 4 byte DOS (GMT) Time when download finished (funny -though... it's a few 1000's of a second longer than bytes 17-24) -8. Bytes 193-200 = 8 byte Windows Date / Time when the download -finished. (same as bytes 17-24) -9. Byte 469 = Start of download URL "http". -10. Three hex "00" in a row separate the end of the download URL from -the beginning of the location saved to on the hard drive. -11. There are three hex "00" at the end of the location where the file -was stored on the hard drive. -12. The remainder of the record, which could be considered 'slack -space' is taken up with hex EF:BE:AD:DE which is "DEADBEEF". - -================================================================ -TESTING -================================================================ - -Bulk_extractor needs a systematic approach to internal unit tests and -overall system tests. - -Unit Tests: - -sbuf_t - tests - - test each constructor & destructors - - test find and copy - -Input/Ouput Testing -regress.py - currently runs bulk_extractor on a few test images - - Add code to validate output - -path-printer - - - Test bulk_extractor program to extract known items from known disk images. - - Use the nps-emails disk iamge - case 1 - output a given page - case 2 - output a subset of a given page - case 3 - output a forensic path with a GZIP - case 4 - output a forensic path with a BASE64 - -open source memory testing tools - -Input / Output Validation: Validate that with a given known input that the output has been properly produced. --IO Test Case 1: (Based on B. Allen's suggestion) Start with a union data set - i.e. collect the results of all -BE identified features, then using BEViewer to inspect the features. --- Goals: Identification of error rates: false positives, false negatives - -Performance Testing: -- PT Test Case 1: Enabled All --- Objective: Test the overall performance of bulk extractor with regards to memory utilization, cpu utilization, -and overall execution time on a chosen data set ---- Goals: Characterization of Bulk Extractor and all scanners enabled - -- PT Test Case 2: Individual Scanner --- Objective: Test the individual scanner with bulk extractor to characterize memory utilization, cpu utilization, -and execution time on a chosen data set ---- Goals: Characterization of individual scanners to ascertain the performance of an individual scanner - -Security Evaluation Testing: -- SET Test Case 1: Fortify Testing --- Objective: Taking bulk extractor source code and evaluating if the code baseline has vulnerabilities. --- Goals: Identification and corrections of any security issues - - - diff --git a/doc/ROADMAP_1.5.md b/doc/ROADMAP_1.5.md deleted file mode 100644 index c71a0241a..000000000 --- a/doc/ROADMAP_1.5.md +++ /dev/null @@ -1,311 +0,0 @@ -============================================================================== -Bulk Extractor 1.5.4 Feature Freeze: TBD Release: TBD -============================================================================== - -OTHER REFERENCES: -- https://github.com/simsong/bulk_extractor/issues - -DOCUMENTATION: -- document how to write a new scanner and add it to the mainstream. - -BUGFIXES: -- scan_net occasionally throws exceptions. Find out why and stop it. -- scan_net does not properly report timestamps -+ scan_pdf should use multiple strategies for extracting text. - -FEATURES: -+ Inverting bytes - -+ Work with windows raw-device (e.g. \\.\physicaldrive0 ) - when run as Administrator. - - http://msdn.microsoft.com/en-us/library/aa363858(v=vs.85).aspx - -+ Track number of bytes processed - -+ Construction of a stop-list from standard installs of OS and Apps - -+ Replaced hacky XML reading in restart with a proper Expat-based parser. - -+ Fixed exception throwing in MyFlexLexer.h so that msg is properly passed as *what(). - -On Hold: - -- scanner for emails and usernames. "Simson L. Garfinkel" - -- improved testing and validation of CMU LIFT software - -- Support for checkpointing using BLCR. - -- slg: scan_net.cpp - replace all buffer arithmetic with sbuf pointer get. - - Figure out why this is causing a assertion failure: - - /Users/simsong/domex/src/bulk_extractor/trunk/src/bulk_extractor -Z -o out4 -j1 -Y 7805599744 /corp/nps/drives/nps-2011-2tb/nps-2011-2tb.E01 -- simplify beregex_vector, word_and_context_list, and regex_list into a single structure. - -- Integrate Digital Assembly video carving - -- Filter mode - reads from stdin and writes from stdout. -- It's not BASE64 unless you have at least X characters from above 16; -- It's not BASE16 unless you have at least X characters from above 10 -- It's not BASE85 unless you have at least X characters from above 64 - -- scan_rar — integrate JHUAPL code - detect the presence of RAR-compressed data, report it, - and recursively re-process it. Handles both RAR and RAR2 - -- represent all files examined in report.xml file (.001,.002, etc.) - -- Windows shortcut files & IE history - -- Improved regression testing for release: - - bulk_diff.py - - identify_files.py - - Benchmark testing for execution against reference disk images - -- Escape processing to search term histogram - -- Improved restarting, so that each page is retried once. - (Retry it if we see a single start in the XML file but not two starts.) - -- Make sure identify_filenames will not process histogram files and it should produce an excel file. - -- Performance optimization - -- Add NIST hacking case to regression testing. - -- UTF-16 email addresses sometimes have the last character removed; figure out why and fix. - -- Add the classification label of media from .E01 files into the Feature file as a comment. -EWF files have a Notes field in which a classification label may be placed. -This field may be filled with classification labels such as UNCLASSIFIED//FOUO. -bulk_extractor may detect this field and forward a corresponding comment -in generated Feature files such as "# CLASSIFICATION: UNCLASSIFIED". -Classification comments may also be inserted into Feature files using the "-b" banner option. - -BEViewer (Requested but not assigned): -- Display the file path, if there is one, of selected Features. -We may use fiwalk and identify_filenames to additionally display the file -associated with the Feature that is currently navigated to. - -- Revise, document and deploy multi-drive correlator - -================================================================ -Bulk Extractor 1.5: Sometime in 2014 -================================================================ - - -- scan_windir: - - Add support for MBR and GPRT decoding (can we just hijack the SleuthKit code?) - - - -============================================================================== -Bulk Extractor 2.0. Sometime in 2016? -============================================================================== - - -- Source code scanner - - Will this be part of scan_lift? - -- Carvers: - - MPEG carving (Integrate results of Digital Assembly work) - - AVI carving - - Carve iCalendar entries - -- 7Zip Scanner (scan_lzma) - -- Timestamp scanner - -- scan_lzma — detect the presence of LZMA-compressed data, report it, - and recursively re-process it. (Model scan_zip). - -- scan_bzip2 — detect the presence of bzip2-compressed data, report - it, and recursively re-process it. (Model scan_zip). - -- scan_msi — detect the presence of MSI-compressed data, report it, - and recursively re-process it. Find the code for MSI compression in - The Unarchiver. (Model scan_zip). - -- scan_cab — detect the presence of CAB-compressed data, report it, - and recursively re-process it. Find the code for CAB compression in - The Unarchiver. (Model scan_zip). - -- scan_ntfs — detect the presence of NTFS-compressed data, report it, - and recursively re-process it. This is especially difficult because - NTFS compression has no magic numbers, so trial compression needs - to be done! (Model scan_hiber). - -- scan_mime — Some way to handle two MIME quoting problems — =\n - should be replaced by “”, and =40 should be replaced by “@”. But - should all “=” escapements be handled? - - This will handle: - - user@loc= - alhost - - user=40localhost - - loc^M - alhost - -- scan_sqlite — Find, identify, and validate sqlite databases. Ideally - does carving of unallocated SQLITE pages. - -- Modify DFXML so that absolute path of disk image is reported. - http://stackoverflow.com/questions/143174/c-c-how-to-obtain-the-full-path-of-current-directory - -- make feature_recroder::get_name raise an exception rather than aborting? - -- Update scan_net to carve PPP packets (alegedly common with 3G and 4G modem cards) - -- Python bridge, so scanners can be written in python - - Requires that each Python interperter be run in its own address space, - as python is not thread-safe - -- C# bridge, so scanners can be written in C# - -- Codepage / CJKV identification - - typically Windows-Codepage 1252 and / or UTF-8 - -- Human Language identification. - - Identify the kind of language that's present. - - http://sourceforge.net/projects/la-strings/ - - http://lucene.apache.org/nutch/apidocs-0.8.x/org/apache/nutch/analysis/lang/LanguageIdentifier.html - - http://github.com/vcl/cue.language - - http://alias-i.com/lingpipe/demos/tutorial/langid/read-me.html - - http://textcat.sourceforge.net/ - -- Explore integration of http://itextpdf.com/itext.php for PDF text extraction. - - rewrite scan_pdf? - -- Allow bulk_extractor to scan just unallocated area. - - Unallocated lists can come from: - 1 - Real-time analysis of disk using sleuthkit - 2 - DFXML file - 3 - list of blocks from sleuthkit blk_find - - Not clear we want to this in bulk_extractor, rather than just having it scan from stdin? - -- More options for suppression: - - Suppress known sectors (hash list of sector hashes?) - -- Improve documentation - - Document the feature file syntax - - The syntax of Feature files will be documented. - - Basically: We have Feature Files and Histogram Files. - - These files have tab-delineated data. - - BOM is ignored. - - Lines starting with "#" are ignored. - - Entries in most Feature files contain three fields: - - 1) Offset in decimal or else a forensic path, - - 2) the Feature (which might be XML) - - 3) the "context." (which might be XML) - - Entries in gps.txt and exif.txt contain three fields: 1) offset, 2) MD5SUM, 3) formatted content. - - Entries in Histogram files contain two fields: - - 1) histogram count prefixed by "n=" and - - 2) the Feature. - - All bytes below space (" ") are converted to Octal and are escaped with "\". - -- scan_winprefetch - - Add ability to extract executable's location from prefetch hash value - http://www.woanware.co.uk/?page_id=173 - - - Ability to detect and analyze SuperFetch files - https://forensics.wiki/superfetch - -- scan_plist - - create. Give it the ability to find and decode Mac plist files (binary and XML) - -- scan_im: - - Skype - - Pidgon - - Google Talk - - Yahoo! Messenger including decryption (XOR of the @yahoo account name) - - QQ Messenger including decryption (Blowfish with the key being the QQ account number?) - - etc. - -- Windows Jump List scanner? - -- VM detection? ie: - - VirtualBox; VMware; QEMU/KVM; Parallels; Virtual PC - -============================================================================== -Possible Projects -============================================================================== -- new scanner for Windows iedownloadhistory index.dat file contents -File /users//appdata/roaming/microsoft/windows/iedownloadhistory/index.dat -contains download history and timestamp information from IE9. - -Here is the data structure, contributed by Fornzix on linux_forensics on 6/26/12: -1. Records show up as gibberish until the computer is restarted for -some reason. Even shutting down IE9 didn't help. After the restart, -the records are readable. -2. Individual download records are sized in multiples of 128 bytes -(896,1024,1152,1280,.....). -3. Individual downloads start with "URL" (bytes 1-3). -4. Byte 4 = unknown. -5. Byte 5-6 = These two bytes make a 16 bit Integer which is the -length of the record in 128 byte chunks (i.e. hex 0B 00 = 11, and 11 x -128 = 1408, which is the total record length from "URL" to #12 below). -6. Bytes 17-24 = 8 byte Windows Date / Time when the download -finished. -7. Bytes 81-84 = 4 byte DOS (GMT) Time when download finished (funny -though... it's a few 1000's of a second longer than bytes 17-24) -8. Bytes 193-200 = 8 byte Windows Date / Time when the download -finished. (same as bytes 17-24) -9. Byte 469 = Start of download URL "http". -10. Three hex "00" in a row separate the end of the download URL from -the beginning of the location saved to on the hard drive. -11. There are three hex "00" at the end of the location where the file -was stored on the hard drive. -12. The remainder of the record, which could be considered 'slack -space' is taken up with hex EF:BE:AD:DE which is "DEADBEEF". - -================================================================ -TESTING -================================================================ - -Bulk_extractor needs a systematic approach to internal unit tests and -overall system tests. - -Unit Tests: - -sbuf_t - tests - - test each constructor & destructors - - test find and copy - -Input/Ouput Testing -regress.py - currently runs bulk_extractor on a few test images - - Add code to validate output - -path-printer - - - Test bulk_extractor program to extract known items from known disk images. - - Use the nps-emails disk iamge - case 1 - output a given page - case 2 - output a subset of a given page - case 3 - output a forensic path with a GZIP - case 4 - output a forensic path with a BASE64 - -open source memory testing tools - -Input / Output Validation: Validate that with a given known input that the output has been properly produced. --IO Test Case 1: (Based on B. Allen's suggestion) Start with a union data set - i.e. collect the results of all -BE identified features, then using BEViewer to inspect the features. --- Goals: Identification of error rates: false positives, false negatives - -Performance Testing: -- PT Test Case 1: Enabled All --- Objective: Test the overall performance of bulk extractor with regards to memory utilization, cpu utilization, -and overall execution time on a chosen data set ---- Goals: Characterization of Bulk Extractor and all scanners enabled - -- PT Test Case 2: Individual Scanner --- Objective: Test the individual scanner with bulk extractor to characterize memory utilization, cpu utilization, -and execution time on a chosen data set ---- Goals: Characterization of individual scanners to ascertain the performance of an individual scanner - -Security Evaluation Testing: -- SET Test Case 1: Fortify Testing --- Objective: Taking bulk extractor source code and evaluating if the code baseline has vulnerabilities. --- Goals: Identification and corrections of any security issues diff --git a/doc/ROADMAP_1.6.md b/doc/ROADMAP_1.6.md deleted file mode 100644 index 82ec7bc84..000000000 --- a/doc/ROADMAP_1.6.md +++ /dev/null @@ -1,261 +0,0 @@ -============================================================================== -Bulk Extractor 1.6.0 Release: November 2019 -============================================================================== - -See Also --------- - -- https://github.com/simsong/bulk_extractor/issues - -ON THE ROADMAP --------------- - -DOCUMENTATION: -- document how to write a new scanner and add it to the mainstream. - -SCANNER FEATURES: - -- scanner for emails and usernames. "Simson L. Garfinkel" - -- simplify beregex_vector, word_and_context_list, and regex_list into a single structure. - -- Filter mode - reads from stdin and writes from stdout. - -- Sanity checks for BASE16, BASE64 and BASE85scanners - -- scan_rar — Upgrade to RAR5 - -- Windows shortcut files & IE history - -- Improved regression testing for release: - - bulk_diff.py - - identify_files.py - - Benchmark testing for execution against reference disk images - -- Escape processing to search term histogram - -- Make sure identify_filenames will not process histogram files and it should produce an excel file. -- Performance optimization - -- Add NIST hacking case to regression testing. - -- UTF-16 email addresses sometimes have the last character removed; figure out why and fix. - -- Rewrite BEViewer as HTML/JavaScript application - -- Display the file path, if there is one, of selected Features. -We may use fiwalk and identify_filenames to additionally display the file -associated with the Feature that is currently navigated to. - -- Revise, document and deploy multi-drive correlator - -- scan_windir: - - Add support for MBR and GPRT decoding (can we just hijack the SleuthKit code?) - -- Carvers: - - MPEG carving (Integrate results of Digital Assembly work) - - AVI carving - - Carve iCalendar entries - -- 7Zip Scanner (scan_lzma) - -- Timestamp scanner - -- scan_lzma — detect the presence of LZMA-compressed data, report it, - and recursively re-process it. (Model scan_zip). - -- scan_bzip2 — detect the presence of bzip2-compressed data, report - it, and recursively re-process it. (Model scan_zip). - -- scan_msi — detect the presence of MSI-compressed data, report it, - and recursively re-process it. Find the code for MSI compression in - The Unarchiver. (Model scan_zip). - -- scan_cab — detect the presence of CAB-compressed data, report it, - and recursively re-process it. Find the code for CAB compression in - The Unarchiver. (Model scan_zip). - -- scan_ntfs — detect the presence of NTFS-compressed data, report it, - and recursively re-process it. This is especially difficult because - NTFS compression has no magic numbers, so trial compression needs - to be done! (Model scan_hiber). - -- scan_mime — Some way to handle two MIME quoting problems — =\n - should be replaced by “”, and =40 should be replaced by “@”. But - should all “=” escapements be handled? - - This will handle: - - user@loc= - alhost - - user=40localhost - - loc^M - alhost - -- Python bridge, so scanners can be written in python - - Requires that each Python interperter be run in its own address space, - as python is not thread-safe - -- C# bridge, so scanners can be written in C# - -- Codepage / CJKV identification - - typically Windows-Codepage 1252 and / or UTF-8 - -- Human Language identification. - - Identify the kind of language that's present. - - http://sourceforge.net/projects/la-strings/ - - http://lucene.apache.org/nutch/apidocs-0.8.x/org/apache/nutch/analysis/lang/LanguageIdentifier.html - - http://github.com/vcl/cue.language - - http://alias-i.com/lingpipe/demos/tutorial/langid/read-me.html - - http://textcat.sourceforge.net/ - -- Explore integration of http://itextpdf.com/itext.php for PDF text extraction. - - rewrite scan_pdf? - -- Allow bulk_extractor to scan just unallocated area. - - Unallocated lists can come from: - 1 - Real-time analysis of disk using sleuthkit - 2 - DFXML file - 3 - list of blocks from sleuthkit blk_find - - Not clear we want to this in bulk_extractor, rather than just having it scan from stdin? - -- More options for suppression: - - Suppress known sectors (hash list of sector hashes?) - -- Improve documentation - - Document the feature file syntax - - The syntax of Feature files will be documented. - - Basically: We have Feature Files and Histogram Files. - - These files have tab-delineated data. - - BOM is ignored. - - Lines starting with "#" are ignored. - - Entries in most Feature files contain three fields: - - 1) Offset in decimal or else a forensic path, - - 2) the Feature (which might be XML) - - 3) the "context." (which might be XML) - - Entries in gps.txt and exif.txt contain three fields: 1) offset, 2) MD5SUM, 3) formatted content. - - Entries in Histogram files contain two fields: - - 1) histogram count prefixed by "n=" and - - 2) the Feature. - - All bytes below space (" ") are converted to Octal and are escaped with "\". - -- scan_winprefetch - - Add ability to extract executable's location from prefetch hash value - http://www.woanware.co.uk/?page_id=173 - - - Ability to detect and analyze SuperFetch files - https://forensics.wiki/superfetch - -- scan_plist - - create. Give it the ability to find and decode Mac plist files (binary and XML) - -- scan_im: - - Skype - - Pidgon - - Google Talk - - Yahoo! Messenger including decryption (XOR of the @yahoo account name) - - QQ Messenger including decryption (Blowfish with the key being the QQ account number?) - - etc. - -- Windows Jump List scanner? - -- VM detection? ie: - - VirtualBox; VMware; QEMU/KVM; Parallels; Virtual PC - -============================================================================== -Possible Projects -============================================================================== -- new scanner for Windows iedownloadhistory index.dat file contents -File /users//appdata/roaming/microsoft/windows/iedownloadhistory/index.dat -contains download history and timestamp information from IE9. - -Here is the data structure, contributed by Fornzix on linux_forensics on 6/26/12: -1. Records show up as gibberish until the computer is restarted for -some reason. Even shutting down IE9 didn't help. After the restart, -the records are readable. - -2. Individual download records are sized in multiples of 128 bytes -(896,1024,1152,1280,.....). - -3. Individual downloads start with "URL" (bytes 1-3). - -4. Byte 4 = unknown. - -5. Byte 5-6 = These two bytes make a 16 bit Integer which is the -length of the record in 128 byte chunks (i.e. hex 0B 00 = 11, and 11 x -128 = 1408, which is the total record length from "URL" to #12 below). - -6. Bytes 17-24 = 8 byte Windows Date / Time when the download -finished. - -7. Bytes 81-84 = 4 byte DOS (GMT) Time when download finished (funny -though... it's a few 1000's of a second longer than bytes 17-24) - -8. Bytes 193-200 = 8 byte Windows Date / Time when the download -finished. (same as bytes 17-24) - -9. Byte 469 = Start of download URL "http". - -10. Three hex "00" in a row separate the end of the download URL from -the beginning of the location saved to on the hard drive. - -11. There are three hex "00" at the end of the location where the file -was stored on the hard drive. - -12. The remainder of the record, which could be considered 'slack -space' is taken up with hex EF:BE:AD:DE which is "DEADBEEF". - -================================================================ -TESTING -================================================================ - -Bulk_extractor needs a systematic approach to internal unit tests and -overall system tests. - -Unit Tests: - -sbuf_t - tests - - test each constructor & destructors - - test find and copy - -Input/Ouput Testing -regress.py - currently runs bulk_extractor on a few test images - - Add code to validate output - -path-printer - - - Test bulk_extractor program to extract known items from known disk images. - - Use the nps-emails disk iamge - case 1 - output a given page - case 2 - output a subset of a given page - case 3 - output a forensic path with a GZIP - case 4 - output a forensic path with a BASE64 - -open source memory testing tools - -Input / Output Validation: Validate that with a given known input that the output has been properly produced. --IO Test Case 1: (Based on B. Allen's suggestion) Start with a union data set - i.e. collect the results of all -BE identified features, then using BEViewer to inspect the features. --- Goals: Identification of error rates: false positives, false negatives - -Performance Testing: -- PT Test Case 1: Enabled All --- Objective: Test the overall performance of bulk extractor with regards to memory utilization, cpu utilization, -and overall execution time on a chosen data set ---- Goals: Characterization of Bulk Extractor and all scanners enabled - -- PT Test Case 2: Individual Scanner --- Objective: Test the individual scanner with bulk extractor to characterize memory utilization, cpu utilization, -and execution time on a chosen data set ---- Goals: Characterization of individual scanners to ascertain the performance of an individual scanner - -Security Evaluation Testing: -- SET Test Case 1: Fortify Testing --- Objective: Taking bulk extractor source code and evaluating if the code baseline has vulnerabilities. --- Goals: Identification and corrections of any security issues - - - diff --git a/doc/ROADMAP_2.0.md b/doc/ROADMAP_2.0.md deleted file mode 100644 index a2b53f81d..000000000 --- a/doc/ROADMAP_2.0.md +++ /dev/null @@ -1,324 +0,0 @@ -Bulk Extractor 2.0. Planned Release: January 1, 2022 -========================================================= - -This is the planning document for bulk_extractor version 2.0. - -Bulk_extractor was a funded project of the US Government from 2006 -through 2014. The project's development is being continued by the -development team on a volunteer basis. As such, the goals and intended -feature set of version 2.0 is being significantly re-scoped. - -Goals for the 2.0 release -------------------------- - -* Production quality. Version 1.x of bulk_extractor was a research - tool that also found usefuleness in operational settings. Verison - 2.x is a production tool. As such: - - - Research scanners have been removed from the master branch. They - can still be researched by making them shared-libraries and using - the bulk_extractor plug-in system. - - - Unit tests have been added. - -* Improved software development practices. - - - Continious integration is employed to validate each commit. - - - Development will take place in feature branches which will be - added to the master branch only if CI tests pass. - -* Sensible defaults for production operation. With the undertanding - that most users do not understand command-line options, - bulk_extractor now runs with fewer command-line options. - -* Standards-compliant. Where possible, we are adopting C++14 features - that are now widely available. - -* Experimental features have been removed. Experiments are now - conducted with plugin-s. - -* BE2.0 will be released as a pure command-line tool. The user - interface with the windows installer (and embedded CLI) will be - released afterwards. - -* SQL will be turned on by default and the program will provide the - user with instructions on how to use it. Performance will be - analyzed to determine the fastest way to create the text feature - files, the SQLite3 database, and the histograms. - -* Include other easy-to-output feature files by default, such as - collect all email messages. - -* Integration with The Sleuth Kit for file enumeration - -# Architecture Changes -- [ ] be13_api will build a .a library -- [ ] be13_api will include tests and have its own makefile system. -- [ ] dfxml will be a submodule of be13_api -- [ ] Find all of the files first, scan them in file-block order, then scan all of the remaining blocks. We can do this on a 1TB disk with 4K sectors. That's 256 million 4k blocks, or 32MiB of RAM, which is fine. - -Release 2.0 Implementation Plan -------------------------------- -Bulk_extractor depends on the [be13_api](https://github.com/simsong/be13_api) and on [dfxml](https://github.com/simsong/dfxml). In the past, functionaly requirements were specified, code was written, testing was done, and reasonably reliable software was pushed out to users. - -BE2 is adopting test-driven-development and continious integration. The goal is to make testing more efficient (because it will happen all the time) move more effort into the design and development, allowing more time for performance improvements and regression testing. - -Most of the development on BE was done from 1993 to 2010. As a result, the code-base is a combination of legacy C, C++, and C++ with the Standard Template Library. Between 2005 and 2015 there was significant development of the C++11 and C++14 standards. Support for C++14 is now widespread, so BE2 will require C++14 for compiling and deployment. This means that many lines of code that were written for BE1 can now be replaced by a single line of code, thanks to C++14. - -BE2 is not a complete rewrite of BE1, but it is a **significant** rewrite. - -Organizational change: - -* [DFXML](https://github.com/simsong/dfxml) can now be independently tested with unit tests. This is done by `make check` and is also done by the CI system. - -* [be13_api](https://github.com/simsong/be13_api) can now be independently tested. Because BE13_API depends on DFXML, dfxml is a sub-module of be13_api. This is a change from 1.0, in which be13_api referenced DFXML but didn't include it. - -* Because BE doesn't need two copies of DFXML, it no longer has DFXML as a sub-module. - -Here is the proposed plan for development of Release 2.0. This list will be transferred to a GitHub issue at some point. - -- [ ] Complete stand-alone tests for DFXML. -- [ ] Port DFXML to C++14. -- [ ] Complete stand-alone tests for BE13_API with no modules actually being tested. -- [ ] Expand the coverage of BE13_API tests for each *.cpp file. -- [ ] Complete a stand-alone tester for a BE scanner. -- [ ] Create a unit test for each BE2 scanner -- [ ] Create a unit test for the BE2 framework. -- [ ] Create a unit test for the full BE2 system to compile -- [ ] Implement additional scanners as desired. -- [ ] Begin end-to-end regression testing, making sure that all features identified by BE1.5 are still identified by BE2.0 -- [ ] Begin performance testing. - -Features Removed from Release 2.0 -================================= -## Remove Scanners -These scanners are removed. Each can be used if you make it a plug-in: -* scan_lift -* scedan -* hashdb - -## Other removed functionality -* Support for AFF and AFF4 input file formats. - -## Compiling on non-standard platforms -Where possible, C++14 - -Release 2.0 Features -==================== - -## New scanners: -* iCalendar carver -* Email message carver. - -## Ideas on hold -These would make good capstone or master's projects: - -- MPEG carver - -- AVI carver - -- 7Zip Scanner (scan_lzma) - -- Timestamp scanner - -- scan_windir: - - Add support for MBR and GPRT decoding (can we just hijack the SleuthKit code?) - -- scan_lzma — detect the presence of LZMA-compressed data, report it, - and recursively re-process it. (Model scan_zip). - -- scan_bzip2 — detect the presence of bzip2-compressed data, report - it, and recursively re-process it. (Model scan_zip). - -- scan_msi — detect the presence of MSI-compressed data, report it, - and recursively re-process it. Find the code for MSI compression in - The Unarchiver. (Model scan_zip). - -- scan_cab — detect the presence of CAB-compressed data, report it, - and recursively re-process it. Find the code for CAB compression in - The Unarchiver. (Model scan_zip). - -- scan_ntfs — detect the presence of NTFS-compressed data, report it, - and recursively re-process it. This is especially difficult because - NTFS compression has no magic numbers, so trial compression needs - to be done! (Model scan_hiber). - -- Update scan_net to carve PPP packets (alegedly common with 3G and 4G modem cards) - -- C# bridge, so scanners can be written in C# - -- Codepage / CJKV identification - - typically Windows-Codepage 1252 and / or UTF-8 - -- Human Language identification. - - Identify the kind of language that's present. - - http://sourceforge.net/projects/la-strings/ - - http://lucene.apache.org/nutch/apidocs-0.8.x/org/apache/nutch/analysis/lang/LanguageIdentifier.html - - http://github.com/vcl/cue.language - - http://alias-i.com/lingpipe/demos/tutorial/langid/read-me.html - - http://textcat.sourceforge.net/ - -## Ideas for 2.0: - -- scan_mime — Unquote MIME messages and re-process them. - -- Modify DFXML so that absolute path of disk image is reported. - http://stackoverflow.com/questions/143174/c-c-how-to-obtain-the-full-path-of-current-directory - -- make feature_recorder::get_name raise an exception. - -- Python bridge, so scanners can be written in python - - Requires that each Python interperter be run in its own address space, - as python is not thread-safe - -- Explore integration of http://itextpdf.com/itext.php for PDF text extraction. - - rewrite scan_pdf? - -- Allow bulk_extractor to scan just unallocated area. - - Unallocated lists can come from: - 1 - Real-time analysis of disk using sleuthkit - 2 - DFXML file - 3 - list of blocks from sleuthkit blk_find - - Not clear we want to this in bulk_extractor, rather than just having it scan from stdin? - -- More options for suppression: - - Suppress known sectors (hash list of sector hashes?) - -- Improve documentation - - Document the feature file syntax - - The syntax of Feature files will be documented. - - Basically: We have Feature Files and Histogram Files. - - These files have tab-delineated data. - - BOM is ignored. - - Lines starting with "#" are ignored. - - Entries in most Feature files contain three fields: - - 1) Offset in decimal or else a forensic path, - - 2) the Feature (which might be XML) - - 3) the "context." (which might be XML) - - Entries in gps.txt and exif.txt contain three fields: 1) offset, 2) MD5SUM, 3) formatted content. - - Entries in Histogram files contain two fields: - - 1) histogram count prefixed by "n=" and - - 2) the Feature. - - All bytes below space (" ") are converted to Octal and are escaped with "\". - -- scan_winprefetch - - Add ability to extract executable's location from prefetch hash value - http://www.woanware.co.uk/?page_id=173 - - - Ability to detect and analyze SuperFetch files - http://www.forensicswiki.org/wiki/SuperFetch - -- scan_plist - - create. Give it the ability to find and decode Mac plist files (binary and XML) - -- scan_im: - - Skype - - Pidgon - - Google Talk - - Yahoo! Messenger including decryption (XOR of the @yahoo account name) - - QQ Messenger including decryption (Blowfish with the key being the QQ account number?) - - etc. - -- Windows Jump List scanner? - -- VM detection? ie: - - VirtualBox; VMware; QEMU/KVM; Parallels; Virtual PC - -- IE Download history [See notes](iehistory.txt) - -Testing -======= - -Bulk_extractor needs a systematic approach to internal unit tests and -overall system tests. - -## Unit Tests: - -sbuf_t - tests - - test each constructor & destructors - - test find and copy - -path-printer - - - Test bulk_extractor program to extract known items from known disk images. - - Use the nps-emails disk iamge - case 1 - output a given page - case 2 - output a subset of a given page - case 3 - output a forensic path with a GZIP - case 4 - output a forensic path with a BASE64 - -## End-to-end tests - -- Input/Ouput Testing - - regress.py - currently runs bulk_extractor on a few test images - - Add code to validate output - -- open source memory testing tools - -Input / Output Validation -Validate that with a given known input that the output has been properly produced. --IO Test Case 1: (Based on B. Allen's suggestion) Start with a union data set - i.e. collect the results of all -BE identified features, then using BEViewer to inspect the features. --- Goals: Identification of error rates: false positives, false negatives - -Performance Testing: -- PT Test Case 1: Enabled All --- Objective: Test the overall performance of bulk extractor with regards to memory utilization, cpu utilization, -and overall execution time on a chosen data set ---- Goals: Characterization of Bulk Extractor and all scanners enabled - -- PT Test Case 2: Individual Scanner --- Objective: Test the individual scanner with bulk extractor to characterize memory utilization, cpu utilization, -and execution time on a chosen data set ---- Goals: Characterization of individual scanners to ascertain the performance of an individual scanner - -Security Evaluation Testing: -- SET Test Case 1: Fortify Testing --- Objective: Taking bulk extractor source code and evaluating if the code baseline has vulnerabilities. --- Goals: Identification and corrections of any security issues - - -- scanner for emails and usernames. "Simson L. Garfinkel" - -- simplify beregex_vector, word_and_context_list, and regex_list into a single structure. - -- Sanity checks for BASE16, BASE64 and BASE85scanners - -- scan_rar — Upgrade to RAR5 - -- Windows shortcut files & IE history - -- Improved regression testing for release: - - bulk_diff.py - - identify_files.py - - Benchmark testing for execution against reference disk images - -- Escape processing to search term histogram - -- Make sure identify_filenames will not process histogram files and it should produce an excel file. -- Performance optimization - -- Add NIST hacking case to regression testing. - -- UTF-16 email addresses sometimes have the last character removed; figure out why and fix. - -- Rewrite BEViewer as HTML/JavaScript application - -- Display the file path, if there is one, of selected Features. -We may use fiwalk and identify_filenames to additionally display the file -associated with the Feature that is currently navigated to. - -- Revise, document and deploy multi-drive correlator - - - - - - -See Also -======== - -https://github.com/simsong/bulk_extractor/issues - diff --git a/src/TECH_DEBT.md b/src/TECH_DEBT.md index 496122b24..1e9cf824e 100644 --- a/src/TECH_DEBT.md +++ b/src/TECH_DEBT.md @@ -507,11 +507,9 @@ rather than retained beside current material. ### Historical planning/status documents -- `doc/ROADMAP_1.4.md`, `ROADMAP_1.5.md`, and `ROADMAP_1.6.md` are historical - records but are not labeled archival. -- `doc/ROADMAP_2.0.md` still says C++14, `master`, be13_api, working plug-ins, - SQL by default, and a dependency topology contradicted by the source. It also - mixes completed plans with unchecked work. +- The historical 1.4, 1.5, 1.6, and 2.0 roadmaps are consolidated into + `doc/RELEASE_NOTES.md`; its historical source map links immutable, + commit-specific copies for research. - `src/TODO.md` contains old crash traces and untriaged claims rather than an actionable current backlog. - `src/be20_api/TODO.md` is a 2021 work log containing completed, superseded, @@ -803,8 +801,8 @@ appropriate validation in its GitHub issue. - [x] Replace the obsolete programmer manual with the 2.x scanner and build manual. - [x] Remove the version-1.x user manuals, worked examples, and captured help. - [x] Remove the obsolete `src/be13_api` include path from `doc/Makefile.am`. -- [ ] Mark the 1.4, 1.5, and 1.6 roadmaps as historical. -- [ ] Mark or replace `ROADMAP_2.0.md`, which mixes obsolete plans and completed work. +- [x] Consolidate the 1.4, 1.5, 1.6, and 2.0 roadmaps into the release notes, + retaining commit-specific historical copies in its source map. - [ ] Convert root and be20_api TODO/status work logs into a current owned backlog or archive them. - [ ] Reconcile root and be20_api NEWS/ChangeLog histories into a coherent release history. - [x] Rewrite the be20_api README so it does not claim an unchanged 1.3-era API. From b7326e09cae2b1e1c589f40ec540629ab5f7e4f8 Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 18:27:19 -0400 Subject: [PATCH 7/9] Remove obsolete HTML overview --- doc/RELEASE_NOTES.md | 2 ++ doc/bulk_extractor.html | 64 ----------------------------------------- 2 files changed, 2 insertions(+), 64 deletions(-) delete mode 100644 doc/bulk_extractor.html diff --git a/doc/RELEASE_NOTES.md b/doc/RELEASE_NOTES.md index 969cc6392..c2bd38cac 100644 --- a/doc/RELEASE_NOTES.md +++ b/doc/RELEASE_NOTES.md @@ -216,6 +216,8 @@ through the project's normal pull-request and CI process. ([PR #561](https://github.com/simsong/bulk_extractor/pull/561), [PR #558](https://github.com/simsong/bulk_extractor/pull/558), [PR #557](https://github.com/simsong/bulk_extractor/pull/557)). +- Removed the unmaintained standalone HTML overview; the current LaTeX guide + and published documentation site are the supported user documentation. - Documented a controlled release procedure and release-issue template, with isolated artifact assembly, macOS and container `distcheck` gates, and source-level downstream submission paths for Debian/Kali and diff --git a/doc/bulk_extractor.html b/doc/bulk_extractor.html deleted file mode 100644 index 08ecb273b..000000000 --- a/doc/bulk_extractor.html +++ /dev/null @@ -1,64 +0,0 @@ -

bulk_extractor is a C++ program that scans a disk image, a file, or a -directory of files and extracts useful information -without parsing the file system or file system -structures. The results are stored in feature files that can be easily inspected, parsed, or processed with automated tools. bulk_extractor also created a histograms of features that it finds, as features that are more common tend to be more important.

- -

In addition to bulk_extractor, we have made available a small number of python programs that perform automated processing on the feature files.

- -

bulk_extractor is distinguished from other forensic tools -by its speed and thoroughness. Because it ignores file system structure, bulk_extractor can process different parts of the disk in parallel. In practice, the program splits the disk up into 16MiByte pages and processes one page on each available core. This means that 24-core machines process a disk roughly 24 times faster than a 1-core machine. bulk_extractor is also thorough. That's because bulk_extractor automatically detects, decompresses, and recursively re-processes compressed data that is compressed with a variety of algorithms. Our testing has shown that there is a significant amount of compressed data in the unallocated regions of file systems that is missed by most forensic tools that are commonly in use today.

- -

Another advantage of ignoring file systems is that bulk_extractor can be used to process any digital media. We have used the program to process hard drives, SSDs, optical media, camera cards, cell phones, network packet dumps, and other kinds of digital information.

- -

Output Feature Files

-

bulk_extractor now creates an output directory that has the following layout:

-
-
alerts.txt
Processing errors.
-
ccn.txt
Credit card numbers
-
ccn_track2.txt
Credit card "track 2" informaiton, which has previously been found in some bank card fraud cases.
-
domain.txt
Internet domains found on the drive, including dotted-quad addresses found in text.
-
email.txt
Email addresses.
-
ether.txt
Ethernet MAC addresses found through IP packet carving of swap files and compressed system hibernation files and file fragments.
-
exif.txt
EXIFs from JPEGs and video segments. This feature file contains all of the EXIF fields, expanded as XML records.
-
find.txt
The results of specific regular expression search requests.
-
ip.txt
IP addresses found through IP packet carving.
-
rfc822.txt
Email message headers including Date:, Subject: and Message-ID: fields.
-
tcp.txt
TCP flow information found through IP packet carving.
-
telephone.txt
US and international telephone numbers.
-
url.txt
URLs, typically found in browser caches, email messages, and pre-compiled into executables.
-
url_searches.txt
A histogram of terms used in Internet searches from services such as Google, Bing, Yahoo, and others.
-
url_services.txt
A histogram of the domain name portion of all the URLs found on the media.
-
wordlist.txt
A list of all "words" extracted from the disk, useful for password cracking.
-
wordlist_*.txt
The wordlist with duplicates removed, formatted in a form that can be easily imported into a popular password-cracking program.
-
zip.txt
A file containing information regarding every ZIP file component found on the media. This is exceptionally useful as ZIP files contain internal structure and ZIP is increasingly the compound file format of choice for a variety of products such as Microsoft Office
-
- -For each of the above, two additional files may be created: -
-
*_stopped.txt
bulk_extractor supports a stop list, or a list of items that do not need to be brought to the user's attention. However rather than simply suppressing this information, which might cause something critical to be hidden, stopped entries are stored in the stopped files.
-
*_histogram.txt
bulk_extractor can also create histograms of features. This is important, as experience has shown that email addresses, domain names, URLs, and other informaiton that appear more frequently on a hard drive or in a cell phone's memory can be used to rapidly create a pattern of life report.
-
- -Bulk extractor also creates a file that captures the provenance of the run: -
-
report.xml
A Digital Forensics XML report that includes information about the source media, how the bulk_extractor program was compiled and run, the time to process the digital evidence, and a meta report of the information that was found.
-
- -

Post-Processing

- -We have developed four programs for post-processing the bulk_extractor output: -
-
bulk_diff.py
This program reports the differences between two bulk_extractor runs. The intent is to image a computer, run bulk_extractor on a disk image, let the computer run for a period of time, re-image the computer, run bulk_extractor on the second image, and then report the differences. This can be used to infer the user's activities within a time period.
-
cda_tool.py
This tool, currently under development, reads multiple bulk_extractor reports from multiple runs against multiple drives and performs a multi-drive correlation using Garfinkel's Cross Drive Analysis technique. This can be used to automatically identify new social networks or to identify new members of existing networks.
-
file_location.py
In the bulk_extractor feature file, each feature is annotated with the byte offset from the beginning of the image in which it was found. The program takes as input a bulk_extractor feature file and a DFXML file containing the locations of each file on the drive (produced with Garfinkel's fiwalk program) and produces an annotated feature file that contains the offset, feature, and the file in which the feature was found.
-
make_context_stop_list.py
Although forensic analysts frequently make "stop lists"---for example, a lsit of email addresses that appear in the operating system and should therefore be ignored---such lists have a significant problem. Because it is relatively easy to get an email address into the binary of an open source application, ignoring all of these email addresses may make it possible to cloak email addresses from forensic analysis. Our solution is to create context-sensitive stop lists, in which the feature to be stopped is presented with the context in which it occures. The make_context_stop_list.py program takes the results of multiple bulk_extractor runs and creates a single context-sensitive stop list that can then be used to suppress features when found in a specific context. One such stop list constructed from Windows and Linux operating systems is available on the bulk extractor website. -
- - -

Demonstrations

-We have demonstrations of how bulk_extractor runs on several disks images from the NPS Realistic Corpus: - - From fed5755dbc67739822398cd09c60c2216d78c3fc Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 18:30:05 -0400 Subject: [PATCH 8/9] Update man page for bulk_extractor 2.2 --- doc/RELEASE_NOTES.md | 2 + man/bulk_extractor.1 | 448 ++++++++++++++++++++++--------------------- 2 files changed, 230 insertions(+), 220 deletions(-) diff --git a/doc/RELEASE_NOTES.md b/doc/RELEASE_NOTES.md index a4e56fa8a..5d2531ea2 100644 --- a/doc/RELEASE_NOTES.md +++ b/doc/RELEASE_NOTES.md @@ -212,6 +212,8 @@ through the project's normal pull-request and CI process. ([PR #561](https://github.com/simsong/bulk_extractor/pull/561), [PR #558](https://github.com/simsong/bulk_extractor/pull/558), [PR #557](https://github.com/simsong/bulk_extractor/pull/557)). +- Rewrote the installed `bulk_extractor(1)` manual for the 2.2 command-line + interface, current logging, scanner controls, and supported documentation. - Documented a controlled release procedure and release-issue template, with isolated artifact assembly, macOS and container `distcheck` gates, and source-level downstream submission paths for Debian/Kali and diff --git a/man/bulk_extractor.1 b/man/bulk_extractor.1 index ad5e50ed7..c666d8358 100644 --- a/man/bulk_extractor.1 +++ b/man/bulk_extractor.1 @@ -1,228 +1,236 @@ -.\" Process this file with -.\" groff -man -Tascii foo.1 -.\" -.TH BULK_EXTRACTOR 1 "OCT 2013" "User Manuals" +.TH BULK_EXTRACTOR 1 "2026-08-02" "bulk_extractor 2.2.0-DEVELOP" "User Commands" .SH NAME -bulk_extractor \- Scans a disk image for regular expressions and other content. +bulk_extractor \- extract forensic features from disk images, devices, files, +and directories .SH SYNOPSIS -.B bulk_extractor -o -.I output_dir -[options] -.I [ image | -R dir ] -.SH DESCRIPTION -.B bulk_extractor -scans a disk image (or any other file) for a large number of pre-defined regular expressions and -other kinds of content. These items are called -.I features. -When it finds a feature, -.B bulk_extractor -writes the output to an output file. Each line of the output file -contains a byte offset at which the feature was found, a tab, and the actual -feature. Features therefore cannot contain the end-of-line character. - -.B bulk_extractor -includes native support for EnCase (.E01) and AFFLIB (.aff) files, if -it compiled and linked on a system containing those -libraries. Alternatively, the -R option can be used to recursively -scan and process a directory of individual files (disk images in such -a directory will be treated as files, not as disk images). - -.B bulk_extractor -is multi-threaded. By specifying the -.B -j -option, multiple copies of the program can be run. Each thread writes -its results into its own feature file. The files are then combined by -the primary thread when all of the secondary threads complete. - -.B bulk_extractor -is a two-phase program. In phase 1 the features are extracted. In -phase 2 a histogram is created of relevant features. - -.B bulk_extractor -will also create a -.I wordlist -of all the words that are found in the disk image. This can be used as -a dictionary for cracking encryption. - -The options are as follows: -.IP "-o outdir" -Specifies the output directory, which will be created by -.B bulk_extractor -if necessary. -If the output directory contains data from a partial .B bulk_extractor -run, -.B bulk_extractor -will attempt to resume where the previous run left off. - -.IP "-b bannerfile.txt" -Read the contents of -.I bannerfile.txt -and stamp it at the beginning of each output file. This might be useful if -you have some kind of privacy banner that needs to be stamped at the -top of all of your files. - -.IP "-r alert_list.txt" -Specifies an -.I alert list, -(or red list), which is a list of terms that, if found, will be specifically flagged -in a special -.I alert file -that begins with the letters -.I ALERT. -The alert list may contain individual terms, which -must be found in their entirity and are case-sensitive, or wildcards with standard Unix -globbing (e.g. *@company.com). Globbed terms are case-insensitive. - -.IP "-w stop_list.txt" -Specifies a -.I stop list, -(or white list), -which is a list of terms that, if found, will be placed in a special -.I "stopped" -file (rather than in the main file). The whitelist may also contain -globbed terms. - -.IP "-s frac[:passes]" -Specify -.I random sampling -parameters. - -.IP "-p path/format" -Open a disk image and print the information found at -.I path. -The -.I format -specification may be -.I r -for raw output and -.I h -for hex output. -.br -Specify -.I -p - -for interactive mode. +.B -o +.I outdir +.RI [ options ] +.I image .br -Specify -.I -p -http -for HTTP mode. - -.IP "-F " -Specifies a file of regular expressions to be used as search terms. - -.IP "-f " -Specifies a regular expression to be used as a search term. - -.IP "-q nn" -Quiet mode. Only prints every -.I nn -status reports. -.br -Specify -.I -1 -for no status. - -.IP "-Wn1:n2" -The -.I scan_wordlist -scanner should only extract words that are between -.I n1 -and -.I n2 -characters in length. - -.SH TUNING -These commands are useful for tuning operation: - -.IP "-C NN" -Specifies the size of the context window. - -.IP "-S fr::window=NN" -specifies context window for recorder to NN. - -.IP "-S fr::window_before=NN" -specifies context window after to NN for recorder - -.IP "-S fr::window_after=NN" -specifies context window before to NN for recorder - -.IP "-G NN" -specify the page size - -.IP "-g NN" -Specifies the size of the margin in bytes. - -.IP "-j NN" -Use -.I n -threads for analysis. Normally you do not need specify this, as the -default is the number of processors on the current computer. - -.IP "-m NN" -Have .B bulk_extractor -wait at most NN minutes for scanners to finish after all data have been read. - -.SH DEBUGGING -The following commands are useful for debugging: - -.IP "-V" -Print the version number - -.IP "-R outdir" -Restarts the program from where it left off for a particular directory. - -.IP "-B nn" -Set the dedup Bloom filter to -.I nn -bits. This is used by the -.I scan_wordlist -scanner. - -.IP "-M nn" -Specifies a maximum recursion depth of -.I nn. - -.IP "-z pagenum" -Start on page number -.I pagenum. - -.IP "-Y [-]" -Start at input offset -.I o1 -optionally ending at offset -.I o2 - -.IP "-dN" -Enable debugging level -.I N. - -.SH SCANNER CONTROL -Finally, you can control scanners with these options: - -.IP "-P " -Specifies a directory in which to find plugins. - -.IP "-E scanner" -Turns off all scanners, then enabled scanner scanner. - -.IP "-e scanner" -Enables a scanner. - -.IP "-x scanner" -Disables a scanner. - - -.SH HISTORY -.BR "bulk_extractor" -is based on a feature extractor and named entity -recognizer developed for SBook in 1991. The feature extractor was -repurposed for disk images in 2003. The stand-alone -.B bulk_extractor -program was rewritten in 2005 and publicly released in 2007. The -multi-threaded +.B -o +.I outdir +.B -R +.I directory +.RI [ options ] +.SH DESCRIPTION .B bulk_extractor -was released in May 2010. +scans an image, file, raw device, or recursively selected directory for forensic +features without parsing the file system. It writes feature files, histograms, +diagnostic records, and a DFXML report to +.IR outdir . +. +Scanners can recursively inspect recognized derived data, such as compressed or +encoded content. The exact scanners and their defaults depend on the build; +use +.B -H +to list them and their scanner-specific settings. +. +An existing output directory containing +.B report.xml +is resumed automatically. Pages recorded as in progress when the prior run +ended are deliberately skipped so that a data-dependent failure is not repeated. +Use a new output directory for an independent scan, or +.B -Z +to remove an existing directory's contents before starting. +.SH REQUIRED ARGUMENTS +.TP +.B -o, --outdir \fIOUTDIR\fR +Write results to +.IR OUTDIR . +This option is required, including when using +.BR -p . +.SH INPUT AND OUTPUT OPTIONS +.TP +.B -R, --recurse +Treat the input as a directory and recursively scan its files. Do not use this +option for a directory that represents one split or E01 image; supply the image +itself instead. +.TP +.B -Z, --zap +Recursively remove the contents of an existing output directory before scanning. +Use with care. +.TP +.B -Y, --scan \fISTART\fR[\-\fIEND\fR] +Scan only the specified byte-offset range. +.TP +.B -z, --page_start \fIPAGE\fR +Begin scanning at page number +.IR PAGE . +.TP +.B -p, --path \fIPATH\fR +Print a forensic path instead of performing a normal scan. +.I PATH +may include an optional length and +.B /h +for hexadecimal or +.B /r +for raw output. Use +.B -p - +for interactive input or +.B -p -http +for HTTP input. +.TP +.B -A, --offset_add \fIBYTES\fR +Add +.I BYTES +to reported feature locations. +.TP +.B -b, --banner_file \fIFILE\fR +Prepend the contents of +.I FILE +to feature files. +.TP +.B -r, --alert_list \fIFILE\fR +Read the alert list from +.IR FILE . +.TP +.B -w, --stop_list \fIFILE\fR +Read the stop list from +.IR FILE . +.SH SCANNER AND SEARCH OPTIONS +.TP +.B -e, --enable \fISCANNER\fR +Enable a scanner. May be specified more than once. +.TP +.B -x, --disable \fISCANNER\fR +Disable a scanner. May be specified more than once. Use +.B -x all +to disable every scanner before selectively enabling scanners. +.TP +.B -E, --enable_exclusive \fISCANNER\fR +Disable all scanners except +.IR SCANNER . +.TP +.B -P, --scanner_dir \fIDIR\fR +Search +.I DIR +for loadable scanner modules. May be specified more than once. The default +search path also includes the directories in the +.B BE_PATH +environment variable. +.TP +.B -S, --set \fINAME=VALUE\fR +Set a scanner or feature-recorder configuration value. May be specified more +than once. Use +.B -H +to discover supported names, and see +.I doc/carving.md +for carving settings. +.TP +.B -f, --find \fIPATTERN\fR +Search for a pattern. May be specified more than once. +.TP +.B -F, --find_file \fIFILE\fR +Read search patterns from +.IR FILE . +May be specified more than once. +.TP +.B --find-case-sensitive +Make patterns supplied by +.B -f +and +.B -F +case-sensitive. Case-insensitive matching is the default. +.SH PERFORMANCE OPTIONS +.TP +.B -j, --threads \fIN\fR +Use +.I N +worker threads. +.TP +.B -J, --no_threads +Read and process data in the primary thread. +.TP +.B -G, --pagesize \fIBYTES\fR +Set the primary page size. +.TP +.B -g, --marginsize \fIBYTES\fR +Set the page margin size. +.TP +.B -M, --max_depth \fIN\fR +Set the maximum recursive-processing depth. +.TP +.B -s, --sampling \fIFRACTION\fR[:\fIPASSES\fR] +Set random-sampling parameters. +.TP +.B --max_minute_wait \fIMINUTES\fR +Set the maximum time to wait for processing and shutdown after input is read. +.TP +.B --max_bad_alloc_errors \fIN\fR +Set the number of allocation failures tolerated before the scan stops. +.SH LOGGING AND INFORMATION OPTIONS +.TP +.B -d, --debug +Enable debug-level diagnostic logging. Numeric debug masks are no longer +supported. +.TP +.B --log-level \fILEVEL\fR +Set the diagnostic log level to +.BR trace , +.BR debug , +.BR info , +.BR warning , +.BR error , +.BR critical , +or +.BR off . +.TP +.B --log-file \fIFILE\fR +Write diagnostics to +.I FILE +instead of the default +.IR outdir /bulk_extractor.log . +.TP +.B -q, --quit +Suppress status and performance output. +.TP +.B -0, --no_notify +Disable real-time notifications. +.TP +.B -1, --version1 +Use version-1-style console notifications. +.TP +.B --notify_main_thread +Display notifications in the main thread; useful with ThreadSanitizer. +.TP +.B --notify_async +Display notifications asynchronously (the default). +.TP +.B -H, --info_scanners +Report information about each scanner, including scanner-specific +.B -S +settings. +.TP +.B -V, --version +Print the program version and exit. +.TP +.B -h, --help +Print command-line help and exit. +.SH ENVIRONMENT +.TP +.B BE_PATH +A colon-separated list of directories in which to search for loadable scanner +modules. +.TP +.B LOG_LEVEL +Default diagnostic log level when +.B --log-level +is not supplied. +.SH FILES +.TP +.I outdir/report.xml +DFXML report describing the scan. +.TP +.I outdir/bulk_extractor.log +Default diagnostic log file. +.SH SEE ALSO +.IR README.md , +.IR doc/carving.md , +.IR doc/logging.md , +and the published manuals at +.B https://simsong.github.io/bulk_extractor/ . .SH AUTHOR Simson Garfinkel - From 2d2df8dd0595a1701ba125a1d80d4d3c6bddd91e Mon Sep 17 00:00:00 2001 From: Codex AI Assistant Date: Sun, 2 Aug 2026 18:35:18 -0400 Subject: [PATCH 9/9] Remove obsolete performance notes --- doc/RELEASE_NOTES.md | 2 + doc/performance.txt | 111 ------------------------------------------- 2 files changed, 2 insertions(+), 111 deletions(-) delete mode 100644 doc/performance.txt diff --git a/doc/RELEASE_NOTES.md b/doc/RELEASE_NOTES.md index a4e56fa8a..cf147b4aa 100644 --- a/doc/RELEASE_NOTES.md +++ b/doc/RELEASE_NOTES.md @@ -212,6 +212,8 @@ through the project's normal pull-request and CI process. ([PR #561](https://github.com/simsong/bulk_extractor/pull/561), [PR #558](https://github.com/simsong/bulk_extractor/pull/558), [PR #557](https://github.com/simsong/bulk_extractor/pull/557)). +- Removed an unmaintained version-1 performance notebook with obsolete + benchmarks, platforms, and SQL tuning guidance. - Documented a controlled release procedure and release-issue template, with isolated artifact assembly, macOS and container `distcheck` gates, and source-level downstream submission paths for Debian/Kali and diff --git a/doc/performance.txt b/doc/performance.txt deleted file mode 100644 index a17e41fcb..000000000 --- a/doc/performance.txt +++ /dev/null @@ -1,111 +0,0 @@ -CURRENT PERFORMANCE TUNING NOTES - -v1.5.0-dev0 -R4 (32-core Dell 760) ---no-featuresql: -writing to feature file files alone: 497 sec (86.25 MB/sec) (≈60 sec spent writing wordlist) - 492 sec (86.55 MB/sec) - ---no-featuresql - "PRAGMA synchronous = OFF", - "PRAGMA journal_mode=MEMORY", - "PRAGMA temp_store=MEMORY", - "PRAGMA cache_size = 200000", // 10x normal cache - Elapsed time: 431.621 sec; 99.5079 - - "PRAGMA synchronous = OFF", - "PRAGMA journal_mode=MEMORY", - //"PRAGMA temp_store=MEMORY", - "PRAGMA cache_size = 200000", - Elapsed time: 409.784 sec. - - "PRAGMA synchronous = OFF", - "PRAGMA journal_mode=MEMORY", - //"PRAGMA temp_store=MEMORY", - //"PRAGMA cache_size = 200000", - Elapsed time: 474.042 sec. - - "PRAGMA synchronous = OFF", - //"PRAGMA journal_mode=MEMORY", - //"PRAGMA temp_store=MEMORY", - "PRAGMA cache_size = 200000", - Elapsed time: 501.194 sec., - - - -HISTORIC PERFORMANCE STATISTICS - -This section tracks how performance of bulk_extractor has changed over -time. We update it with each new release - -NOTE: We recommend compiling Version 1.4 with -O3 under GCC. Please -use version GCC 4.7 or above. One a Mac you may wish to use clang. - -%% BEGIN NO FILL - - -Use these configure flags to compile with a different optimization: - - --without-opt Drop all -O C flags - --without-o3 Do not force O3 optimization; use default level - - -Disk image: /corp/nps/drives/nps-2009-ubnist1/ubnist1.gen3.E01 - /corp/nps/drives/nps-2009-ubnist1/ubnist1.gen3.E02 - - Media size: 1.9 GiB (2106589184 bytes) - MD5: 49a775d8b109a469d9dd01dc92e0db9c - -Hardware: MacBook Pro 2 Ghz Intel Core i7, 8GB 1333 Mhz DDR3 - 512GB SSD (Simson's Laptop "Mucha"), - -Current and Historic Times with no tuning [1]: - -MacOS 10.8.0; LLVM build 2336.11.00; -O3 - -version 1.4: 144 seconds (14.59 MBytes/sec) (-O3; ) -version 1.3: 185 seconds (11.34 MBytes/sec) (-O3; ) -version 1.2.0: 141 seconds (14.9 MBytes/sec) (-O3; ) -version 1.1.3: 171 seconds (12.3 MBytes/sec) (-O3; AES disabled) -version 1.0.7: 256 seconds (8.22 MBytes/sec) (-O3; AES disabled) - -MacOS 10.7.8; LLVM build 2336.1.00; No optimization - -version 1.2.0: 350 seconds (5.72 MBytes/sec) -version 1.1.3: 468 seconds (4.28 MBytes/sec) - -Windows 7, same hardware ("boot camp"): - -version 1.4: TBD -version 1.3: 198 seconds (10.6 MBytes/sec) (-O3; AES enabled; 32bit) -version 1.3: 186 seconds (11.33 MBytes/sec) (-O3; AES enabled; 64bit) -version 1.2.0: 207.4 seconds (9.69 MBytes/sec, [2]) - - -Normalized Testing Results: - - -v1.4.0: - -Windows 7 43.5 MBytes/sec; 5.43 MBytes/s/c (CPU Bound) 2013-09-10 [A] -Windows 7 48.4 MBytes/sec; 6.05 MBytes/s/c (CPU Bound) 2013-09-13 [B] -MacOS gcc 4.2.1 43.7 MBytes/sec; 5.46 MBytes/s/c (CPU Bound) 2013-09-10 [A] -MacOS clang 4.0 46.5 MBytes/sec; 5.81 MBytes/s/c (CPU Bound) 2013-09-10 [A] -MacOS clang 4.0 48.0 MBytes/sec; 6.01 MBytes/s/c (CPU Bound) 2013-09-13 [B] -MacOS gcc 4.2.1 51.2 MBytes/sec; 6.40 MBytes/s/c (CPU Bound) 2013-09-14 [B] -MacOS gcc 48 53.1 MBytes/sec; 6.64 MBytes/s/c (CPU Bound) 2013-09-14 [B] -F19 gcc 4.8.1 52.2 MBytes/sec; 6.53 MBytes/s/c (CPU Bound) 2013-09-27 [B] - -Testing hardware and image: - -[A] Macbook Pro 'DF8Y 2.0Ghz i7 16GiB RAM SSD (Mucha); domexusers -[B] Macbook Pro 'DV7P 2.4Ghz i7 8GiB RAM 7200RPM HD (NPS 15"); domexusers - - -Notes: -1 - Times are for running bulk_extractor with default tuning, no options. - -1 - Times reported are the fastest of three consecutive runs. [We care -about maximum possible performance, not typical performance.] - -2 - bulk_extractor 1.2.0 scan_exiv was disabled under Windows