Bring the test-data convention over from iLEAPP

.github/pull_request_template.md

@@ -0,0 +1,16 @@
+## What this changes
+
+<!-- One or two sentences. What does this add, fix or change? -->
+
+## For a new or changed artifact
+
+<!-- Delete this section if the PR does not touch scripts/artifacts. -->
+
+- [ ] Ran the tool against a real extraction and confirmed the row counts, not just that it imports.
+- [ ] Ran `python admin/scripts/check_artifact_output.py <report folder>` on that report and **fixed or documented every finding**. An empty or constant column is often a real result (no group chats, coarse location denied); the fix for those is to say so in the artifact's `notes`, which is also what stops the checker reporting them.
+- [ ] Checked it against a second app data directory where the platform provides one (a second user, account, or container). `--compare <multi-container report>` reads the scaling for you: it should be exactly double.
+- [ ] `notes`, `description` and `sample_data` say only what the data shows, and the numbers were re-derived from the finished run.
+
+## Anything reviewers should know
+
+<!-- Undocumented codes reported as stored, a store that could not be decrypted, a validation gap, etc. -->

.github/workflows/request_test_data.yml

@@ -0,0 +1,42 @@
+name: Request Test Data
+
+# Asks external contributors for test data when a PR changes artifact modules
+# without any. pull_request_target so the comment can be posted on fork PRs;
+# safe here because the job checks out and runs only the base branch's code.
+# The contributor's files are fetched as text by the script and never executed.
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+    paths:
+      - 'scripts/artifacts/**'
+
+permissions:
+  contents: read
+  issues: write
+  pull-requests: write
+
+jobs:
+  request-test-data:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout base branch
+        uses: actions/checkout@v4
+        with:
+          # Blob-less fetch; the script needs only admin/ sources, not the
+          # committed test fixtures under admin/test/cases/data.
+          sparse-checkout: |
+            /*
+            !/admin/test/cases/data
+          sparse-checkout-cone-mode: false
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Check the PR for test data
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: python admin/scripts/check_pr_test_data.py

.github/workflows/test_cases.yml

@@ -0,0 +1,51 @@
+name: Test Cases
+
+# Runs every committed test case (admin/test/cases) and compares the output
+# against the recorded baselines (admin/test/results). Baselines are ordinary
+# committed files: a PR that deliberately changes a parser's output re-records
+# the affected snapshot in the same PR, so the reviewer sees the row-level
+# diff next to the code change. Units excluded from gating are listed with
+# reasons in admin/test/cases/known_failures.json.
+#
+# This job needs the committed fixtures, so it does NOT use the sparse
+# checkout the other PR workflows use.
+
+on:
+  pull_request:
+    paths:
+      - 'scripts/**'
+      - 'admin/test/cases/**'
+      - 'admin/test/results/**'
+      - 'admin/test/scripts/**'
+      - 'requirements.txt'
+      - '.github/workflows/test_cases.yml'
+  push:
+    branches:
+      - main
+    paths:
+      - 'scripts/**'
+      - 'admin/test/cases/**'
+      - 'admin/test/results/**'
+      - 'admin/test/scripts/**'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test-cases:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+
+      - name: Install runtime dependencies
+        run: python -m pip install -r requirements.txt
+
+      - name: Run test cases against recorded baselines
+        run: python admin/test/scripts/run_test_cases.py

admin/docs/testing/create_module_test_cases.md

@@ -0,0 +1,191 @@
+# Creating Module Test Cases
+
+This document describes the process of creating test cases for LEAPP modules using the `make_test_data.py` script.
+
+## Overview
+
+The `make_test_data.py` script is designed to generate test data for LEAPP modules. It processes input files (zip, tar, or tar.gz) to extract relevant files based on the module's artifact patterns and creates structured test cases.
+
+## Usage
+
+To create test cases for a module, use one of the following commands:
+
+```bash
+python make_test_data.py <module_name> --image <image_name>
+python make_test_data.py <module_name> --case <case_number> --input <input_file>
+python make_test_data.py <module_name> --image-prompt
+```
+
+Arguments:
+- `<module_name>`: Name of the module (e.g., keyboard or keyboard.py)
+- `--image <image_name>`: Name of the image from the manifest
+- `--case <case_number>`: Case number for the test data
+- `--input <input_file>`: Path to the input file (zip, tar, or tar.gz)
+- `--image-prompt`: Prompt for image selection from the manifest
+
+## Process
+
+1. The script imports the specified module and retrieves artifact information.
+2. It creates or updates a JSON file with test case metadata.
+3. The script processes the input archive file, searching for files matching the artifact patterns.
+4. For each artifact, it creates a zip file containing the matching files.
+5. The JSON file is updated with information about the created test data.
+
+## Output
+
+The script generates the following outputs:
+
+1. A JSON file containing test case metadata.
+    - `admin/test/cases/testdata.<module_name>.json`
+2. Zip files for each artifact, containing the relevant test data files.
+    - `admin/test/cases/data/<module_name>/testdata.<module_name>.<artifact_name>.<case_key>.zip`
+
+## Example
+
+```bash
+python make_test_data.py keyboard --image ios_15_image
+```
+
+This command will create test data for the keyboard module, using the specified image from the manifest.
+
+## Notes
+
+- The script supports zip, tar, and tar.gz input files.
+- Test data is stored in the `admin/test/cases/data` directory.
+- JSON metadata files are stored in the `admin/test/cases` directory.
+- Always review and update the generated JSON file with additional test case details as needed.
+
+## Test Case JSON File Structure
+
+The script generates a JSON file (e.g., `testdata.<module_name>.json`) that contains metadata and information about the test cases. This file is crucial for defining the scenarios that `test_module.py` will execute. Here's an explanation of its structure:
+
+```json
+{
+  "case_ios12_basic": {
+    "description": "Tests basic data extraction for iOS 12.x.",
+    "maker": "Your Name",
+    "os_name": "iOS",
+    "os_version": "12.5.5",
+    "make_data": {
+      "input_data_path": "/path/to/iOS12_image.tar.gz",
+      "os": "macOS-14.0-...",
+      "timestamp": "2024-10-14T10:17:49.432528",
+      "last_commit": {
+          "hash": "abcdef123...",
+      }
+    },
+    "artifacts": {
+      "get_photosMetadata": {
+        "search_patterns": [
+          "*/mobile/Media/PhotoData/Photos.sqlite*"
+        ],
+        "file_count": 1
+      },
+      "another_artifact_function_if_any": {
+      }
+    }
+  }
+}
+```
+
+- `case_ios12_basic`: A unique identifier for each test case. This key is used by `test_module.py` to find the corresponding input data ZIP and golden file.
+  - `description`: A brief description of what this test case covers (to be filled in manually).
+  - `maker`: The person who created or last verified the test case (to be filled in manually).
+  - `os_name`: Include the name of the operating system this case data originated from (iOS, Android, etc)
+  - `os_version`: The specific OS version string (e.g., "12.5.5", "14.1") that this test case represents. `test_module.py` will use this to mock `iOS.get_version()`.
+  - `make_data`: Information about the `make_test_data.py` run that generated this case's input data.
+    - `input_data_path`: The path to the original full image/archive used.
+    - `os`: The operating system on which `make_test_data.py` was run.
+    - `timestamp`: The date and time when the input data ZIP was created.
+    - `last_commit`: Information about the module's Git commit at the time of data creation.
+  - `artifacts`: A dictionary where keys are artifact function names from the module.
+    - `artifact_name` (e.g., "get_photosMetadata"):
+      - `search_patterns`: The file patterns from the module's `__artifacts_v2__` block used to find relevant files.
+      - `file_count`: The number of files found and included in the input ZIP for this artifact and case.
+
+Multiple test cases (e.g., "case_ios12_basic", "case_ios14_advanced") can be included in a single `testdata.<module_name>.json` file. It's recommended to use descriptive case keys.
+
+## Image Manifest
+
+The script uses an `image_manifest.json` file located in the `admin` directory. This manifest contains information about available test images, including their names, descriptions, and local paths.
+
+## Known Issues and Limitations
+
+### Dynamic File Searching in Modules
+
+Some modules, such as `sms.py`, use a two-step process for file searching that can limit the effectiveness of automated test case creation:
+
+1. The module provides an initial search pattern to locate a primary database file.
+2. After processing the database, the module performs additional file searches based on data extracted from the database.
+
+For example, the `sms.py` module:
+1. First searches for the SMS database file.
+2. Then searches for attachment files based on paths stored in the database.
+
+This approach presents challenges for automated test case creation:
+
+- The `make_test_data.py` script only uses the initial search patterns defined in the module.
+- It cannot anticipate or include files that would be found by secondary searches within the module.
+
+#### Impact on Test Coverage
+
+This limitation may result in incomplete test data for modules that employ this dynamic searching technique. The created test cases might not include all the files that the module would process in a real-world scenario.
+
+#### Future Improvements
+
+To address this issue and improve test coverage, we need to consider the following approaches:
+
+1. Enhance the `make_test_data.py` script to simulate the module's dynamic file searching behavior.
+2. Modify the module structure to separate file searching from data processing, allowing for more comprehensive initial search patterns.
+3. Implement a two-pass system in the test case creation process to capture files found by secondary searches.
+
+Until these improvements are implemented, be aware that modules using dynamic file searching may require manual intervention to ensure comprehensive test data.
+
+## Updating the JSON File
+
+After creating test cases with `make_test_data.py`, it's important to manually update the `testdata.<module_name>.json` file with:
+
+1. A meaningful `description` for each test case.
+2. Your name as the `maker`.
+3. The relevant `os_name` and `os_version` string so `test_module.py` can simulate it.
+
+This information, along with the golden files generated by `test_module.py`, is crucial for understanding test coverage and validating results.
+
+## Baselines and CI
+
+Every committed case runs in CI against a recorded snapshot. The workflow
+`.github/workflows/test_cases.yml` calls `admin/test/scripts/run_test_cases.py`,
+which re-runs each artifact against its case zip and fails when the output no
+longer matches the latest snapshot under `admin/test/results/<module>/`.
+
+Baselines are ordinary committed files. When a PR deliberately changes a
+parser's output, re-record the affected snapshot in the same PR:
+
+```bash
+python admin/test/scripts/test_module.py <module> -a all -c all
+```
+
+Commit the new snapshot and delete the superseded one, so the reviewer sees the
+row-level diff next to the code change that caused it. Rows are compared as
+unordered sets, so a change in result order alone does not fail the check.
+
+Units that cannot gate yet are listed with a reason per unit in
+`admin/test/cases/known_failures.json`. They still run and report, but do not
+fail CI. When one of them passes again, the run says so; remove its entry in
+the same PR that fixed it.
+
+## Requirements
+
+### __artifacts_v2__ Block
+
+The test case creation process requires modules to use the `__artifacts_v2__` block for defining artifacts. This is because the v1 artifact definition has some limitations that can cause issues with the test harness.
+
+If you want to apply this testing process to a v1 script before converting the entire code, you can update the artifact block to v2 format first to capture the results.
+
+## Git Integration
+
+The script now includes git integration to capture information about the last commit that modified the module file. This information is stored in the JSON metadata file for each test case.
+
+## Performance Considerations
+
+The script includes performance optimizations and progress indicators for processing large archive files. It also provides timing information for various stages of the test data creation process.