Skip to content

Feat: Add OpenEoX Hardware Lifecycle Scanner - #907

Open
PrudhviChanda wants to merge 21 commits into
cisagov:mainfrom
PrudhviChanda:feat/openeox-hardware-lifecycle-scanner
Open

Feat: Add OpenEoX Hardware Lifecycle Scanner#907
PrudhviChanda wants to merge 21 commits into
cisagov:mainfrom
PrudhviChanda:feat/openeox-hardware-lifecycle-scanner

Conversation

@PrudhviChanda

Copy link
Copy Markdown

Add OpenEoX Hardware Lifecycle Scanner for NetBox 4.4+

🗣 Description

This PR introduces a new NetBox script (hardware_lifecycle_auditor.py) designed to ingest standard OpenEoX JSON feeds to automate the identification of End-of-Support (EOS) hardware.

Additionally, this PR updates netbox/scripts/netbox_init.py to programmatically stage and register custom scripts via the Django ORM. This bypasses the NetBox 4.4+ syncdatasource symlink restrictions and Docker ConfigMap read-only file system errors (rsync: mkstemp failed: Read-only file system (30)) that previously prevented custom Malcolm scripts from loading.

Features of the Auditor:

  • Parses OpenEoX JSON from authenticated remote URLs (via an Auth Header input) or local file uploads.
  • Dynamically applies Lifecycle: End of Support and Lifecycle: Supported NetBox tags to active inventory.
  • Generates an inline, formatted CSV report of compliant and non-compliant devices.

💭 Motivation and context

This change is driven by CISA BOD 26-02(https://www.cisa.gov/news-events/news/end-just-beginning-better-security-enhanced-vulnerability-management-openeox), which mandates agencies to identify and remove End-of-Support edge devices.

By natively tagging devices in NetBox based on the OpenEoX standard, Malcolm's backend Logstash workers can automatically pull these tags and enrich Zeek/Suricata network traffic logs in OpenSearch. This bridges the gap between physical asset vulnerabilities and live network traffic monitoring.

🧪 Testing

Environment: Malcolm Docker deployment with NetBox 4.x.

Steps Performed:

  1. Validated the netbox_init.py shim correctly copied the script from /usr/local/bin/ to the writable /opt/netbox/netbox/scripts/ directory and successfully registered the ScriptModule without triggering Docker permission crashes.
  2. Created a mock C1111 Cisco device in NetBox and marked it as Active.
  3. Executed the script against a remote OpenEoX JSON payload hosted on a GitHub Gist.
  4. Verified HTTP 404/JSON parsing error handling.
  5. Confirmed the script respected the NetBox "Dry Run" database protections until the Commit changes box was checked.
  6. Verified the correct application of the red Lifecycle: End of Support tag to the device profile and the generation of the CSV output block.

📷 Screenshots (if appropriate)

  1. The Script UI:
Screen Shot 2026-02-24 at 11 03 02 AM
  1. The CSV Output:
Screen Shot 2026-02-24 at 11 04 24 AM
  1. The Device Tag:
Screen Shot 2026-02-24 at 11 05 01 AM

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.
  • Bump major, minor, patch, pre-release, and/or build versions as appropriate via the bump_version script if this repository is versioned and the changes in this PR warrant a version bump.
  • Create a pre-release (necessary if and only if the pre-release version was bumped).

✅ Pre-merge checklist

  • Revert dependencies to default branches.
  • Finalize version.

✅ Post-merge checklist

  • Create a release (necessary if and only if the version was bumped).

mmguero and others added 8 commits February 19, 2026 20:36
- Apply multiple enhancements to `clean-processed-folder.py` so that it runs fast
enough to keep up with the generation of log files in pipeline capture mode. These
changes increased the file processing rate by a factor of 100.
  - Preprocess the filebeat registry into a format for checking file presence using the
    `in` operator.
  - Replace regular expression pattern matching for mime file types with list searching.
  - Refactor running `fuser` with the subprocess module for increased speed.
- Treat zero-length files, which have no mime type, as eligible log files.
- Update logging to improve the ability to monitor script performance.
- Fix the search for Suricata log files which was excluding log files created in the
pipeline mode.
- Run `clean-processed-folder.py` every minute in order to minimize the risk of
overflowing the partition where the log files are stored especially in the pipeline
capture mode.
@mmguero mmguero self-assigned this Feb 25, 2026
@mmguero mmguero added the netbox Related to Malcolm's use of NetBox label Feb 25, 2026
@mmguero mmguero added this to Malcolm Feb 25, 2026
@mmguero mmguero moved this to Review in Malcolm Feb 25, 2026
@mmguero mmguero added this to the v26.03.0 milestone Feb 25, 2026
@mmguero mmguero removed their assignment Mar 11, 2026
@mmguero mmguero modified the milestones: v26.03.0, v26.04.0 Mar 16, 2026
mmguero and others added 11 commits April 13, 2026 14:55
- Apply multiple enhancements to `clean-processed-folder.py` so that it runs fast
enough to keep up with the generation of log files in pipeline capture mode. These
changes increased the file processing rate by a factor of 100.
  - Preprocess the filebeat registry into a format for checking file presence using the
    `in` operator.
  - Replace regular expression pattern matching for mime file types with list searching.
  - Refactor running `fuser` with the subprocess module for increased speed.
- Treat zero-length files, which have no mime type, as eligible log files.
- Update logging to improve the ability to monitor script performance.
- Fix the search for Suricata log files which was excluding log files created in the
pipeline mode.
- Run `clean-processed-folder.py` every minute in order to minimize the risk of
overflowing the partition where the log files are stored especially in the pipeline
capture mode.
- Apply multiple enhancements to `clean-processed-folder.py` so that it runs fast
enough to keep up with the generation of log files in pipeline capture mode. These
changes increased the file processing rate by a factor of 100.
  - Preprocess the filebeat registry into a format for checking file presence using the
    `in` operator.
  - Replace regular expression pattern matching for mime file types with list searching.
  - Refactor running `fuser` with the subprocess module for increased speed.
- Treat zero-length files, which have no mime type, as eligible log files.
- Update logging to improve the ability to monitor script performance.
- Fix the search for Suricata log files which was excluding log files created in the
pipeline mode.
- Run `clean-processed-folder.py` every minute in order to minimize the risk of
overflowing the partition where the log files are stored especially in the pipeline
capture mode.
@mmguero mmguero self-assigned this Apr 24, 2026
@mmguero mmguero moved this from Review to In Progress in Malcolm Apr 24, 2026
@mmguero

mmguero commented Apr 24, 2026

Copy link
Copy Markdown
Collaborator

Hi, @PrudhviChanda, let me update you as to where I'm at with this:

  • I have manually pulled this PR into my release prep branch (retaining your commit history and authorship, so it will show up as you)
  • I then took your cool custom script loading code in netbox-init and made it more generic and dynamic so it could be used by other users for their own scripts. Going forward, users will be able to drop their own custom .py scripts in Malcolm's netbox/custom-scripts/ directory and they will automatically be loaded up as you coded it in netbox-init.py.

So, here's what I'm going to do for now, for the v26.04.2 release (next week):

  • I have created a feature to track the custom script registration code you wrote (allow users to provide custom netbox scripts to be automatically registered on startup #954) and which I made more generic
  • I'm not going to pull in hardware_lifecycle_auditor.py just yet, but you (or anybody else) will be able to run it simply by dropping it in netbox/custom-scripts/ and restarting Malcolm.
    • I like the reasoning and concept behind it, but think it needs a little bit of work and hardening. I'm doing some testing on it now: partially for the script itself, but partially for getting some solid examples.
    • If you're able to provide references or links to actual OpenEoX feeds in the wild (or even static manifests/payloads) and to the specifications, that would help me feel more comfortable about being able to release it. I'd like to be able to confirm and test your script against some real OpenEoX data.

Let me know what you think. At the very least, thanks for your contribution of figuring out how to register the custom scripts correctly (as I said, I have pulled it in to my development branch, which will be merged here to the official release branch next week). In the meantime, after that release you'll be able to run your custom hardware_lifecycle_auditor.py script along with any other custom netbox script.

@mmguero mmguero modified the milestones: v26.04.2, v26.05.0 Apr 24, 2026
@PrudhviChanda

Copy link
Copy Markdown
Author

Hi @mmguero

Thanks for pulling in the custom script loader and making it generic for the v26.04.2 release. Glad to see that feature making it in.

Regarding the OpenEoX testing: I faced the exact same wall. Real payloads are locked behind vendor auth portals, so public test data basically doesn't exist. Because of that, I realized hardcoding schema keys was a limiting design. I just pushed a refactor that drops the static keys and uses a recursive crawler instead. The script doesn't care about the JSON structure anymore. it just recursively checks for known model and date identifiers in any payload (Cisco, Palo Alto, OpenEoX, etc.).To make testing easier, I’ve put together a sample JSON file that combines a few different vendor schemas that I was able to find in various configs. Please find the URL for the JSON data below:

https://gist.githubusercontent.com/PrudhviChanda/3c1a1ac4912b7e241f37304cfb51944c/raw/27e4deeb6bf00daf5912f84d5d264299d32da40d/cisco_openeox_test.json

mmguero added a commit to idaholab/Malcolm that referenced this pull request Apr 29, 2026
…neric and dynamic so it can be used for other custom scripts as well
mmguero added a commit to idaholab/Malcolm that referenced this pull request Apr 29, 2026
…neric and dynamic so it can be used for other custom scripts as well
mmguero added a commit to idaholab/Malcolm that referenced this pull request Apr 29, 2026
…neric and dynamic so it can be used for other custom scripts as well
@mmguero mmguero modified the milestones: v26.05.1, v26.06.0 May 6, 2026
@mmguero
mmguero force-pushed the main branch 2 times, most recently from fb19199 to 0b3b9ef Compare May 20, 2026 21:40
@mmguero mmguero modified the milestones: v26.06.0, v26.07.0, v26.06.1 May 27, 2026
@mmguero mmguero moved this from In Progress to Todo (develop) in Malcolm Jul 29, 2026
@mmguero mmguero modified the milestones: v26.08.0, v26.09.0 Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

netbox Related to Malcolm's use of NetBox

Projects

Status: Todo (develop)

Development

Successfully merging this pull request may close these issues.

3 participants