Feat: Add OpenEoX Hardware Lifecycle Scanner - #907
Conversation
- Apply multiple enhancements to `clean-processed-folder.py` so that it runs fast
enough to keep up with the generation of log files in pipeline capture mode. These
changes increased the file processing rate by a factor of 100.
- Preprocess the filebeat registry into a format for checking file presence using the
`in` operator.
- Replace regular expression pattern matching for mime file types with list searching.
- Refactor running `fuser` with the subprocess module for increased speed.
- Treat zero-length files, which have no mime type, as eligible log files.
- Update logging to improve the ability to monitor script performance.
- Fix the search for Suricata log files which was excluding log files created in the
pipeline mode.
- Run `clean-processed-folder.py` every minute in order to minimize the risk of
overflowing the partition where the log files are stored especially in the pipeline
capture mode.
- Apply multiple enhancements to `clean-processed-folder.py` so that it runs fast
enough to keep up with the generation of log files in pipeline capture mode. These
changes increased the file processing rate by a factor of 100.
- Preprocess the filebeat registry into a format for checking file presence using the
`in` operator.
- Replace regular expression pattern matching for mime file types with list searching.
- Refactor running `fuser` with the subprocess module for increased speed.
- Treat zero-length files, which have no mime type, as eligible log files.
- Update logging to improve the ability to monitor script performance.
- Fix the search for Suricata log files which was excluding log files created in the
pipeline mode.
- Run `clean-processed-folder.py` every minute in order to minimize the risk of
overflowing the partition where the log files are stored especially in the pipeline
capture mode.
- Apply multiple enhancements to `clean-processed-folder.py` so that it runs fast
enough to keep up with the generation of log files in pipeline capture mode. These
changes increased the file processing rate by a factor of 100.
- Preprocess the filebeat registry into a format for checking file presence using the
`in` operator.
- Replace regular expression pattern matching for mime file types with list searching.
- Refactor running `fuser` with the subprocess module for increased speed.
- Treat zero-length files, which have no mime type, as eligible log files.
- Update logging to improve the ability to monitor script performance.
- Fix the search for Suricata log files which was excluding log files created in the
pipeline mode.
- Run `clean-processed-folder.py` every minute in order to minimize the risk of
overflowing the partition where the log files are stored especially in the pipeline
capture mode.
|
Hi, @PrudhviChanda, let me update you as to where I'm at with this:
So, here's what I'm going to do for now, for the v26.04.2 release (next week):
Let me know what you think. At the very least, thanks for your contribution of figuring out how to register the custom scripts correctly (as I said, I have pulled it in to my development branch, which will be merged here to the official release branch next week). In the meantime, after that release you'll be able to run your custom |
…peneox-hardware-lifecycle-scanner
|
Hi @mmguero Thanks for pulling in the custom script loader and making it generic for the v26.04.2 release. Glad to see that feature making it in. Regarding the OpenEoX testing: I faced the exact same wall. Real payloads are locked behind vendor auth portals, so public test data basically doesn't exist. Because of that, I realized hardcoding schema keys was a limiting design. I just pushed a refactor that drops the static keys and uses a recursive crawler instead. The script doesn't care about the JSON structure anymore. it just recursively checks for known model and date identifiers in any payload (Cisco, Palo Alto, OpenEoX, etc.).To make testing easier, I’ve put together a sample JSON file that combines a few different vendor schemas that I was able to find in various configs. Please find the URL for the JSON data below: |
…neric and dynamic so it can be used for other custom scripts as well
…neric and dynamic so it can be used for other custom scripts as well
…neric and dynamic so it can be used for other custom scripts as well
fb19199 to
0b3b9ef
Compare
Add OpenEoX Hardware Lifecycle Scanner for NetBox 4.4+
🗣 Description
This PR introduces a new NetBox script (
hardware_lifecycle_auditor.py) designed to ingest standard OpenEoX JSON feeds to automate the identification of End-of-Support (EOS) hardware.Additionally, this PR updates
netbox/scripts/netbox_init.pyto programmatically stage and register custom scripts via the Django ORM. This bypasses the NetBox 4.4+syncdatasourcesymlink restrictions and Docker ConfigMap read-only file system errors (rsync: mkstemp failed: Read-only file system (30)) that previously prevented custom Malcolm scripts from loading.Features of the Auditor:
Lifecycle: End of SupportandLifecycle: SupportedNetBox tags to active inventory.💭 Motivation and context
This change is driven by CISA BOD 26-02(https://www.cisa.gov/news-events/news/end-just-beginning-better-security-enhanced-vulnerability-management-openeox), which mandates agencies to identify and remove End-of-Support edge devices.
By natively tagging devices in NetBox based on the OpenEoX standard, Malcolm's backend Logstash workers can automatically pull these tags and enrich Zeek/Suricata network traffic logs in OpenSearch. This bridges the gap between physical asset vulnerabilities and live network traffic monitoring.
🧪 Testing
Environment: Malcolm Docker deployment with NetBox 4.x.
Steps Performed:
netbox_init.pyshim correctly copied the script from/usr/local/bin/to the writable/opt/netbox/netbox/scripts/directory and successfully registered theScriptModulewithout triggering Docker permission crashes.C1111Cisco device in NetBox and marked it as Active.Commit changesbox was checked.Lifecycle: End of Supporttag to the device profile and the generation of the CSV output block.📷 Screenshots (if appropriate)
✅ Pre-approval checklist
bump_versionscript if this repository is versioned and the changes in this PR warrant a version bump.✅ Pre-merge checklist
✅ Post-merge checklist