Skip to content

Support S3-backed Zeek log inputs - #156

Closed
keilogic wants to merge 1 commit into
SuperCowPowers:mainfrom
keilogic:support-s3-log-input
Closed

keilogic wants to merge 1 commit into
SuperCowPowers:mainfrom
keilogic:support-s3-log-input

Conversation

@keilogic

@keilogic keilogic commented Jun 1, 2026

Copy link
Copy Markdown

Summary

  • parse Zeek log headers through local files or optional fsspec URI paths
  • let LogToDataFrame and JSONLogToDataFrame pass remote paths such as s3://... through to Pandas once the matching fsspec backend is installed
  • add zat[s3] packaging/docs and memory-backed regression coverage for URI input paths

Fixes #141.

Testing

  • PYTHONPATH=. python -m pytest zat\utils\field_info.py zat\log_to_dataframe.py zat\json_log_to_dataframe.py -q -> 3 passed
  • python -m black --check zat\utils\field_info.py zat\log_to_dataframe.py zat\json_log_to_dataframe.py
  • python -m isort --check zat\utils\field_info.py zat\log_to_dataframe.py zat\json_log_to_dataframe.py
  • python -m flake8 zat\utils\field_info.py zat\log_to_dataframe.py zat\json_log_to_dataframe.py
  • git diff --check
  • Manual memory:// LogToDataFrame load returned (360, 3)
  • PYTHONPATH=. python -m pytest zat -q -> 19 passed, 3 skipped, 2 unrelated local Windows failures:
    • zat/utils/cache.py::test hit the existing /tmp/zat_file_storage\my_test_cache rename collision
    • zat/utils/signal_utils.py::test uses signal.SIGQUIT, which is not available on Windows

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Loading Data from S3

2 participants