Skip to content

update waf datetime parsing for different servers #5293

Description

@rshewitt

User Story

In order to accurately derive file modified dates from waf source, datagov wants to update waf datetime parsing to handle different formats/servers

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

  • GIVEN a waf source
    AND a datetime format not like "2025-01-22 23:21" (e.g. "05-Jun-2025 17:17")
    WHEN waf datetime parsing happens
    THEN the other format is parsed correctly \

Background

  • we consider the modified date in waf sources to avoid unnecessary downloads (if the source document is older than what we have then skip it)
  • currently, our waf datime parsing function only handles "2025-01-22 23:21" format but we want to make sure other formats are supported to. that function will pad with a placeholder datetime but if we can correctly parse the date on the page that would be better.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

  • identify different datetime formats (based on server? what if the server type isn't in the response header?)
  • implement those datetime formats based on the server type? or just try them all

Activity

  1. moved this to 📥 Queue in data.gov team boardon Jul 2, 2025
  2. FuhuXia commented on Jul 2, 2025

    @FuhuXia
    Member

    To traverse WAF and read path/filenames and file timestamps, we can borrow code from ckanext-spatial.

  3. moved this from 📥 Queue to New 📤 Queue in data.gov team boardon Sep 16, 2025
  4. moved this from New 📤 Queue to 📥 Queue in data.gov team boardon Sep 16, 2025
  5. moved this from 📥 Queue to No status in data.gov team boardon Feb 25, 2026
  6. moved this from 📥 Queue to No status in data.gov team boardon Apr 23, 2026
  7. moved this from O&M Backlog to 📥 Queue in data.gov team boardon Jun 24, 2026
  8. moved this from 📥 Queue (up next) to 📟 Sprint Backlog (Top Priority) in data.gov team boardon Sep 8, 2026
  9. moved this from 📟 Sprint Backlog (Top Priority) to 📥 Queue (up next) in data.gov team boardon Sep 10, 2026
  10. self-assigned this
    on Sep 17, 2026
  11. moved this from 📥 Queue (up next) to 📟 Sprint Backlog (Top Priority) in data.gov team boardon Sep 17, 2026
  12. moved this from 📟 Sprint Backlog (Top Priority) to 🏗 In Progress in data.gov team boardon Sep 21, 2026
  13. moved this from 🏗 In Progress to 👀 Needs Review in data.gov team boardon Sep 22, 2026
  14. moved this from 👀 Needs Review to ✔ Done in data.gov team boardon Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions