Skip to content

Fix: Ford - PAS Dev Loc not Parsing - #132

Open
TLane4N6 wants to merge 2 commits into
abrignoni:mainfrom
TLane4N6:VLEAPP
Open

Fix: Ford - PAS Dev Loc not Parsing#132
TLane4N6 wants to merge 2 commits into
abrignoni:mainfrom
TLane4N6:VLEAPP

Conversation

@TLane4N6

Copy link
Copy Markdown

Extraction from Sync3 did not parse locations due to format with semicolon separator.
Example data: 8/07/2026 5:56:51.842/23156/23/NAV_FRAMEWORK_IF/dev_loc_results/682/=pos_result.Tcount:11565508 GPS Input Lon:-42.123472 Lat:32.123444 Alt:199.000000 Hd:160.383911

Also received access denied error and found that script was attempting to open a directory "pas_debug.log.11" instead of the file with the same name that was within.

The script appeared to stop processing after the above error and didn't attempt to parse additional logs.

All but 2 or 3 of the 25+ logs were stored in gzip archives which were not parsed.

Script vibe coded with Claude.

Updated script went from 0 locations to over 52,000. I also successfully tested the script with a Sync4 extraction. The Sync4 extraction used the same syntax with the semicolon, but did not have any gzip and no directories with same name as the file..

1st time submitting a pull request so let me know if I missed anything.

@abrignoni

Copy link
Copy Markdown
Owner

Thanks for the contribution!

This PR changes artifact modules without test data for them. A small fixture with each artifact change lets reviewers run the module against real data, and the committed case keeps guarding the module after merge.

  • pasDeGeo.py: please include a fixture with this PR.

Adding a fixture

Generate it from your extraction with the helper (details in create_module_test_cases.md):

python admin/test/scripts/make_test_data.py <module> --case <case_number> --input <extraction.zip>

It writes admin/test/cases/testdata.<module>.json and one zip per artifact under admin/test/cases/data/<module>/.

Size rules:

  • Under 10 MB per zip: commit the files in this PR.
  • 10 to 25 MB: commit the case JSON in the PR and attach the zip to a comment here.
  • Over 25 MB: say so here and a maintainer will arrange a handoff.

If your extraction cannot be shared:

  • If the app appears on a public research image, generate the fixture from that instead. public_corpus_images.md lists the images and where to download them.
  • Or sanitize the real file in place: keep the file the app wrote and overwrite only the personal values, which keeps the format honest.
  • Or script a known session: install the app on a test device with a throwaway account, perform documented actions, and extract that.

If none of those fit, say so here and we will work it out. The PR can still be reviewed and merged with the gap recorded in the artifact's notes.

This is a request, not a gate. Nothing here blocks review.

@abrignoni abrignoni added the needs-test-data Artifact PR without test data for the changed modules label Aug 25, 2026
@TLane4N6 TLane4N6 closed this Aug 26, 2026
@TLane4N6 TLane4N6 reopened this Aug 26, 2026
@TLane4N6

Copy link
Copy Markdown
Author

Thanks, I will work on that. I have been unable to find any public images for Ford Sync and the data I have does contain personal info. I am in the process of sanitizing it and will add it when I'm able.

@abrignoni

Copy link
Copy Markdown
Owner

Thanks for this, and welcome. Strong first pull request. Both problems you found are real and I reproduced them here. The seekers hand an artifact the directory path as well as the file inside it, and opening that directory stops the loop, so every log after it goes unparsed. The gzip handling and the Lon: format are both needed.

Two things to fix before it goes in:

  1. Timestamps. timeorder() does not zero pad, so a line with a single digit month or hour, like the one in your description, builds a string fromisoformat rejects. _ts returns the raw text into a column typed datetime, LAVA stores it as text, and those rows sort to the end of the table. Pad month, day and hour.

  2. Truncated gzip. EOFError is not an OSError, so a .gz that ends early still stops the artifact, which is the same failure you just fixed for directories. Adding EOFError and zlib.error there also keeps the lines read before the truncation. PermissionError is already covered by OSError.

Three smaller ones:

  • Log the skip. The module does not import logfunc, so an unreadable log disappears with nothing in the output.
  • source_path is still the last file only. The wrapper splits that element on newlines, so join the logs you read and the "located at" line names all of them.
  • Bump version and last_update_date on the eight artifacts, and put the new format plus the gzip and directory handling in notes.

One to watch: \s* also loosens the existing patterns, which had inconsistent spacing, so counts on older logs can move. Worth a run against a log in the old format so we can see whether they do.

Only the test data bot has run so far. The rest of CI needs my approval on a first pull request and I will kick it off. No rush on the fixture, and sanitizing in place is the right call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-test-data Artifact PR without test data for the changed modules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants