Skip to content

fix(parser): Parse more than utf8#1072

Merged
math-fehr merged 1 commit into
mainfrom
sishtiaq/954_parse_more_than_UTF8
Jul 15, 2026
Merged

fix(parser): Parse more than utf8#1072
math-fehr merged 1 commit into
mainfrom
sishtiaq/954_parse_more_than_UTF8

Conversation

@sishtiaq

@sishtiaq sishtiaq commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

fix(parse): Extend parser to parse more than UTF-8 (#954).

sqlite3.c has a char table that needs to be parsed. This PR allows non-UTF-8 to be parsed too. The three chars {\n,\t,"} are still printed as human-readable, even if their corresponding hex codes are parsed.

The main change is that Parser.parseStringLiteral now returns a ByteArray, rather than a String. Some of the previous uses (AttrParser, MlirParser) converted to Byte anyway; these calculations have now gone.
On the other hand, UnitTest/Parser has a wrapper to convert to String for comparison.

@sishtiaq sishtiaq force-pushed the sishtiaq/954_parse_more_than_UTF8 branch from d079d69 to bc31ba6 Compare July 13, 2026 14:39
@sishtiaq sishtiaq requested a review from GZGavinZhao July 13, 2026 14:40
@sishtiaq sishtiaq self-assigned this Jul 13, 2026
@sishtiaq sishtiaq requested a review from math-fehr July 13, 2026 14:42
@sishtiaq sishtiaq marked this pull request as ready for review July 13, 2026 14:44

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VeIR Benchmarks

Details
Benchmark suite Current: fcbeb63 Previous: 1971543 Ratio
add-fold-worklist/create 1881000 ns (± 69769) 2264000 ns (± 89652) 0.83
add-fold-worklist/rewrite 3599000 ns (± 57399) 4027000 ns (± 79991) 0.89
add-fold-worklist-local/create 1863000 ns (± 27568) 2225000 ns (± 106345) 0.84
add-fold-worklist-local/rewrite 3362000 ns (± 86170) 3716000 ns (± 61682) 0.90
add-zero-worklist/create 1868000 ns (± 62051) 2226000 ns (± 51046) 0.84
add-zero-worklist/rewrite 2235000 ns (± 20681) 2600000 ns (± 51267) 0.86
add-zero-reuse-worklist/create 1576000 ns (± 55623) 1889000 ns (± 90843) 0.83
add-zero-reuse-worklist/rewrite 1866000 ns (± 94180) 2127000 ns (± 49138) 0.88
mul-two-worklist/create 1932500 ns (± 96028) 2209000 ns (± 62608) 0.87
mul-two-worklist/rewrite 5095500 ns (± 185066) 5620000 ns (± 248024) 0.91
add-fold-forwards/create 1896000 ns (± 90378) 2279000 ns (± 74547) 0.83
add-fold-forwards/rewrite 2715000 ns (± 61249) 3063000 ns (± 51863) 0.89
add-zero-forwards/create 1899000 ns (± 38004) 2163000 ns (± 99039) 0.88
add-zero-forwards/rewrite 1790000 ns (± 24509) 1940000 ns (± 16682) 0.92
add-zero-reuse-forwards/create 1551000 ns (± 32898) 1902000 ns (± 49987) 0.82
add-zero-reuse-forwards/rewrite 1402000 ns (± 19942) 1574000 ns (± 33890) 0.89
mul-two-forwards/create 1868000 ns (± 10040) 2239000 ns (± 86176) 0.83
mul-two-forwards/rewrite 3279000 ns (± 71965) 3723000 ns (± 102624) 0.88
add-zero-reuse-first/create 1583500 ns (± 55205) 1837000 ns (± 97092) 0.86
add-zero-reuse-first/rewrite 10000 ns (± 2251) 8000 ns (± 1821) 1.25
add-zero-lots-of-reuse-first/create 1568000 ns (± 57395) 1924000 ns (± 83197) 0.81
add-zero-lots-of-reuse-first/rewrite 790500 ns (± 61209) 801000 ns (± 13405) 0.99

This comment was automatically generated by workflow using github-action-benchmark.

Comment thread Veir/IR/Attribute.lean Outdated

@math-fehr math-fehr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot! I added a few comments, but otherwise all good.

Comment thread UnitTest/Parser.lean
Comment thread Veir/IR/Attribute.lean Outdated
Comment thread Veir/IR/Attribute.lean Outdated
Comment thread Veir/Parser/MlirParser.lean Outdated
Comment thread Veir/Parser/Parser.lean Outdated
@sishtiaq sishtiaq force-pushed the sishtiaq/954_parse_more_than_UTF8 branch from 0bb46b4 to 35e600d Compare July 14, 2026 10:10
@sishtiaq

Copy link
Copy Markdown
Collaborator Author

I've addressed all comments @math-fehr @GZGavinZhao. If you can have a check, and put the PR in to the merge queue please.

@math-fehr math-fehr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that looks good to me, I'll just wait for @GZGavinZhao to review it then I can merge it!

Comment thread Veir/Parser/MlirParser.lean Outdated

@GZGavinZhao GZGavinZhao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks for implementing this. Just a small error message improvement suggestion, then should be good to merge!

@sishtiaq sishtiaq force-pushed the sishtiaq/954_parse_more_than_UTF8 branch from 35e600d to 603bbe1 Compare July 15, 2026 09:17
\{\t,\n,"\} and their corresponding hex codes \{\09,\0A,\22\} are both printed as human-readable \{\t,\n,"\}.
@sishtiaq sishtiaq force-pushed the sishtiaq/954_parse_more_than_UTF8 branch from 603bbe1 to fcbeb63 Compare July 15, 2026 17:25
@math-fehr

Copy link
Copy Markdown
Collaborator

Thanks!

@math-fehr math-fehr added this pull request to the merge queue Jul 15, 2026
Merged via the queue into main with commit d5b22ce Jul 15, 2026
5 checks passed
@math-fehr math-fehr deleted the sishtiaq/954_parse_more_than_UTF8 branch July 15, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants