Add zig implementation by michaelortmann · Pull Request #100 · juditacs/wordcount

michaelortmann · 2024-02-12T06:42:15Z

i know that this repo is no longer maintained, so i dont expect this PR to be merged. This is the first zig code I ever wrote, so although it is the fastest in my benchmark, please bear with me.
zig 1:06.52:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount >out-zig.txt
	Command being timed: "zig/zig-out/bin/wordcount"
	User time (seconds): 62.23
	System time (seconds): 4.10
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:06.52
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8755192
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 28546
	Voluntary context switches: 12501
	Involuntary context switches: 190
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

java 1:07.34:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v java -Xms2G -Xmx8G -classpath java:java/zah-0.6.jar WordCountOptimized >out-java.txt
	Command being timed: "java -Xms2G -Xmx8G -classpath java:java/zah-0.6.jar WordCountOptimized"
	User time (seconds): 90.77
	System time (seconds): 4.06
	Percent of CPU this job got: 140%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:07.34
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 5846176
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 46388
	Voluntary context switches: 7262
	Involuntary context switches: 1776
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360232
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

c++ 1:49.37:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v cpp/wordcount_clang >out-cpp.txt
	Command being timed: "cpp/wordcount_clang"
	User time (seconds): 105.26
	System time (seconds): 4.00
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:49.37
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8046060
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 1300918
	Voluntary context switches: 2
	Involuntary context switches: 337
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360232
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

rust 1:53.38:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v rust/wordcount/wordcount >out-rust.txt
	Command being timed: "rust/wordcount/wordcount"
	User time (seconds): 108.92
	System time (seconds): 3.84
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:53.38
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 9205972
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 13829
	Voluntary context switches: 47317
	Involuntary context switches: 711
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

verify:

$ sha256sum out-*
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-cpp.txt
71ff0ac64c298edbaff79c81fe9c86e2bdcbc250b3df3a99781241d5db162eee  out-java.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-rust.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig.txt

so, the java implementation has a bug
versions:

$ zig version
0.11.0
$ javac --version
javac 21.0.2
$ clang --version
clang version 16.0.6
$ rustc --version
rustc 1.76.0

michaelortmann · 2024-02-13T05:08:48Z

threaded zig 1:00.70:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount_threaded >out-zig-threaded.txt
	Command being timed: "zig/zig-out/bin/wordcount_threaded"
	User time (seconds): 61.24
	System time (seconds): 4.10
	Percent of CPU this job got: 107%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:00.70
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8755372
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 28501
	Voluntary context switches: 21566
	Involuntary context switches: 160
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360232
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

verify:

$ sha256sum out-zig-threaded.txt 
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig-threaded.txt

juditacs · 2024-02-15T21:23:32Z

Thank you, this looks interesting.

We might have a new version of this repo on the way: #97

michaelortmann · 2024-09-10T07:35:54Z

Meanwhile rust 1.81.0 got a faster sort implementation (https://blog.rust-lang.org/2024/09/05/Rust-1.81.0.html#new-sort-implementations), 25% faster in this test, wow!, and zig also got faster. C optimized from #93 is fastest, using only half the memory and doing only 1 voluntary context switch. New benchmarks:
c optimized 0:50.57:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v c/wordcount_opt >out-c-opt.txt
	Command being timed: "c/wordcount_opt"
	User time (seconds): 47.13
	System time (seconds): 3.32
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:50.57
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3779520
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 861087
	Voluntary context switches: 1
	Involuntary context switches: 274
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

threaded zig 0:58.24:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount_threaded >out-zig-threaded.txt 
	Command being timed: "zig/zig-out/bin/wordcount_threaded"
	User time (seconds): 59.46
	System time (seconds): 3.99
	Percent of CPU this job got: 108%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:58.24
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8755944
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 23766
	Voluntary context switches: 23083
	Involuntary context switches: 327
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360232
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

zig 1:02.59:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount >out-zig.txt
	Command being timed: "zig/zig-out/bin/wordcount"
	User time (seconds): 58.14
	System time (seconds): 3.90
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:02.59
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8756132
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 24181
	Voluntary context switches: 23039
	Involuntary context switches: 344
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

rust 1:24.05:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v rust/wordcount/wordcount >out-rust.txt
	Command being timed: "rust/wordcount/wordcount"
	User time (seconds): 79.35
	System time (seconds): 3.74
	Percent of CPU this job got: 98%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:24.05
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 9218764
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 14040
	Voluntary context switches: 71074
	Involuntary context switches: 460
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

verify:

$ sha256sum out-*
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-c-opt.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-rust.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig-threaded.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig.txt

versions:

$ clang --version
clang version 18.1.8
$ zig version
0.13.0
$ rustc --version
rustc 1.81.0

michaelortmann · 2024-09-10T20:47:20Z

Faster less than string compare. New benchmarks:
threaded zig 0:52.48:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount_threaded >out-zig-threaded.txt
	Command being timed: "zig/zig-out/bin/wordcount_threaded"
	User time (seconds): 53.37
	System time (seconds): 3.96
	Percent of CPU this job got: 109%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:52.48
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8756044
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 23768
	Voluntary context switches: 23026
	Involuntary context switches: 317
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

zig 0:57.28:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount >out-zig.txt
	Command being timed: "zig/zig-out/bin/wordcount"
	User time (seconds): 52.86
	System time (seconds): 3.86
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:57.28
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8756124
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 24180
	Voluntary context switches: 23177
	Involuntary context switches: 339
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

michaelortmann · 2024-09-11T07:40:41Z

Tune StringHashMap capacity to avoid resize / rehash. Threaded version is faster than C optimized. New benchmarks:
threaded zig 0:49.62:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount_threaded >out-zig-threaded.txt
	Command being timed: "zig/zig-out/bin/wordcount_threaded"
	User time (seconds): 49.84
	System time (seconds): 3.87
	Percent of CPU this job got: 108%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:49.62
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8756084
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 17985
	Voluntary context switches: 22984
	Involuntary context switches: 314
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360232
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

zig 0:55.88:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount >out-zig.txt
	Command being timed: "zig/zig-out/bin/wordcount"
	User time (seconds): 51.59
	System time (seconds): 3.76
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:55.88
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8756000
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 18380
	Voluntary context switches: 23096
	Involuntary context switches: 383
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

verify:

$ sha256sum out-*
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig-threaded.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig.txt

michaelortmann · 2024-09-12T12:35:33Z

Removed buffered reader overhead and tuned stdin read initial buffer size to avoid resize / rehash. Also uses 7% less memory in this benchmark. New benchmarks:

threaded zig 0:47.80:

	Command being timed: "zig/zig-out/bin/wordcount_threaded"
	User time (seconds): 49.15
	System time (seconds): 2.02
	Percent of CPU this job got: 107%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:47.80
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 7861084
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2
	Minor (reclaiming a frame) page faults: 3863
	Voluntary context switches: 94139
	Involuntary context switches: 317
	Swaps: 0
	File system inputs: 512
	File system outputs: 2360240
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

zig 0:52.72:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount >out-zig.txt
	Command being timed: "zig/zig-out/bin/wordcount"
	User time (seconds): 49.39
	System time (seconds): 2.03
	Percent of CPU this job got: 97%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:52.72
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8109792
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 4228
	Voluntary context switches: 94217
	Involuntary context switches: 280
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

verify:

$ sha256sum out-*
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig-threaded.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig.txt

michaelortmann · 2024-09-12T22:56:25Z

Removed a layer of indirection. Unthreaded version also is faster than C optimized. By 12%. New benchmarks:

threaded zig 0:40.52:

	Command being timed: "zig/zig-out/bin/wordcount_threaded"
	User time (seconds): 40.53
	System time (seconds): 2.10
	Percent of CPU this job got: 105%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:40.52
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8196972
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 4030
	Voluntary context switches: 94217
	Involuntary context switches: 241
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

zig 0:44.33:

$ cat data/huwiki-latest-pages-meta-current.xml | /usr/bin/time -v zig/zig-out/bin/wordcount >out-zig.txt
	Command being timed: "zig/zig-out/bin/wordcount"
	User time (seconds): 41.02
	System time (seconds): 2.01
	Percent of CPU this job got: 97%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:44.33
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8447200
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 4259
	Voluntary context switches: 94164
	Involuntary context switches: 218
	Swaps: 0
	File system inputs: 0
	File system outputs: 2360224
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

verify:

$ sha256sum out-*
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig-threaded.txt
ea81dd93280cfc6e64d0037ba63388cc0fad9f9ac325393c625e8838c8607ed5  out-zig.txt

Michael Ortmann added 2 commits February 11, 2024 21:19

Add zig implementation

e5d702a

Add threaded zig implementation

6b1ba24

Migrate build.zig to zig 0.12.0

87b8b64

Faster less than string compare

6641d76

Tune StringHashMap capacity to avoid resize / rehash

8ecabc4

Tune stdin read initial buffer size to avoid resize / rehash

cd94f04

Remove a layer of indirection

a2d9b38

Navigation Menu

Add zig implementation - #100

Add zig implementation#100
michaelortmann wants to merge 7 commits into
juditacs:masterfrom
michaelortmann:zig

michaelortmann commented Feb 12, 2024 •

edited

Loading

Uh oh!

michaelortmann commented Feb 13, 2024

Uh oh!

juditacs commented Feb 15, 2024

Uh oh!

michaelortmann commented Sep 10, 2024 •

edited

Loading

Uh oh!

michaelortmann commented Sep 10, 2024

Uh oh!

michaelortmann commented Sep 11, 2024

Uh oh!

michaelortmann commented Sep 12, 2024

Uh oh!

michaelortmann commented Sep 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

michaelortmann commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelortmann commented Feb 13, 2024

Uh oh!

juditacs commented Feb 15, 2024

Uh oh!

michaelortmann commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelortmann commented Sep 10, 2024

Uh oh!

michaelortmann commented Sep 11, 2024

Uh oh!

michaelortmann commented Sep 12, 2024

Uh oh!

michaelortmann commented Sep 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

michaelortmann commented Feb 12, 2024 •

edited

Loading

michaelortmann commented Sep 10, 2024 •

edited

Loading