Reproducing Table 1 Sudoku results from the released checkpoints

[SamC12321]

Hello, and thank you for releasing NeurASP and the pretrained Sudoku models.

I am trying to reproduce the results in Table 1, Section 3.2 by evaluating the six released checkpoints without retraining them. I pinned the repository to commit c3d8b48057991d1b75169f5816da1a683b9d8c1d, downloaded the six checkpoints linked from examples/sudoku/README.md, and used the released 75-board test split.

I can reproduce some of the reported values closely, but several of the released-checkpoint results differ from Table 1, particularly for the 19- and 23-board models. I would appreciate clarification on the exact artifacts and evaluation procedure used for the published table.

Environment

• Python 3.7.6
• PyTorch 1.4.0+cpu
• Torchvision 0.5.0+cpu
• Clingo 5.4.0 (py37lua53hf484d3e_0, Potassco channel)
• NumPy 1.18.4
• SciPy 1.4.1
• scikit-learn 0.22.2.post1
• Pillow 7.1.2

The core Python, PyTorch, and Clingo versions follow the versions documented as supported in the paper-era README. Dependencies not pinned by the authors were fixed to releases available by the commit date. In particular, I used Clingo 5.4 rather than 5.5, so the compatibility issue discussed in #1 does not occur.

SHA-256 values of the downloaded checkpoints

cfb2dcead401cb5c8dc3f96bde7cd655422e3b9679cf1dc9267eeca3f359347a  model_data15.pt
75da6d29fd32962e99abfd62b1b4ac4c9d90a17370d2cc7319c6926cedee1109  model_data17.pt
3c6c71d6e4887107b8966c756d4d1b6010f3a9308805bca451d64d5f46165c7c  model_data19.pt
822e72878cffd17060ba33900f1bbda55e0e4e234851403cc0df9c554a883168  model_data21.pt
9b081a748ccb956219ab63f2691c2521953439de73fa2d0830e25f0842da0745  model_data23.pt
791a352198c6a057e486b4addbc458a7e33c3d1fca2b04ede1372d636d4190c8  model_data25.pt

Results

I first ran the released evaluator from examples/sudoku using:

python test.py

Two runs were byte-identical, with output SHA-256:

fd1b2754547cb1578a0426f2dc0a331c6a4a297801f61c33834f0e0c870e6e5c

The table below uses paper / reproduced notation. Reproduced percentages or ranges are measured on the same 75 released test boards.

Training boards	Raw Accidentify	Accidentify, Pi_sudoku without r	Full Accidentify	Direct Accsol
15	15% / 17.33%	49% / 49.33%	71% / 66.67%	71% / 72.00%
17	31% / 37.33%	62% / 64.00–65.33%	80% / 78.67%	80% / 86.67–88.00%
19	72% / 69.33%	90% / 81.33%	95% / 88.00%	95% / 94.67%
21	85% / 82.67%	95% / 92.00%	98% / 98.67%	98% / 97.33%
23	93% / 84.00%	99% / 94.67%	100% / 96.00%	100% / 97.33%
25	100% / 98.67%	100% / 100.00%	100% / 100.00%	100% / 100.00%

As an additional cross-check, the 17.33% raw whole-board accuracy and 98.02% raw cell-level accuracy I obtain for the 15-board checkpoint exactly match the values independently reported in #1 before that report encounters its Clingo 5.5 error.

Evaluation of measurements not produced by test.py

The released test.py prints raw whole-board accuracy, raw cell accuracy, and full-program identification accuracy. I could not find released evaluation code for the Pi_sudoku-without-r column or for direct solution accuracy, so I evaluated those externally without modifying the pinned source.

Pi_sudoku without r

I used the exact program from test.py, removing only the following rule:

{a(R,C,N): N=1..9}=1 :- identify(Pos, img, empty), R=Pos/9, C=Pos\9.

I then scored the resulting identify atoms against all 81 labels of each incomplete input board, following the behavior of testInferenceResults.

Direct Accsol

I first verified with Clingo that each of the 75 ground-truth test puzzles has exactly one solution.

I then used the unchanged full program, extracted its 81 a(R,C,N) assignments, and compared the resulting board directly with the unique ground-truth solution.

Variation between runs

I also observed one-board variation in some of the externally evaluated runs.

The pinned implementation constructs Clingo using eight threads and returns the last collected model:

clingo.Control(['--warn=none', '-t', '8'])
...
return [models[-1]]

If multiple tied or alternative optimal models are produced, this appears capable of making the selected model sensitive to parallel execution. I therefore report the observed ranges above rather than selecting whichever run happens to be closest to the paper.

Questions

Could you please clarify the following?

1. Are the six checkpoints currently linked from examples/sudoku/README.md the exact checkpoints used to produce Table 1?
2. What exact test split, random seeds, and inference settings were used for the published Table 1 experiment?
3. Is the original evaluation code used to compute the Pi_sudoku-without-r and Accsol columns available?
4. How were ties or multiple most-probable stable models handled in the Table 1 experiment?
5. If available, could you provide checksums for the original Table 1 checkpoints?

I am happy to provide the complete logs, environment specification, checkpoint hashes, and the two small external evaluation scripts if they would be useful.

Thank you for any guidance.

[SamC12321]

Update: results from 30 fresh-process evaluations

I repeated the evaluation 30 times in fresh processes using commit c3d8b48, the released 75-board test split, the six checkpoints currently linked from examples/sudoku/README.md, Python 3.7.6, PyTorch 1.4.0, and Clingo 5.4.0.

Values below are mean ± sample standard deviation across the 30 runs. Each entry is shown as paper / released-checkpoint result.

Training boards | Raw NN | Without r | Full identification | Direct Accsol -- | -- | -- | -- | -- 15 | 15 / 17.33 ± 0.00 | 49 / 49.38 ± 0.24 | 71 / 66.76 ± 0.78 | 71 / 72.31 ± 0.83 17 | 31 / 37.33 ± 0.00 | 62 / 65.11 ± 0.51 | 80 / 77.51 ± 0.58 | 80 / 87.47 ± 0.97 19 | 72 / 69.33 ± 0.00 | 90 / 81.33 ± 0.00 | 95 / 88.36 ± 0.69 | 95 / 95.07 ± 0.71 21 | 85 / 82.67 ± 0.00 | 95 / 92.00 ± 0.00 | 98 / 97.47 ± 0.41 | 98 / 97.56 ± 0.51 23 | 93 / 84.00 ± 0.00 | 99 / 94.67 ± 0.00 | 100 / 95.78 ± 0.51 | 100 / 97.33 ± 0.00 25 | 100 / 98.67 ± 0.00 | 100 / 100.00 ± 0.00 | 100 / 100.00 ± 0.00 | 100 / 100.00 ± 0.00

1. Which artifacts were used for the raw NN column?

The raw neural-network results are deterministic across all 30 runs and differ from Table 1 for all six checkpoints.

These measurements do not involve Clingo, so solver tie-breaking or parallel model selection cannot explain the differences.

Could you clarify whether the six checkpoint files currently linked from examples/sudoku/README.md are the exact checkpoint weights used to produce the raw neural-network column of Table 1?

If not, and the original Table 1 checkpoints are still available, could you provide those weights or their checksums?

If the currently linked checkpoints are the exact Table 1 checkpoints, then it would be helpful to know what else differed in the original evaluation setup, particularly:

• the exact test split/test data used for Table 1,

• the preprocessing applied to the inputs,

• and any data-generation or evaluation code that differed from the released version.

This would help isolate whether the discrepancy comes from the checkpoint weights, test data, preprocessing, or another part of the original evaluation pipeline.

2. How was Accsol computed?

I also found that directly measured solution accuracy is not generally equal to full-program identification accuracy.

For each board, I computed both quantities from the same inferred stable model. A board can be identified incorrectly but still be solved correctly. For example, a clue can be predicted as empty, after which rule r can assign the correct digit using the Sudoku constraints.

The 17-board checkpoint shows this clearly:

Full Accidentify: 77.51 ± 0.58%
Direct Accsol:    87.47 ± 0.97%

The implication

correct identification -> correct solution

therefore does not imply that Accidentify = Accsol, because the reverse implication need not hold.

The released test.py computes identification accuracy, but I have not found code in the repository that independently computes completed-solution accuracy.

Could you clarify exactly how the Accsol column in Table 1 was computed?

In particular:

• Was Accsol evaluated independently by comparing the completed a(R,C,N) grid with the ground-truth Sudoku solution?

• Or was Accsol taken to be equal to full-program Accidentify?

• If it was evaluated independently, could you share the original Accsol evaluation code and the solution labels/test solutions used?

At this point, the two main things I am trying to determine are:

1. what exact checkpoint/data/preprocessing artifacts are needed to reproduce the raw NN values in Table 1, and

2. how the published Accsol values were actually calculated.