Skip to content

Track package and paper metrics weekly - #137

Merged
sbfnk merged 14 commits into
mainfrom
stats-tracking
Aug 31, 2026
Merged

Track package and paper metrics weekly#137
sbfnk merged 14 commits into
mainfrom
stats-tracking

Conversation

@sbfnk-bot

@sbfnk-bot sbfnk-bot commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Adds a weekly job that records download counts, GitHub stars and citation counts, and shows the download figure on the software page.

Issue #85 asked for this to support grant applications. The value is the history, which only accumulates from the day the job starts, so this collects the numbers now and keeps the presentation minimal.

What it collects

_data/package-stats.csv gets one row per package per week: stars, forks, open issues, whether the package is on CRAN, and CRAN downloads for the last month. The list of packages comes from the r-universe registry plus _data/software-extras.yml, so it tracks exactly what the software page shows and needs no list of its own.

_data/paper-citations.csv gets one row per paper per week, with citation counts from OpenAlex. The DOIs come from _data/papers.bib, which the publications workflow already maintains.

Rerunning on the same day corrects the rows rather than duplicating them.

What it shows

One addition to the software page: the download figure joins the existing metadata line, reading R · Updated Aug 2026 · 598 CRAN downloads/month. Naming CRAN matters because only the ten packages on CRAN have a figure, and without it a blank could be read as "not used" rather than "not on CRAN".

Nothing else is rendered. In particular there is no citation count anywhere on the site.

Why citations are collected but not shown

Citation counts belong in a grant application, not on a public page. Beside each paper in a list sorted by year they become a column the eye reads down and ranks, including work by different people in the group, and recent papers legitimately show zero: the ten 2025 papers have 19 citations between them, which is normal and looks poor in a table.

The numbers are also concentrated. Of 5,240 citations across 39 papers, 2,830 belong to a single 2020 paper, and the median paper has 15. OpenAlex indexes fewer sources than Google Scholar, so these are a lower bound.

Packages carry no citation column at all. Of sixteen, three declare a citable reference, and one of those is a CRAN package DOI that is almost never cited, so the column would have been blank or misleading nearly everywhere.

Stars are collected but not displayed for the same reason. They measure visibility rather than use, and the two diverge: scoringutils has 934 downloads and 64 stars, EpiNow2 has 598 and 140, contactsurveys has 263 and 2.

Notes

The job commits the CSVs directly to main rather than opening a weekly pull request, which needs permissions: contents: write. That commit triggers a site build, so the download figures on the software page refresh themselves.

If _data/package-stats.csv is absent the page still renders, but every package then falls to the "On r-universe" branch rather than showing a download figure, so the file is expected to be present once the job has run.

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
@sbfnk
sbfnk marked this pull request as ready for review August 26, 2026 09:03
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Comment thread _automation/get_stats.R Outdated
Comment thread software.qmd Outdated
Comment thread .github/workflows/update-stats.yaml Outdated
sbfnk-bot and others added 2 commits August 26, 2026 10:43
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Comment thread _automation/get_stats.R
Comment thread software.qmd Outdated
sbfnk-bot and others added 2 commits August 26, 2026 10:54
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Comment thread _automation/get_stats.R Outdated
Comment thread software.qmd Outdated
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Comment thread software.qmd Outdated
Comment thread _automation/get_stats.R Outdated
sbfnk-bot and others added 5 commits August 26, 2026 12:42
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
…age name

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
…right name

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
…sitory

Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Comment thread .github/workflows/update-stats.yaml Outdated
echo "No change this week"
else
git commit -m "Weekly package and paper metrics"
git push

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A second, more serious problem on this line, separate from the rebuild one already open above: the push will be rejected outright, so the job never records anything.

repos/.../rules/branches/main returns ["deletion","non_fast_forward","pull_request"] — an active ruleset requiring a pull request on the default branch. GITHUB_TOKEN bypasses that only if listed as a bypass actor, and the evidence says it is not: every github-actions[bot] commit on main arrived as a PR merge (#131, #124, #103, #96, #92), never a direct push; update-publications.yaml, the repo's other weekly job writing to _data/, routes through peter-evans/create-pull-request; and release-reminders.yaml uses an App token rather than GITHUB_TOKEN.

Failure scenario: the PR merges, Monday's job collects fine, git commit succeeds, git push is rejected, the run goes red and nothing is written. That repeats weekly with no signal beyond a failed run. Worse, it interacts with the 35-day window added in 5ea6dd2: once the single committed date falls outside it, the page drops all ten CRAN figures and renders "On r-universe" links instead, with nothing to say why. The feature quietly un-does itself about five weeks after merge.

Not fixing it — .github/** stays a human's call. Both options that work are already patterns in this repo: create-pull-request as in update-publications.yaml (fixes the push, not the rebuild), or an App token as in release-reminders.yaml (fixes both, since a push under a non-Actions identity also triggers build-and-deploy).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 772e351, at the maintainer's direction — this file was left alone through the review rounds on purpose, and is changed now because it was asked for rather than because a reviewer suggested it.

The job now generates a token from the same GitHub App release-reminders.yaml uses, hands it to actions/checkout so git push carries it, and passes it to the R code as GITHUB_PAT. That addresses both findings on this line: an App is a distinct identity from github-actions[bot], so its push triggers build-and-deploy, and Apps are what ruleset bypass is normally granted to. The permissions: contents: write block is gone, since the App token carries its own.

Two things this still needs, neither of which is visible from here — both org settings:

  1. The App must have Contents: read and write on this repository.
  2. The App must be a bypass actor on the "Protect default branches" ruleset.

Until both are true the push will still be rejected. The cheap check is to dispatch this workflow once from the branch and see whether it pushes.

sbfnk-bot and others added 2 commits August 31, 2026 09:03
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
Co-authored-by: sbfnk <sebastian.funk@lshtm.ac.uk>
@sbfnk
sbfnk merged commit c73ca41 into main Aug 31, 2026
3 checks passed
@sbfnk
sbfnk deleted the stats-tracking branch August 31, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants