Scottish Parliament syncer

[ajparsons]

This is a new bridging script that will update people.json from the official Scottish Parliament API as new people are loaded there.

It works by first creating a membership table from the various SP datasets - and then comparing this against our data.

• Calculating start and end reasons from alignment/not alignment between membership and election periods.
• Creating posts for constituencies that don't currently exist.
• Adding people who are present after 2021 that we don't have the ID for.
• Adding/closing memberships when the API has more up to date information about that.
• Flagging mismatches between the two sets of data wider than a 5 day threshold.

I've updated people.json in two ways:

• two people didn't have their sp id.
• added identifers between our memberships and theirs.

This should work for the election, but also should detect new MSPs/party changes if we add it to the daily update at some point.

I've done an experiment deleting a load of current MSPs and then letting it recreate them and this seemed to work ok.

Runs with poetry run python -m pyscraper.rep_syncer sp run [--people PATH] [--dry-run] [--quiet]

It currently also picks up some historical mismatches in membership dates - have decided these don't need to be fixed for the moment, but could pick up later.

This doesn't have the email exporter - but the email API bit is described there so that's not a big job.

As noted in the readme, we actually already have a process for the Senedd that should work if someone is updating wikidata.