`oai.islandora.archipelago` - 3 collections harvested at the same time include overlap of records (but are from distinct oai sets)

[christinklez]

Harvesting successful, but these collections seem to be picking up a handful of the same item records
(Note: The linkback to CA-R results in a 404 - this is a separate issue that CA-R has now resolved.)

27828: Ya-Ka-Ama Indian Education Inc.
https://calisphere-stage.cdlib.org/collections/27828/
https://californiarevealed.org/oai/request?verb=ListRecords&set=group:209324&metadataPrefix=oai_dc, 185 results

27771: Society of California Pioneers
https://calisphere-stage.cdlib.org/collections/27771/
https://californiarevealed.org/oai/request?verb=ListRecords&set=group:127243&metadataPrefix=oai_dc, 631 results

27773: Gilb Museum of Arcadia Heritage
https://calisphere-stage.cdlib.org/collections/27773/
https://californiarevealed.org/oai/request?verb=ListRecords&set=group:127241&metadataPrefix=oai_dc, results 1090

For example, you can see that all three of these collections have the “1 miscellaneous paper with list on it” document as the first item. These are likely duplicates that should only appear as part of 27773 Gilb Museum of Arcadia Heritage. The record should also point to this page: https://californiarevealed.org/do/009cd742-e16e-421a-8926-a56eb9174a34
Oai for example item: https://californiarevealed.org/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:californiarevealed.org:node-516734

Note from CA-R:

This reminds me of an issue where I was trying to page through the results without copying the resumption token each time, I was incrementing the final number by one manually. After seeing the same results multiple times, I looked at the resumption token, and it showed that I needed to be about ten numbers higher than I was.

When I viewed these sets, I did not see duplication of the Gilb entries in the other partners. I’m not clear on why this would have happened. I’ll need to pass this on to our developer group. Can you tell me anything else about these harvests? Like did they throw any weird errors, or were multiple sets being harvested at the same time?

My question:

Is this an issue on our harvester/mapper side? Or is this an issue on the archipelago system side?

[barbarahui]

When I try harvesting these 3 collections locally:

Collection 27828: I can fetch 13 pages of metadata (10 records per page) and then there is an error when fetching the 14th page. This is definitely an issue on their end:

requests.exceptions.RetryError: HTTPSConnectionPool(host='californiarevealed.org', port=443): Max retries exceeded with url: /oai/request?verb=ListRecords&resumptionToken=25476 (Caused by ResponseError('too many 500 error responses'))

I can harvest the other 2 collections successfully and I don't see duplicated records when I search across the vernacular and mapped metadata.

Can you try re-harvesting these one after another and see if you get the same results?

I'm wondering if there's some bug in their resumption token code.

[christinklez]

@barbarahui - I've experienced that issue as well, as indicated here! https://github.com/ucldc/archives_ops/issues/416 (btw, I tried to go back to the logs, but they time out and are not retrievable.)

I asked them to try adding Rikolti as a user agent, but they actually asked for more information:

We heard back from our development group today. They are having difficulty pinpointing this behavior and related events in our logs.
 
Can you provide the following:
Any logging on your side that includes the date/time of request and URL for the failed/max request?
The IP address(es) of the crawler, assuming they are static(ish).

I'm pretty certain that the AWS IP address changes like everytime, so that won't be helpful. Can you provide more details from that error log? (Date/time, and any particular associated URL?) I can send this info over to them.

In the meantime, I'll also try to reproduce this so that we can send additional data points over. Thanks!!

[barbarahui]

Here's the whole error complete with date/time:

[2026-07-09, 22:09:47 UTC] {{taskinstance.py:1824}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 889, in urlopen
    return self.urlopen(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 889, in urlopen
    return self.urlopen(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 889, in urlopen
    return self.urlopen(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 879, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='californiarevealed.org', port=443): Max retries exceeded with url: /oai/request?verb=ListRecords&resumptionToken=25476 (Caused by ResponseError('too many 500 error responses'))

I am harvesting from my local machine, but I don't immediately see a way to get the IP address that they would see in their logs. Airflow is running inside a docker container and I'm not sure which IP address they would ultimately see.

[christinklez]

Thank you! I'll send this over to them!

I'm pretty certain the IP address changes everytime (something about picking a new AWS machine or whatever everytime we harvest) so I'll just let them know that.

[barbarahui]

You're right, there's no single IP address associated with our requests to their API, regardless of whether we're running the harvest in AWS or from a developer's local machine.