FW: [OAI-implementers] Open Archives Initiative Protocol for
Meta data Harvesting Version 2 news
Thu, 14 Feb 2002 10:20:04 -0500 (EST)
On Fri, 8 Feb 2002, Xiaoming Liu wrote:
> Because the datastamp is always increasing in OAI, I think Alan's
> method (high resolution date stamps and results is ordered by
> time) will also work, but not necessarily monatomically, if the DP could
> return all records of a specific datestamp in one reply. But it did
> put some dangers to harvester as Walter suggested, if suddenly DP creates
> 10K records with same datestamp, it has to return them in one response, it
> quite possibly will break the harvester.
This is pretty much the solution I adopted for arXiv. The resumptionToken
is essentially a query to resume the list and where possible this follows
day granularity. However, we have a few days with large numbers of updates
(migrations) so to cope with this I add an index within the day.