[OAI-implementers] Resumption 'from' date.

Xiaoming Liu liu_x@cs.odu.edu
Sun, 10 Feb 2002 21:17:51 -0500 (EST)


On Mon, 11 Feb 2002, Alan Kent wrote:

> If a goal of OAI is to keep it simple and not change radically (which
> I can appreciate), then I would revert to my simpler proposed extension

Based on previous discussions in this list, I think there is one way to
implement a consistent/stateless view (from the harvester) without
modifying protocol.

The suggested way is to encode the resumptionToken with the query
parameter, at the same time data provider always uses datestamp to sort
the query result.

The format should be almost same as I suggested before, but the cursor
will be the current processed datestamp (borrowed from your solution).
likely:

resumptionToken=sets:from:until:metadataformat:processed_datestamp.

The resumptionToken is transparent to service provider, but when data
provider sees this resumptionToken, it will re-create the query, and use
this "processed_datestamp" as a "from" date.

And whether to implement such a mechanism is totally depended on the data
provider, a large /frequently changed data provider may want to use such a
mechanism to support a consistent view to harvester. 

regards,
liu


> which is to allow a server to return an optional addtional date/time in
> ListRecords/ListIdentifiers responses indicating that "the client can
> use this as a 'from' date to resume if the response token times out".
> 
> For the very simple implementations (or small data volume sites), the
> server just omits this value.
> 
> For more sophisticated implementations with a database engine behind
> the scenes (for example, so it can easly sort the records), then for
> each packet it can say "I am guaranteeing to at least have returned
> everything up to this date". This allows a harvester client hitting a
> large site for a first time to not have to start again from scratch if
> something goes wrong (resumption token time out etc). Date resolution
> is fine here (getting some entries a second time is not the problem -
> the problem is starting again from the very beginning).
> 
> My first attempt at a client harvester for example took about a day to
> go to several sites and download everything. It hung several times
> (unknown network issues), meaning I had to restart it on some large
> sites from scratch. Many other sites it failed for (I can post a list
> if people are interested - many seemed to only support GET and not POST).
> 
> Alan
> _______________________________________________
> OAI-implementers mailing list
> OAI-implementers@oaisrv.nsdl.cornell.edu
> http://oaisrv.nsdl.cornell.edu/mailman/listinfo/oai-implementers
>