[OAI-implementers] Perl oai modules
tdb01r at ecs.soton.ac.uk
Fri Feb 25 20:55:03 EST 2005
Ed Summers wrote:
> On Fri, Feb 25, 2005 at 09:28:47PM +0000, Tim Brody wrote:
>>These modules represent very similar functionality to Ed Summer's
>>Net::OAI::Harvester for harvesting, but with support for constructing
>>OAI responses (the repository side). I hope (!) they are also more
>>resilient to repository weirdness.
> Heh, could you be more specific about the "repository weirdness". Nice
> work by the way.
UTF-8 errors are probably the most common, but I've "developed out" my
capability to handle those. Ideally I'd like a PerlIO layer that handles
bad UTF-8, otherwise I'm sure Perl5.8's UTF support should be usable to
fix bad chars (or spot latin and recode).
A couple of sites have bad baseURLs in their Identify response, which I
now check for and ignore - in principle a harvester should follow
baseURL like a redirect, but some sites have relative URLs!
One site indexed in Celestial (an OAI1.1) expects the verb as the first
argument in the query, so I've added a special case for that.
Another site has a bad implementation of timestamps - it only accepts
date+time. So something else to add would be to retry a request with
appended to from/until T00:00:00Z (probably application layer rather
I've added support for oai_dc: or dc: prefixed tags in Dublin Core,
although I guess I could just check for the namespace.
Other than that; graceful handling of retry-after, redirect, resumption
token repetition (to one depth only), and retry-on empty response.
All the best,
More information about the OAI-implementers