[OAI-implementers] How to verify a download worked?
Mon, 4 Mar 2002 15:14:44 +1100
I was wondering if anyone has good schemes for verifying if a download
of metadata 'worked'. For example, I crawled the arXiv site and got
around 60,000 records. However, it turns out the site actually has
190,000 or so records. So I only got 1/3 of the site!
Has anyone used any clever tricks to verify how well a crawl worked?
I now have to work out if my crawler has been discarding one in three
Alan Kent (mailto:firstname.lastname@example.org, http://www.mds.rmit.edu.au)
Postal: Multimedia Database Systems, RMIT, GPO Box 2476V, Melbourne 3001.
Where: RMIT MDS, Bld 91, Level 3, 110 Victoria St, Carlton 3053, VIC Australia.
Phone: +61 3 9925 4114 Reception: +61 3 9925 4099 Fax: +61 3 9925 4098