View source for Preliminary test results for Fedora Ingestion Service

==Usage results==

==='''Notice regarding GSearch Service'''===

During the tests, we noticed that if the Ingest Service stops for a reason (RPC errors, Internet connection problems, operator side cancellation of task), GSearch will delete the index file, and indexes only the last successfully ingested objects, after the incident. 
* Proposed solution: to manage Fedora Commons messaging service in this cases, in order to send a notification to the Repository Admin, or, better, for the massive ingest, to stop the GSearch indexing before the ingest, and fire it after that again, in an background separate thread.
* 1,462,351 milliseconds manually indexing time for 212,713 objects, because the index file was deleted therefore as a console cancellation of the work flow.
* 1,920,544 milliseconds manually indexing time for 258,493 objects after a massive ingest. The messaging service has been disabled.
* 3,433,129 milliseconds manually indexing time for 335,010 objects.

==='''Memory used by Ingestion Service application'''===

** The Ingestion Service uses maximum amount of 320MB of memory and 50% from a processor of 1,8 GB, because of using of [http://xml.apache.org/xalan-j/ Xalan-java 2.7.1 XSLT processor].
** The preparation for Ingest uses the maximum memory, effective ingest uses only 60MB of memory.

==='''Tests results'''=== 

After testing of the Ingest Service on the biggest collection [http://www.keytonature.eu/wiki/Vascular_plants_(UNITS) Vascular plants (UNITS)], reports as follows:
** Preparing for ingest takes: 52 minutes. Files validation and splitting multiple valued items takes 35 minutes from this total time for 49851 objects
** Effective ingest took from 12:50:00 PM to 17:45:00 PM, about 10,000/hour.
** The final amount of objects in K2N Fedora Commons repository at 2009-09-27 5:45 PM is 335,010 digital objects.

==='''Metadata quality. Notices for data providers'''===

* It is strongly recommended to have a complete set of metadata: providers are advised to provide as many metadata as possible for a digital resource;
* It is strongly recommended to provide a unique Resource ID metadata, to improve metadata management;
* Please, do not copy/paste from HTML formatted sources (html pages, TaxonPages, other Internet resources). This metadata are html pre-formatted and our tools for parsing, analyzing and search, work hard to avoid HTML tags in different phases;
* Please, try to keep the metadata coherence, specially when you submit large collection with attachments: if the collection is changing in time and the attachment is changing also, please remove the attachment reference from wiki collection pages in which the attachment doesn't belong anymore;
* Keeping the coherence between Metadata template, used for collection and Infobox Organization template, used for providers description. See [http://www.keytonature.eu/wiki/NatureGate NatureGate] wiki page where the application initial cannot find the provider automatic until a manually correction have been performed instead.



(Return to [[MediaWiki_based_ingest_tool]])