Content Transfer, Session #5
NDIIPP Annual Meeting
Wednesday, July 9, 2008
10:30 a.m. – 12:00 p.m.

Presenters:  
Stephen Abrams; Keith Johnson; Jane Mandelbaum; Justin Mathena; Joe Pawletko; and Brian Vargas

Attendees: 26

Discussion Overview
LC Perspective
•	Year of Content for NDIIPP in 2008
o	Every partner is participating in content transfer to LC
•	Negotiation of what content to send
•	Selection of content to preserve
•	Selection of tools and methods for transfer
•	Bagging and Tagging the data
o	Builds on the experience of each partner
o	Important to import and export as testing a crucial aspect of the digital preservation architecture 	
•	LC experience is that transfer is still hard
•	Goals for moving forward include:
o	Automate the transfer process
o	Continue to reduce overhead
o	Transfer fewer large files
o	Parallelize transfers whenever possible
o	Measure all phases of the transfer

California Digital Library Perspective 
•	CDL is experiencing a direct growth curve of about 100 mb/month of data submission within their own organization
o	Indirect submission tends to be mixed media, handled manually, slower to process, and backlog piles up more easily
•	Worked with LC to develop protocol for transfer: 
o	BagIt spec http://www.cdlib.org/inside/diglib/bagit/bagitspec.html
•	Allows for low overhead transfer
•	Minimal metadata and file system structure
•	Currently working on defining protocol for publishing that an organization has bags available for transfer. 

Stanford University Perspective
•	Stanford has been working with LC to test the BagIt spec and further refine it
•	Batches have been transferred in 500 GB packages
•	Stanford has been working on a content transfer best practices document to share experience and knowledge. Key points include:
o	Checksum should reflect the file at a point-in-time. The original, authoritative checksum should always be the reference throughout the transfer process
o	There must be tolerance in tools. Especially over a network it is especially important that the tools be graceful in restarting sessions 
o	Communication is of the utmost importance throughout the process. 

UCSB Perspective
•	UCSB discussed how transfer is often just as challenging within a single architecture system as it is when transferring between institutions. 
o	NGDA plans to have a 100 year system, that necessitates the transfer of data over many systems
•	Even when it’s called ingest, transfer is a lot of work
o	Flexibility leads to complexity
•	Mathena postulate and Ingest corollary
o	An archive is only as good as its collection; the bigger the collection the better the archive usually is. 
o	The quality of the ingest system has a direct effect on the value of an archive
•	Working together is hard: coordinating efforts, sharing data, not replicating work, and the other aspects of collaboration are not easy. However, our greatest strength is that collaboration. The future needs more open source models, more collaboration, and shared resources (particularly in building long-lived tools)

NYU Perspective
•	NYU is transferring more than 20 hours of public television to the Library of Congress
•	NYU creates a unix mapping to generate sim links in their Storage Resource Broker (SRB) system
•	They froze the collections in SRB during bag pull to ensure the data is there to pull from 
•	Thus far, NYU has transferred 31 bags, 24 hours of content, about 900 GB of data
•	Average bag size 29 – flatten filed 8 megabits per second – each bag contains one episode