January 5, 2011 - While the saying "New Zealand is far from everywhere" may be true, distance is not an issue regarding its digital cultural collections and how efficiently the National Library of New Zealand makes them available over the Internet. For a small country (population approximately 4,367,700 as of June 2010), it has made wise choices that have resulted in an exemplary cyberinfrastructure and national digital library.
Over recent years , the NLNZ has moved towards aggregating its online collections (external link) and high-tech resources under an initiative called the National Digital Heritage Archive (external link). On the front end, the NDHA build their own Web tools and design clean interfaces to make users' experience easy and intuitive. And on the back end they have partnered with Ex Libris (external link) and Sun (now part of Oracle (external link)) to develop an OAIS compliant (external link) repository.
The NDHA also has a great deal of support from their government, which is upgrading the country's overall cyberinfrastructure to help improve their economy and make New Zealand competitive worldwide.
To begin with, the government has mandated legal deposit (external link). As part of the 2003 National Library of New Zealand Act (external link), the NLNZ is permitted to collect digital content, which enables them to avoid the mire of rights wrangling and get right to the job of gathering and archiving their country's cultural heritage.
Publishers are required to submit publications to the library. "And we were given the right to request assistance from publishers in the collection of material under legal deposit if we came up with problems," said Steve Knight, program director for NLNZ's Preservation Research & Consultancy division.
Under the 2003 act, websites were declared to be public documents, so they too were subject to legal deposit. However, there is still the outstanding problem of how they manage password protected, commercial and dynamic or database driven websites.
Similarly, NDHA is researching ways to capture newspaper content. "Maybe some kind of mechanism, some RSS or ping that alerts us to changes in the newspaper," Knight said. "But then do we download the whole newspaper again or can we isolate only the changes and how do we then manage the display and delivery of fragmentary changes?"
As NDHA ponders those technological challenges and solutions, it is still refining its relationships with publishers. A few years ago the NDHA conferred with some of New Zealand's larger publishers about developing an efficient digital archiving workflow from the publishers to the NDHA archives, but Knight said that the time wasn't right.
Steve Knight
"They still hadn't really understood what some of their potential business models were or how to leverage their digital investments," he said. "We offered to embargo any non-public material for as long as they like. Hopefully we all are ready to have that conversation again."
The NDHA also has the right to circumvent technological protection mechanisms for the purposes of digital preservation. For example, if it acquires a DVD or CD-ROM that is "locked," it is permitted by law to unlock it and access the content in order to preserve it.
The NDIIPP Preserving Virtual Worlds (external link) project faces the same challenge: how can you archive and preserve the code and contents of a digital game when you are forbidden by copyright law to crack the game open…even if the game is no longer commercially available?
NDHA's technologists are savvy toolmakers (external link), mindful of the user experience and the usability of an interface. For example, on the DigitalNZ site (external link), users can customize their own personal search tool with a few clicks of a mouse. And tools such as the Web Deposit Tool (external link) make it convenient for publishers or average citizens to deposit their work with the NDHA.
For digital library professionals, there is the Web Curator Tool (external link), a sort of content management system for selecting, crawling and archiving websites. The WCT enables a user to enter descriptive and administrative metadata for a website, schedule and run a web crawl on that site and review the archived content. The WCT uses the Internet Archive's Heritrix (external link) as the crawler.
Indigo is a tool NDHA developed for internal use to load fine-grain digital objects into the NDHA's preservation system, in contrast to massive Web crawls. "Indigo is for the pointy end of material where you want to do boutique activity, such as on unpublished manuscripts or journals," said Knight. "Indigo gets it, packages it into a SIP, creates the METS structure, does fixity activities, does synchronization with the appropriate collection management system, updates the relevant producer records, packages it all up into a simple repository readied SIP and fires it up into the permanent repository."
The NDHA's Metadata Extraction Tool (external link) extracts preservation metadata from a range of file formats and outputs the metadata to an XML file.
Rosetta (external link), developed with Ex Libris, is a digital preservation management system, the heart of their repository. "We've actually developed an integration pipeline from the Web Creator Tool directly into the Rosetta software," said Knight.
The NDHA spent about 18 to 20 months working out business and functional requirements specifications before they sought a developer and found Ex Libris. Knight said, "We wanted to be very, very clear that we had exhausted our own knowledge – and what was then global knowledge – about what a digital preservation system should be able to do."
The NDHA also decided to develop commercial software for Rosetta instead of open-source software. "It was partly because there were more guarantees that we would get what we wanted, as defined in the specification process," said Knight. "And we wanted to have a life cycle and an enhancement process. What you don't get out of most open-source activity is finished product. You don't often get the interfaces, the reporting, the monitoring, the back end and the back-office configurations." He said that these features distinguish an engineer-oriented tool from a tool that is usable by a broader group including internal staff users.
But just because Rosetta is proprietary doesn't necessarily mean that the NDHA has taken a position either way regarding open-source and proprietary. Knight said that some of the tools the NDHA uses – such as Indigo, Droid and the NDHA metadata extractor – are all open source. He said, "By being locked into one or the other, you automatically dismiss what might be good solutions or diminish what you're trying to achieve."
NLNZ has also been developing portals as a single point of access to digital collections from several different participants. Within New Zealand, they worked on Matapihi (external link), a portal to various New Zealand cultural institutions. Later they created an interface, called KRIS (external link), that provided access to a number of university repositories.
They've expanded their portal efforts to reach other national libraries. For example, the Iberoamérica Digital project is a collaboration between the NLNZ and the National Library of Spain. Their goal is to share collections among Spanish- and Portuguese-speaking national libraries.
Despite the NDHA's successes at home and abroad, Knight says they need to keep up their momentum and continue to forge ahead. "There's an enormous amount we've achieved but we have a lot still to achieve," said Knight. "So we need to be not sitting too much on our laurels or we may quickly find ourselves losing traction. We expect complexity to increase in the short term."
