State Records Home
Personal tools
You are here :: Home → Recordkeeping in the NSW public sector → Topics → Digital recordkeeping → Digital records preservation discussion paper → Digital records preservation requirements
 

Digital records preservation requirements

The New South Wales Government's approach to digital records preservation must support the keeping of authentic and accessible records for use now and by future generations. It must therefore be shaped by requirements for archival records, legislative rules and by environmental issues specific to the New South Wales public sector. The approach chosen must be:

  • feasible - public offices and State Records must have hardware and software capable of implementing the method
  • sustainable - either the method can be applied indefinitely into the future or there are credible grounds for asserting that another path will offer a logical sequel to the method, should it cease being sustainable
  • practical - implementation should be within reasonable limits of difficulty and expense, and
  • appropriate - for the types of objects to be preserved and the specific objectives of preservation. [1]

Accordingly, various New South Wales government requirements influencing our choice of digital records preservation approach are discussed in this section. These are:

  • the need for authenticity
  • legislative requirements affecting digital records preservation
  • the size and complexity of the New South Wales public sector
  • technologies in use now for making and managing digital records, and
  • the need to preserve both State archives and temporary records.

The need for authenticity

A key requirement for the preservation of digital records in particular - given their transient nature and vulnerability to compromise - is authenticity.

Government and the community expects that the custodian of its archives will protect and preserve them in such a way that they can be relied upon to reflect an accurate picture of Government's activities and can be trusted as 'official', conveying the meaning and intentions of the records creators. This expectation can only be met by ensuring the authenticity of the records being preserved.

Archives and records professionals have been considering how to preserve authentic digital records for some time. A significant project that examined this question was the International Research on Permanent Authentic Records in Electronic Systems (InterPARES), coordinated by the University of British Columbia, Canada. An InterPARES Preservation Task Force Report from 2001 notes that:

"Naively, preservation may be seen as a process that keeps records free from change. However, it can be easily shown that it is practically impossible for any record to remain absolutely unaltered or immutable over time…The archival requirement for integrity, and therefore authenticity, depends on the message intended by the record creator." [2]

This is a widely accepted view in archival thinking. Digital records preservation programs need to be concerned as much with defining how they will preserve the intent and meaning of the record as they are with preserving its content. This is no easy task as the characteristics that give a record its meaning can vary from case to case.

For example:

  • in a Word document, meaning might be conveyed via bolding of certain words;
  • in a digital map, meaning might be conveyed via the colours used;
  • in a report from a database meaning might be conveyed in the way the data is laid out.

In their green paper An Approach to the Preservation of Digital Records (2002), the National Archives of Australia describe the concept of a record's 'essence' - the combined characteristics of a record which are essential to understanding how its creator meant for it to be understood. The National Archives of Australia's digital preservation program, now operational, seeks to determine and preserve the essence of the digital records in their custody so they can adequately recreate their meaning for future users. However they do not propose to do this by recreating precisely the original 'look and feel' of every record. By contrast, the Public Record Office Victoria's VERS Project regarded the original appearance of documents to users as a significant consideration in the selection of their preservation technique, in part as a result of the requirements of local evidence laws. [3] The New South WalesEvidence Act 1995 does not impose such requirements (see section 5.2).

The InterPARES Project's findings emphasise the importance of the creation and capture of metadata as a fundamental support to a record's authenticity, including metadata that should come with records, documenting the immediate context of their creation and the manner in which they have been handled and maintained. The findings also indicate that the preserver (such as an archival institution) must be able to demonstrate that:

  • an unbroken chain of custody of the records is maintained
  • security and control procedures are implemented and monitored
  • the content of the record and any required annotations and elements of documentary form remain unchanged after preservation, and
  • they have documented the preservation process. [4]

Some of these requirements will be met by enforcing strict controls over the security of the transfer process. Implementing standards such as the New South Wales Recordkeeping Metadata Standard and the Australasian Digital Recordkeeping Initiative's (ADRI) Digital Record Export Standard [5] will assist us in specifying that information which we will require at a minimum for preserving authentic digital records.

In summary:

  • Adequate recordkeeping metadata must be linked to or otherwise associated with any digital record being preserved.
  • The approach to digital preservation used must be:
  • capable of demonstrating that the content of the record and essential characteristics remain unchanged as a result of preservation processes
  • capable of recording an appropriate audit trail of preservation activities linked to or otherwise associated all records that are preserved
  • Compliance with digital recordkeeping standards supports authenticity.

Legislative requirements affecting digital records preservation

There are a number of pieces of legislation that contain provisions which must be considered in the selection of an approach to digital records preservation, including:

  • the requirements of the State Records Act 1998 relating to the making and keeping of records by public offices and the management of the State’s archives
  • the requirements of the Evidence Act 1995 for the provision of copies of records from electronic systems, and
  • the requirements of the Electronic Transactions Act 2000 for the retention of records of business conducted electronically.

State Records Act 1998

Under section 14 of the Act, public offices are obliged to 'maintain accessibility to equipment / technology dependent records', for as long as they are required.

The Act also requires public offices to make and keep full and accurate records. The Standard on Full and Accurate Records, issued in support of this requirement was developed to reflect the qualities that a record should possess as described by the Australian and International Standard AS ISO 15489-2002 Records Management, that is, that records must:

  • be made
  • be accurate
  • be authentic
  • have integrity, and
  • be useable.

These requirements need to apply to records both before and after they have undergone digital preservation processes in any public office. The key requirements in relation to records going though preservation processes are authenticity, useability and integrity. The minimum compliance requirements for these are:

Records must be authentic:

  • records are routinely captured into official recordkeeping systems;
  • appropriate metadata is created and captured, or otherwise associated with, records.

Records must have integrity:

  • unauthorised access, alteration, deletion or destruction of records is forbidden by office policy and practice;
  • recordkeeping systems and storage facilities are designed, implemented and monitored to protect records from unauthorised access, alteration, deletion or loss;
  • records are uniquely identified;
  • migration of records from one system to another is controlled and documented.

Records must be useable:

  • records are linked to the business context;
  • records relating to the same business activity or transaction are linked to each other;
  • location and use of records is recorded and tracked;
  • records are accessible for as long as they are required.

In addition, the State Records Act, along with the Freedom of Information Act 1989 and the Privacy and Personal Information Protection Act 1998 provide for public access to government records meaning that records must be accessible and useable for as long as they are required.

Evidence Act 1995

This Act, mirrored in most other Australian jurisdictions, abolished the 'best evidence' rule, meaning that copies of records from electronic systems may be provided as evidence in courts of law. The responsible public office must, however, ensure the records' integrity so they will not be challenged by the opposing side. Measures recommended for ensuring digital records will not be regarded as poor evidence includes:

  • written policies and procedures should be in place to define operations for development, maintenance and use of electronic recordkeeping systems;
  • training and support should be provided to reinforce policies and procedures to staff;
  • system controls should be developed to ensure the accuracy and reliability of the records produced;
  • system audit trails should be implemented to show the history of the record, i.e. the creation, use and context;
  • routine and regular tests of the systems performance should be conducted, including hardware and software;
  • adequate security should be provided to limit access [6].

Electronic Transactions Act 2000

The Electronic Transactions Act 2000, section 11 (2) says:

'If, under a law of this jurisdiction, a person is required to retain, for a particular period, a document that is in the form of paper, an article or other material, that requirement is taken to have been met if the person retains, or causes another person to retain, an electronic form of the document throughout that period, where:

(a)  having regard to all the relevant circumstances at the time of the generation of the electronic form of the document, the method of generating the electronic form of the document provided a reliable means of assuring the maintenance of the integrity of the information contained in the document, and

(b)  at the time of the generation of the electronic form of the document, it was reasonable to expect that the information contained in the electronic form of the document would be readily accessible so as to be useable for subsequent reference, and

(c) if the regulations require that the electronic form of the document be retained on a particular kind of data storage device-that requirement has been met throughout that period.'

These requirements from the Act indicate that where information is made and kept in support of business transactions there is an expectation that its integrity is maintained. There is also a need for that information to be '…readily accessible and useable, both immediately and for subsequent reference'.

In summary:

  • The method of preserving digital records must not damage the integrity, useability, accessibility and authenticity of the records.
  • The preservation process that the digital records go through must be accountable itself.

The size and complexity of the New South Wales public sector

New South Wales 'public offices' include State government agencies, local government, the public health system, universities and State owned corporations. The New South Wales Parliament, the Courts and the Office of the Governor are covered by the Act by agreement. Over five hundred public offices are included in State Records' compliance monitoring activities.

The sector encompasses a wide variety of organisations of varying sizes. Surveys conducted by State Records have shown that a large proportion of Government agencies and councils are small or very small - small meaning fewer than 80 full time staff, and very small meaning fewer than 20 full time staff [7]. These small and very small agencies are often lacking in technical capabilities and specialised staff in support areas like ICT and records management. While most Government agencies have their head offices in metropolitan Sydney, the majority of councils are located outside of metropolitan Sydney. Of the Universities, more than half are located in regional centres.

Like all other government jurisdictions, New South Wales frequently experiences administrative change, with government bodies merging and splitting and functions transferring from one to the other. This change can often result in dilemmas for those responsible for transferring records from one organisation to another, especially where the records are in proprietary formats.

In summary:

  • The approach taken to digital records preservation must be able to be implemented in public offices of a range of different sizes including small and very small public offices.
  • The approach taken to digital records preservation should not require large investments in additional personnel, technology or infrastructure for public offices choosing to implement it 'in house' to preserve temporary value records.
  • It would be desirable if the approach taken to digital records preservation simplified the process of transferring records between agencies in the event of administrative change.

Technologies in use now for making and managing digital records

Despite its size and complexity there is some homogeneity across the New South Wales public sector in terms of applications currently in use. The majority of respondents to State Records' 2005 Information Survey on Digital Recordkeeping advised that they used Microsoft Office for their standard desktop applications such as word processing, presentations and spreadsheets. Most agencies use either Outlook (Microsoft) or Groupwise (Novell) for email. SAP is widely used for business process management, financial and HR management, along with specialised products such as CHRIS and Sybase. The New South Wales Government ICT policy People First has directed a move towards further centralisation of ICT systems such as websites, exploring standardising 'back office applications' and investigation of open source options for government ICT.

However the New South Wales Government is also characterised by a wide variety of 'home grown' applications built for specific purposes such as case management systems such as the Department of Corrective Services' Offender Information Management System or the Department of Corrective Services' 'KiDS' system.

The nature of systems implemented specifically for keeping records is a critical issue for the success of a digital preservation program. In a survey conducted amongst Government agencies on behalf of the Government Chief Information Office in 2004, it was found that 93% of respondents had implemented a records management system, 47% had implemented a document management system and 40% a web content management system [8]. In terms of records / document management systems, the most commonly used product is TRIM Captura (1) and Context (2), with Objective running at third place behind these. It is important to note, however, that there is still a wide range of other products in use in New South Wales Government, many of which are not on the current GSAS RIMS contract, and that the quality of the implementations of these systems is patchy. It would be unrealistic to expect all digital records that are eligible for transfer as archives to have rich and well defined metadata linked to them. In some cases we will need to accept records that have been managed in less than perfect recordkeeping environments, with minimal metadata.

In summary:

  • State Records' digital preservation approach must be able to cope with a wide variety of formats, including non-documentary formats.
  • State Records' digital preservation approach must be flexible enough to cope with records from EDRM applications, especially TRIM and Objective, and from custom built applications such as case management systems.
  • State Records must be prepared to accept records for preservation as archives that have minimal contextual information with them regarding their management and use.

The need to preserve both State archives and temporary value records

State Records has a responsibility to both preserve digital State archives and also to support and assist public offices in the preservation of temporary value records, some of which may need to be retained for very long periods of time.

State archives

The State Records Act 1998 permits archives to be physically stored with State Records or with a public office under a distributed management agreement. The approach we take to preserving archives and other records in digital formats must be able to be implemented under either of these models; that is:

  • by a public office physically transferring digital records to State Records to be preserved as State archives, or
  • by a public office or regional repository retaining digital records as State archives in their custody.

The documentation of context for digital records brought under the control of State Records as digital archives will need to be subject to the same standards as any other archives under the same control. State Records employs the series system for describing and managing archives, which requires documentation of the functional and administrative contexts that records are created in [9]. In addition, digital State archives will need to be processed and accessible quickly, in order to meet public office requirements for retrieval.

Temporary value records

Many records that are not required as State archives have very long retention requirements. For example records relating to the environment, health or utilities and infrastructure. Where these records are kept in digital formats, the State Records Act requires that the responsible public office ensures that the records remain accessible for as long as they are required. Currently, public offices can follow the guidelines on ensuring the accessibility of digital records that are offered in the State Records publication Future Proof [10]. These suggest a combination of strategies with an emphasis on migration. There is, however, a need for more practical assistance and tools for public offices that need to maintain digital records over long periods of time. By providing such tools and support, there is also an opportunity to reduce costs that could be incurred by many public offices across government potentially implementing their own digital records preservation projects.

In summary:

  • Records may need to be preserved as digital State archives when they are in a public office's custody, a regional repository's custody or State Records' custody.
  • Preserved digital archives must be able to be managed under State Records' existing archival control systems regardless of their physical location.
  • The method of preservation must allow for fast processing of transferred archives, to ensure access requirements are met as soon as practicable.
  • There is a need for digital records preservation tools and advice for public offices that wish to preserve records 'in house'.

Footnotes

[1] Thibodeau, Kenneth, 'Overview of technological approaches to digital preservation and challenges in coming years', The State of Digital Preservation: An International Perspective, Conference proceedings (WashingtonDC: Council on Library and Information Resources

[2] InterPARES Preservation Task Force Report, Appendix 6: 'How to preserve authentic electronic records' http://www.interpares.org/ip2/ip2_products.cfm

[3] Public Record Office Victoria, Victorian Electronic Records Strategy Final Report, 1999. http://www.prov.vic.gov.au/vers/pdf/final.pdf

[4] InterPARES Authenticity Task Force Report, Appendix 2: 'Requirements for assessing and maintaining the authenticity of electronic records', March 2002. http://www.interpares.org/book/interpares_book_k_app02.pdf http://www.interpares.org/ip2/ip2_products.cfm

[5] The ADRI Digital Record Export Standard prescribes the physical representation of electronic records that are to be transferred to an archive. It specifies the metadata that must be transferred with each file, optional metadata and the way the record is physically represented as an XML document. Available in draft form from www.adri.gov.au

[6] National Archives of Australia, Archives Advice 23: Providing records in evidence, 2003. http://www.naa.gov.au/recordkeeping/rkpubs/advices/advice23.html

[7] The survey consulted for this data was the 2004 Records Management Survey available at http://www.records.nsw.gov.au/documents/recordkeeping-surveys/2004 Records Management Survey Report.pdf This survey did not include area health services.

[8] Review of Content Management for NSW Government, conducted by James Robertson, Step 2 Designs, April 2004, p22

[9] For more information see State Records, Archives in Brief 52 - Introducing Archives Investigator http://www.records.nsw.gov.au/state-archives/guides-and-finding-aids/archives-in-brief/archives-in-brief-52

[10] State Records, Future Proof: Ensuring the accessibility of equipment / technology dependent records, 2004.  http://www.records.nsw.gov.au/recordkeeping/guideline_14_future_proof_5051.asp