Managing digital records overview (Guideline 22)
- Purpose of this guidance
- Make digital recordkeeping achievable
- Keep your digital records in recordkeeping systems
- Use recordkeeping metadata for digital recordkeeping
- Effectively manage the migration of your digital records
- Target specific record formats that are causing you problems
- Definitions and annotated bibliography
Overloaded networks, flooded email systems, large and complex record formats, a multitude of mandatory recordkeeping rules to comply with, system change and continual technological obsolescence – those managing digital records have a lot to deal with. To complicate matters further, all the problems associated with digital records have to be dealt with now. You can’t defer the problem of their management because doing nothing about your digital records will consign them to oblivion just as surely as doing the wrong thing. This guidance is intended to help you get control of your digital records and manage them effectively.
Digital records will only have value for your organisation if they are accessible and if you can preserve their authenticity and integrity.
To maintain digital record authenticity, integrity and accessibility you should try to:
- make digital recordkeeping achievable for your organisation
- keep your digital records in recordkeeping systems
- use recordkeeping metadata for digital recordkeeping
- effectively manage the migration of your digital records, and
- target specific record formats that are causing you problems.
These five strategies are explained briefly below, with links to more information.
Due to their sheer quantity, diversity and complexity, managing digital records can be a very challenging task. You can apply numerous strategies to make the management of digital records achievable.
Share responsibility for managing digital records
It’s not all up to one group of people in the organisation. It is important to recognise that the success of digital records management strategies require a number of stakeholders to work in partnership. | More |
Limit the number of file formats used in your organisation
It is estimated in the business environment that 4500 types of record formats exist today. It greatly simplifies the management of digital records and minimises costs if you identify a minimum set of formats which meet business needs and longevity concerns and restrict data creation to those formats. | More |
Use open formats
Open formats are less at risk of becoming inaccessible because of changes to vendor arrangements and are easier to migrate. Using these for record creation or migrating records to these formats can save you a lot of money and effort in the long term. | More |
Standard templates for different business processes can be developed in your organisation. These can promote business efficiency but can also help with long term digital record management. For example, using templates can help you migrate records to new platforms, facilitating large scale preservation projects. Using templates will assist you to generate high quality XML if you want use this durable file format for your long term preservation. | More |
Apply standard creation rules
There are a number of creation practices specific to the type of document being created that can either promote or endanger the long term preservation of records. You may want to consider implementing rules about these in your organisation. | More |
Keeping digital records is expensive. You will save money long term by destroying digital records when it is appropriate to do so, under the terms of your approved retention and disposal authorities.
Beware of those who argue for keeping all digital records because storage is cheap. While this may be so:
- preserving records is expensive! Public offices have an obligation to maintain accessibility to equipment/technology dependent records under s.14 of the State Records Act. Maintaining accessibility over time is an expensive undertaking and should only be considered for records that are truly needed and are required to be kept for long term retention or as State archives
- some records should be routinely destroyed as authorised e.g. those containing personal or sensitive information in order to respect the rights of individuals and comply with privacy laws
- retaining records unnecessarily leads to information confusion - staff are presented with too many hits for each information request. This can result in loss of productivity as staff try to filter information requests and find the most relevant information
- retaining records unnecessarily can also inhibit the ability to locate and retrieve information quickly and efficiently for Freedom of Information requests, Discovery Orders
- retaining large numbers of digital records may reduce server speed resulting in a user response lag time
- at some point, even with cheap storage, information will need to be deleted to improve performance and make way for new records. If you have not tagged records from their creation with authorised disposal information, the process of deciding what to delete will be labour intensive and difficult.
| More |
Implement a 'technology watch'
Digital records are particularly vulnerable to obsolescence. It is necessary for all organisations with digital records to mount a ‘technology watch’ to monitor their condition. This will require you to be aware of the systems and formats used within your own organisation and to be alert to changes by vendors or other factors that may affect organisational systems and digital resources. | More |
Plan for the costs of digital preservation
It is important that you allocate adequate resources for digital preservation. Costs of digital preservation will vary according to the strategies chosen, past and present controls and the complexity of the digital environment. Careful planning can help you to reduce costs significantly. | More |
Case study: Be aware of what you have stored on removable storage media
One organisation used a large quantity of removable storage media to store records that it needed to keep long term but did not have space to store on its online servers. The storage media used were not clearly labelled and they were stored together with the organisation’s backup tapes. After several months staff had forgotten what was actually on these storage media and they were included in the cycle of backup tapes. The data they contained was quickly overwritten and the records that had long term value to the organisation were lost. As they were not on the server there were no backups available. | More |
What is a recordkeeping system?
Digital records will only have value for your organisation if they are accessible and if you can preserve their authenticity and integrity. A recordkeeping system is a system that captures, protects and provides access to records over time. Recordkeeping systems therefore make records accessible and also employ the necessary controls that can ensure record authenticity and integrity. They are necessary business tools for the use and preservation of your digital records.
Recordkeeping systems don’t have to be big and they don’t need to be comprised of purpose built software. They can be fairly simple systems, as long as they provide the appropriate controls.
Capturing or saving digital records into a recordkeeping system means that records are:
- stable and fixed in content and form
- easily accessible
- linked to metadata detailing their technological dependencies
- managed according to specified business rules
- protected against unauthorised action
- linked to their business context, and
- retained for the correct length of time.
| More |
Any system in your organisation that is used to perform business and which is required to maintain records of that business should be configured as a recordkeeping system.
|A recordkeeping system can be:||For example:|
| More |
What systems are not recordkeeping systems?
Backup systems and network environments are often treated like they are recordkeeping systems. Backup systems allow system servers to be restored after a catastrophic failure, such as a disk crash or natural disaster. They are not recordkeeping systems and cannot be relied upon as the means to maintain records. They are difficult to access, they are not maintained long term, they may not be complete, they contain a lot of redundancy and records cannot be managed in this environment. Networks cannot be used as recordkeeping systems because they have no capacity to actually manage records and do not protect record integrity and authenticity. | More |
What if we don't have recordkeeping systems?
If you don’t have recordkeeping systems in your organisation you have two options: you can either purchase a dedicated records management software application from Government contract to use as the basis for your recordkeeping system, or you can reconfigure your current business applications to incorporate recordkeeping functionality. To do this you will need to:
- understand your organisation and the business it performs
- identify the records that will need to be captured within the system, and
- develop means to capture records within the system.
| More |
Plan for disaster protection and recovery of your systems
As part broader planning for the security of electronic information you should identify threats to records and recordkeeping systems, and the business functions, operations and services that depend on them. You also need to design means of protecting your records and implement disaster and continuity management strategies. | More |
What is the value of metadata?
Metadata is any component in your business systems or the functionality inherent in that component that helps you to control, identify, describe, use or manage your information.
The importance of developing good metadata cannot be overestimated. Having good structure and functionality in your system driven by metadata is critical to good business practice as well as good records management.
Spending money now on metadata will save you money later. Incomplete or poor quality metadata, or ad hoc metadata that is created without due consideration of business requirements, will require costly remediation actions in the long term, will complicate migration and will limit the useability and effectiveness of data throughout the lifespan of your systems. | More |
Create metadata schemas for your business systems
Spend time and money on system development and make sure you design a metadata schema for each business system that supports your business needs. This work will pay off with easier daily use of the system and greater manageability in the long term.
Your schemas should:
- specify the metadata fields used in the system
- provide a definition for each of the fields, indicating what can and what cannot be applied within them, and
- identify the encoding schemes or ‘picklists’ that are going to be used to provide data values in the system.
Migration, the transfer of records from one software or hardware configuration into another, is easier to achieve if metadata description is standardised. Uniform description avoids having to translate multiple record types into the target system. This saves time, effort and money and results in fewer problems during migration activities. | More |
There are a number of key points to consider when developing metadata schemas.
- Metadata must support business. What system structure and functionality that can be enabled by metadata is required? Schemas must be based on specific knowledge of the business that needs to be performed within the system.
- Metadata must support recordkeeping requirements, as well as business requirements. To ensure the accessibility and integrity of the records it is critical that the mandatory properties outlined in State Records’ Standard on digital recordkeeping be included, along with the properties that document the management processes you perform on your records.
- Metadata schemas may not be uniform across your organisation. Business will vary from section to section within your organisation and the inclusion of specific metadata fields for example to drive particular workflows may be needed tosupport individual business requirements.
- Metadata can be scalable and can be applied to a range of different things. Metadata can describe records, at different levels of aggregation. It can also be usedto describe other things such as people, workgroups, organisations, business transactions, activities and functions, as well as mandates such as laws, regulations, policies and business rules. You should determine howmetadata to describe different entities could be of use to your organisation.
- Develop good encoding schemes or’ picklists’. You should develop tools known as encoding schemes that provide a controlled list of all the acceptable values that can be applied within a certain metadata field. Use of encoding schemes can be critical for enabling interoperability between systems. They also promote standardisation, automation, consistency and accuracy.
| More |
Consider key points when implementing metadata schemas
There are a number of key points to consider when implementing metadata schemas.
- Plan for metadata automation, inheritance and reuse. Automating of as much metadata capture as possible makes life easier for staff in your organisation, enables more consistent metadata application and can also save time and money and improve efficiency.
- Metadata must start to be captured from the moment a record is created. More metadata must also be added to the record when it is used in different processes or as different management actions are performed upon it.
- Metadata itself is a key record. Metadata needs to be maintained in context through time because it attests to a record’s integrity by documenting its context and management history. It also contains important business information.
- Systems are only as good as the people who use them. Spend time and money explaining the roles of the different metadata fields in your business systems and explaining how the particular encoding schemes that provide values for these fields operate. This is critical to the effective operation of your systems and for the long term sustainability and integrity of the systems.
| More |
Facilitate preservation with metadata
Use metadata as a tool to flag technological dependence, manage record destruction and manage record migration.
| More |
Recognise that migration is a high risk process
All your significant organisational records with long term value will one day have to be migrated. With business system evolution and ever-present organisational change, that one day will come around sooner than you think.
Migration is a preservation activity that transfers records from one hardware or software configuration to another, or from one generation of technology to another.
Migration is necessary because the many protocols and software components that enable records to be read and used are constantly evolving. Their evolution is rapid and often does not retain compatibility, especially over periods longer than a few years. Without migration, access to important organisational records would be lost.
Migration, however, is a high risk process. It changes data and therefore threatens the authenticity, integrity and even the existence of records. All migration projects should therefore be specifically designed to help mitigate the risks associated with migration and to protect record authenticity, integrity and accessibility. | More |
The defining features that you need to understand before you plan for and implement migration operations are that:
- records are complex
- metadata is critical
- essential characteristics must be preserved.
| More |
Use migration to preserve the long term stability of digital records
Migrating records to a stable long term storage format is a means to achieve long term stability and to protect high value records. A stable long term format is a widely-used, non-proprietary, platform-independent format that is either uncompressed or lossless with, where possible, freely available specifications. These formats tend to have a long expected life which helps to minimise the number of migrations that a record will be subject to. See 1. Making digital recordkeeping achievable for a list of the stable long term formats recommended for records of long term value.
Migrating records to stable long term formats can also save time and money because this approach minimises the number of migrations that you will potentially be required to perform and could save you time and money in the management of your digital assets.
You should consider which of your organisational records could benefit from being migrated to a stable long term storage format. In general, records are migrated to long term storage formats when they are no longer required for active business operations. Records can however be migrated to standard storage formats earlier in their lifespan if this meets the needs of your organisation.
Be aware of triggers for migration and different migration types
You need to be aware of the various different triggers for migration and for the different levels of planning that are required by the different migration types. | More |
Plan for migration
Depending on the type of migration you are undertaking, migration projects can require a lot of planning. The complexity of large system migrations may require a lead time of at least six months.
Each form of migration will require project planning. Migration to a complete new system will require extensive planning. An extensive list of issues to be considered for all migration types is provided here. | More |
What about contractors?
If contractors are performing migrations on your behalf, you need to be very specific about your requirements. | More |
Perform pre migration testing
Once you have used your planning process to develop a migration method and then configured your target system, you need to perform a test migration on a small sample of duplicated records. | More |
Tip: Duplicate data prior to commencing migration
You may want to create a copy of the data you want to migrate just prior to migration. This gives you a full set of data to rely on if the migration results in the loss or corruption of data.
| More |
Once your migration is complete, post migration testing must confirm that:
- all records requiring migration are migrated, including those that are stored off line, in non active systems or secondary storage environments
- the entire record, including all necessary metadata, is migrated
- all necessary business rules, functionality and essential characteristics have been preserved
- users are satisfied with the authenticity, completeness, accessibility and useability of the migrated record.
Once post migration testing is complete, obtain sign off from the Chief Information Officer or management official with appropriate authority. | More |
Make records of your migration
The entire migration process and associated project planning should be documented. | More |
Keep source records for at least six months
Following their successful migration source records, the records that were used as the input to the migration, must be kept for at least six months. Retaining the source records for at least this period will enable the migration to be repeated if it is discovered that some or all of the migrated records do not meet quality control standards or business requirements. See General Retention and Disposal Authority - Source records that have been migrated (GA33) for more information. | More |
Think strategically about migration
Migration is an opportunity for business enhancement and business improvement and for really contributing to an improvement in your organisation’s bottom line. Try to see how migration can improve the way your business operates. | More |
Checklist for migration
This checklist may help you to plan your migration projects. | More |
Particular types of digital records cause management problems for many organisations. Advice is provided on:
- managing email - technical issues; policy and procedural issues; staff training issues and strategies
- managing 'born' digital images - still photographs - preparation for the capture of images; equipment and file formats for creating images; Primary, Original and Working image creation and optimisation; retention or disposal of images; file formats for storing images; media/systems for storing images; compression; file formats for delivery; titling; version control; useability and accessibility; system and application integration; maintaining context; enforcing standards; digital rights management
- managing CAD files - CAD file size; version control; useability and accessibility; maintaining context; system integration; managing file storage; enforcing filing standards
- managing web records -convincing management of the need to manage web records; retention and disposal of web records; options for capture and management; obtaining more information.
Also provided is an Annotated bibliography which includes the sources cited in this guideline and other useful information on the issues discussed in this guideline.