Scope and purpose
Migration is necessary because the many protocols and software components that enable records to be read and used are constantly evolving.
This guidance is intended to outline the important considerations when migrating records and information between systems, and to help records and information managers provide appropriate guidance on migration projects. The guidance outlines an important role for records and information managers in system migration projects, but is not intended to be a comprehensive methodology for system migration.
Migration is, for the purposes of this guidance, any activity undertaken to move data from one system or platform to another. Migration may occur between major version upgrades of a system, or between applications on different platforms.
Migration is a constant factor for any organisation that conducts its business digitally. All digital systems eventually will be unsuitable for ongoing business purposes or for long term preservation of digital information. This may be because:
- they are built using technologies which become unsupported or difficult to support
- they become expensive to keep relative to alternatives
- they become superseded by technology with more advanced features
- they are rationalised onto a single technology platform as the result of an organisational merger
- the business function they perform is outsourced
- the business function they perform moves to a shared service arrangement
- the business function they perform ceases to exist.
Even the best designed and best implemented systems will become obsolete for one or more of these reasons. As such, it is important that your organisation’s records and information management strategy incorporates an understanding of the key technologies on which high value and high risk records are dependent.
Any major system migration is a complex project. It requires the input and support of a broad range of people in your organisation. Effectively managing a system migration depends on:
- understanding the information in your current system, the way it is managed, the way it is used in business processes, and the dependencies it has on other data sources
- the new system having appropriate functionality to accommodate the core data, metadata and functionality of the existing system
- undertaking a rigorous migration process which is well planned and tested before and after data migration is performed, to have confidence that the data was migrated as intended, the functionality exists as planned, and that the data can continue to be relied on as a record
- a robust quality assurance process.
Records and information managers have a central role in a migration project. This includes assessing and understanding the information in the current system, providing requirements specifications for the implementation of the new system, determining which information must be migrated to the new system, and which information may be able to be managed in another way, and assisting with the development of testing procedures to ensure records and information are successfully migrated.
Any system migration is a complex and usually expensive process, involving multiple internal and external parties. Key roles that a records and information manager would interact with during a migration process include:
- project manager – responsible for management and overall delivery of a project
- business analysts – IT staff who analyse business requirements and translate these into system requirements
- architects – IT staff who do a detailed overall design of a system’s technical functionality
- test managers and/or testers – responsible for defining, overseeing and performing tests to confirm whether processes and systems operate as intended
In smaller or more straightforward projects, a records and information manager might have significant responsibility for some of these functions. However, any best practice migrations are likely to be performed in accordance with standard project management methodologies. This guidance focuses on core considerations and responsibilities for records and information managers, and does not detail a recommended methodology for conducting migration projects. Organisations planning for the transfer of digital records as State archives, or undertaking a migration project specifically for digital preservation purposes are referred to the State Records Digital Archives migration methodology.
Records have certain defining features that must be supported during migration operations. Understanding these features is critical to maintaining record authenticity, integrity, reliability and useability during migrations.
The defining features that you need to understand before you plan for and implement migration operations are:
- records are complex
- metadata is critical
- essential characteristics must be preserved.
Records are complex
Records are not simply data. In order to serve as evidence and information they are comprised of a complex of related information:
- structure – the form and layout of the record
- content – the informational value of the record – this could be simple text or a complex aggregation, such as a word processed document containing a spreadsheet or a web page containing images
- context – information about who created the record, why and when they created it, how it has been managed and what other records it is related to.
All of this information must be maintained during migration to preserve the evidential and informational value of records.
Metadata is critical
Metadata is used to describe, access and manage records. It is generally the means by which much of a record’s context is documented and is the ultimate means by which the integrity, trustworthiness, and authenticity of a record can be proven. It is essential that metadata is preserved and that connections between a record and its metadata are maintained during migration.
Metadata relationships that must be safeguarded during migration include:
- structural – that is within the document, such as a document containing a linked spreadsheet or images
- between records – e.g. between records documenting related aspects of business
- between records and/or ‘containers’ – e.g. documents aggregated to files/folders
- between records and other entities – e.g. between records and creating agents
- between records and control tools such as Business Classification Schemes, Retention and Disposal Schedules, access and security controls and mandates.
[Source: Queensland State Archives, Migrating Digital Records: A Guideline for Queensland Public Authorities, June 2012, p.18]
Importantly, metadata documenting the process of migration must also be captured.
Essential characteristics must be preserved
Essential characteristics are those features that are critical to a record’s meaning, use or organisational value. All migration operations should be designed to preserve the essential characteristics of the records being migrated.
Each organisation needs to determine the essential characteristics that apply to their own specific records. Generally, essential characteristics will differ according to record type and the business purpose served by the record.
For example, a report contains a map where colours are used to signify different agricultural areas, these colours have meaning and the report could not be interpreted accurately if these colours were lost through migration to another format. For this report, the colours are a critical characteristic and any migration performed on this report must ensure that they are maintained.
Once you know a system is going to be upgraded, there are numerous issues to consider when developing your plan of how your migration will be performed. The following table identifies some of the data and system issues you may need to consider.
Data issues to consider in migration planning
|Data quality and completeness||
[These points were principally drawn from Part II, Section 1.2 of ARMA International, ANSI/ARMA 16-2007, The Digital Records Conversion Process: Program Planning, Requirements, Procedures]
|Understand user behaviour and requirements||
|Format and compatibility requirements||
|Older records that have undergone multiple migrations||
|Comprehensive systems testing||
Where relevant, plans should be made to address these and any other issues identified before your migration is conducted.
Once your planning is complete you will have a thorough understanding of your records as well as your current and target systems. This will enable you to:
- begin to address the issues that must be resolved before migration can commence
- identify the desired target state of your records post migration
- contribute to the development of a migration method that will convert your records, including all metadata and essential characteristics, from their current state to the target state.
Migration should be undertaken in accordance with best practice project management methodologies. Comprehensive records of the project should include:
- a description of the records being migrated and the business function they relate to
- the identified essential characteristics of the records
- all system configurations, including metadata definitions and mappings
- a description of the factors triggering the migration project
- all reports that compare original system functionality to target system functionality
- all decisions, including decisions not to migrate certain metadata components of a record
- risk assessments
- details of any disposal and data clean-up performed prior to migration, including records of decisions and the processes undertaken
- records of management approval of migration plans
- the technical requirements of the original and target systems
- technical reports on the migration itself and the personnel involved
- details of all testing, pre and post migration, and comprehensive test reports which confirm the system is functioning successfully and that all expected data and metadata has been migrated.
- documentation of any necessary variation in records structure, design, metadata, format or content that will or have resulted from the migration
- details of the disposal of the source records used, once the appropriate quality assurance period has elapsed.
[This list draws on Queensland State Archives’ guide to recordkeeping documentation contained in Migrating Digital Records: A Guideline for Queensland Public Authorities, pages 33-34.]
The six month retention period should begin to be calculated from the conclusion of successful post migration testing, where the migration and all outstanding issues associated with it have been signed off by the appropriate senior manager (often the Chief Information Officer).
The general retention and disposal authority requires that the public office has properly planned, tested and documented the migration. In some scenarios retaining source records for longer may be an appropriate risk mitigation strategy. However, the core concern for high value and high risk records and information is ensuring that the migration planning, testing and documentation is extremely robust. Increasing the retention period of the source system will not substitute for performing those processes inadequately.
For further advice about the issues associated with the retention of source records see the general retention and disposal authority for source records that have been migrated (GA33).
Also see Transition Guidelines: Managing Legacy Data and Information from the Department of Finance, Services and Innovation.
Published January 2009 / revised February 2015