4. Effectively manage the migration of your digital records (Guideline 22)

Records management controls are disposal authorities, security classifications and record classification tools.

Much functionality may be lost and much expense incurred if these tools and the functionality they deliver is not adequately identified and migrated to the target system.

It is important to be aware that complex relationships can exist between one or more of these tools and some or all records in the system. These relationships must be preserved during migration.

Identify the different varieties and aggregations of records in the system and interdependencies that exist between them 

The records in your system may be individual documents, aggregations of related documents (files or volumes) or aggregations of files (series). These aggregations and the implicit relationship connections that exist between them must be safeguarded during the migration as these relationships are part of the evidentiary context of the records.

Aggregations can also be used to apply management rules. For example, if they are uniform, disposal or other rules can be specified at the file or series level and a business rule used to identify that these rules should apply to all records nested within the file or series container.

In such situations, this functionality must be replicated in the target system or the implicit metadata represented by the aggregations must be made explicit in the target system.

• metadata identifying:
  • the name of the creator and creating organisation
  • the business process that produced the record
  • its date of creation
  • its specified relationships to other records
  • its original and current file formats
  • a history of the recordkeeping actions performed on the record.

•  all content – text, images or otherwise. This includes:
  • page numbers
  • headers and footers
  • automatically created content such as tables of contents and indexes
  • document properties
  • comments that may be attached.

•  the logical relationships that exist between the various components of the text document when rendered onscreen

• features of the record appearance which convey meaning. These may include:
  • bolding
  • italics
  • underlining
  • font size/style

Other aspects of the appearance of the migrated record may change from the appearance of the original provided that the meaning of the record is left unchanged.

• any functionality in the record that ensures it is understood such as the ability to view embedded graphics
• behaviour that is counter to ongoing integrity of the record such as an automatically updating date field should be 'turned off'.

• essential header elements:
  • the email address, the organisation and the full name of the sender
  • the email address, the organisation and the full name of all recipients
  • for outgoing messages, the date and time the message was sent
  • for incoming messages, the date and time the message was received, as well as the date and time the message was sent
  • the subject of the message
  • security and/or confidentiality settings
  • the file name and the file format of any attachments
  • data about original and current file formats
  • a description of the business process that produced the record
  • any specified relationships to other records
  • a history of the recordkeeping actions performed on the record.

• all content – this includes:
  • extra files such as photos or images inserted in the original message
  • links and hyperlinks.

• the structure of the original message
• attachments and inserted items.

• the meaning of the message, but the appearance of the message does not need to remain the same.

• the ability to open and access attachments
• links to other documents, including the title of the document.

• metadata identifying:
  • the name of the creator and creating organisation
  • the business process that produced the record
  • its date of creation
  • its specified relationships to other records
  • its original and current file formats
  • a history of the recordkeeping actions performed on the record.

• the content of the spreadsheet, including the content of any possible charts and other inserted objects. This includes:
  • both the standard display of the worksheets (the entered values and the results of calculations using formulae) and the underlying layer containing formulae and inserted functions

• worksheets in the correct sequence with the correct names
• the existing row and column structure of the document which provides each cell with its unique designation – ie migration must not change the coordinates of a cell from A3 to B3 as this could cause the formulae embedded in the document to produce a result that differs from the original value
• inserted charts or objects
• links to related spreadsheets. Related spreadsheets must also be preserved if these provide values to cells
• links between calculations and the formulae that produce them.

• the appearance of inserted objects.

The appearance of the spreadsheet and charts may differ from the originals.

•  the ability of formulae to carry out a calculation or recalculation. If records are no longer actively used for business, this functionality will need to be disabled so that the record is not constantly altered each time it is opened.

It should be noted that European testing showed that newer versions of spreadsheet applications sometimes use mathematical formulae in a manner different to the older version, thereby yielding a different result. The ability to apply formulae should therefore usually be disabled when the record is captured into a recordkeeping system. Before migration, it should be checked that all formulae are actually disabled.

Remember, the preservation of an authentic record requires the retention of its content of the record at the time it played a role in the business process. This is particularly important in relation to spreadsheets which are easily subject to change.

• metadata identifying:
  • the name of the creator and creating organisation
  • the business process that produced the record
  • its date of creation
  • its specified relationships to other records
  • its original and current file formats
  • the name and version of the query languages that are used
  • a history of the recordkeeping actions performed on the record.

• the actual content of the tables
• the queries used so that the required content can be represented.

• the physical structure of the database. This includes:
  • the tables
  • relationships between the tables, including the constraints
  • the views
  • field attributes
  • the structural composition of the data as presented onscreen.

The logical structure of the database can be preserved in an entity relationship diagram or an XML schema.

• the onscreen representation.

• the behaviour of the user application
  • this can be preserved in the form of descriptions of system-supported functions, as well as screenshots of the displays used for entering and amending data, generating reports, etc. It can also be preserved in system documentation and user manuals.

• one internal organisational business system to another 

• changing business needs that lead to the adoption of a new business system
• technological changes that require the update of business systems
• the need to transfer records from active business systems to long term storage systems
• other internal and external drivers.

• one organisation to another 

• administrative change that requires the transfer of functions or activities from one organisation to another
• the adoption of shared service arrangements that require the transfer of records from one organisation to another
• other internal and external drivers.

• one organisation to State Records 

• requirements for the management of State archives, where State Records is required to take control and custody of State archives
• other business drivers.

Project planning, checking of provided export/import gateways that will perform migration, pre and post migration testing

• What is the quality of back end tables? Are the staffing details in these tables up to date? Are people who have left the organisation still recorded there and listed as still holding records? You do not want to migrate this data. You should clean it up first.
• Does a file audit need to be conducted to identify current locations of paper-based records?
• Is the latest version of retention and disposal authorities referenced – for example GA28 and not GDA2 for administrative records? If not, do you want to rectify this prior to migration?

• Sometimes people decide not to migrate all metadata components of a record. This must only be the case for those metadata elements that have no ongoing business or accountability relevance for the organisation. If for technical, capacity or management reasons you decide not to migrate all metadata elements, these decisions should be fully considered and comprehensively documented.

• One large public sector organisation held ‘Records Week’ in the week preceding their records system migration. Staff were required to clean out shared directories and email systems, file all records and return hard copy files to the registry. This helped to ensure that as many records as possible were in the system prior to migration.

• Can certain records be disposed of rather than migrated? It is not efficient to migrate records that can be disposed of.
• Migration is a good opportunity to see what records you have stored in secondary storage areas and to cut down on these records if possible.

• Staff frequently maintain personal systems that they use to control business records. Migration can be used as a trigger to bring these rogue or independent systems within broader business systems and to gain greater control and accessibility to all business records of the organisation. 

• Ensure you understand the hardware, software and format requirements of all the records requiring migration.

• Be aware of any non standard records that you will need to migrate and ensure these are incorporated in your plans.

• Check the earliest records in your system. Look at their quality, check their dependencies, look at what is going on with these records and what is required to support their ongoing maintenance.

• Ensure that all records requiring migration are actually migrated. Individual records should not be inadvertently left out of the migration action. You should ensure that all relevant online and offline records, and those at secondary storage or data warehousing facilities are included where appropriate.

• Data may need cleansing or massaging to get it from an older system to a newer one.
• Cleansing requirements may be format related – ie some older formats may need formatting removed and poorly written code cleaned up, usually through a vendor approved or custom script.
• Cleansing requirements can also be system related – older systems tend to store large records as separate pages while newer systems tend to save them as one object. You may choose to clean these older records by converting them to a single document before migration, or you may leave them and build transformation rules into the target system.
• Any data cleansing needs to be fully documented to help prove the integrity of the migrated records. [12]

• One very large organisation chose not to migrate all its records. For example, its old system contained 150 000 records of destroyed files.
• These were translated into one file and this was migrated and maintained, rather than each of the individual records. Any change of this type needs to be well documented to help maintain integrity.

• Importing movement history as active metadata is very difficult as it involves migrating and maintaining a 20 year history of staff. At migration it may be easier to maintain movement histories as a record and start movement tracking afresh in the new system. Again, any such changes need to be documented. This approach may be particularly useful if you have not conducted a records audit prior to migration to identify missing records or records still marked out to staff that have left the organisation.

• Some systems do not store sender, receiver and date received metadata from email messages in separate fields. You need to establish this yourself if required. Searching and accessibility can be badly limited if this functionality is not enabled.
• Some metadata is not explicit. For example, some disposal metadata is applied through inheritance from a parent record. At migration, all metadata values should be made explicit or the functionality preserved to allow these values to be inherited from a parent record.

• Talk to users of the system to make sure that you are planning to keep all the information they need to keep using the records.

• Determine whether any records need to be decompressed or unencrypted before they can be migrated.

•  Identify whether any digital signatures need to be migrated and how this will be achieved.

•  In many systems, picklists are used to populate a significant proportion of your data. When moving to a new system you need to review the quality of your existing picklists and make sure that they still reflect current requirements. Check that the available values are actually the values that bring the most benefit. Your new system may have increased functionality and may have the capacity for new or expanded picklists so make sure you investigate this possibility.
• If you alter or remove a picklist that was present in your old system, make a record of what it was and document any changes in a report or file note.

• Look at how people are using existing data elements. Are people using specific fields in a variety of different ways? Does your title element contain disposal, location and agent information, as well as title information? This is a common problem that really limits system usability.
• You can fix this problem by issuing very clear procedures with your new system that explain exactly how the system should be used. Then you need to monitor system useage and follow up with those who use the system inappropriately.

• It is important to note that fields such as the title field described above that contain a variety of different information elements can cause problems during the migration. Migration is generally a one to one mapping of a data element in the old system to a data element in the new system. In some situations it may not be possible to map multiple values in one element to various different elements in your target system.
• If you encounter a situation like this, you may need to look at manually adding data to some fields to compensate, migrating some data to a Notes field or undertaking a more complex migration than you originally planned.

•  If your current system has implemented disposal triggers and has already started calculating retention periods based on these, you must make sure that these calculations are maintained unchanged in your migration. For example, if you have a group of records that are to be kept for five years after audit and the audit was conducted two years ago and your system has calculated the destruction due date based on these events, then the system countdown must be maintained accurately and not start again following migration. This is a common problem at migration that can result in a significant amount of resentencing work, so make sure it does not become a problem during your migration project.

• If your system is large and if it has been in operation for a long time it is likely that it has been used in different ways by different people over time. You should run a variety of different reports to enable you to get a comprehensive overview of all the data in the system and the different ways it has been recorded. This will help you to ensure that your migration plans are comprehensive and accurate and will help you to avoid any potential data loss in migration.
• As an example, the migration planning team in one large organisation assumed that all business units used the Title field to document record title information, but ran reports and generated extracts to confirm their assumptions. In the process they discovered that one business unit had been using a minor descriptive field to record title information and this descriptive field was not scheduled for migration. The team subsequently changed their migration plans to ensure that this field was migrated to the new system. Without their checks, necessary data would have been lost from the system.

• Dates are very important in recordkeeping. Records systems generally have the capacity to document a lot of dates about record creation, registration, disposal, access or forthcoming recordkeeping events.
• New systems have greater capacities for date application than old systems. In addition, applying date metadata was not done as rigorously as it should have been in some old systems. Both of these scenarios can lead to problems in migration.
• New systems can often have a default date setting as standard out-of-the-box functionality. If, for example, you did not add 'creation date' to all your files in your old system and if 'creation date' is a mandatory field in your new system, at migration the system may automatically apply the current date to all blank date fields, irrespective of when the file was actually created. This can cause problems with accurate searching, disposal, legal requests etc.
• To prevent these problems you need to check the comprehensiveness of your existing date metadata and the business rules that apply to dates in your new system. This will enable you to anticipate any problems and take steps to resolve them. For example, you may decide to perform some data cleansing prior to migration by back capturing accurate date information. This is a more cost effective approach than allowing inaccurate data to be recorded in the system.
• In most systems you can run a report to identify any blank fields. You should do this to identify whether there is any data that is missing or that will be automatically rewritten by the new system at migration.

• One organisation was using its migration to consolidate records stored across different databases into one database. In the multiple database structure, records in each database were given a prefix based on the name of the section that owned the database. For example, the Finance and Administration records all had identifiers beginning with 'F', ie F07/1234 etc. Migration planning staff realised that the records of the Property division and the Planning division had the same prefix and consequently all had the same record identifiers. At migration they ran a business rule to change all the identifiers in the Planning database from 'P' to 'PL' to ensure that all records in the consolidated database had unique and searchable post-migration identifiers.

• Prior to migration, you or the contractors performing the migration will create an extract of data from your system to model how data will be removed, what it will look like post migration and where it will be presented in the new system.
• It is very important to go through this extract thoroughly. It is likely that you will discover that in the first iteration that not all data has been extracted accurately. For example, some picklists may not be carried through or some data elements may not have been picked up. Another common problem is for some data elements to be merged together, ie the extract may merge the title and notes fields. It is important that you identify all such problems and work out what is causing them prior to performing your migration.

Metadata mapping is a critical part of migration planning. Metadata mapping between the old and new systems will help to identify all your critical metadata and will make sure that it is all migrated. Ensuring all metadata can be migrated is necessary for maintaining functionality and can also be critical for ensuring that integrity and authenticity are preserved.

• Make sure you have a correct set of rules that define your metadata elements. It is very important to make sure that the meaning of elements is maintained across migrations.
• If your organisation uses a range of identifiers for its information and if it needs to maintain a record of previous record identifiers, make sure these can be kept.
• Make sure appropriate date detail is maintained. In some migrations the only date information carried across is the date of the migration. This becomes the default date for all recordkeeping actions, including date of creation. This severely impacts recordkeeping processes such as disposal and access and has significant impact on the useability and integrity of records. Therefore the meaning of each date field must be maintained following migration.
• If you are using contractors to perform the migration, make sure that their metadata mapping is critical to signing off on your approval of the pre migration work they have done.
• Business rules for the same function may be handled differently in the original and target systems. For example, access rights may be handled using caveats in one system and using user profiles in the other. It is necessary to plan a strategy for the migration of the required functionality using the different business rules.

• Remember that migration is an opportunity to improve system functionality. Look realistically at the problems and shortcomings in your current system. Don't simply bring them forward. Instead look at how you can use migration as an opportunity for system improvement.

• You need to ensure that you understand the full functionality of the target system. This includes recordkeeping functionality as well as other types of functionality.

• Systems can have different rules for the character length of fields. This can raise significant problems with migration. For example, in one organisation, their target system in its title field created a data string of the old creator and title fields. It also limited the title field to 100 characters. Following the migration, the first 100 characters of each record were taken up by the creator information (branch name and staff member name) and there was no space left for the actual record titles. This was completely unworkable and the migration had to be repeated which was costly, time consuming and led to further system downtime.
• Is there adequate server capacity for the target system? If the system is being rolled out to regional offices, is there sufficient bandwidth? Are response times adequate?

• If systems are talking to each other, what happens when one is upgraded?
• Are there live connections between the system requiring migration and the agency website or intranet? For example, is the latest version of a position description retained in the records system and linked live to the intranet? Do online transactional systems link to the business system being migrated? How can necessary functionality be supported post migration?

• New target systems often have greater functionality than the systems they are replacing. In one organisation, data appeared in the new system that was not present in the old system. This was because the new system had greater capacity to be able to access data contained within the record. It is important to be aware of this possibility, identify the effects it will have and determine whether these are required by your organisation.

• You may need new procedure documentation if your new system is significantly different to its predecessor. User training is also critical so ensure that you have adequate time and resources for this.

Identified migration triggers that signify a migration needs to be performed? Common triggers include:

• the adoption of new business systems
• impending technological change or obsolescence
• the need to move high value records to stable long term formats
• administrative change
• the adoption of shared service arrangements
• outsourcing of business functions
• the need to transfer records to State Records.  

Understood the unique characteristics of records that need to be preserved during migration? These are:

• record complexity
• metadata
• essential characteristics.  

Determined the type of migration required? This could be:

• minor functionality change
• medium levels of functionality change
• migration to a complete new system.  

Appropriately planned for all relevant data issues in the migration? These could include:

• performing data cleaning
• implementing a records clean up
• maintaining all appropriate metadata
• destroying records where appropriate
• identifying independent systems that need to be captured into an organisational recordkeeping system and migrated
• understanding the hardware, software and format requirements of the records requiring migration
• identifying non standard records
• checking older records that have undergone multiple migrations
• identifying all records requiring migration, including records in online and offline storage, records in all business environments, including shared service environments, and secondary storage environments
• initiating data cleansing
• performing necessary record consolidation
• planning for the management of aspects of record context that are difficult to manage through migration, such as movement histories
• making implicit metadata explicit
• talking to users to ensure that all their business requirements for records can be met by the proposed migration
• managing records that are encrypted or contain digital signatures
• checking the quality of encoding schemes
• verifying user understanding of existing metadata fields
• determining whether multiple values have been used in single metadata fields
• ensuring existing count downs remain in place
• comprehensively assessing metadata used in the system
• verifying use of date information
• identifying all records across databases that will be consolidated
• verifying which fields will be migrated and how and where.
  

Appropriately planned for all relevant system issues in the migration? These could include:

• completing an extensive metadata mapping, to ensure that all necessary metadata elements, their corresponding meaning and functionality, relevant business rules and links to other systems can be migrated between systems
• assessing how current functionality could be improved
• ensuring you understand the capacities of the target system
• assessing capacity issues in the target system
• identifying interdependencies between systems that need to be maintained
• identifying potential issues with the increased functionality of the target system.

Addressed the issues that need to be resolved before migration can commence?  

Configured the target system so that it meets all your business requirements?

Identified what you will require from any contractors who perform migration on your behalf? This could include all information concerning:

• all metadata mapping and documentation of issues requiring remediation
• all data cleansing and record consolidation
• all business rules applied in the original and target systems
• any user consultation
• any disposal performed
• all plans for maintenance of connections between systems
• full migration testing, and
• full migration implementation.  

On a sample that represents an adequate proportion of the total number of records migrated, ensured and tested that:

• complete records, including all necessary metadata, has been migrated?
• all necessary business rules have been retained?
• all necessary functionality has been retained?
• all essential characteristics have been retained?
• users are satisfied with the authenticity, completeness, accessibility and useability of the migrated records?

• the records being migrated
• the trigger for the migration
• relevant research
• all decisions
• risk assessments
• the identified essential characteristics
• the technical requirements of the original and target systems
• the formal migration or transfer process
• the date and time of the migration and all personnel involved
• all system configurations, including metadata definitions and mappings
• all testing
• all reports that compare original system functionality to target system functionality
• all sign offs
• any data cleanup performed
• any variations to plans
• any necessary variation in records design, metadata, format or content that will result from the migration
• the disposal of the source records used. 

Stored source records in an appropriate environment where they:

• cannot be accessed by users?
• cannot be mistaken for the official version of the records?
• are secure?
• can be accessed by technical staff in order to repeat all or portions of the migration as required?