Quality Assurance of Administrative Data (QAAD)

1. Background to the Statistics

Youth Training Statistics and Research Branch (YTSRB) use data supplied by the Department for the Economy’s Trainee and Apprentice Management System (TAMS). TAMS is the Department’s central training management system used to administer ApprenticeshipsNI, Skills for Life and Work and other work-based learning programmes. 

TAMS is used to:

• Register and manage participant records for ApprenticeshipsNI, Skills for Life and Work and other training programmes. 
• Record Delivery Agreements and supporting documentation between training providers, employers and apprentices. 
• Capture participant details, including existing employee status, programme participation, disabilities and support requirements. 
• Support funding administration, including approval of disability supplements and processing of programme claims and milestone payments., 
• Provide compliance and audit evidence, ensuring training providers meet Operational Requirements and maintain required records. 
• Manage programme monitoring and reporting, including participant progression and claims processing. 

In particular, the TAMS system is used to provide starts, leavers, occupancy and qualification data for ApprenticeshipsNI and Skills for Life and Work.

ApprenticeshipsNI plays a vital role in developing skills and creating opportunities across a wide range of sectors throughout Northern Ireland. It offers individuals the chance to gain valuable, hands-on experience while working towards recognised qualifications, supporting both career progression and business growth. The minimum working hours required for an apprenticeship program in Northern Ireland is 21 hours per week on a permanent contract, which includes any time allocated for off-the-job training.  ApprenticeshipsNI aims to provide participants with the opportunity to take part in a Level 2, Level 2/3 or Level 3 Apprenticeship where the apprentice, in paid employment from day one, works towards achieving an industry-approved Level 2/Level 3 Apprenticeship Framework.

In September 2021, the Department for the Economy introduced Skills for Life and Work to replace the Skills for Life and Skills for Work Level 1 of the Training for Success programme. Skills for Life and Work is for 16 and17 year olds who left school, or under 22 years for those with a disability, and under 24 years for those who qualify under the Children (Leaving Care) Act (NI) (2002).  It is suitable for young persons who need qualifications at Entry Level or Level 1 and assists in providing essential work experience. This then allows the participant to move on to higher training such as a Traineeship or an Apprenticeship or full or part-time employment. Skills for Life and Work includes a significant amount of time spent in a workplace (usually every week) as well as training for the participant. They work towards qualifications in personal development and employability, and a qualification in the vocational (work) area they are employed in. Participants can also work towards Essential Skills qualifications in literacy, numeracy and ICT.  Participants may be able to follow one or more Level 2 qualifications while on Skills for Life and Work, depending on their individual needs and circumstances.

This report outlines the statistical production process from initial collection through to the output of the release. It identifies potential risks in data quality and outlines how those risks are mitigated.

2. Quality Assurance of Administrative Data Assessment

The assessment of our administrative data source has been carried out in accordance with the Office for Statistics Regulation’s Quality Assurance of Administrative Data Toolkit.

The relevant administrative data source has been assessed according to the toolkit’s risk and profile matrix, which reflects the level of risk to data quality and the public interest profile of the statistics.

Public Interest Profile

Level of risk of quality concerns	Lower	Medium	Higher
Low	Statistics of lower quality concern and lower public interest [A1]	Statistics of low quality concern and medium public interest [A1/A2]	Statistics of a low quality concern and higher public interest [A1/A2]
Medium	Statistics of medium quality concern and lower public interest [A1/A2]	Statistics of medium quality concern and medium public interest [A2]	Statistics of medium quality concern and higher public interest [A2/A3]
High	Statistics of higher quality concern and lower public interest [A1/A2/A3]	Statistics of higher quality concern and medium public interest [A3]	Statistics of higher quality concern and higher public interest [A3]

The risk of quality concerns in relation to the administrative source has been assessed as ‘medium’ due to the fact that the data are collected from multiple sites and there is an intermediary data supplier. However, there are safeguards in respect of financial audit and operational checks. The public interest profile for this administrative data source has been assessed as being ‘low’ due to a niche user base and limited media interest.

3. Areas of Quality Assurance of Administrative Data

The four practice areas for assurance are as follows:

• Operational Context and Administrative Data Collection
• Communication with Data Supply Partners
• QA Principles, Standards and Checks applied by Data Suppliers
• Producer’s QA Investigations and Documentation

3.1 Operational Context and Administrative Data Collection (QAAD Score A1)

The primary purpose of the TAMS database is for contracted training suppliers to maintain a record of ApprenticeshipsNI and Skills for Life and Work participants’ enrolment and progress on the programme, in order make claims for the relevant output related funding. 

Strengths

• Data extracts are received on a monthly basis.
• Full coverage of participants on the programmes.
• Detailed operational requirements ensure suppliers use identical collection and coding systems.

Weaknesses

• Data are subject to revision, particularly as there may be a delay in recording some information in the administrative system.
• Due to rigidities in the TAMS system, data must go through certain manual interventions during the validation process.
• Where the provision of certain information is not directly related to funding, there may be a lack of coverage in the database.

3.2. Communication with Data Supply Partners (QAAD Score A1)

The DfE Youth Training Statistics and Research Branch team is in regular contact via email, telephone and meetings with the Business Process and Application Services IT team responsible for managing the delivery of data. There is a formal agreement in place between DfE and the data supplier in relation to the delivery of TAMS data.

YTSRB is satisfied that this working relationship is sufficient and that there is adequate communication with data supply partners.

3.3 Quality Assurance Principles, Standard and Checks by Data Suppliers (QAAD Score A1)

Data are input to the administrative system by contracted training suppliers, who are responsible for ensuring the accuracy of the data held on the system.  Contractors are assessed twice a year on compliance with the terms of the contract, including all operational requirements. This ensures that contractors have all the necessary processes, policies and systems in place to permit the delivery of services in accordance with the obligations in their contract.

Strengths

• Contractors required to provide regular assurance regarding accuracy of data held on the system.
• DfE audits contractor compliance with operational guidelines.

3.4. Producers Quality Assurance investigations (QAAD Score A1)

In addition to the built-in validation processes implemented by the data supplier, the DfE YTSRB team conducts further quality assurance on the data received before using it in the production of outputs. This takes the form of various ‘sense-checks’ of the data via frequencies and cross-tabulations. Certain previously identified errors in the data are corrected at this stage.

If any there are any apparent discrepancies or errors in the data, the YTSRB team will contact the Business Process and Application Services IT team and do the following:

• Send a query to the data supplier via the responsible DfE IT team.
• Use statistical and domain area knowledge to inform a quality adjustment to the data.
• Request re-delivery of the data once identified issues have been addressed.

Once YTSRB has completed the quality assurance of the TAMS data, they are used in the production system to produce outputs relating to the contract and performance management of the ApprenticeshipsNI and Skills for Life and Work programmes.  If this process generates questions or issues, queries may be raised with the data supplier and this process repeated.

YTSRB keeps an audit trail of previous versions of the database on an internal drive. If there are any revisions, YTSRB is able to identify where these have taken place as well as the extent of any change.

Strengths

• Consistent validation process and sense checks
• Audit trail – historic files are stored

Weaknesses

Validation processes are time consuming and have an impact on timeliness of outputs.

4. Summary

As a result of investigations into the administrative source for ApprenticeshipsNI and Skills for Life and Work data, YTSRB considers the main strengths of the data for our purpose to be:

• High level of coverage (100% of the population).
• Comprehensive guidance provided to training providers in respect of inputting data to the system.
• Operational and Financial Audits carried out by DfE.
• Additional quality checks carried out by DfE YTSRB team.

We believe the current limitations of the dataset are:

• Certain fields may be under-populated where the information contained in these is not directly related to receipt of funding.
• Rigidities within the system mean that the same errors must be corrected each month during the data validation process.
• Time lags in updating the database, together with time consuming validations, have an impact on the timeliness of outputs.