Quality Assurance of Administrative Data (QAAD) report for Training for Success and Apprenticeships NI: September 2018

Data quality procedures for Administrative data.

1. Background to the Statistics

Youth Training Statistics and Research Branch uses data supplied by the Department for the Economy’s Client Management System (CMS), which is installed in Jobs and Benefits offices throughout Northern Ireland. This is an IT system used to facilitate the interface with the Department’s customers. It maintains a basic client record; allows the preferred occupation stated by clients to be matched against suitable vacancies; records actions such as interviews, referrals to training opportunities, placing into jobs etc. In particular, it has been significantly enhanced to record ApprenticeshipsNI 2013/2017 and Training for Success 2013/2017 specific actions, and certain permitted data fields are populated automatically whenever Training Suppliers update their records on the Trainee Management System software.

The relevant data are extracted from CMS each month to form the ApprenticeshipsNI 2013/2017 and Training for Success 2013/2017 databases maintained by the Department’s Youth Training Statistics and Research Branch within Analytical Services. These databases are used to produce the biannual Training for Success and Apprenticeships NI statistical bulletins.  

This report outlines the statistical production process from initial collection through to the output of the release. It identifies potential risks in data quality and outlines how those risks are mitigated.

2. Quality Assurance of Administrative Data assessment

The assessment of our administrative data source has been carried out in accordance with the Office for Statistics Regulation’s Quality Assurance of Administrative Data Toolkit.

The relevant administrative data source has been assessed according to the toolkit’s risk and profile matrix, which reflects the level of risk to data quality and the public interest profile of the statistics.

Level of risk of quality concerns                                                          Public interest profile

Public interest profile

Lower

Medium

Higher

Low

Statistics of lower quality concern and lower public interest
[A1]

Statistics of low quality concern and medium public interest
[A1/A2]

Statistics of a low quality concern and higher public interest
[A1/A2]

Medium

Statistics of medium quality concern and lower public interest
[A1/A2]

Statistics of medium quality concern and medium public interest
[A2]

Statistics of medium quality concern and higher public interest
[A2/A3]

High

Statistics of higher quality concern and lower public interest
[A1/A2/A3]

Statistics of higher quality concern and medium public interest
[A3]

Statistics of higher quality concern and higher public interest
[A3]

The risk of quality concerns in relation to the administrative source has been assessed as ‘medium’ due to the fact that the data are collected from multiple sites and there is an intermediary data supplier. However, there are safeguards in respect of financial audit and operational checks. The public interest profile for this administrative data source has been assessed as being ‘low’ due to a niche user base and limited media interest.

3. Areas of Quality Assurance of Administrative Data

The four practice areas for assurance are as follows:

  • Operational Context and Administrative Data Collection
  • Communication with Data Supply Partners
  • QA Principles, Standards and Checks applied by Data Suppliers
  • Producer’s QA Investigations and Documentation

3.1 Operational Context and Administrative Data Collection (QAAD Score A1)

In September 2007, the Department for the Economy introduced Training for Success (TfS) to replace the Jobskills programme. TfS is a programme designed for young people aged 16-17 (or under 22/24 years for those with extended eligibility entitlement) which provides training to give them the tools and skills they need to get a job. The training provides young people with relevant qualifications as well as the required personal and behavioural skills to progress into work.

ApprenticeshipsNI aims to provide participants with the opportunity to take part in a Level 2/Level 3 Apprenticeship where the apprentice, in paid employment from day one, works towards achieving an industry-approved Level 2/Level 3 Apprenticeship Framework.

The primary purpose of the CMS database is for contracted training suppliers to maintain a record of participants’ enrolment and progress on the programme, in order make claims for the relevant output related funding. Data are input into the system via the Training Management System (TMS).

Strengths

  • Data extracts received on a monthly basis.
  • Full coverage of participants on the programmes.
  • Detailed operational requirements ensure suppliers use identical collection and coding systems.

Weaknesses

  • Data are subject to revision, particularly as there may be a delay in recording some information in the administrative system.
  • Due to rigidities in the TMS system, data must go through certain manual interventions during the validation process.  
  • Where the provision of certain information is not directly related to funding, there may be a lack of coverage in the database.

3.2 Communication with Data Supply Partners (QAAD score A1)

The DfE Youth Training Statistics and Research Branch team is in regular contact via email, telephone and meetings with the DfE IT team responsible for managing the delivery of data. There is a formal agreement in place between DfE and Fujitsu in relation to the delivery of TfS and AppsNI data.

YTSRB is satisfied that this working relationship is sufficient and that there is adequate communication with data supply partners.

3.3 Quality Assurance Principles, Standards and Checks by Data Supplier (QAAD Score A1)

Data are input to the administrative system by contracted training suppliers, who are responsible for ensuring the accuracy of the data held on the system. Contractors are required to complete an annual Record of Controls return which is assessed and approved by DfE to provide assurance that the Contractor continues to have all the necessary processes, policies and systems in place to permit the delivery of services in accordance with the obligations in their contract. 

Strengths

  • Contractors required to provide regular assurance regarding accuracy of data held on the system.
  • DfE audits contractor compliance with operational guideline

3.4 Producers Quality Assurance Investigations (QAAD Score A1)

In addition to the built-in validation processes implemented by Fujitsu, the DfE YTSRB team conducts further quality assurance on the data received before using it in the production of outputs. This takes the form of various ‘sense-checks’ of the data via frequencies and cross-tabulations. Certain previously identified errors in the data are corrected at this stage.

If any there are any apparent discrepancies or errors in the data, the YTSRB team will contact the IT Branch and do the following:

  • Send a query to the data supplier via the responsible DfE IT team.
  • Use statistical and domain area knowledge to inform a quality adjustment to the data.
  • Request re-delivery of the data once identified issues have been addressed. 

Once YTSRB has completed the quality assurance of the CMS data, they are used in the production system to produce outputs relating to the contract and performance management of the TfS and AppsNI programmes. If this process generates questions or issues, queries may be raised with the data supplier and this process repeated.

YTSRB keeps an audit trail of previous versions of the database on an internal drive. If there are any revisions, YTSRB is able to identify where these have taken place as well as the extent of any change.

Strengths

  • Consistent validation process and sense checks
  • Audit trail – historic files are stored

Weaknesses

  • Validation processes are time consuming and have an impact on timeliness of outputs.

4. Summary

As a result of investigations into the administrative source for TfS and AppsNI data, YTSRB considers the main strengths of the data for our purpose to be:

  • High level of coverage (100% of the population).
  • Comprehensive guidance provided to training providers in respect of inputting data to the system.
  • Operational and Financial Audits carried out by DfE.
  • Additional quality checks carried out by DfE YTSRB team.

We believe the current limitations of the dataset are:

  • Certain fields may be under-populated where the information contained in these is not directly related to receipt of funding.
  • Rigidities within the system mean that the same errors must be corrected each month during the data validation process.
  • Time lags in updating the database, together with time consuming validations, have an impact on the timeliness of outputs.
Back to top