Our team ran a randomised control trial to increase the number of classes that apprentices and trainees (‘learners’) attend. We significantly increased course attendance by sending employers a text message detailing what their learners are studying at their vocational training organisation (TAFE).

The following post summarises my team’s recent report.

Summary

• Half of all apprentices in New South Wales cancel their contracts within their first year
• Our research shows that there is a disconnect between what apprentices learn in class and what they do at work
• We tested whether a behavioural intervention could improve class attendance. Teachers sent employers a weekly SMS about what their apprentices were learning in class and encouraging them to have a conversation about the lesson, or try out the skill on-the-job
• Our intervention significantly increased class attendance. Students whose employers received these SMS were more likely to still be studying by the end of the semester.

Background

TAFE New South Wales (NSW) is Australia’s largest provider of training and vocational education. Each year it trains more than 500,000 students. However, some apprentices were not attending all their TAFE NSW classes and not completing their apprenticeships.

Almost half (48%) of apprentices in New South Wales cancel their contracts within the first year, and 77% will cancel within two years. That’s a tremendous personal cost to these students, which also translates to $91 million loss of the state’s economy in cancellations alone, and upwards of $348 million in related revenue.

Our previous fieldwork revealed two key barriers to apprenticeship and traineeship completion: a disconnect between formal study and on-the-job learning, and insufficient employer support.

Significant disconnect between what happens at TAFE and what happens at work

At work, learners often do not communicate what they are learning to their employers. Consequently, employers do not value the training their learners received.

Employers are also often unaware of what their learners study at TAFE so they cannot give them a chance to practice their skills at work. Due to this disconnect, both employers and learners have negative perceptions about the value of the training they receive at TAFE.

Insufficient employer support

While some employers strongly support their learners, others are not providing adequate support.

We found that some employers are reluctant to invest time and training for their learners until the learners can prove that they will be valuable employees. The lack of employer support may have reduced the opportunity for the learners to apply the skills that they have learnt at work. As such, they may not value their learning at TAFE NSW and hence quit early.

We also conducted a literature review to identify key barriers and drivers of attendance and completions.

What We Did

In partnership with TAFE South Western Sydney, we worked with teachers to send weekly SMS (mobile phone text messages) to employers about what their apprentices were learning at their education institute.

Over the semester, employers were encouraged to have conversations with their apprentices about their curriculum and to give them an opportunity to try these skills on the job.

How We Did It

The aim of our trial was to test whether a behavioural intervention to expand communication about TAFE courses could positively improve apprentice and trainees’ class attendance.

Our secondary interests were to see if this would enhance interactions between learners and employers, and advance completions.

We ran a randomised control trial with one set of learners in the Treatment group – the group who received our behavioural intervention. Their employers received text messages for one semester. They were encouraged to have conversations about what the students were learning in class, or to let students apply this learning at work. Each week, TAFE NSW teachers and head teachers wrote to our project team with a brief summary of what they taught in class. We then used this content to craft a message, either prompting employers to give the apprentice a chance to practice that skill on the job, or to talk to their apprentice about that topic. These messages were designed to:

• reinforce the value of the formal training
• remind the employer that it is critical for the apprentice to be in class each week
• encourage employers to connect the formal training with the on-the-job training, and to
  rotate the work program of their learners.

The other set of students were in the Control group. They benefited from the same level of service all students already receive, but without these text messages.

We included 48 classes in four fields: electro-technology, plumbing, carpentry, and allied timber trades. Students in our trial were randomly allocated into Treatment or Control based on the classes they were enrolled in.

What We Found

Our trial found that apprentices receiving this additional support were 3.1 percentage points more likely to still be studying by the end of the semester. Plus, there were additional benefits for other students. This translates to 147 extra classes that were attended for the 48 groups in the trial.

There are currently discussions with TAFE NSW and other stakeholders to roll this simple but cost-effective use of SMS communication to motivate learners across the state.

Potentially, this behavioural technique to strengthen the connections between teachers, learners and employers can enhance outcomes for thousands of apprentices and trainees (the latest available data show there are over 80,000 apprentices and trainees currently in training across New South Wales). 

Recommendations

Use timely text messages with actions employers can take

• Create stronger links to translate curriculum (including theory) to on-the-job-learning.
• Enhance communication with employers using SMS at optimum times, detailing what learners are learning and providing succinct tips, ‘cheat-sheets,’ checklists and other actionable messages to boost class attendance.

Draw on behavioural science to disseminate best practice to employers

• Make behaviour change easy. Reduce the hassle for employers to learn better communication and supervision skills, with mobile-accessible resources and support.
• Design attractive rewards and sanctions. Boost communication between learners and employers using incentives and disincentivise poor practices.
• Reinforce social norms of best practice. Disseminate best practice from employers with high completion rates. Make completions data available from top employers, using social comparison methods from behavioural science. This will stimulate peer influence and promote changed behaviour among low-performing employers.
• Maintain timeliness of messages. Effects of messages are short-lived and require ongoing reinforcement during difficult periods where learners may be considering quitting.

Apply lessons from the trial to better use data on completions

• Get access to data early and achieving consistency in reporting attendance and cancellations. Better use of data allows for superior monitoring of negative patterns, targeting and timing of interventions. Proactively support learners using machine learning to identify problem patterns in class attendance and target interventions.
• Support employers’ supervision capability using electronic prompts. SMS can be used to update records and inform employers on learners’ ongoing outcomes and class attendance.

TAFE NSW will be incorporating our recommendations as part of broader activities to enhance customer experience. 

As TAFE NSW has moved to the ‘One TAFE’ model and looks to standardise course content in 2019, our findings will support their new strategic plan.

How Social Science Helped

• Social science theory helped us to scope the problem, by identifying what was already documented about vocational training outcomes and delving deeper on what is happening on-the-ground in New South Wales. We did this through fieldwork, site visits to training organisations and employers, and interviews with educators, students, employers and providers
• Social science methods enabled us to co-design a solution, to test actual behavioural change. Many other programs rely on surveys and interviews with students to see what they thought about a new change. We did this too but went one step further: we delivered a behavioural intervention (encouraging conversations with apprentices and their employers) and then we followed class attendance over time, to see whether behaviour had actually changed. We then surveyed employers to see what they thought about the intervention.

Notes

Read our final report.

This post draws on our initial results originally published in our 2018 report. I have updated it with our final results published in 2019. I have also added additional context and the final section on social science.