Toward an Evidence-based Supervision Model: Training Students in Independent Fieldwork

Why the Eight Hour Supervisor Training May Not Be Enough

The talk opens with a graph of BCBA counts by year, running from 1999 through 2026 to date. The line climbs steeply over the past ten years. More new analysts means more people who need a good supervisor. Many of those supervisors are new as well.

The BACB asks a supervisor to finish an eight hour training first, and that course lists nine learning goals. Three of them are giving feedback, building in ethics, and setting up structured content. The list was never meant to cover every skill a supervisor needs. It also takes for granted that the BCBA can already do each one well.

That is the gap. Giving feedback is not one simple act. It is a chain of many small steps. A BCBA can sit through the course and still miss most of them. Knowing what to do and doing it well are not the same thing.

Therefore, it is possible these learning objectives may not be enough information to adequately prepare supervisors.

From the talk — Dr. Allison Bowhers-Peutin

Three Supervisor Skills Pulled From OBM Research

If the required course is thin, the next place to look is OBM. That stands for organizational behavior management (ABA applied to workplaces). Three supervisor skills come out of that work: feedback, goal setting, and prompting.

Feedback came first. In 1980, Sulzer-Azaroff and colleagues ran a feedback program at a large industrial site. Supervisors gave feedback twice a week on the job hazards from that week, plus tips on how to avoid them. Hazards fell about 60 percent from baseline (the level before the change).

Goal setting came second. In 2008, Amigo and colleagues used goals to cut the time it took to clean tables at restaurants. The manager showed each worker their own data and a target to reach. Workers hit that goal in two of five chances. Then the team added feedback on top of the goal, and clean up times fell well below the target.

Prompting came third. Most prompt research in our field studies clients with developmental disabilities. Johnson and Sulzer-Azaroff instead compared three kinds of prompts with graduate students: informational, thematic, and non-informational. The non-informational prompts taught the skill better and held up better over time. A supervisor also has to fade prompts so the trainee works without help.

For goal setting to work effectively, the intervention must be combined with a behavioral consequence.

From the talk — Dr. Allison Bowhers-Peutin

How the Study Broke Each Skill Into Scored Steps

Everyone in the study worked at a private school for children with autism, ages 3 to 22. The BACB had approved each supervisor to supervise, and some had years of practice while others had just started. A trainee could join if they had 50 or fewer supervised hours and their own supervisor was in the study.

Three skills served as the dependent variables (the behaviors that got scored). Each was cut into steps, and each step was marked right or wrong. Goal setting had nine steps. The supervisor reviews each score on the form and picks a goal to raise or hold. They say the goal out loud in clear terms and spell out what the skill should look like. They name current performance, set the target, and ask the trainee for input. Then they answer questions and show the trainee their graph.

Feedback had eight steps. Name a skill that got better and praise it in specific terms. Name a skill done wrong and spell out how it should be done. Give the feedback close in time to the work itself. Ask for questions, say what comes next, and tie it back to the trainee goals.

Prompting had nine steps. Spot the chance to prompt and act right away. Use the right prompting steps and record the data on the right sheet. Praise a correct response. Review the most intrusive prompt used and how to fade it. Then take questions and review the task list graph. The team also scored how well each supervisor followed policy. They also counted every booster session (a short one on one retraining).

We also had both supervisors and trainees complete social validity surveys to solicit feedback on their experience in the supervision program.

From the talk — Dr. Allison Bowhers-Peutin

The Performance Objective Matrix as a Goal Picking Tool

The main tool was the supervision competency form, filled in by the supervisor during each session with the trainee. It scores how the trainee did. Those scores then feed into a POM, or performance objective matrix (a weighted scorecard grid). The math is automatic. The supervisor picks a goal from data, not from memory of the week.

The top of the grid holds six goals for the trainee. These are work with clients, being ready for meetings, taking feedback, use of technical terms, professionalism, and project work. Under each one sits a score in steps of 20, up to 100 percent. Under the score sits a weight, which says which areas matter most at your company.

Two areas carried the top weight of 25 here: taking feedback and use of technical terms. Client work and being ready for meetings carried 15 each, while professionalism and project work carried 10 each. The bottom row of the grid is the performance index.

Here is the example from the talk. A trainee scores 40 on professionalism and 60 on project work, and both areas carry a weight of 10. So the index is 400 for one and 600 for the other. The next goal goes to professionalism, the weak spot once weight is counted. Change the weights and your priorities move with them.

This is calculated by the actual score times the weight.

From the talk — Dr. Allison Bowhers-Peutin

The Training Package: Lecture Baseline Versus BST

First came a preference assessment (a ranking of things they like), with items such as food and gift cards. In each supervision period, a supervisor earned a point for each step done right. A higher score bought a pick from the higher ranked items, so baseline came with a real reward on top.

Baseline then began with a group meeting. The trainer went over policies, contracts, and what was expected. The group walked through how to set goals, give feedback, and prompt a trainee. Each expectation came with steps, examples, and non examples. After that meeting, supervisors taped every one on one session with their trainee. In short, they were told what to do and paid to do it.

The trainings used BST, or behavioral skills training (tell, show, practice, then coach). If a supervisor scored 67 percent of steps or lower in a later session, a booster followed. In a booster, they practiced new role play scenes with the trainer and got more feedback. Experiment 1 used a multiple baseline design across the three skills (training staggered across skills).

During each training, supervisors were provided direct instruction on the skill followed by a video model of the skill and then an opportunity to practice until mastery was demonstrated.

From the talk — Dr. Allison Bowhers-Peutin

Experiment 1 Results: Feedback Was the Hardest Skill

Experiment 1 ran with six supervisors and six trainees. Two scorers rated the same sessions, or IOA (a check on scoring agreement). It ranged from 89 to 98 percent. Treatment fidelity (was the training run right) was checked in a third of trainings and averaged 96 percent.

Take Supervisor 2. In baseline they did 0 to 10 percent of the goal setting steps. After the workshop they needed one booster. Then they ran 80 to 100 percent, and their maintenance probe (a check weeks later) sat near 90 percent. Feedback looked different. Baseline swung from about 10 percent up to 70. The trend rose after the workshop, but three boosters were needed along the way. They ended at 100 percent, then scored about 50 at the probe. Prompting was flat near 45 to 70 percent, rose to about 90, and hit 100 at the probe.

Supervisor 3 tells much the same story. Goal setting sat at 0 percent for three straight baseline sessions. It climbed after the workshop, with one big dip and one booster. Feedback took two boosters in the first two meetings after the workshop. Prompting swung from 20 to 70 percent in baseline, then reached 100 percent.

The booster counts make the pattern plain. Supervisors needed an average of 1.7 boosters for goal setting, 3.6 for feedback, and 0.5 for prompting. Feedback took 11 boosters in all, and three of the six posted their lowest probe score on that skill. The surveys stayed high anyway. Supervisors averaged 4.2, 4.1, and 4.2 across the three trainings, and trainees averaged 4.6.

I would have thought setting goals would have been the highest in terms of booster sessions.

From the talk — Dr. Allison Bowhers-Peutin

Two Flaws in the Feedback Training, and Three Fixes

A weak result is worth a second look at the training itself. Two things set the feedback workshop apart from the other two.

First, no one told supervisors when their feedback would be watched. Goal setting and prompting happen at one clear moment in a meeting. Feedback can happen many times in an hour. No one named the exchange that would be scored, so supervisors were left guessing.

Second, the role play scenes were not alike. Goal setting scenes all sat on the POM and data collection, so they looked the same each time. Feedback scenes ranged across many kinds of talks. That range has real value on the job, since a BCBA has to handle all of them. Inside a training package, it made the practice uneven.

Experiment 2 changed three things. Supervisors were told just when to give feedback in the watched session. Trainings moved to a fixed monthly date instead of waiting for steady state responding (stable data before the next change). And the order flipped: prompting, then goal setting, then feedback.

We found there were two inconsistencies how the feed back training was developed when we compared it to the workshops for goal setting and providing prompts.

From the talk — Dr. Allison Bowhers-Peutin

Experiment 2 and What to Take Back to Your Agency

Experiment 2 ran with two supervisors. IOA ranged from 88 to 91 percent, and fidelity was checked in half the trainings at 100 percent. Supervisor 7 got better on all three skills and needed no boosters, but left the company before any probe. Supervisor 8 went from about 55 percent on prompting to 80 to 100 percent. Goal setting rose from 0 to 48 percent up to 90 to 100. Feedback sat near 70 percent in baseline, then hit 90 to 100. Every probe for Supervisor 8 came in at 100 percent, and neither one needed a single booster.

Read that with the limits in view, because two people is a small sample. Experiment 2 was a systematic replication (a repeat with changed steps). Both supervisors were newly certified and had never had a trainee before. Their history was not the same as the six who came before them.

The headline holds up anyway. Being told what to do, with a reward on top, did not get these BCBAs to mastery, and BST did. The monthly date fixed a real problem too. In Experiment 1, some supervisors started in February or March and waited until November or December for the last skill. Months of waiting let them practice errors.

That is the part to carry home. Steady state rules protect experimental control, but they stall the training. In an agency, a fixed monthly cycle is a thing you will really do. BST still costs effort: weeks to build the tools, then time with staff around billable hours. Other forms of behavioral teaching, such as PSI (self paced units with checks), may take less effort. The takeaway tool here is the competency form with its built in POM.

We do know that being told to do something isn't always the best way to teach someone.

From the talk — Dr. Allison Bowhers-Peutin