Article Club: Treatment Integrity Errors During Discrete Trial Instruction

What Treatment Integrity Means During Discrete Trial Instruction

Discrete trial instruction (teaching in short, planned steps) carries most early autism programs. The session opened by naming its parts. Staff set the child up to learn. They give one clear instruction, or SD (the cue to respond). A controlling prompt (help that assures a right answer) stands ready. The child responds. A consequence follows. Then comes the intertrial interval (a short pause between trials). Teams use this chain to teach naming, social, and school skills.

Treatment integrity (how closely staff follow steps) is the score for that chain. The article treats it as precision in how a plan gets run. The speakers also used the phrase procedural fidelity (the same idea, other name). Older work had already shown that weak fidelity drains a strong plan. Two steps stood out in that work: giving the controlling prompt, and not leaking extra help.

The speakers also flagged a gap in those older studies: ecological validity (how well it matches real life). Many of them used errors that staff may never make. Some did not even study children in real settings. So the authors asked a plainer question. What do the errors staff truly make do to learning? They ran three studies in an early intensive behavioral intervention clinic (intensive early autism therapy).

treatment integrity is the degree to which an independent variable is implemented with precision.

From the talk — Pauli Catanese & Sam Mueller

What Nine Classroom Staff Actually Did Across 160 Trials

Study one just watched. Nine staff took part: four special education teachers, one general education teacher, one speech pathologist, and three paraprofessionals. Five children with autism were the learners. Observers sat in on one-on-one and small group school work. They scored eight staff steps, from setting up ready behavior to handling problem behavior. The math was simple: correct trials divided by chances, times 100. Watching ran in blocks of 12 trials, three to six blocks per visit.

The staff were told the point was to see how the child responded. They also knew they would get tips afterward. That cover story aims to cut reactivity (people change when they are watched). The group noted that a BCBA (the supervising behavior analyst) in the room shifts things anyway. Still, the shift is smaller when the focus seems to sit on the child.

Then the numbers. Across 160 trials, a tangible or edible item followed a correct answer on 21% of chances. A controlling prompt showed up on 41% of trials. The instruction was given only once on 55% of trials. The best steps, clear instructions and ready behavior, sat near 60%. The group still called that weak. The three lowest steps became the errors used in studies two and three.

The room pushed back in useful ways. One speaker asked if the tangible score tracked a reinforcement schedule (how often a reward follows a response). It did not. The code only asked whether an item followed a correct response. Others asked how a missing prompt was scored when the child needed no help. The talk left that open for future research. The group's own worry sat earlier in the chain. If you never win attention, the trial tells you nothing about the skill.

so as you can see that the teachers who work with kids with ASD don't always consistently implement all of the procedures with integrity, which can be pretty concerning.

From the talk — Pauli Catanese & Sam Mueller

Three Errors Stacked on One Trial Stalled Learning

Study two put those three errors to work. Six children with autism took part: four were four years old, one was six, one was nine. Sessions ran in a private room. Most targets were expressive identification (naming what you see): actions, jobs, and letters. One child worked on play skills with real items, like paper and a marker. Each child had one set taught well and a matched set taught poorly.

The design was an adapted alternating treatments design (sets taught side by side). It held a high integrity set, a low integrity set, and a control set. Training used a constant prompt delay (a fixed wait before help). It started at zero seconds. After two sessions with prompted answers on 90% of trials, the wait grew to five seconds. Mastery meant 90% correct on two sessions in a row. A set was dropped if it ran three times longer than another. Agreement between observers hit 93%, and integrity of the planned steps hit 99%.

Low integrity meant errors on 67% of trials, spread evenly. On an error trial, all three errors hit at once. The therapist added a second instruction with new words. No model or physical prompt followed a wrong answer or silence. No praise or preferred item followed a correct answer.

Every child mastered the set taught with full integrity. One child needed twice as many sessions on the sloppy set. Others never got there at all. When staff switched that same set to clean teaching, it moved fast. One child needed two extra sessions back at the zero second wait. So the errors did not merely slow learning. For some learners, they stopped it.

So we found out that these components are all important.

From the talk — Pauli Catanese & Sam Mueller

One Error at a Time: Effects Differed by Learner

Study three split the errors apart. Three learners came back, each one slow or stuck on the low integrity set. The design matched study two, but added three single error conditions. In one, the therapist repeated the instruction with words not in the protocol. In another, no model prompt followed an error or a no response. In the third, reinforcement was held back after correct and prompted answers. Agreement ran 98% to 99%, and planned integrity ran 99%.

The results split. Two of the three learners mastered every training set, though not at the same speed. For one learner, the fastest sets were clean teaching and the missing reinforcer condition. The slow sets were the extra instruction and the missing prompt. For two learners, the deepest drop came from no prompt plus reworded instructions. For another, the missing reinforcer did the most harm.

One result flips the story. A learner who got through the triple error set in study two did not master a single error set in study three. The authors noted those targets may have been harder. That fits a wider point from the talk. Hard skills and bigger arrays leave less room for error.

For supervisors, that is both a problem and a gift. You cannot rank these errors once and be done, since the worst error moved by learner. What held up across learners was the stack. Several errors at once was the worst case every time. So chip away at the count. Fewer errors per session is a goal you can actually coach.

integrity errors can be more problematic when difficult skill or multiple target stimuli are trained.

From the talk — Pauli Catanese & Sam Mueller

Why Error-Prone Trials Keep Getting Reinforced

The sharpest practice point in the talk came off a graph. One learner mastered the sloppy set anyway, just slowly. That is a trap for the adult in the room. Progress still shows up, so the staff member keeps getting reinforced for loose teaching.

The story a team tells itself then shifts. The learner gets labeled slow. The teaching never gets a second look. Rate of learning, not pass or fail, is where these errors show up first.

That reframe also helps with buy-in. One speaker said data moved them more than a rule did. Seeing a flat line beats hearing that errors can hurt learning. The group noted this is not only a new-hire issue. Staff who have been around for years still drift. For them, the why has to come with the what: the client's long term goals and what this specific skill is building toward.

So then as an instructor, you kind of get reinforced for some of your students are making progress.

From the talk — Pauli Catanese & Sam Mueller

A Trial Checklist You Rotate Instead of Rewrite

Most teams already train error correction. Far fewer train the plain outline of a trial. That was the gap the group named. A general checklist would cover five things: get ready behavior, win attention, give one clear instruction, prompt after an error, deliver the reinforcer (praise or a preferred item).

The upside is that this list barely changes across programs. One speaker ran integrity checks on a skill target and found nearly the same steps each time. Arrays are the exception. Shifting card positions may need its own short training. A per-client checklist, by contrast, goes stale fast. Programs change, and the paper in the folder does not.

So keep the general list and rotate the focus. Run the trial fidelity check one month. Run a consequence strategy check the next. Score it, write one line, and file it. Hand staff their own copy to scan before a session. Picture prompts work for this too. During shadowing, flip it around. Give the trainee the checklist and ask them to mark your errors. If a new staff member takes over a case, add a phase change line (a note on the graph) so later data still reads clearly.

I think we need a discrete trial instruction checklist, which I haven't seen floating around.

From the talk — Pauli Catanese & Sam Mueller

Training and Feedback That Actually Moves Fidelity

The common failure mode is quiet shadowing. New staff watch for a week or two. No one models, no one asks them to practice, no one gives feedback. The first real feedback lands at supervision, which can get cut short. Behavior skills training (teach, model, practice, then coach) fixes that order. Role play before the client, not after.

Pyramidal training (trained staff then train others) spreads the work. A supervisor trains two staff to run the steps, then trains them to train. Senior staff and fieldwork trainees are the natural fit. The group noted that teams often expect this without ever teaching it.

Volume matters as much as content. Cap feedback at one or two targets per session, five at the very most. A new RBT (the person running daily sessions) is already tracking the client, the room, and the data system. Pick the one lever that unlocks the rest, often attention or the reinforcer. Give it, circle back, then add the next.

Two moves from the discussion are easy to copy. First, ask the staff member their goal for the session before you share yours. One supervisor heard a goal about getting materials set up, and that turned out to be the real prerequisite. Second, run a preference assessment (ask what feedback style works best) on feedback itself. Some want coaching in the moment, some want a model, some want a written summary. Teams have used a short form for this. Then say out loud that questions are welcome, because staff months into a case may be scared to admit a gap.

I've also had really positive results with asking an RBT what their goal is.

From the talk — Pauli Catanese & Sam Mueller