Treatment Integrity Is Not a Checkbox: Building Meaningful Fidelity Systems
Presented by Dr. Jessica Osos
Treatment integrity is foundational to effective and ethical applied behavior analysis (ABA), yet contemporary research reveals a persistent disconnect between its valued importance and its actual measurement, analysis, and use in practice: most behavior analysts observe staff and provide feedback, but far fewer consistently collect, track, analyze, or act upon fidelity data in a systematic way, leaving fidelity treated as a compliance activity rather than a mechanism for improving clinical quality and client outcomes (Bergmann et al., 2024; Colón et al., 2024). This continuing education session reframes treatment integrity as an active, dynamic component of evidence-based practice and supervision. Attendees will examine the current state of practice — who collects fidelity data, how, and how often — alongside common barriers and misconceptions, including fidelity drift and overreliance on global scores, and will then be guided through a practical framework for building feasible, scalable fidelity systems: task-analyzing procedures into observable components, selecting appropriate measurement systems, prioritizing fidelity targets, and embedding performance feedback and behavioral skills training (BST) (Morris et al., 2024). Clinical examples across service settings will illustrate how treatment integrity data can inform supervision, staff training, ethical decision-making, and program modification without increasing administrative burden.
Sign in to buy — $15One-time purchase, permanent access. Cert delivered to your inbox the moment you pass the quiz.
About this CEU
Treatment integrity is foundational to effective, ethical ABA — yet recent practitioner surveys reveal a persistent gap between how much behavior analysts value fidelity and how consistently they measure, analyze, and act on it. This session reframes treatment integrity as an active driver of clinical quality rather than a compliance checkbox, and walks participants through a practical, six-step framework for building fidelity systems that are feasible, scalable, and genuinely useful in supervision.
Abstract
Treatment integrity is foundational to effective and ethical applied behavior analysis (ABA), yet contemporary research reveals a persistent disconnect between its valued importance and its actual measurement, analysis, and use in practice: most behavior analysts observe staff and provide feedback, but far fewer consistently collect, track, analyze, or act upon fidelity data in a systematic way, leaving fidelity treated as a compliance activity rather than a mechanism for improving clinical quality and client outcomes (Bergmann et al., 2024; Colón et al., 2024). This continuing education session reframes treatment integrity as an active, dynamic component of evidence-based practice and supervision. Attendees will examine the current state of practice — who collects fidelity data, how, and how often — alongside common barriers and misconceptions, including fidelity drift and overreliance on global scores, and will then be guided through a practical framework for building feasible, scalable fidelity systems: task-analyzing procedures into observable components, selecting appropriate measurement systems, prioritizing fidelity targets, and embedding performance feedback and behavioral skills training (BST) (Morris et al., 2024). Clinical examples across service settings will illustrate how treatment integrity data can inform supervision, staff training, ethical decision-making, and program modification without increasing administrative burden.
Credits
1 supervision CEU · 54 min
From the talk
What was covered
Why fidelity data break down in real ABA (applied behavior analysis) practice. A six step way to build a treatment integrity system that fits your caseload.
- Treat fidelity as the answer to "why is this client stuck?" rather than as a form you file.
- Score each component of a plan on its own. A single global percentage hides the one step that is failing.
- Aim to observe about 40% of sessions. Use indirect tools only to fill the gap, never as your main measure.
- Break one procedure into small, scorable steps (a task analysis), then turn it into an operational checklist this month. Make every item something two supervisors would score the same way.
- Cut reactivity (people acting differently when watched). Observe often, start the moment you walk in, and keep your schedule hard to predict.
- Fix the system before you fix the person: most reported barriers are time, tools, and missing expectations.
Why Treatment Integrity Protects Your Data and Your Clients
Treatment integrity (how closely staff follow the plan) is easy to define. It is the extent to which a plan is used as it was designed. Gresham defined it in 1989, and later work carried it forward. Say your plan calls for a prompt on every trial. Fidelity data tell you whether that prompt really happened.
Dr. Osos makes the case from three angles. The first is internal validity (proof the plan caused the change). When behavior changes, fidelity data let you say the plan did it. Without those data, you are guessing. The second angle is daily decision making. Picture a client who is not making progress. Your first question should be whether the plan was run as written. High fidelity points to a plan that needs revision. Low fidelity points to a staff member who needs more training.
The third angle is ethics. Checking and re-checking our work runs all through the BACB ethics code (the rules behavior analysts must follow). It also sits in the CASP practice guidelines (a provider group's standards of care). Research backs this up. Errors show up among teachers, aides, and technicians. One 2013 study looked at discrete trial training (teaching in short, set trials). Fidelity ran from 21% to 74% in real settings. A 2007 study found students were off task less often when teachers ran plans well. When fidelity slips, plans work less well and more slowly.
And so those two different measurements lead to completely different actions.
From the talk — Dr. Jessica Osos
Omission and Commission Errors Do Different Kinds of Damage
Fidelity errors fall into two buckets, and the split matters for how you respond. Errors of omission (a required step was skipped) mean the staff member left something out. The prompt was due, and it never came. One research team found omission errors reliably slowed skill gains.
Errors of commission (adding a step the plan lacks) are the flip side. The staff member adds something the plan never called for. That might be praise the protocol does not specify. Dr. Osos says she often sees a reprimand that is not in the plan. Another team found commission errors were more harmful when the target was problem behavior.
Effects are also idiosyncratic (the same error hits clients differently). In one study, three participants faced the same commission error. One was basically unaffected. One learned the skill after extra sessions. One never learned it. That is the whole argument for measuring. You cannot predict from your chair which error will matter for which client. It also takes very little to count as low. Praising every other correct response is already 50% on that item. So read fidelity data next to the client's own data, not alone.
So the takeaway isn't to memorize which is worse, right?
From the talk — Dr. Jessica Osos
What Two 2024 Surveys Say About Real Practice
Two recent surveys of practicing BCBAs (board certified behavior analysts) tell the same story. Bergeman et al. (2024) surveyed 203 BCBAs across settings and populations. Cologne et al. (2024) surveyed 137 BCBAs about training and barriers. Nearly all had been trained in fidelity, and nearly all said they value it. Yet most collect data on only a small slice of their sessions. The steps most often skipped are the data steps: tracking, analyzing, and acting.
So the fix is not a pep talk about why fidelity matters. The fix is time and systems. Look at who collects the data and you can see one lever right away. About half of respondents collect it themselves. About a third have a supervisee collect it. Smaller shares go to staff they do not supervise.
Training your supervisees to collect fidelity data does double duty. It expands how many sessions you can cover. It also builds a real skill that sits on the task list and prepares them to supervise later. Dr. Osos notes that staff training time can often count as unrestricted fieldwork hours. The method news is good too. About 99% of respondents used direct assessment. About 80% observed live sessions, in person or by live stream. Dr. Osos guessed the rest watched recorded video.
the problem is downstream of belief
From the talk — Dr. Jessica Osos
Pick the Right Grain of Fidelity Checklist
Direct observation means watching the person run the procedure and scoring it against a written checklist. Live or recorded both count. But checklists are not all the same, and they sit on a spectrum. The most global is an all or none whole session score. That is one yes or no for the entire session, and very few people use it. It throws away nearly everything you would need for a next step.
A generic checklist breaks the plan into parts but is not built for one client. About 20% use that. A specific checklist is built for that client and program, and another 20% or so use that. Trial by trial component fidelity, where you score every part at every chance, is the gold standard. Only about 15% use it. Over half of respondents use some kind of component level checklist.
More granular means more sensitive and better at catching the exact error. It also costs more effort. Good clinical judgment is picking the right point on that line for the item and the moment. Here is why grain matters. Say monitoring runs at 100% and praise is delivered on two of five chances, which is 40%. Average them and you report 70%, which looks fine. That single number hides a component running at 40%. A 2015 study showed exactly this masking effect. Component scores let you say what was strong and what to target next.
But never throw away the component level detail.
From the talk — Dr. Jessica Osos
How Often to Measure, and Where Indirect Tools Fit
This is the uncomfortable part. About 63% of respondents collect direct fidelity data on 20% or fewer of their supervision sessions. Only about 8% do it for more than 80%. Gresham and colleagues (2017) looked at how much sampling you need to trust the number. Their work points to about 40% of sessions as a floor. Very few practitioners clear that bar. If you sample one session in five, your number may not describe the other four.
Dr. Osos holds her own clinic to the same mirror. Her team runs a fidelity check on each RBT (a front line ABA technician) once a month. She says plainly that this likely sits below the 40% mark, even if it beats what many teams do.
Indirect methods have a place, but they need care. Only about 30% of respondents use any at all. Permanent products (what the session leaves behind) means checking a finished data sheet. It can only capture what leaves a trace, and it cannot tell you how well those trials were run. Supervisee interviews, used by about 25%, depend on memory and on how people want to look. Self-reports, used by about 22%, tend to run high, because people rate themselves generously. One workaround she has seen: record the session, then watch it with the staff member and score it together.
You really want to avoid them being your primary measure of fidelity and you want to pair them with direct observation.
From the talk — Dr. Jessica Osos
Steps One and Two: Task Analyze, Then Assign a Measure
The practical core of the talk draws on a six step framework from Morris et al. (2024). Task analyze the procedure. Assign a measure to each part. Plan the observation. Collect the data. Analyze and interpret it. Take action. The framework is not tied to one intervention. You run the same loop for anything you prescribe.
Step one is a task analysis (breaking a task into small steps). Split the plan into parts you can score on their own. Include the ordered steps and the conditional ones, the if this then that items. Then turn fuzzy words into observable behavior. The test is interobserver agreement (two scorers land on the same answer). If two supervisors would score an item differently, it is not written well enough yet.
Step two assigns a measure, because not every item is a yes or no. Behavior has three measurable dimensions. Repeatability is count, as in deliver praise at least five times. Temporal extent is duration, as in how long an activity runs. Temporal locus is timing, such as latency (time from cue to response), as in redirect within 15 seconds. The skill is matching the measure to what actually matters about that component.
A huge part of this step is operationalizing fuzzy terms.
From the talk — Dr. Jessica Osos
Steps Three and Four: Plan the Observation and Manage Reactivity
Direct observation costs supervisor time, so design it to be efficient. Four choices shape the plan. Prioritize the high impact and complex parts first. Still sample the simple ones, because drift (a slow slide away from the plan) can start anywhere. Choose the moment, so you watch when the target part actually happens. If novel instruction runs in the morning, move your schedule or record it. Right size the length, long enough to be representative and short enough that you keep doing it. Then set the frequency, and spread it across strong and weak performers so observation is not a punishment.
Dr. Osos rotates two things at her clinic each month. She changes which supervisor runs the check and which client the RBT is with. That widens the sample without adding sessions.
Step four is collecting the data, and reactivity is the catch. A 1982 paper flagged it. A 2021 paper offers four fixes. Monitor often, so your presence stops being an event. Start scoring the moment you enter, before anyone can settle in. Run some unannounced return visits. Keep the overall schedule unpredictable. Balance that against rapport. Greet the staff member and say what you are measuring, so the check feels supportive instead of like a trap.
So staff often perform differently when a supervisor is watching.
From the talk — Dr. Jessica Osos
Steps Five and Six: Read the Data, Then Fix the System
Summarize to a global score if it helps you track trends, and weight the items that matter most. Keep the component scores either way, and show the staff member the item level sheet. Then act on the pattern. Low fidelity with no progress means retrain the implementer, not rewrite the plan. High fidelity with no progress means revise the plan or swap the intervention. Sometimes the data surface a barrier instead. If nobody can hit a 15 minute interval because of how the room runs, ask them why. Then change the item. If every RBT misses the same step, the step is the problem.
The reported barriers back that up. In one survey, lack of resources like time, staff, and equipment was the biggest at nearly 29%. About one in five said their employer simply does not require fidelity. Limited supervision time and heavy caseloads followed. The other survey found the same top three: no time, other tasks first, and no system in place.
The training data close the loop. Definitions, the ethics code, and when and how to collect are well covered, around 80%. Tracking, analyzing, and giving feedback are covered far less. Those are exactly the steps practitioners skip. The levers are organizational, which is the hopeful part. Build efficient checklists. Move to a system that graphs for you if you can. Train supervisees to collect. Write fidelity into your formal supervision expectations. Train new staff with BST (tell, show, practice, then feedback). Use the same checklist you will score them on, so nothing is a surprise. Ready made templates already exist in Morris et al. (2024).
the top barriers aren't, I don't know how to, or I don't think it's worth it.
From the talk — Dr. Jessica Osos
Common questions
What is treatment integrity in ABA?▾
It is the extent to which an intervention is carried out the way it was designed. Gresham defined it in 1989, and it is also called procedural fidelity. If your plan says to prompt on every trial, fidelity data tell you how often that prompt actually happened. Without it, you cannot say whether the plan or something else caused a change.
How often should I collect fidelity data?▾
Gresham and colleagues (2017) suggest about 40% of sessions before you can trust the estimate. Survey data show most BCBAs sample 20% or fewer, and only about 8% get past 80%. If direct observation for 40% is out of reach, blend direct observation with indirect methods to raise your coverage. Just keep direct observation as the primary measure.
Are self-reports and completed data sheets good enough?▾
They are fine as a supplement and risky as your main measure. Permanent products only show what leaves a trace, and they cannot tell you how well the trials were run. Interviews depend on memory and self-presentation, and self-raters tend to score themselves high. Pair any indirect method with direct observation.
The client is not progressing. Do I change the plan or retrain the staff?▾
Check fidelity first, because the two cases call for opposite actions. Low fidelity with no progress means the plan was never really tested, so retrain and support the implementer. High fidelity with no progress means the plan itself needs revision or replacement. Component level scores tell you which item to target.
Do I have to share fidelity data with insurers?▾
Dr. Osos does not report scores to funders. Her team checks the treatment integrity and IOA (agreement between two observers) box on the CASP protocol modification note template. In the narrative section, they note strengths and areas to improve. She expects funder scrutiny to grow, so she recommends measuring now in case it is ever requested. Fidelity data can also help explain slow progress at reauthorization.
What goes on a real fidelity checklist?▾
Her clinic uses a first page of plus or minus scoring for each DTT (discrete trial teaching) component, trial by trial, plus an IOA sample on two to four programs. Later pages use a three point rating for broader session skills: environmental arrangement, managing antecedents, skill acquisition, play and NET (natural environment teaching), behavior plan, transitions, and group activities. The last page totals points earned out of points available. Every check ends with things to maintain and up to three things to focus on.
About the speaker
Dr. Jessica Osos is a doctoral level BCBA. Her research covers generative learning strategies, toilet training, and clinical quality in ABA. She is the EIBI (early intensive behavior intervention) and ABA clinical program manager for Intermountain Children's Health ABA services in Utah. She also consults independently through Copper Consulting Group, LLC. In this session, she shared the fidelity checklist and monthly monitoring system her own clinic uses. She also covered the research behind it.
This summary was generated from the recording’s transcript. Quotes are taken word for word from the talk.
What you'll learn
- 1Explain the functional relationship between treatment integrity, intervention effectiveness, and data‑based decision making, including ethical implications when fidelity is not directly assessed. Identify common organizational, contextual, and behavioral variables that contribute to fidelity drift, including competing contingencies, training gaps, and system failures. Describe and apply at least three empirically supported strategies for developing sustainable treatment integrity systems, including task analysis of procedures, direct fidelity measurement, and performance‑based feedback.
Concepts in this CEU
Course materials
Related talks
Talks that cover the same concepts
Rad N Bad: Dosage by Tradition: What the Research Actually Says About ABA Hours, and the Contingencies Behind Them
Hickory Learning Group
Reflective Supervision and Client Outcomes
The Behaviorist Bookclub
Toward an Evidence-based Supervision Model: Training Students in Independent Fieldwork
Willow Mentorship & Consulting, LLC
Teaching RBTs to Think Clinically: Training In-the-Moment Decision Making
Copper Consulting Group
Building a Supervision System That Actually Works
The Behaviorist Bookclub
Supervision Articles Deep Dive
The Behaviorist Bookclub
More from Copper Consulting Group
Crash Course in Behavior Analytic Toilet Training: Translating Research to Practice
Copper Consulting Group
The Use of Matrix Training to Teach Multi-Component Tacting Through Telehealth
Copper Consulting Group
Developing Team Culture Through Psychological Safety
Copper Consulting Group
OpenCEU is supported by advertising from third-party sponsors. Sponsors are clearly labeled, have no influence over course content, instructors, or CEU decisions, and never appear on certificates. The BACB does not sponsor, approve, or endorse OpenCEU sponsors or their products.