No Man's Land: Is Your Data Real? Clock Time Without IOA and Procedural Integrity Is Unfalsifiable
$10 · 1 Supervision CEU · certificate included
One-time purchase, permanent access. Claim your certificate as soon as you pass the quiz.
What you’ll learn
- Explain why a dependent variable without adequate operational clarity and interobserver agreement fails the behavioral dimension of ABA, and identify practices that produce unreliable session data. Describe why clock time without measured procedural integrity makes clinical outcomes unfalsifiable, distinguishing treatment failure from implementation failure, and summarize research on commission vs omission integrity errors. Analyze organizational and supervision contingencies that can shape reconstructed data and zero-fidelity 97155 time, and identify Ethics Code-relevant risks when reports rest on unverified data. Design a within-overlap calibration plan that collects parallel IOA and a brief PI checklist, and apply a two-by-two IOA×PI decision matrix. Describe how single-case replication and generality with verified caregiver fidelity can guide titration and step-down instead of calendar dates.
Presented by Sean Yocum and Mike Carrero

About this CEU
Time on a billing sheet is not an independent variable, and guesses entered in the parking lot are not a dependent variable. Mike and I rip the shiplap off everyday ABA data culture and ask whether graphs without interobserver agreement and interventions without procedural integrity are still science—or just unfalsifiable clock time. We walk the research on IOA, integrity errors, and floor contingencies, then give a Monday decision matrix you can run inside existing 97155 overlap. Episode 57: No Man's Land: Is Your Data Real? Instructors: Sean Yocum, MA, BCBA, LBA (#1-22-57936) and Michael Carrero, BCBA, LBA (#1-22-57932). Provider: Hickory Learning Group (BACB ACE OP-24-10923). Modality: recorded/asynchronous online. CEUs: 1.0 total (0.5 Learning, 0.0 Ethics, 0.5 Supervision). Price: $10. Theme: Ink and Coral (Rad N Bad). Target air date: Fri 2026-11-13 9:00 AM ET. Completion statement: Listening to the podcast alone does NOT earn BACB CEU credit. Credit is earned only by completing this OpenCEU course and passing the knowledge checks. Disclosures: Sean Yocum owns Hickory Learning Group (HLG), the CE provider. Affiliated with Homestead, Hi Rasmus, and Rad N Bad Behavior Consultants; Michael Carrero is co-host/co-instructor. No guest.
Credits
1 CEU (0.5 learning, 0.5 supervision) · 58 min
From the talk
What was covered
Why graphs without IOA and plans without procedural integrity cannot be tested, plus a 15 minute calibration block and a decision matrix.
- Score parallel IOA with your RBT on the same probe sheet and the same tablet layout during existing supervision overlap.
- Keep a five to six item procedural integrity checklist on the core steps: MO, SD, prompt timing, differential reinforcement, and error correction.
- If IOA sits under 80%, stop reading the graph and rebuild the definition and the measurement method first.
- If IOA is solid but integrity is low, retrain with behavior skills training and immediate feedback instead of rewriting the program.
- Check timestamps in the data app to catch sheets that cannot be scored live during a real session.
- Tie step down to replication across people and places, not to a date on the calendar.
Two Studs in the Wall: A Reliable DV and a Verified IV
Sean Yocum starts with a flipped house. From the curb it looks great. The counters gleam. Then you turn on the shower. Warm water pours through the dining room wall. You peel back the shiplap and find no real plumbing. He asks our field the same question. How much of our science is load bearing? How much is fresh paint over dry rot?
His answer is two studs. You need a reliable dependent variable, or DV (the learner's behavior you measure). It has to be backed by real interobserver agreement, or IOA (two people score the same thing). You also need a measured independent variable, or IV (the treatment you actually deliver). That one gets verified with procedural integrity, or PI (proof the plan was run right).
The hosts do not pick a side on hours. Two hours a week or forty hours in a clinic: that fight is not this episode. Sean cares about what happens inside the hours. He holds each claim up to single case design standards (one learner studied over time). Horner et al. (2005) set the bar for the DV. They asked for repeated measurement with IOA of at least 80%, across at least 20% of sessions in every condition. Baer, Wolf, and Risley (1968) called the first dimension behavioral. We measure what people do, not what someone recalls.
Sean also shares an informal poll. He asked about ten BCBAs across five companies one question. Do you track or score IOA and PI? The answer was no. He calls that anecdotal, not data. It is still why they built this episode.
if you claim to practice single case design, you need two non-negotiable studs in the wall.
From the talk — Sean Yocum
Car Seat Data: Why End of Session Guessing Breaks the DV
Michael Carrero teaches definitions with three Cs: clear, concise, and complete. You should know what the behavior looks like and sounds like. You should know its temporal parts (when it starts and stops). Most of all, scorers must agree. Hawkins et al. (1977) said a definition has to be objective, clear, and complete. Mike's test is simple. How often have you read a definition and asked whether something counts?
Now picture a floor at 4:50 p.m. One RBT (the front line therapist) has twelve active programs. The app has forty drop downs. A learner bolted out the door twice. Every graph still needs a data point before anyone goes home. So the numbers get rebuilt from memory on the drive out. Sean calls that car seat data.
Memory is a weak instrument. Mike points to police lineup research, where confident people pick the wrong face. He points to those old classroom videos where you miss what moves on screen. You may get the gist right. The rate, the time, and the intensity drift. Powell et al. (1977) showed a second path to error. Picking the wrong interval method bends the DV even with live observers. Sean also cites a meta-analysis of trained visual analysts. Looking at the same single case graph, they agreed about 76% of the time.
Mike's fix starts with the method, not the staff. Data belong in the record in the moment. If frequency counts cannot be scored with integrity, change the measure. Partial interval or duration may hold up better in that session.
A graph with 0 IOA and end-of-session guessing violates the behavioral dimension of ABA before we even talk about treatment.
From the talk — Sean Yocum
The Uncollectible Data Sheet Is a Design Failure, Not an RBT Failure
Both hosts refuse the bad actor story. Nobody is faking science on purpose. The system builds traps instead. LeBlanc et al. (2016) is the anchor here. Assign continuous trial by trial recording across a dozen programs. Add high rate problem behavior to block. You just built competing contingencies (two demands that cannot both win).
Sean offers an audit you can run today. Open the learner's program book or data app. Check the timestamp logs on scored trials. Say you find 80 trials across nine acquisition targets, all clicked between 4:52 and 4:56 p.m. That is not an integrity problem on the floor.
Mike's fix is training plus load. Use behavior skills training, or BST (teach, model, practice, then feedback). Get everyone on the same page before the session starts. Then tailor the target load to the staff, the family, and the setting. Running a target just to run it is fluff. Four programs done well beat twelve done fast.
Mike also names the culture piece. Staff have to hear that skipping a target is safe. No retaliation, no reprisal. The rest of the data will show the session was real. Sean adds the supervision angle. Take parallel IOA on the sheet you assigned, and twelve minutes may prove it cannot be scored live.
That's a BCBA design failure.
From the talk — Sean Yocum
Clock Time Without Procedural Integrity Makes Outcomes Unfalsifiable
The IV is the treatment itself: your procedures and your instructional notes. Baer, Wolf, and Risley called one dimension technological. Procedures must be named and described in full. Another dimension is analytic. You have to show a believable functional relation (proof your treatment caused the change). Horner et al. asked for overt, frequent measurement of treatment fidelity on the IV.
Fryling et al. (2012) named the cost of skipping that step. Without integrity data, clinical outcomes become unfalsifiable (impossible to prove wrong). A flat graph for six months then tells you nothing. Was it treatment failure, where the plan does not work? Or implementation failure, where the plan never ran as written? Sean says we flip a coin instead. Progress means ABA worked. A plateau means the diagnosis was severe, or we ask the payer for more hours.
Mike ties this to medical necessity (proof the service is needed). He uses a tourniquet (a tight wrap that stops bleeding). The standard is two inches above the wound, and never over a joint. That rule lets him check his own work. Without the rule, he is wrapping and hoping. His point lands hard. Asking for more hours means little if you cannot say what you did. You also need to show how well you did it.
Sean then guards against the easy clip. The hosts are not calling community ABA a racket. If payers heard that, they would not fund better IOA and PI. They would cut hours and deny care. Sean leans on CASP's 2020 practice guidelines here. A hospital that finds sloppy surgical technique does not announce that surgery fails. It enforces the surgical checklist. Our checklist belongs inside the supervision hour. Police the IV ourselves, or payers will do it with a chainsaw.
if we aren't measuring procedural integrity on the independent variable, aren't we just billing for clock time and calling it science?
From the talk — Sean Yocum
Commission Errors Hurt Learners More Than Omission Errors
Integrity errors are not all the same size. St. Peter Pipkin et al. (2010) studied this during differential reinforcement of alternative behavior. That is DRA (reward a better replacement behavior) for short. Errors of commission mean you reinforce the problem behavior anyway, even part of the time. Those errors degrade outcomes far worse than errors of omission (skipping a planned step). Commission errors also broke the functional relation.
Carroll et al. (2013) looked at discrete trial instruction (short, repeated teaching trials). Routine integrity errors slowed skill gains. Some learning stalled out. The plan on paper looked fine the whole time.
Here is the Monday version. You write a functional communication training plan (teach a better way to ask) with DRA. The implementer reinforces screaming 30% of the time. You did not water the plan down. You put screaming on a thin, intermittent schedule and made it stronger. Then the spike shows up on your graph. Ten minutes of PI data is the difference between fixing the plan and rewriting a plan that never ran.
you're basically going to look at that spike on the graph and you might rewrite the whole behavior plan when the plan wasn't even running.
From the talk — Sean Yocum
The Contingencies That Keep Producing Zero IOA and Zero PI
Sean drops the moral defect story. He runs a functional analysis (a test of why behavior keeps happening) on our own systems. Vollmer et al. (2008) wrote about the practical side of data reliability and integrity monitoring. Reliable data and high integrity do not maintain by magic. They follow the same reinforcement rules as any other behavior.
So look at what the floor reinforces. Turn in a clean, complete sheet at 4:59 p.m. and you get green check marks and zero hassle. Now say the hard thing instead. The session blew up for 40 minutes, so half the trial data are missing. You get a look, an email about trials per hour, maybe a performance plan. Sean calls it a Jersey diner. The line cook gets praised for plating in 90 seconds. Nobody checks whether the eggs are raw.
Supervision was supposed to catch all this. Dixon et al. (2016) looked at outcomes across 638 children in community ABA. Treatment intensity accounted for some variance in mastered learning objectives. Piling on raw supervision hours did not move the needle. Supervisor credentials and clinical quality did. Clock time billed under 97155 (the supervision billing code) is not calibration. Sean also names the ethics code. It covers accuracy in service billing and reporting. It also covers collecting and using data, and continuing evaluation with performance monitoring and feedback. Certify reports built on unverified data and it stops being only a science problem.
Try his 30 second test. Open the last three reauthorization packets you signed and sent to a funder. Look at the graphs that justified another six months. Can you point to one IOA score for the DV? Can you point to one task analyzed PI score near 80% to 85% for the IV? If both answers are no, ask what functional relation you certified. Mike adds one note of hope. North Carolina changed its authorization reports and asked providers to show percentages, and providers adjusted.
sitting in the corner on your laptop, billing 97155 while taking zero IOA and zero PI does nothing for the kid.
From the talk — Sean Yocum
Build a 15 Minute Calibration Block, Then Run the IOA by PI Matrix
None of this needs an unbilled hour. Step one: shrink the DV to human bandwidth. Drop continuous trial by trial recording across 15 targets. Use first trial probes or cold probes (score only the first try). Momentary time sampling (short checks at set moments) also works. Staff should be able to score it live with one tap.
Step two: turn part of your existing overlap into a 15 minute calibration block. Put the laptop down. Score parallel IOA with the RBT on the same probe sheet and the same tablet layout. Then score a five to six item task analyzed PI checklist on the core mechanism. Sean's items are the MO (what makes the reward worth it) and the SD (the cue or instruction). He also checks prompt timing, differential reinforcement, and error correction. Give immediate performance feedback right after you watch. Research shows that feedback raises and holds staff integrity on both antecedent and consequence steps.
Now you have two real numbers, and Mike says that is a roadmap. IOA under 80%: stop reading the graph, rewrite the definition, and simplify the measure. IOA good and PI low: do not rewrite the program, retrain with feedback and BST until PI clears 85%. IOA at 80% and PI at 85% with a flat line means something else. The DV is real and the IV ran, so change the plan now. Both numbers good with progress: you have a functional relation, so replicate it and start to fade.
Mike also names the social payoff. Numbers make feedback objective instead of personal. Feedback paralysis fades. The supervisor steps out of the hierarchy and sits in the front seat with the staff, reading the same map.
if you supervise 10% to 20% of a learner's hours, you are already in the room for the exact 20% session threshold that research is requiring.
From the talk — Sean Yocum
Replication and Generality Beat Calendar Dates for Step Down
One change in one corner of the clinic proves little. Single case standards ask for replication across at least three demonstrations. Think behaviors, settings, and implementers. That is how you rule out threats to internal validity (other reasons the change happened). Sean also points to a stable baseline (data taken before treatment starts). He cites Sidman (1960) for this idea. He also points to visual analysis of level, trend, variability, and immediacy of change.
Generality (skills that hold up elsewhere) is the seventh dimension. A change counts when it lasts over time, moves across settings, and holds with everyday people in the learner's world. Sean describes pyramidal training (one trainer trains the next trainer). Train the clinician to high PI first. The clinician then trains the caregiver or the teacher to that same verified criterion. Measure both sides: the coach's fidelity and the caregiver's fidelity.
Mike pushes back on fake generality. Three rooms down the same hallway is not the community. The skill has to mean something for the learner and the people around them. The real test is whether it holds where no support exists.
That reframes step down. Sean's sample titration plan has three phases. Phase one establishes the functional relation. Phase two transfers and replicates it across implementers. Phase three targets generality and discharge. When behavior holds under natural contingencies with verified caregiver fidelity, you titrate (lower hours as skills hold). Sean notes CASP 2020 and the BACB code support anchoring transitions in replication. The high hour versus low hour fight loses its heat when the data decide.
It has to actually have meaningful outcomes for the individual and those around them, not just could you replicate it in that environment, right?
From the talk — Michael Carrero
Common questions
Does listening to the Rad N Bad podcast episode earn BACB CEUs?▾
No. Listening on Spotify, Apple, or YouTube does not earn credit. You earn credit only by completing the on-demand CE event on OpenCEU.
How much IOA and procedural integrity do I actually need?▾
The hosts use Horner et al. (2005) as the bar for the dependent variable. They ask for IOA of at least 80%, across at least 20% of sessions in every condition. For the independent variable, Sean points to task analyzed integrity scores in the 80% to 85% range. Below those numbers, the graph is not ready to analyze.
How do I tell treatment failure from implementation failure?▾
You cannot without integrity data. Fryling et al. (2012) is the reference the hosts use: unmeasured integrity makes outcomes unfalsifiable. A flat graph plus a real PI score tells you something important. It shows whether the plan is weak, or whether the plan never ran as written.
Where does the time for IOA and PI come from if I only get 10% supervision?▾
Sean's point is that the time already exists inside your supervision overlap. Spend 15 minutes of it scoring parallel IOA with the RBT on the same sheet. Then score a short PI checklist. If you supervise 10% to 20% of hours, you are already present for the 20% of sessions the standard asks for.
What goes on a brief procedural integrity checklist?▾
Keep it to five or six items on the core mechanism of the plan. Sean lists the MO, the SD, prompt timing, differential reinforcement, and error correction. Score it live, then give immediate performance feedback right after you watch.
Will reporting low IOA and PI give payers a reason to cut hours?▾
The hosts argue the opposite risk is bigger. If the field never verifies its own data, payers will act on that gap themselves. Their analogy: a hospital that finds a sloppy surgeon enforces the surgical checklist instead of closing the operating room. They cite CASP's 2020 practice guidelines in that discussion.
About the speaker
Sean Yocum co-hosts the Rad N Bad Podcast for Hickory Learning Group with Michael Carrero. In this episode he walks the research on interobserver agreement (two people scoring the same session) and procedural integrity (proof the plan was followed). He also covers commission versus omission integrity errors. Then he builds a 15 minute calibration plan and a two by two decision matrix. He also traces how our own contingencies produce reconstructed data and zero fidelity supervision time. He ties step down to replication instead of calendar dates.
This summary was generated from the recording’s transcript. Quotes are taken word for word from the talk.
Concepts in this CEU
More from Hickory Learning Group
Intersection of Cultural Ontogeny, Ecological FBA and the Behavior Instructional Plan
Hickory Learning Group
Rad N Bad: Your Error Correction Sucks — Error-Correction Assessments, Trials to Mastery, and Learner-Informed Procedures
Hickory Learning Group
Rad N Bad: Your Prompting Hierarchy Sucks — Individualized Prompt Selection, Written Fading Rules, and Preventing Prompt Dependency
Hickory Learning Group
OpenCEU is supported by advertising from third-party sponsors. Sponsors are clearly labeled, have no influence over course content, instructors, or CEU decisions, and never appear on certificates. The BACB does not sponsor, approve, or endorse OpenCEU sponsors or their products.