Interobserver Agreement (IOA): How to Trust Your Data
Interobserver agreement (IOA) shows if two observers record behavior the same way. Learn the 80% rule, IOA methods, and how BCBAs use it in practice.
Key takeaway
Interobserver agreement (IOA) is how closely two independent observers match when they record the same behavior. When two people watch the same session and write down the same thing, we can believe the data more.

No Man's Land: Is Your Data Real? Clock Time Without IOA and Procedural Integrity Is Unfalsifiable
Free learning path on this topic
Supervision, Done Right
Four supervision CEUs that build on each other. What reflective supervision is, how to build a supervision system, how to assess supervisee competence instead of counting hours, and how supervision affects provider outcomes. Each course is free and issues its own certificate.
Experts who cover this
7 recordings on openceu cover interobserver agreement.
- Featured talk1 CEUs (0.5 Learning, 0.5 Supervision)
No Man's Land: Is Your Data Real? Clock Time Without IOA and Procedural Integrity Is Unfalsifiable
Sean YocumHickory Learning Group
The hosts argue IOA is the non-negotiable standard verifying the dependent variable, citing the 80% across 20% of sessions benchmark from Horner et al. (2005)…
- 1 PDU
Trustworthy Indicators and Believability of Data
Dr. Celia HeymanCelia Heyman Exam Prep
Dr. Heyman presents IOA as the practical substitute for establishing true values in behavior analysis, then walks through roughly nine IOA calculation methods…
- 0.5 Learning CEUs
Rad N Bad: 30–40 Hours of ABA — Dosage, Titration, and the Data Behind Service Intensity
Sean Yocum, LBA & Michael CarreroHickory Learning Group
IOA is framed as essential proof that data presented to insurance funders is valid and reliable, with the hosts arguing it should be a required reporting…
- 1 Supervision CEU
Toward an Evidence-based Supervision Model: Training Students in Independent Fieldwork
Dr. Allison Bowhers-PeutinWillow Mentorship & Consulting, LLC
Total count IOA was collected across both experiments to verify reliability of scoring, ranging from 89-98% in experiment one and 88-91% in experiment two.
- 1 Supervision CEU
Dunder Mifflin’s Guide to BCBA Supervision: Lessons from The Office
Mellanie PageThe ABA Collective, LLC.
Mentioned as one of the concrete, trackable metrics supervisors should give RBTs regular feedback on, and as something an RBT can collect on themselves while…
- 1 Learning CEU
Research to practice - extending past the pages
Matt HarringtonThe Behaviorist Bookclub
Named as one of the first, most parsimonious things to rule out when a supervisor only observes a fraction of sessions and the data seems off, before jumping…
- 1 Supervision CEU
Article Club: Treatment Integrity Errors During Discrete Trial Instruction
Pauli Catanese & Sam MuellerAchieve Behavior Partners
Reliability metric reported across the three studies (93-99%) to confirm consistent scoring of participant and therapist responses between observers.
On this page · 15 sections▾
Interobserver agreement (IOA) is how closely two independent observers match when they record the same behavior. When two people watch the same session and write down the same thing, we can believe the data more. When they do not match, the graph may be telling a story that never happened.
Behavior analysts lean on IOA because our field rarely has a "true value" to check against. This page pulls together what practitioners and researchers said about IOA across several talks. You will learn what it measures and how to pick a method. You will also learn what the 80 percent rule means and why some experts treat IOA as a hard rule while others leave room for judgment.
What interobserver agreement actually measures#
Dr. Celia Heyman starts with a simple problem. In some sciences, you can check your number against a known answer. That known answer is called a true value. In behavior analysis, we usually do not have one. So we cannot easily prove that our data are accurate.
Heyman separates three ideas that people often mix up. Validity means you measured the right thing. Accuracy means your number matches the true value. Reliability means you get the same number again, or that another person gets the same number as you. In her example, two people measure the same surface and both arrive at the same answer. We still do not know if the answer is accurate. But we know it is reliable, because two people agree.
That is the job IOA does. It does not prove the truth. It shows that the measurement is consistent across people. Heyman puts it plainly.
we use something called inter-observer agreement, IOA.
From the talk — Dr. Celia Heyman
For a clinician, this matters every day. If your data come from one person's eyes, you are trusting one person's habits. IOA lets you check those habits against a second observer.
Why IOA is a load-bearing wall#
Sean Yocum frames IOA with a house-building image. He says any practice that claims to use single case design needs two "studs in the wall." One is a reliable measure of the learner's behavior. The other is a measured treatment that was actually delivered as written. In research language, the learner's behavior is the dependent variable, or the thing you measure to see change. The treatment is the independent variable, or the thing you change.
A reliable dependent variable that is backed by real inter-observer agreement and a measured independent variable verified with procedural integrity.
From the talk — Sean Yocum
Yocum traces this back to the start of the field. He points to Baer, Wolf, and Risley in 1968, who named "behavioral" as the first dimension of ABA. That means we measure what people actually do, not what someone remembers. He also cites Hawkins and colleagues in 1977, who said a behavior definition must be objective, clear, and complete. A good definition lets two observers score the exact same response.
Without IOA, Yocum argues, you are not running a science. He calls it a theater prop. The graph looks real, but nothing holds it up.
The 80 percent across 20 percent benchmark#
Yocum gives a specific number. He points to Horner and colleagues in 2005, who laid out quality signs for single case research. Their standard for the learner's behavior was repeated measurement backed by IOA.
the non-negotiable standard for the dependent variable was repeated measurement backed by inter-observer agreement, or IOA, of at least 80% across at least 20% of sessions in every condition.
From the talk — Sean Yocum
So there are two parts. First, agreement should reach at least 80 percent. Second, IOA should be collected in at least 20 percent of sessions. Note the last phrase too: "in every condition." That means baseline and treatment both need checks, not just one phase.
Yocum then paints a picture of a typical late afternoon in a clinic. One tired technician or caregiver is juggling twelve programs and a tablet with forty drop-down menus. Every graph needs a data point before anyone goes home. What happens? He calls it "car seat data." People rebuild the session from memory at the end of the day.
He adds a sobering point. Even trained experts do not always agree when they read the same graph.
trained visual analysts looking at the exact same single case graph, and they only agreed about 0.76 of the time or 76% of the time.
From the talk — Sean Yocum
If experts agree only three out of four times on real data, that is already a problem. Guessed data with zero IOA gives them even less to work with.
What to do when IOA comes back low#
Yocum offers a clear decision rule. He treats low IOA as a stop sign for the whole graph.
If your IOA is less than 80%, stop looking at your graph. Your dependent variable is unreliable.
From the talk — Sean Yocum
His fix is not to change the treatment. His fix is to change the measurement. He says to rewrite the operational definition, which is the written rule for what counts as the behavior. He also says to simplify the measurement system. If staff cannot agree, the definition may be too vague, or the system may ask too much at once.
He then walks through what happens after IOA is good. If IOA is fine but the treatment was not delivered well, you still cannot judge the program. In that case, he suggests feedback and behavioral skills training for staff, not a program rewrite. Only when IOA is at 80 percent and the treatment is delivered well can you trust a flat graph. Then the data are real, and you can make a fair call about the plan.
This order matters. It keeps teams from blaming a learner or a program when the real problem is the data sheet. IOA comes first because every other decision rests on it.
Choosing the right IOA method#
There is more than one way to calculate IOA. Heyman admits the list can feel like a lot.
Now, there are about nine of these. Okay? And they might feel like it's so overwhelming to remember everything.
From the talk — Dr. Celia Heyman
Her shortcut is to look at how the data were collected before choosing a formula. Ask one question. Did the data come from event recording, from timing, or from time sampling? Event recording means counting each time a behavior happens. Timing means measuring how long something lasts or how long it takes to start. Time sampling means splitting a session into intervals and marking whether the behavior happened in each one.
Once you answer that, most options drop away. Heyman notes that if you are working with a count, it came from event recording. That leaves you with about four IOA choices instead of nine.
She also explains why time sampling is different. With whole interval recording, a behavior that lasts only half an interval does not get scored. So some real instances go uncounted. That is why time sampling is called discontinuous measurement. The other methods are continuous, because they try to catch every instance.
One technical report describes a spreadsheet tool for many of these formulas. It covers total count, exact agreement, trial-by-trial, interval-by-interval, scored-interval, unscored-interval, and duration methods. Tools like this can take some of the math burden off busy clinicians.
Interval-by-interval, scored, and unscored methods#
Heyman spends time on the three time sampling options. Each one counts agreement a little differently.
Interval-by-interval IOA compares every interval. If both observers marked the same thing for an interval, that is an agreement. Scored-interval IOA looks only at intervals where at least one person said the behavior happened. Unscored-interval IOA looks only at intervals where at least one person said it did not happen. In her worked example, only one interval had both observers marking "no." Six intervals had at least one "no." So unscored IOA was one divided by six.
Why would anyone use scored or unscored methods? Heyman says the reason has to do with how often a behavior happens. The scored and unscored methods are built for behavior that happens at certain rates.
But in research, you often do not know ahead of time how often a behavior will occur. So researchers mostly stick with one method.
the most popular time sampling IOA used in publication is interval by interval.
From the talk — Dr. Celia Heyman
Heyman adds that she has not found published articles using scored or unscored IOA for time sampling data. For practitioners, that means interval-by-interval is the common default. Still, knowing the others helps you spot when a high number might hide a problem.
IOA belongs in clinical practice, not just research#
Stephanie Peterson hears one excuse often from practitioners who want to publish.
A lot of times what I hear is, well, we can't publish this because we don't have IOA data and we just don't have capacity to get IOA data.
From the talk — Stephanie Peterson
Her answer flips the question around. If you are not collecting IOA in daily practice, that is already a concern.
Shouldn't we be taking IOA data anyway on our clinical practice? Like if we're not taking IOA data on our clinical practice, that's probably not a good thing.
From the talk — Stephanie Peterson
Peterson points to the stakes. Clinicians use data to decide whether to remove part of a treatment. They use data when a team weighs whether a child goes on or off medication. They use data to decide on a more restrictive intervention. These are big choices for a real person. If the data behind them have no IOA, that choice may not be trustworthy.
She points to a 2017 article in Behavior Analysis in Practice on the importance of IOA. She credits that work to Vollmer and colleagues. Her main point is that the bar for research and the bar for good care should not be far apart. If a measure is good enough to change a child's medication, it should be good enough to check with a second observer.
Using video to make IOA possible#
Many clinics struggle to have two people in the room at once. Video can help. Several published studies scored IOA from recordings after the session ended.
Lindgren and colleagues (2024) video-recorded all sessions in a telehealth study. Trained research assistants then reviewed recordings to score IOA for at least 33 percent of sessions. They picked which sessions to check with an online random number generator. Random picks matter, because they stop people from only checking their best sessions. Sherman and colleagues (2025) also videotaped every session and scored IOA afterward. They reached 99 percent agreement across 32 percent of sessions.
Mellanie Page shows how video can serve supervision too. She describes a self-evaluation step for technicians who may have blind spots. The supervisor writes out objective criteria for the plan. The technician watches a recording of their own session. Then both compare what they saw against the criteria.
You can ask them to take IOA data as they watch with you so they can pinpoint areas of consistency with expectation and then areas where they need to make improvements.
From the talk — Mellanie Page
This turns IOA into a teaching tool. The technician sees with their own eyes where their data match and where they drift.
IOA as a supervision and feedback tool#
Page places IOA inside behavioral skills training, or BST. BST is a four-step teaching method: instruction, modeling, rehearsal, and feedback. During her talk, collecting IOA alongside a new staff member came up as one practical way to apply BST in the field.
Collecting IOA data, placing them with one of the kids with BST.
From the talk — Mellanie Page
The idea is simple. When a supervisor and a technician score the same session, any gap shows up in the numbers. That gap points to what needs more training. It might be the definition, the timing, or the way the behavior looks in real life.
Yocum makes a related point about feedback. He describes a kind of "feedback paralysis" that can happen between supervisors and staff. Feedback can feel personal and top-down. But when feedback is tied to IOA and treatment delivery data, it becomes objective. Yocum says it takes the supervisor out of the hierarchy and puts both people side by side. They are looking at the same numbers, working toward the same goal.
For supervisors, this is a big win. IOA does not just protect the data. It also gives a neutral starting point for hard conversations about performance.
Ruling out data problems first#
Matt Harrington ties IOA to parsimony. Parsimony means you rule out the simplest explanations before you reach for complex ones. When a case is not going well, the first simple question is whether the data are even right.
is my data accurate as a supervisor? I don't get to see a hundred percent of the sessions. I get to see 20%, 30% if I can slide it through Medicaid.
From the talk — Matt Harrington
Harrington's point is that a supervisor sees only a slice of the work. The rest of the sessions come in as numbers on a screen. Is that data telling the real story? Or is the supervisor overreacting to numbers that do not match what is happening?
He lays out an order of checks. First, check the data. Next, check whether the treatment is delivered as planned across settings. Then look at values, motivation, and skills. His plan for the first step is short and direct.
I'm going to take IOA for a little bit. Give me a week. Let me rule out IOA.
From the talk — Matt Harrington
This keeps teams from building a complex clinical theory on top of a data error. A week of IOA can save months of chasing the wrong problem.
IOA and reporting to funders#
On a separate podcast episode, Sean Yocum and Michael Carrero take IOA into the insurance world. Clinics present graphs to funders to show progress and justify hours of service. Yocum says treatment delivery data and IOA are central to proving the treatment is having an impact.
if pi and ioa are huge to prove that our intervention is actually having the impact on what is happening
From the talk — Sean Yocum
The hosts raise a hard question. If providers do not record or report IOA, how can anyone be sure the data reflect what really happened? Saying "it is true because I signed off on the session" is not enough, they argue. Carrero is direct about the fix.
EI and IOA data. it needs to be reported to funders.
From the talk — Michael Carrero
The hosts go a step further. They say providers with nothing to hide should welcome outside review. They call for insurance reform. They also suggest that someone who knows how to read graphs, maybe a BCBA, should look at the data. In their view, IOA is not just an internal quality step. It is part of being honest with the people paying for care.
What IOA looks like in published studies#
Several speakers reported IOA when they presented research. These numbers give a feel for what strong agreement looks like in practice.
Sam Mueller presented a study on treatment errors during discrete trial instruction. That is a structured teaching format with clear trials. The team collected data on both learner responses and therapist responses. Learner data included correct, prompted, incorrect, and no response. Therapist data included steps like securing attention and delivering a reinforcer.
The IOA, um, was 93%.
From the talk — Sam Mueller
Dr. Allison Bowhers-Peutin shared a study on training BCBA supervisors in skills like goal setting, feedback, and prompting. Her team used total count IOA, which compares the totals two observers recorded.
Total count IOA was collected for experiment one, and the average agreement range... between 89 to 98% for all the supervisory skills.
From the talk — Dr. Allison Bowhers-Peutin
In her second experiment, total count IOA ranged from 88 to 91 percent. Both studies cleared the 80 percent bar. Notice too that IOA was used on the behavior of adult staff and supervisors, not only on learners. Any behavior you measure can be checked with a second observer.
Where experts disagree#
All the speakers value IOA. The real split is about how strict the rule should be.
Sean Yocum takes the firmest line. He calls IOA "non-negotiable." He says that a graph with zero IOA and end-of-session guessing breaks the behavioral dimension of ABA before treatment even comes up. In his view, there is no trustworthy graph without IOA.
Matt Harrington agrees that IOA should be ruled out early. But he adds a twist. He says parsimony also means ruling things out conceptually, not only by collecting new data. His example is self-injury that is very clear, has no hidden parts, and happens four or five times an hour. A technician with basic skills and a clicker would likely get high exact IOA on that. So a supervisor may reasonably reason past IOA for such a behavior, instead of always pausing to collect it.
Amber Valentino shares a real exception from her own work. Her team taught a very private and personal skill. The parents ran the sessions, and the definitions were simple.
This is the only study I've ever had published. It didn't have IOA, which I know sounds a little funny to you, but it was a sensitive thing. We were teaching something very private and personal, and we just couldn't justify having another person in there or video recording.
From the talk — Amber Valentino
Valentino called it a very unique exception and said she does not expect it to happen again in her career. Stephanie Peterson, on the same panel, still pushes hard for routine IOA. So the debate is not whether IOA matters. It is whether dignity, privacy, or a very clear behavior can ever justify going without it.
What the research says#
Published studies add useful detail on how IOA is reported and how much is enough.
Essig and colleagues (2022) reviewed two major behavior-analytic journals from 2017 to 2021. About 96.4 percent of articles reported IOA for participant behavior. But only 17.7 percent of those reporting treatment delivery data also checked IOA on that data. The authors call this gap unfortunate and ask journal reviewers to weigh it.
Hausman and colleagues (2022) reanalyzed functional analysis data at different cutoffs for how many sessions had IOA. With highly trained observers in a structured clinic, the cutoffs made no significant difference. IOA did shift with the overall rate of behavior. The authors suggest the consistency of data may matter more than the number of sessions checked.
Anckner and Jacobs (2024) had untrained visitors use a free smartphone tool to record wolf behavior. All reached at least 80 percent IOA. But kappa, a stricter agreement score, fell below 0.6 for some visitors tracking two behaviors at once. Simple rules and clear target behaviors helped.
Sherman and colleagues (2025) scored trial-by-trial IOA, meaning they checked agreement on every single trial, from video. They covered 32 percent of sessions in a staff training study and reached 99 percent agreement.
FAQ#
What is a good IOA score? Sean Yocum cites the Horner and colleagues 2005 standard. IOA should be at least 80 percent, collected in at least 20 percent of sessions, in every condition. If IOA falls below 80 percent, Yocum says to stop reading the graph and fix the definition or the measurement system first. The Hausman and colleagues (2022) study suggests the number of sessions checked may matter less than steady data. This held at least with well-trained observers.
Which IOA formula should I use? Dr. Celia Heyman says to start with how the data were collected. Count data come from event recording, which narrows your options. Timing data and time sampling data each have their own formulas. For time sampling, interval-by-interval is the most common choice in published research.
Can I collect IOA without a second person in the room? Yes. Several studies recorded sessions on video and scored IOA later. Mellanie Page also suggests having technicians score IOA while watching their own recorded sessions with a supervisor. To go deeper, see Trustworthy Indicators and Believability of Data for Dr. Heyman's full walkthrough of IOA methods.
Turn this topic into a CEU
You just studied this. Now get credit for it.
Watch No Man's Land: Is Your Data Real? Clock Time Without IOA and Procedural Integrity Is Unfalsifiable with Sean Yocum and earn 1 free BCBA CEU. Detailed certificate, delivered the moment you finish.
Keep going with a free learning path
Supervision, Done Right
Four supervision CEUs that build on each other. What reflective supervision is, how to build a supervision system, how to assess supervisee competence instead of counting hours, and how supervision affects provider outcomes. Each course is free and issues its own certificate.




