Quality Without Measurement is Just Marketing: A Critical Analysis of Quality Service Delivery

Why Mission Statements Are Not Quality Measures

Open almost any ABA agency website. You will find the word quality. Byra showed slides of real taglines with the agency names blocked out. Highest quality therapy. The gold standard of care. Excellence in everything we do. She told the audience to go check their own site that day. The promise shows up at least once, often many times.

Then she asked the harder question. If you lead with quality, do you track goal mastery rates (how often goals get met)? If you claim best practices, do you supervise staff twice a month? If you make data-driven decisions, do you review that data every 10 sessions? When the answer is no, the slogan is just a claim.

Hollow claims get noticed outside the field. She pointed to a recent New York Times story about clinics billing around children's naps. One staff member reportedly held a nap to seven minutes. Seven minutes is under half of a 15-minute CPT code (a billing unit of service time). That left the remaining eight minutes still billable. She also pointed to her home state. A Medicaid plan in Arizona pulled contracts with two providers over quality concerns. About a thousand children had to find services somewhere else.

Quality is how we deliver on our promises to our families.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA

Fast Growth, Thin Safeguards: The Numbers Behind the Problem

The field grew fast. About 46% of certified BCBAs have less than five years in the role. There are roughly 695 training programs, and their rigor varies a lot. Byra graduated when the field held 3,500 BCBAs. The count is now past 80,000.

Demand still outruns supply. She cited job postings for BCBAs above 100,000 in a recent year, against about 75,000 certificants at the time. Autism prevalence climbed too. It went from about 1 in 125 to about 1 in 36, by her last check of CDC numbers. More access is good news. The open question is what fills the gap.

Leadership timelines moved too. A colleague's poll asked when BCBAs took their first clinical or center director job. Twelve percent were certified less than a year. Thirty-nine percent were one to three years in. Only 10% waited seven years. Byra compared new certification to a new driver's license. You are legal to drive. That does not mean you should take six lanes of Phoenix traffic.

Supervision rules add little protection. A new BCBA can supervise right away with monthly check-ins. The standards say plainly that the supervisor is not responsible for the supervisee. And the advice to find a high-quality supervisor has no test behind it. How do you know they are high quality? They told you so.

But growth without quality standards creates risk.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA

Why "Insurance Approved It" Is Not a Quality Metric

A BCBA once told Byra that a treatment plan must be fine because the funder accepted it. She turned that into a poll question. Most respondents said approval does not equal clinical quality. A handful still called approval a solid quality check.

Her counter is a term worth knowing: administrative authorization (approved without a real clinical review). When a plan reaches a funder, a clock starts in calendar days, not business days. A reviewer may get 20 plans a day and finish 15. The backlog grows all week. When the days run out, plans get waved through.

She put that share near 20%, based on funders she knows. No coaching. No partial denial. No real read. The youngest clients are first in that line. If one plan gets waved through, it is the four-year-old, not the 14-year-old. So a high approval rate is a weak quality signal, especially on a preschool caseload. She tied it to Goodhart's law, the idea that a measure stops working once it becomes the target.

So approved does not mean effective, authorized doesn't mean equal quality, and then billed doesn't mean it's beneficial.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA

What the Existing Quality Checklists Miss

Quality tools already exist, and Byra credited them. Dr. Mark Sundberg published a quick assessment and a long one in 2015, posted on avbpress.com. She called it a reasonable place to start. It leans toward school readiness rather than early intensive behavioral intervention (early, high-hour ABA for young children).

She also pointed to a checklist of about 30 quality indicators from Dr. Amanda Kelly. Dr. Stephanie Peterson and Becky Eldridge built on Sundberg's work with another indicator set. Her critique is about measurement, not intent. An item like heavy focus on social skills leaves too much room. What counts as heavy? If social skills get a heavy focus and language does too, are we now at 200%?

The wider system has the same weakness. Most quality assurance is reactive. Agencies look back at last month's supervision numbers and self-report the miss. Regulators and funders do not have the bandwidth to read everything that crosses a desk. Large organizations hold piles of aggregate outcome data that nobody shares. So compliance gets the attention and outcomes wait.

Not to say that these tools aren't helpful and don't give some good frameworks.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA

The Five Domains and the 28 Yes-or-No Items

Byra's answer is a 28-item assessment with yes or no answers only. No partial credit and no half points. You either do the thing consistently or you do not. The items spread across five domains, four to six per domain. Gaps come back with action statements. Results also compare your agency to others of similar size, with no names shown.

She recommends more than one rater. Have brand-new BCBAs take it. Have graduate trainees take it. Have clinic directors and regional leaders take it too. Leadership often believes a practice is routine. The BCBAs running the cases often say it is not. That mismatch is useful data on its own.

The five domains are daily sessions, treatment fidelity (running the plan as written), data analysis, caregiver guidance, and supervision. The items are concrete. Session areas stay organized and materials stay ready. Preference assessments (quick checks of what a learner wants) happen daily. Fidelity checks on skill goals happen at least every two weeks. New goals get introduced with BST, or behavior skills training (instruct, model, practice, feedback), with every technician on the case. Skill data gets reviewed every 10 sessions. Caregiver guidance happens at least monthly, with satisfaction surveys every six months. Supervision happens twice a month, with a structured plan in hand.

She was blunt about that last one. She described supervisors rolling down the hall with a laptop. They told a technician to keep doing what they were already doing. One supervisor fell asleep during a session. Later, that same supervisor looked up marinade recipes for a weekend barbecue. Then the supervisor asked the technician which goal should come next.

Let's make sure we are not doing any of that.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA

The 50 Learning Opportunities Per Hour Benchmark

One item draws the most pushback: a trial count of at least 50 learning opportunities per hour on average. Byra holds the line on it. ESDM (Early Start Denver Model, a play-based teaching approach) points to about 60 opportunities per hour. Discrete trial teaching (short, structured teaching trials) runs much higher, into the 200 range. So 50 sits below both numbers. It is a floor, not a stretch goal.

The count includes more than brand-new targets. Generalization and mastery items count too. The point is active engagement, not novelty. Of roughly 165 people who had taken the assessment at the time, this item usually came back as a no.

Her argument is arithmetic. The VB-MAPP transition assessment (a check on readiness to leave services) asks how many learning opportunities a learner needs. A kindergarten class might cover the letter B in one day and move to C. Many children in ABA need hundreds of repetitions instead. If a target needs 200 opportunities and a session delivers 10 per hour, mastery is years away. Multiply that by every open target in your data system. That is one reason children stay in care so long.

Technicians often welcome the change. When trial counts went up during clinic-wide projects, staff told her the day went faster. They reported less challenging behavior, because everyone stayed engaged and reinforcement rates went up.

We should be actively working ourselves out of a job.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA

The Five-Step Course Correction: Assess, Prioritize, Implement, Monitor, Reassess

Step one is an honest baseline. Answer yes only when the practice happens all the time. Average scores run near 60%. Nobody has scored 100 yet. One agency came close, and she knows that team well enough to trust the answers. Gaps are not a verdict on your work.

Step two is restraint. Pick one or two domains, not five. She asks four questions. Where are the biggest gaps? Where is the biggest impact on client outcomes? Which change spills over into other domains? What does your team most need to build? A narrow focus raises your odds.

Step three is real implementation. Do not send an email and call it training. Use BST with your clinicians the way you would with technicians. Write concrete, measurable changes instead of vague intentions. Document what you changed and why.

Step four is monitoring. Set a timeline and hold monthly check-ins on the change you picked. Use simple metrics. Name who tracks the number and who hears about it. Celebrate the small wins and clear barriers as they appear. Step five is retaking the assessment in three to six months, comparing against baseline, then choosing the next domain.

One change done well beats many done halfway.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA

Marathon Pace, Staff Retention, and Working Without Backup

New agencies get the easier path. Define your quality indicators before you hire the first behavior technician. Build the fidelity monitoring schedule before the first client starts. Take a baseline before the first month ends. Review data against your benchmarks before the first quarter ends. Culture gets built in month one, not year three.

For everyone else, pace is the whole game. Byra argued for a marathon mindset over a sprint. A two-pack-a-day smoker does not quit cold turkey. A non-runner does not run 26 miles tomorrow. Staff will quit before a full overhaul finishes. Small changes, repeated over time, produce large effects.

Quality also keeps people. Byra ran a LinkedIn survey of roughly 200 BCBAs. In it, 48% said they left a position over the quality of services they could deliver. Retained staff produce better client outcomes. Better outcomes build reputation and referrals. A strong reputation then attracts stronger clinicians.

She closed with advice for the lone holdout in a clinic. Explain the why to technicians: here is my expectation, here is the impact. That moves behavior better than citing company policy. Look for allies rather than quitting tomorrow, since leaving does not improve the agency and the seat gets refilled. She drew a line at egregious problems like fraudulent billing. And job seekers can ask an interviewer to fill out the assessment with them. A refusal answers the question.

I will say it's harder when you are standing alone.

From the talk — Kristen Byra, Ph.D., BCBA-D, LBA