Generate: Learner Centered Skill Assessments

Why Skill Acquisition Assessment Fell Behind

Our field has sharp tools for problem behavior. Skill acquisition (teaching a person new skills) has not kept up. Malady started with the subject matter itself. It is the interaction between a person and their world. The basic unit is the operant (a response inside its context). He is not a fan of the ABC metaphor. The B in the middle pulls your eye toward action alone. The science then turns into a practice about action, not interaction.

He prefers a construct from J.R. Cantor's interbehaviorism (a whole event view of behavior). It puts the organism on one side and stimulus objects on the other. You work inward from response to response function, and from object to stimulus function. Malady likes it because it shows where the person sits inside the event. Stimulus response stimulus assumes a person is there. It never points you to them. He also keeps history of reinforcement, genetic endowment, and current environment in view.

In 2012 he nearly left the field. A mentor pulled him into volunteer work with adults with disabilities. He visited a day habilitation program (a daily adult support program) with 110 learners aged 18 to 65. Their plans were skill focused. Nobody was moving. A closer look showed the average person had carried the same skill plan for five years. Needs in that room were wide. Some people needed help to maintain basic biological functioning. Others lived in their own apartments and wanted places to socialize. He went looking for a behavioral assessment that spanned that range and could not find one.

And it was really clear that people were working on these habilitation plans that were really skill focused and skill oriented, but they weren't making any progress on them.

From the talk — Mark Malady

The Gap Between the Evidence and What We Use

Malady walked through a 2025 review by Banda and Hart. It pulls together data that Padea looked at in 2020 and 2021. The table lists each instrument, its age range, type, domains, target skills, and time to administer. It also lists reliability (do the scores repeat) and validity (does it measure what it claims). The last column is reported use in practice.

The scores are humbling. The VB-MAPP does not land in the strong or very strong band. It sits in emerging. The ABLLS is in the same place. Tools built outside behavior analysis often score better. The EFL from McGreevy et al. and the ABLA from Kerr et al. in 1977 both hold up. The ABLA has really strong support. It is also a basic tool, aimed at discrimination skills.

Then flip to use. Padella et al. surveyed about 1,400 behavior analysts. Four instruments carried most of the market. Another 115 named tools never crossed the ten percent mark. The VB-MAPP led, with the ABLLS-R behind it. So the weaker evidence is getting the heavier use. Age is part of the story. Many of these tools are more than ten years old, and funders in the American autism market pay for them. Sometimes they name the tool outright.

Work is underway to close that gap. A paper published this year ran a psychometric evaluation (a study of a test's math) of the ABLLS-R and the AFLS. It drew on a vendor data set of more than 41,000 unique people. Malady's read is fair to the field. Practice ran out ahead of validation because clinicians needed tools right then.

We are not in alignment on our current tools for skill-based assessment being empirically validated as instruments in and of themselves.

From the talk — Mark Malady

What a Skill Assessment Should Actually Do

Malady put six jobs on the screen. Give an accurate account of what the learner can do. Help the learner build a life that means something to them. Show real routes for growth. Name strengths, not just holes. Flag where continued support belongs. Capture the learner's own vision for their future. That last one is easy to skip and hard to replace.

One design rule shapes the rest: no one can master all skills. He described adult team meetings where eight or nine staff sit around one person. Everyone at the table would score 100 percent on the skill assessment in front of them. Only the focal person would not. Time spent on one area is time not spent elsewhere. A tool that shows mastery for every professional in the room is not measuring much.

The support line matters just as much. Behavioral instruments often miss it. Some gaps should become teaching targets. Others should become programmed supports the person carries across a lifespan. Both paths can lead to the life the person wants. An assessment that only produces teaching targets will push you toward one answer every time.

So if our instruments are showing mastery for eight out of nine people, there's a problem in the instrument and the breadth of the skills that are being covered or how deep the measurement criteria goes for that skill.

From the talk — Mark Malady

Four Features to Check in Any Instrument

The Council of Autism Service Providers and the Association of Professional Behavior Analysts published joint guidance for assessing autistic learners. It names four features worth knowing for every tool on your shelf. Direct versus indirect: are you measuring the behavior, or asking someone about it? Standardized versus not: how much can administration vary, and what support exists for that style?

Then comes norm referenced (scored against same age peers) versus criterion referenced (scored against a defined standard). That is really a question about where the targets came from. Norm referencing gets complicated with autistic learners. Comparing a person to peers without the diagnosis drags in questions about indistinguishability (looking no different from peers). The listed limitation is blunt. Scores sit next to same age peers with no room for the person's own circumstances.

The fourth feature is single modal versus multimodal (using several tools together). The guidance says no single instrument has met the need across learners. Build an assessment package for the person instead. Criterion referenced tools carry their own listed misapplication: teaching to the test. If you go that way, put a check in your quality process so programming does not collapse into the item list.

So a big recommendation of the guidelines is that to this point, they have not seen single instruments that can fit the need across learners.

From the talk — Mark Malady

No Cold Probes, Smarter Entry Points, and Assent

Assessment comes first in your relationship with a learner. That is exactly why Malady dropped the cold probe (a test trial without teaching). The logic of a cold probe is that teaching would spoil the data. So your first hour together becomes ask, fail, next item, with no feedback and no reinforcement. Some teams paper over it with a pairing session first. The assessment session still feels cold.

His answer is to teach during the assessment and control for it with measurement. Good measurement can tell you whether a skill is in acquisition, in transition, or at mastery. The tool also offers top down and bottom up entry. A cluster holds several related skills, with a low entry point and a high one. What you already know about the learner decides where you start. Handing someone items far past their reach is a fast way to lose them.

Other design choices follow the same logic. Measurement systems are matched to each pinpoint (a clearly defined skill target), so rate, latency, duration, and rating scales all appear. Assent (the learner agreeing to take part) is built in through a pre-teach step before each pinpoint. Accommodations ride on learning channels from precision teaching (rate based teaching and charting). Timing windows can shrink for learners with low tolerance for instruction, without losing the measure. Malady reported over 400 completed learners, ages 2 to 68, across four outside organizations, with a case comparison study and test-retest data now in review.

Well, it creates a really awkward initial relationship and it can be perceived as punitive by some learners.

From the talk — Mark Malady

Your Graph Teaches People What to Want

Most of us know the standard displays: stacked bar grids, single bars, and triangles that climb with age. Malady's complaint is about what they ask of the viewer. The goal they imply is filling every box by a certain age. For a group of learners who differ from each other by definition, that lands very close to normalization. Families read the graph. Staff read the graph. The graph sets the goal.

Generate uses a radial graph (a spoke shaped data display) instead. The outer edge is the terminal target. A second path is laid on top: the comparison criteria for the milestone that learner picked. Skills can now read as over skilled or under skilled against that specific goal, not against an age.

He showed what that buys you. One adult in 24 hour care wanted to stay home alone for 10 to 15 minutes. The team kept adding prerequisites. The graph compared that person to people who already stay home alone in similar settings. The learner was far past the line. The question changed from what must they learn to what the support team needs to feel comfortable. That person now stays home for periods up to five hours. Other profiles skew: strong in some domains, thin in others. He noted that young adults often look thin on close relationship skills, which may call for access to places, not specialty teaching.

And in each of these, the issue that I have with them is that the goal is to fill up all of the boxes.

From the talk — Mark Malady

Audit Your Own Assessment Package

You may not control the instruments you use. Funders and employers often decide that. Malady's point is that plenty is still in your hands, and none of it breaks procedural fidelity (running a tool as written). Start with a simple audit. Take your last 10 learners. Chart the assessment package each one received. If you use three tools on every learner, you have a multimodal label without multimodal practice.

Next, make selection a real step. Use a semi-structured interview (a guided but flexible interview) at intake to decide what to explore. Note which areas the person is likely strong in already. Note which are not relevant to where they are headed. Write down the choice and the reason. Then you can check whether two assessors would select the same package.

You can also overlay a goal on a standard tool. He showed a learner who wanted ballet classes. Sample the VB-MAPP for the skills that ballet actually requires and highlight those. The tool stays intact. The programming narrows to something the person cares about. For criteria, find competent performers who resemble your learner, benchmark their skills, and set the bar from the lowest, medium, or average performer. Test the criteria on two more competent performers before you use them. Finally, give your report a section for accommodations and assistance, so skill building is not the only answer on the page.

If everybody, if you have one color, that should be a red flag.

From the talk — Mark Malady

Funders, Legacy Tools, and the Research You Could Run

An attendee asked the practical version of all this. How do you run a newer instrument when the payer wants the legacy one? Malady's short term move is to ask for both. Bring it to your rep or your plan reviewer. Ask whether you can run the required tool plus the newer one for a defined set of cases. A named person on the other end of the phone changes that conversation.

Transparency helps too. Share the pie chart of what your learners actually receive. Show the package you wanted to run next to the package you felt forced to run. That gives the payer something concrete to react to. He also pointed at a policy detail most clinicians never see.

The last ask was for research. The evidence base is thin because few people are studying these tools in practice. You do not need a lab to help. Study the assessment process you already run, with and without a procedural change. Questions about administration time, assessor agreement, and learner experience are all open. He offered to brainstorm with anyone who wants to start.

And then the other thing is most insurers have written policies that they don't make available that as long as there's one validity and one reliability study that they have to accept it.

From the talk — Mark Malady