Trustworthy Indicators and Believability of Data

This course is designed to help learners distinguish among three essential indicators of trustworthy data: validity, accuracy, and reliability. It is particularly beneficial for students who struggle to determine which interobserver agreement (IOA) methods are most appropriate for evaluating the believability of behavioral data. Dr. Heyman takes a conceptual approach to teaching IOA, emphasizing that rote memorization of formulas is unnecessary when learners truly understand the purpose of measurement and the dimensions of behavior being assessed. By focusing on the why behind each method rather than simply the how, this course helps learners develop a deeper understanding of IOA and the confidence to select and apply appropriate methods across different measurement scenarios.

Trustworthy data is fundamental to effective clinical decision-making in applied behavior analysis (ABA). However, the concepts of validity, accuracy, and reliability are often misunderstood or used interchangeably, potentially compromising the interpretation of behavioral data. This presentation examines the distinguishing characteristics of these three indicators of data trustworthiness and their implications for measurement and clinical decision-making (Cooper et al., 2020). Particular attention is given to interobserver agreement (IOA), its purpose in evaluating the consistency of behavioral measurement, and its limitations as an indicator of data accuracy and validity. Participants will explore the conceptual rationale underlying commonly used IOA methods, including total count, mean count-per-interval, exact count-per-interval, trial-by-trial, interval-based, duration, and latency agreement. Rather than emphasizing rote memorization of formulas, this talk highlights how the selection of IOA methods should be guided by the dimensions of behavior being measured, the characteristics of the data, and the potential for different calculations to overestimate or underestimate agreement (Cooper et al., 2020; Hausman et al., 2022; Mudford et al., 2009). Through practical examples, participants will develop a deeper understanding of how and why different IOA methods are applied, strengthening their ability to evaluate measurement quality, interpret agreement scores, and make informed, data-based decisions. References: Cooper, J. O., Heron, T. E., & Heward, W. L. (2020). Applied behavior analysis (3rd ed.). Pearson. Hausman, N. L., Javed, N., Bednar, M. K., Guell, M., Schaller, E., Nevill, R. E., & Kahng, S. (2022). Interobserver agreement: A preliminary investigation into how much is enough? Journal of Applied Behavior Analysis, 55(2), 357–368. https://doi.org/10.1002/jaba.811 Mudford, O. C., Taylor, S. A., & Martin, N. T. (2009). Continuous recording and interobserver agreement algorithms reported in the Journal of Applied Behavior Analysis (1995–2005). Journal of Applied Behavior Analysis, 42(1), 165–169. https://doi.org/10.1901/jaba.2009.42-165

1 PDU · 50 min

1. 1Upon completion of this presentation, attendees will be able to: 1. Differentiate among the concepts of validity, accuracy, and reliability as essential indicators of trustworthy behavioral data. 2. Select appropriate interobserver agreement (IOA) methods based on the dimensions of behavior being measured, type of data, and the rationale for evaluating data believability. 3. Explain the conceptual foundations and purposes of various IOA methods, demonstrating an understanding that extends beyond rote memorization of formulas. 4. Distinguish the different behavioral dimensions, measurement procedures, and data products.