Cognitive Science Program

Disentangling prevalence induced biases in medical image decision-making (2021)
Jennifer S. Trueblood, Quentin Eichbaum, Adam C. Seegmiller, Charles Stratton, Payton O'Daniels and William R.Holmes
Cognition, 212

Abstract

Many important real-world decision tasks involve the detection of rarely occurring targets (e.g., weapons in luggage, potentially cancerous abnormalities in radiographs). Over the past decade, it has been repeatedly demonstrated that extreme prevalence (both high and low) leads to an increase in errors. While this “prevalence effect” is well established, the cognitive and/or perceptual mechanisms responsible for it are not. One reason for this is that the most common tool for analyzing prevalence effects, Signal Detection Theory, cannot distinguish between different biases that might be present. Through an application to pathology image-based decision-making, we illustrate that an evidence accumulation modeling framework can be used to disentangle different types of biases. Importantly, our results show that prevalence influences both response expectancy and stimulus evaluation biases, with novices (students, N = 96) showing a more pronounced response expectancy bias and experts (medical laboratory professionals, N = 19) showing a more pronounced stimulus evaluation bias.

A Bayesian approach to parameter inference for a wide class of binary evidence accumulation models (2022)
Matthew Murrow and William R Holmes

Abstract

Many decision-making theories are encoded in a class of processes known as evidence accumulation models (EAM). These assume that noisy evidence stochastically accumulates until a set threshold is reached, triggering a decision. One of the most successful and widely used of this class is the drift-diffusion model (DDM). The DDM however is limited in scope and does not account for processes such as evidence leakage, changes of evidence, or time varying caution. More complex EAMs can encode a wider array of hypotheses, but are currently limited by the computational challenges. In this work, we develop the python package PyBEAM (Bayesian Evidence Accumulation Models) to fill this gap. Toward this end, we develop a general probabilistic framework for predicting the choice and response time distributions for a general class of binary decision models. In addition, we have heavily computationally optimized this modeling process and integrated it with PyMC3, a widely used python package for Bayesian parameter estimation. This 1) substantially expands the class of EAM models to which Bayesian methods can be applied, 2) reduces the computational time to do so, and 3) lowers the entry fee for working with these models. Here we demonstrate the concepts behind this methodology, its application to parameter recovery for a variety of models, and apply it to a recently published data set to demonstrate its practical use.

Urgency, leakage, and the relative nature of information processing in decision-making (2021)
Jennifer S. Trueblood, Andrew Heathcote, Nathan J. Evans and William R. Holmes
Psychological Review, 128 (1), 160-186

Abstract

Over the last decade, there has been a robust debate in decision neuroscience and psychology about what mechanism governs the time course of decision-making. Historically, the most prominent hypothesis is that neural architectures accumulate information over time until some threshold is met, the so-called Evidence Accumulation hypothesis. However, most applications of this theory rely on simplifying assumptions, belying a number of potential complexities. Is changing stimulus information perceived and processed in an independent manner or is there a relative component? Does urgency play a role? What about evidence leakage? Although the latter questions have been the subject of recent investigations, most studies to date have been piecemeal in nature, addressing one aspect of the decision process or another. Here we develop a modeling framework, an extension of the Urgency Gating Model, in conjunction with a changing information experimental paradigm to simultaneously probe these aspects of the decision process. Using state-of-the-art Bayesian methods to perform parameter-based inference, we find that (a) information processing is relative with early information influencing the perception of late information, (b) time varying urgency and evidence accumulation are of roughly equal strength in the decision process, and (c) leakage is present with a time scale of ∼200–250 ms. We also show that these effects can only be identified in a changing information paradigm. To our knowledge, this is the first comprehensive study to utilize a changing information paradigm to jointly and quantitatively estimate the temporal dynamics of human decision-making.

A Joint Deep Neural Network and Evidence Accumulation Modeling Approach to Human Decision-Making with Naturalistic Images (2020)
William R. Holmes, Payton O'Daniels and Jennifer S. Trueblood
Computational Brain and Behavior, 3 1-12

Abstract

Evidence accumulation models (EAM) have proven to be an invaluable tool in probing the dynamical properties of decisions over recent decades. However, much of this literature has studied decisions utilizing simple stimuli where the experimenter has perfect knowledge and control over stimulus properties. Here, we develop and test a new method for studying decisions involving naturalistic stimuli (medical images in this case) where the experimenter has neither perfect knowledge nor control of the stimuli properties. The central challenge in studying such decisions is to extract useful representations of images that can be associated with accumulation or drift rates in EAMs. Here, we couple a deep convolutional neural network (CNN) with the diffusion decision model (DDM) to study how expert pathologists and novices make decisions involving the classification of digital images of blood cells as either normal (non-blast) or cancerous (blast). In our approach, the CNN is the basis of a function that translates each image into a drift rate for use in the DDM. Results of fitting the joint CNN-DDM model to choice and response time data demonstrates that (1) both novices and experts demonstrated substantial speed accuracy tradeoffs, (2) both were susceptible to biases introduced by the presentation of pre-stimulus probabilistic cues, and (3) experts were more adept at extracting useful information from images than novices. These results demonstrate that this is a fruitful approach to studying decisions involving complex stimuli that will open new avenues for studying questions not possible with existing methods. Furthermore, this approach is technically feasible and has the potential to be translated into other domains of decision-making research.

A practical guide to the Probability Density Approximation (PDA) with improved implementation and error characterization (2015)
William R. Holmes
Journal of Mathematical Pyschology, 68-69 13-24

Abstract

A critical task in modeling is to determine how well the theoretical assumptions encoded in a model account for observations. Bayesian methods are an ideal framework for doing just this. Existing approximate Bayesian computation (ABC) methods however rely on often insufficient “summary statistics”. Here, I present and analyze a highly efficient extension of the recently proposed (Turner and Sederberg 2014) Probability Density Approximation (PDA) method, which circumvents this insufficiency. This method combines Markov Chain Monte Carlo simulation with tools from non-parametric statistics to improve upon existing ABC methods. The primary contributions of this article are: (1) A more efficient implementation of this method that substantially improves computational performance is described. (2) Theoretical results describing the influence of methodological approximation errors on posterior estimation are discussed. In particular, while this method is highly accurate, even small errors have a strong influence on model comparisons when using standard statistical approaches (such as deviance information criterion). (3) An augmentation of the standard PDA procedure, termed “resampled PDA”, that reduces the negative influence of approximation errors on performance and accuracy, is presented. (4) A number of examples of varying complexity are presented along with supplementary code for their implementation.