Decision Making Under the Gambler’s Fallacy: Evidence from Asylum Judges, Loan Officers, and Baseball Umpires *

Abstract

I. I ntroduction

Does the sequencing of decisions matter for decision making? Controlling for the quality and merits of a case, we find that the sequence of past decisions matters for the current decision, that is, decision makers exhibit negatively autocorrelated decision making. Using three independent and high-stakes field settings (refugee asylum court decisions in the United States, loan application reviews from a field experiment by Cole, Kanz, and Klapper (2015 ), and Major League Baseball home plate umpire calls on pitches) we show consistent evidence of negatively autocorrelated decision making, despite controlling for case quality, which leads to decision reversals and errors.

In each of the three high-stakes settings, we show that the ordering of case quality is likely to be conditionally random. However, a significant percentage of decisions, more than 5% in some samples, are reversed or erroneous due to negative autocorrelation induced by the behavior of decision makers. The three settings provide independent evidence of negatively autocorrelated decision making across a wide variety of contexts for decision makers in their primary occupations, and across a very large sample size of decisions in some cases. Each field setting offers unique advantages and limitations in terms of data analysis that taken together portray a compelling picture of negatively autocorrelated decision making arising from belief biases.

First, we test whether U.S. judges in refugee asylum cases are more likely to deny (grant) asylum after granting (denying) asylum to the previous applicant. The asylum courts setting offers administrative data on high-frequency judicial decisions with very high stakes for the asylum applicants—a judge’s decision determines whether refugees seeking asylum will be deported from the United States. The setting is also convenient because cases filed within each court (usually a city) are randomly assigned to judges within the court and judges decide on the queue of cases on a first-in-first-out basis. By controlling for the recent approval rates of other judges in the same court, we are able to control for time variation in court-level case quality to ensure that our findings are not generated spuriously by negative autocorrelation in underlying case quality. A limitation of the asylum court data, however, is that we cannot discern whether any individual decision is correct given the case’s merits. We estimate that judges are up to 3.3 percentage points more likely to reject the current case if they approved the previous case. This translates into 2% of decisions being reversed purely due to the sequencing of past decisions, all else being equal. This effect is also stronger following a longer sequence of decisions in the same direction, when judges have “moderate” grant rates close to 50% (calculated excluding the current decision), and when the current and previous cases share similar characteristics or occur close in time (which is suggestive of coarse thinking as in Mullainathan, Schwartzstein, and Shleifer 2008 ). We also find that a judge’s experience mitigates the negative autocorrelation in decision making.

Second, we test whether loan officers are more likely to deny a loan application after approving the previous application by using data from a loan officer field experiment conducted in India by Cole, Kanz, and Klapper (2015) . The field experiment offers controlled conditions in which the order of loan files, and hence their quality, within each session is randomized by the experimenter. In addition, loan officers are randomly assigned to one of three incentive schemes, allowing us to test whether strong pay-for-performance incentives reduce the bias in decision making. The setting is also convenient in that we can observe true loan quality, so we can discern loan officer mistakes. Another advantage of the field experiment setting is that payoffs only depend on accuracy. Loan officers in the experiment are told that their decisions do not affect actual loan origination and they do not face quotas. Therefore, any negative autocorrelation in decisions is unlikely to be driven by concerns about external perceptions, quotas, or by the desire to treat loan applicants in a certain fashion. We find that up to 9% of decisions are reversed due to negative autocorrelation in decision making under the flat incentive scheme among moderate decision makers. The effect is significantly weaker under the stronger incentive schemes and among less moderate decision makers. Across all incentive schemes, the negative autocorrelation is stronger following a streak of two decisions in the same direction. Education, age, experience, and a longer period of time spent reviewing the current loan application reduce the negative autocorrelation in decisions.

Third, we test whether Major League Baseball (MLB) umpires are more likely to call the current pitch a ball after calling the previous pitch a strike and vice versa. An advantage of the baseball umpire data is that it includes precise measures of the three-dimensional location of each pitch. Thus, while pitches may not be randomly ordered over time, we can control for each pitch’s true “quality” or location and measure whether mistakes in calls conditional on a pitch’s true location are negatively predicted by the previous call. We find that umpires are 1.5 percentage points less likely to call a pitch a strike if the previous pitch was called a strike, holding pitch location fixed. This effect more than doubles when the current pitch is close to the edge of the strike zone (meaning it is a less obvious call) and is also significantly larger following two previous calls in the same direction. Put differently, MLB umpires call the same pitches in the exact same location differently depending solely on the sequence of previous calls. We also show that any endogenous changes in pitch location over time are likely to be biases against our findings.

Altogether, we show that negatively autocorrelated decision making in three diverse settings is unrelated to the quality or merits of the cases considered and hence results in decision errors. We explore several potential explanations that could be consistent with negatively autocorrelated decision making, including belief biases such as the gambler’s fallacy and sequential contrast effects, and other explanations such as quotas, learning, and a desire to treat all parties fairly. We find that the evidence across all three settings is most consistent with the gambler’s fallacy and/or sequential contrast effects, and in several tests we are able to reject the other theories.

The “law of small numbers” and the “gambler’s fallacy” are both names for the well-documented tendency of people to overestimate the likelihood that a short sequence will resemble the general population ( Tversky and Kahneman 1971 , 1974 ; Rabin 2002 ; Rabin and Vayanos 2010 ) or underestimate the likelihood of streaks occurring by chance. For example, people often believe that a sequence of coin flips such as “HTHTH” is more likely to occur than “HHHHT” even though each sequence occurs with equal probability. Similarly, people may expect flips of a fair coin to generate high rates of alternation between heads and tails even though streaks of heads or tails often occur by chance. This misperception of random processes can lead to errors in predictions.

In our analysis of decision making under uncertainty, a decision maker who himself suffers from the gambler’s fallacy may similarly believe that streaks of good or bad quality cases are unlikely to occur by chance. Consequently, the decision maker may approach the next case with a prior belief that the case is likely to be positive if she deemed the previous case to be negative, and vice versa. Assuming that decisions made under uncertainty are at least partially influenced by the agent’s priors, these priors then lead to negatively autocorrelated decisions. Similarly, a decision maker who fully understands random processes may still engage in negatively autocorrelated decision making in an attempt to appear fair if she is being evaluated by others, for example promotion committees or voters who suffer from the gambler’s fallacy.

Our analysis differs from the existing literature on the gambler’s fallacy in several ways. First, most of the existing empirical literature examines behavior in gambling or laboratory settings (e.g., Bar-Hillel and Wagenaar 1991 ; Rapoport and Budescu 1992 ; Clotfelter and Cook 1993 ; Terrell 1994 ; Ayton and Fischer 2004 ; Croson and Sundali 2005 ; Asparouhova, Hertzel, and Lemmon 2009 ; Benjamin, Moore, and Rabin 2013 ; Suetens, Galbo-Jorgensen, and Tyran 2015 ) and does not test whether the gambler’s fallacy can bias high-stakes decision making in real-world or field settings such as those involving judges, loan officers, and professional baseball umpires. ¹

Second, our analysis differs from the existing literature because we focus on decisions. We define a decision as the outcome of an inference problem using both a prediction and investigation of the current case’s merits. In contrast, the existing literature on the gambler’s fallacy typically focuses on predictions or bets made by agents who do not also assess case merits. Our focus on decisions highlights how greater effort on the part of the decision maker or better availability of information regarding the merits of the current case can reduce errors in decisions even if the decision maker continues to suffer from the gambler’s fallacy when forming predictions. Our findings support this view across all three of our empirical settings.

Finally, we study the behavior of experienced decision makers making decisions in their primary occupations. In some settings, we have variation in incentives to be accurate and show that stronger incentives can reduce the influence of decision biases on decisions. In addition, and in contrast to the laboratory setting as well as other empirical settings studied in the literature, our decision makers see a large sample of cases—many hundreds for an asylum judge and tens of thousands or more for an umpire—which affords us very large samples of decisions.

Other potential alternative and perhaps complementary explanations appear less consistent with the data, though in some cases we cannot completely rule them out. One potential alternative explanation is that decision makers face quotas for the maximum number of affirmative decisions, which could induce negative autocorrelation in decisions since a previous affirmative decision implies that fewer affirmative decisions can be made in the future. However, in all three of our empirical settings, agents do not face explicit quotas or targets. For example, loan officers in the field experiment are only paid based upon accuracy and their decisions do not affect loan origination. Asylum judges are not subject to any explicit quotas or targets and neither are baseball umpires. Nevertheless, one may be concerned about self-imposed quotas or targets. We show that such self-imposed quotas are unlikely to explain our results by contrasting the fraction of recent decisions in one direction with the sequence of such decisions. In a quotas model, the only thing that should matter is the fraction of affirmative decisions. We find, however, that agents negatively react to extreme recency, holding the fraction of recent affirmative decisions constant. That is, if one of the last N decisions was decided in the affirmative, it matters whether the affirmative decision occurred most recently or further back in time. This behavior is consistent with the sequencing of decisions mattering, and is largely inconsistent with self-imposed quotas unless the decision maker also has very limited memory and cannot remember beyond the most recent decision.

Another related potential explanation is a learning model where decision makers do not necessarily face quotas, but they believe that the correct fraction of affirmative decisions should be some level. The decision makers are unsure of where to set the quality bar to achieve that target rate and therefore learn over time, which could lead to negative autocorrelation in decisions. However, baseball umpires should not have a target rate and instead have a quality bar (the official strike zone) that is set for them. Further, decision makers in all of our settings are highly experienced and should therefore have a standard of quality calibrated from many years of experience. As a consequence, they are probably not learning much from their most recent decision or sequence of decisions. In addition, a learning model would not predict a strong negative reaction to the most recent decision either, especially when we also control for their own recent history of decisions, which should be a better proxy for learning.

Another potential interpretation specific to the baseball setting is that umpires may have a preference to be equally “fair” to both teams. Such a desire is unlikely to drive behavior in the asylum judge and loan officers settings because the decision makers review sequences of independent cases that are not part of “teams.” However, a preference to be equally nice to two opposing teams in baseball may lead to negative autocorrelation of umpire calls if, after calling a marginal or difficult-to-call pitch a strike, the umpire chooses to “make it up” to the team at bat by calling the next pitch a ball. We show that such preferences are unlikely to drive our estimates for baseball umpires. Indeed, we find that the negative autocorrelation remains equally strong or stronger when the previous call was obvious (i.e., far from the strike zone boundary) and correct. In these cases, the umpire is less likely to feel guilt about making a particular call because the umpire probably could not have called the pitch any other way (e.g., he and everyone else knew it was the right call to make). Nevertheless, we find strong negative autocorrelation following these obvious and correct calls, suggesting that a desire to undo marginal calls or mistakes is not the sole driver of our results.

Finally, we investigate several potential explanations closely related to the gambler’s fallacy. Since these are empirically indistinguishable, we present them as possible variants of the same theme, though we argue that they may be less plausible in some of our settings. The first is sequential contrast effects (SCE), in which the decision maker’s perception of the quality of the current case is negatively biased by the quality of the previous case ( Pepitone and DiNubile 1976 ; Simonsohn and Loewenstein 2006 ; Simonsohn 2006 ). For example, Bhargava and Fisman (2014) find that speed dating subjects are more likely to reject the next candidate if the previous candidate was very attractive, and Hartzmark and Shue (2016) find that investors perceive today’s earnings news as less impressive if unrelated firms released good earnings news in the previous day. Theoretically, the gambler’s fallacy and SCE can predict the same patterns in decision outcomes. The distinction is mainly with regard to when the subject makes a quality assessment. Under the gambler’s fallacy, a subject who sees a high quality case will predict that the next case is likely to be lower in quality in a probabilistic sense even before seeing the next case, whereas SCE predicts the subject will make a relative comparison after seeing both cases. While the laboratory or prediction markets may be able to separate these two biases, they will be observationally equivalent when looking at only decision outcomes since we cannot observe what is inside a decision maker’s head. Complicating matters further, it may also be the case that the gambler’s fallacy affects the decision maker’s perception of quality, thus leading to a contrast effect. For example, a subject may believe that the next case is likely to be lower in quality after seeing a high-quality case, and this makes him perceive the next case as indeed being less attractive.

We present suggestive evidence that our results are more consistent with a simple gambler’s fallacy model than the SCE model. The SCE may be less likely to occur in the context of baseball because there is a well-defined quality metric (the regulated strike zone), although SCE may still bias perceptions of quality on the margin. In both the asylum court and loan approval settings, we find that decisions are unrelated to continuous quality measures of the previous case after we condition on the previous binary decision. This is consistent with a simple gambler’s fallacy model in which agents expect binary reversals, and less supportive of a SCE model in which agents should react negatively to the continuous quality of the previous case. However, our tests cannot fully reject SCE because we may measure the true quality of the previous case with error.

Another possibility is that the decision maker is rational but cares about the opinions of others, for example promotion committees or voters, who are fooled by randomness. In other words, it is the outside monitors who have the gambler’s fallacy and decision makers merely cater to it. These rational decision makers will choose to make negatively autocorrelated decisions to avoid the appearance of being too lenient or too harsh. While concerns about external perceptions could be an important driver of decisions, they are unlikely to drive the results in the context of loan approval, which is an experimental setting where monetary payouts depend only on accuracy (and loan officers know this) and the ordering of decisions is never reported to an outside party.

Lastly, a related explanation is that agents may prefer to alternate being “mean” and “nice” over short time horizons. This preference could, again, originate from the gambler’s fallacy. A decision maker who desires to be fair may over-infer that she is becoming too negative from a short sequence of “mean” decisions. However, a preference to alternate mean and nice is unlikely to drive behavior in the loan approval setting where loan officers in the experiment know that they do not affect real loan origination (so there is no sense of being mean or nice to loan applicants).

Overall, we show that belief biases possibly stemming from misperceptions of what constitutes a fair process can lead to decision reversals and errors. While we cannot completely distinguish between variants of the gambler’s fallacy and SCE, our evidence is unique to the literature on decision making biases in its breadth, particularly in terms of studying large samples of important decisions made as part of the decision maker’s primary occupation. We also find heterogeneity in the field data that may have useful policy implications. For example, we find that negative autocorrelation in decisions declines if the current and previous case considered are separated by a greater time delay, consistent with experimental results in Gold and Hester (2008) , in which the gambler’s fallacy diminishes in coin flip predictions if the coin is allowed to “rest.” We further find that education, experience, and strong incentives for accuracy can reduce biases in decisions. Finally, our research also contributes to the sizable body of psychology literature using vignette studies of small samples of judges that suggests unconscious heuristics (e.g., anchoring, status quo bias, availability) play a role in judicial decision making (e.g., Guthrie et al. 2000 ). In addition, our results contribute to the theoretical literature on decision making, for example, Bordalo, Gennaioli, and Shleifer (2014) , which models how judges can be biased by legally irrelevant information.

The rest of the article is organized as follows. Section II outlines our empirical framework and discusses how it relates to theory. Section III presents the results for asylum judges, Section IV presents results for the loan officer experiment, and Section V presents the MLB umpire results. Section VI discusses our findings in relation to various theories, including the gambler’s fallacy, while Section VII concludes.

II. E mpirical F ramework and T heory

We describe our empirical framework for testing autocorrelation in sequential decision making across the three empirical contexts and relate it to various theories of decision making.

II.A. Baseline Specification

In some empirical settings we can also determine whether any particular decision was a mistake. If we include a dummy for the correct decision as part of Controls , then any nonzero estimate of β₁ is evidence of mistakes. In other settings when we cannot definitively determine a mistake, we use β₁ to estimate the fraction of decisions that are reversed due to autocorrelated decision making. For example, in the case of negative autocorrelation bias (what we find in the data), the reversal rate is: $−2β1a(1−a)$ , where a represents the base rate of affirmative decisions in the data (see  Appendix A for details).

Even if the ordering of cases is random within each decision maker, we face the problem that our estimate of β₁ may be biased upward when it is estimated using panel data with heterogeneity across decision makers. The tendency of each decision maker to be positive could be a fixed individual characteristic or slowly changing over time. If we do not control for heterogeneity in the tendency to be positive across decision makers (and possibly within decision makers over time), that would lead to an upward bias for β₁ since the previous and current decision are both positively correlated with the decision maker’s unobserved tendency to be positive.

We control for decision maker heterogeneity in several ways. One simple method is to control for heterogeneity using decision maker fixed effects. However, decision maker fixed effects within a finite panel can lead to negative correlation between any two decisions by the same decision maker, which biases toward β₁ < 0. To remove this bias, we alternatively control for a moving average of the previous n decisions made by each decision maker, not including the current decision. A benefit of this specification is that it also tests whether or not the decision maker reacts more to the most recent decision, controlling for the average affirmative rate among a set of recent decisions. The drawback of using a moving average is that it may imprecisely measure the tendency of each decision maker to be positive due to small samples and hence be an inadequate control for heterogeneity. Thus, we also control for the decision maker’s average decision in all other settings other than the current decision. ³ In our baseline results we report estimates that control for individual heterogeneity using recent moving averages and leave-out-means because these methods do not bias toward β₁ < 0. In the Online Appendix , we show that the results are very similar with the inclusion of decision maker fixed effects, although point estimates tend to be more negative, as expected. Finally, we cluster standard errors by decision maker or decision maker×session as noted.

A second important reason we include control variables is that the sequence of cases considered is not necessarily randomly ordered within each decision maker. To attribute β₁ < 0 to decision biases, the underlying quality of the sequence of cases considered, conditional on the set of controls, should not itself be negatively autocorrelated. We discuss for each empirical setting why the sequences of cases appear to be conditionally random. ⁴

Because many of our regressions include fixed effects (e.g., nationality of asylum applicant), we estimate all specifications using the linear probability model, allowing for clustered standard errors, as suggested by Angrist and Pischke (2008) . However, we recognize there is debate in the econometrics literature concerning the relative merits of various binary dependent variable models. In the Online Appendix , we reestimate all baseline tables using logit and probit models and estimate similar marginal effects.

II.B. Streaks

The predictions under an SCE model are less obvious and depend on the specific assumptions of the model. For instance, if agents only contrast current case quality with the case that preceded it, then the decision in time t – 2 should not matter, so we would expect β₁ = β₂ < β₃ = 0. However, if agents contrast the current case with the previous case and, to a lesser degree, the case before that, a SCE model could deliver similar predictions to those of the gamblers’ fallacy model, implying β₁ < β₂ < β₃ < 0.

A quotas model, on the other hand, yields very different predictions. For quotas, β₁ should be the most negative, since two affirmative decisions in the past puts a more binding constraint on the quota limit than following only one affirmative decision. However, when the decision maker decided in the affirmative for only one out of the two most recent cases, it should not matter whether the affirmative decision was most recent, hence β₂ = β₃ . The learning model also does not predict β₂ < β₃ unless it is a particular form of learning where more weight is given to the most recent decision. We test these various predictions across each of our three settings.

III. A sylum J udges

Our first empirical setting is U.S. asylum court decisions.

III.A. Asylum Judges: Data Description and Institutional Context

The United States offers asylum to foreign nationals who can (i) prove that they have a well-founded fear of persecution in their own countries, and (ii) that their race, religion, nationality, political opinions, or membership in a particular social group is one central reason for the threatened persecution. Decisions to grant or deny asylum have potentially very high stakes for the asylum applicants. An applicant for asylum may reasonably fear imprisonment, torture, or death if forced to return to her home country. For a more detailed description of the asylum adjudication process in the United States, we refer the interested reader to Ramji-Nogales et al. (2007) .

We use administrative data on U.S. refugee asylum cases considered in immigration courts from 1985 to 2013. Judges in immigration courts hear two types of cases: affirmative cases in which the applicant seeks asylum on her own initiative, and defensive cases in which the applicant applies for asylum after being apprehended by the Department of Homeland Security (DHS). Defensive cases are referred directly to the immigration courts while affirmative cases pass a first round of review by asylum officers in the lower level Asylum Offices. For these reasons, a judge may treat these cases differently or, at the very least, categorize them separately. Therefore, we also test whether the negative autocorrelation in decision making is stronger when consecutive cases have the same defensive status (both affirmative or both defensive). ⁵

The court proceeding at the immigration court level is adversarial and typically lasts several hours. Asylum seekers may be represented by an attorney at their own expense. A DHS attorney cross-examines the asylum applicant and argues before the judge that asylum is not warranted. Those who are denied asylum are ordered to be deported. Decisions to grant or deny asylum made by judges at the immigration court level are typically binding, although applicants may further appeal to the Board of Immigration Appeals.

Our baseline tests explore whether judges are less likely to grant asylum after granting asylum in the previous case. To attribute negative autocorrelation in decisions to a cognitive bias, we first need to show that the underlying quality of the sequence of cases considered by each judge is not itself negatively autocorrelated. Several unique features of the immigration court process help us address this concern. Each immigration court covers a geographic region. Cases considered within each court are randomly assigned to the judges associated with the court (on average, there are eight judges per court). The judges then review the queue of cases following a “first-in-first-out” rule. ⁶ In other words, judges do not reshuffle the ordering of cases to be considered.

Thus, any time variation in case quality (e.g., a surge in refugees from a hot conflict zone) should originate at the court level. This variation in case quality is likely to be positively autocorrelated on a case-by-case level and therefore a bias against our findings of negative autocorrelation in decisions. We also directly control for time variation in court-level case quality using the recent approval rates of other judges in the same court and test autocorrelation in observable proxies of case quality in the Online Appendix .

Judges have a high degree of discretion in deciding case outcomes. They face no explicit or formally recommended quotas with respect to the grant rate for asylum. They are subject to the supervision of the Attorney General, but otherwise exercise independent judgment and discretion in considering and determining the cases before them. The lack of quotas and oversight is further evidenced by the wide disparities in grant rates among judges associated with the same immigration court Ramji-Nogales et al. (2007) . For example, within the same four-year time period in the court of New York, two judges granted asylum to fewer than 10% of the cases considered, while three other judges granted asylum to over 80% of cases considered. Because many judges display extreme decision rates (close to 0 or 1), we also present subsample analysis excluding extreme judges or limiting to moderate judges (grant rate close to 0.5). We exclude the current observation in the calculation of moderate status, so our results within the moderate subsample will not spuriously generate findings of negative autocorrelation in the absence of true bias.

Judges are appointed by the U.S. Attorney General. In our own data collection of immigration judge biographies, many judges previously worked as immigration lawyers or at the Immigration and Naturalization Service (INS) for some time before they were appointed. Judges typically serve until retirement. Their base salaries are set by a federal pay scale and locality pay is capped at Level III of the Executive Schedule. In 2014, that rate was $167,000. Based on conversations with the president of the National Association of Immigration Judges, no bonuses are granted. See  Appendix C for more background information.

Our data come from a Freedom of Information Act (FOIA) request filed through the Transactional Records Access Clearinghouse (TRAC). We exclude non-asylum-related immigration decisions and focus on applications for asylum, withholding of removal, or protection under the convention against torture (CAT). Applicants typically apply for all three types of asylum protection at the same time. As in Ramji-Nogales et al. (2007) , when an individual has multiple decisions on the same day on these three applications, we use the decision on the asylum application because a grant of asylum allows the applicant all the benefits of a grant of withholding of removal or protection under the withholding-CAT, while the reverse does not hold. In the Online Appendix we redefine a grant of asylum as affirmative if any of the three applications are granted and find qualitatively similar results. ⁷ We merge TRAC data with our own hand-collected data on judicial biographies. We exclude family members, except the lead family member, because in almost all cases all family members are either granted or denied asylum together.

We also restrict the sample to decisions with known time ordering within day or across days and whose immediate prior decision by the judge is on the same day or previous day, or over the weekend if it is a Monday. Finally, we restrict the sample to judges who review a minimum of 100 cases for a given court and courts with a minimum of 1,000 cases in the data. These exclusions restrict the sample to 150,357 decisions across 357 judges and 45 courthouses.

Table I summarizes our sample of asylum decisions. Judges have long tenures, with a median of eight years of experience. For data on tenure, we only have biographical data on 323 of the 357 judges, accounting for 142,699 decisions. The average case load of a judge is approximately two asylum cases per day. The average grant rate is 0.29; 94% of cases have a lawyer representing the applicant, and 44% are defensive cases initiated by the government. The average family size is 1.21. Time-wise, 47% of hearings occurred between 8 AM and 12 PM, 38% occurred between 12 PM and 2 PM, and 15% occurred between 2 PM and 8 PM. We mark the clock time according to the time that a hearing session opened.

The nonextreme indicator tags decisions for which the average grant rate for the judge for that nationality-defensive category, calculated excluding the current observation, is between 0.2 and 0.8. The moderate indicator tags decisions for which the average grant rate for the judge for that nationality-defensive category, excluding the current observation, is between 0.3 and 0.7. ⁸

III.B. Asylum Judges: Empirical Specification Details

Observations are at the judge × case order level; Y _it is an indicator for whether asylum is granted. Cases are ordered within day and across days. Our regression sample includes observations in which the lagged case was viewed on the same day or the previous workday (e.g., we include the observation if the current case is viewed on Monday and the lagged case was viewed on Friday), and for which we know the ordering of cases considered within the same day. ⁹

Control variables in the regressions include, unless otherwise noted, a set of dummies for the number of affirmative decisions over the past five decisions (excluding the current decision) of the judge. This controls for recent trends in grants, case quality, or judge’s mood. We also include a set of dummies for the number of grant decisions over the past five decisions across other judges (excluding the current judge) in the same court. This controls for recent trends in grants, case quality, or mood at the court level. To control for longer-term trends in judge- and court-specific grant rates, we control for the judge’s leave-out-mean grant rate for the relevant nationality × defensive category, calculated excluding the current observation. We also control for the court’s average grant rate for the relevant nationality × defensive category, calculated by excluding the judge associated with the current observation. In our baseline results, we do not include judge fixed effects because they mechanically induce a small degree of negative correlation between Y _it and Y _i,t−1 . In the Online Appendix we report results using judge’s fixed effects and obtain similar results with slightly more negative coefficient estimates, as expected. Finally, we control for the characteristics of the current case: presence-of-lawyer indicator, family size, nationality × defensive status fixed effects, and time-of-day fixed effects (morning/lunchtime/afternoon). Including time-of-day fixed effects is designed to control for other factors such as hunger or fatigue, which may influence judicial decision making (as shown in the setting of parole judges by Danziger et al. 2011 ).

III.C. Asylum Judges: Results

In Table II , column (1), we present results for the full sample of case decisions and find that judges are 0.5 percentage points less likely to grant asylum to the current applicant if the previous decision was an approval rather than a denial, all else being equal. In the remaining columns we focus on cumulative subsamples in which the magnitude of the negative autocorrelation increases substantially. First, the asylum data cover a large number of judges who tend to grant or deny asylum to almost all applicants from certain nationalities. More extreme judges necessarily exhibit less negative autocorrelation in their decisions. In column (2) of Table II , we restrict the sample to nonextreme judge observations (where nonextreme is calculated excluding the current decision). The extent of negative autocorrelation doubles to 1.1 percentage points.

In column (3) of Table II , we further restrict the sample to cases that follow another case on the same day (rather than the previous day). We find stronger negative autocorrelation within same-day cases. The stronger negative autocorrelation when two consecutive cases occur more closely in time is broadly consistent with saliency and the gambler’s fallacy decision making model because more recent cases may be more salient and lead to stronger expectations of reversals. These results are also consistent with experimental results in Gold and Hester (2008) , which finds that laboratory subjects who are asked to predict coin flips exhibit less gambler’s fallacy after an interruption when the coin “rests.” The higher saliency of more recent cases could also be consistent with stronger SCE, but is less likely to be consistent with a quotas constraint unless judges self-impose daily but not overnight or multiday quotas.

Column (4) of Table II restricts the sample further to cases in which the current and previous case have the same defensive status. Individuals seeking asylum affirmatively, where the applicant initiates, can be very different from those seeking asylum defensively, where the government initiates. In affirmative cases, applicants typically enter the country legally and are applying to extend their stay. In defensive cases, applicants often have entered the country illegally and have been detained at the border or caught subsequently. Judges may view these scenarios to be qualitatively different; the negative autocorrelation increases to 3.3 percentage points.

Hence, from an unconditional 0.5 percentage points, the negative autocorrelation increases sixfold to 3.3 percentage points if we examine moderate judges on same-day cases with the same defensive status. Using the estimate in column (4) of Table II within the sample of nonextreme, same-day, same defensive cases, the coefficient implies that 1.6% of asylum decisions would have been reversed absent the negative autocorrelation in decision making. Table A.I in the Online Appendix reports the extent of the negative autocorrelation among each omitted sample and presents formal statistical tests for whether the estimates in each cumulative subsample significantly differ from one another. We find that the negative autocorrelation among extreme-judge and different-defensive-status subsamples are close to zero and significantly differ from the nonomitted samples. However, the negative autocorrelation is economically substantial even across consecutive days (with insignificant differences), although the effect size doubles when the judge considers cases within the same day.

Finally, column (5) of Table II tests whether decisions are more likely to be reversed following streaks of previous decisions. After a streak of two grants, judges are 5.5 percentage points less likely to grant asylum relative to decisions following a streak of two denials. Following a deny then grant decision, judges are 3.7 percentage points less likely to grant asylum relative to decisions following a streak of two denials, whereas behavior following a grant then deny decision is insignificantly different from behavior following a streak of two denials. In terms of our empirical framework introduced in Section II.B, we find that β₁ < β₂ < β₃ < 0. A formal statistical test of the difference in the β ’s appears at the bottom of the table, where we reject that the betas are all equal and that β₂ = β₃ . (The only insignificant difference is between β₁ and β₂ , though β₁ has, as predicted, a more negative point estimate.) These results are consistent with the gambler’s fallacy affecting decisions and inconsistent with a basic quotas model. Moreover, the magnitudes are economically significant. Using the largest point estimate following a streak of two grant decisions, a 5.5 percentage point decline in the approval rate represents a 19% reduction in the probability of approval relative to the base rate of approval of 29%.

We report robustness tests of our findings in the Online Appendix . Table A.II reports results using logit and probit models. The economic magnitudes are similar. Table A.III reports results using judge fixed effects. As expected, the coefficients are slightly more negative due to a mechanical negative autocorrelation between any two decisions by the same judge induced by the fixed effects. However, the bias appears to be small and the coefficient estimates are of similar magnitude to our baseline results that control for a moving average of each judge’s past five decisions, as well as her leave-out-mean grant rate. In addition, the precision of the estimates does not change much between the two specifications, suggesting that controlling for heterogeneity using the moving average of a judge’s decisions and her leave-out-mean instead of judge fixed effects yields similar identification despite the former containing more measurement error. Table A.IV presents results for an alternative definition of the granting of asylum, where instead of using the asylum grant decision, we code a decision as a grant if the judge granted any of the asylum, withholding of removal, or protection under the U.S. CAT applications. The results are very consistent with slightly smaller point estimates.

Finally, a potential concern with the sample split among moderate and extreme decision makers is that we may mechanically measure stronger negative autocorrelation among moderates. We emphasize that because we do not use the current observation in the calculation of whether a decision maker is moderate, restricting the sample to moderates does not mechanically generate β₁ < 0 if the true autocorrelation is 0 (e.g., a judge who decides based on random coin flips would be classified as a moderate, but would display zero autocorrelation). However, another potential issue that could mechanically generate greater measured negative autocorrelation for moderate judges is our use of a binary statistical model. The autocorrelation of a binary variable is biased away from 1 and the size of the bias increases as the base rate of decisions gets closer to 0 or one. This may lead us to estimate a lower degree for negative autocorrelation for “extreme” decision makers. One method to address this issue is to use the tetrachoric correlation, which models binary variables as functions of continuous (bivariate normal) latent variables. The bivariate probit model extends the tetrachoric correlation to allow for additional control variables. Using the bivariate probit model, Table A.V shows that there is strong negative autocorrelation in decisions for moderate decision makers that is statistically significant and of similar economic magnitude as those from our baseline regressions. Conversely, for extreme decision makers, there is no evidence of any autocorrelation. These results match our estimates from the linear probability, logit, and probit regressions and indicate that the potential mechanical correlation coming from binary models is not driving our results.

Table III explores additional heterogeneity across judges and cases. In this and subsequent tables, we restrict our analysis to the sample defined in column (4) of Table II —observations for which the current and previous case were decided by nonextreme judges on the same day and with the same defensive status. Column (1) of Table III shows that the reduction in the probability of approval following a previous grant is 4.2 percentage points greater when the previous decision corresponds to an application with the same nationality as the current applicant. While there is significant negative autocorrelation when sequential cases correspond to different applicant nationalities, the negative autocorrelation is three times larger when the two cases correspond to the same nationality. This suggests that the negative autocorrelation in decisions may be tied to saliency and coarse thinking. Judges are more likely to engage in negatively autocorrelated decision making when the previous case considered is similar in terms of characteristics, in this case nationality. These results are consistent with the stronger autocorrelation also found when the previous case occurred close in time with the current case or shared the same defensive status (as shown in Table II ).

Columns (2) and (3) of Table III show that moderate judges and judges with less experience display stronger negative autocorrelation in decisions. Judges who have less than the median experience in the sample (eight years) display stronger negative autocorrelation. The fourth column repeats the regression including judge fixed effects. We find that experience is also associated with significantly less negatively autocorrelated decisions for a given judge over time. ¹⁰

Because we measure decisions rather than predictions, reduced negative autocorrelation does not necessarily imply that experienced judges are more sophisticated in terms of understanding random processes. Both experienced and inexperienced judges could suffer equally from the gambler’s fallacy in terms of forming prior beliefs regarding the quality of the current case. However, experienced judges may draw, or believe they draw, more informative signals regarding the quality of the current case. If so, experienced judges will rely more on the current signal and less on their prior beliefs, leading to reduced negative autocorrelation in decisions.

Finally, we present evidence supporting the validity of our analysis. To attribute negative autocorrelation in decisions to cognitive biases and not case quality, we show that the underlying quality of the sequence of cases considered by each judge is not itself negatively autocorrelated. Within a court, the incoming queue of cases is randomly assigned to judges associated with that court, and the judges review the queue of cases following a first-in-first-out rule. Therefore, time variation in case quality (e.g., a surge in refugees from a hot conflict zone) should originate at the court level and is likely to be positively autocorrelated on a case-by-case level. We support this assumption in Table A.VI in the Online Appendix . We find that case quality does not appear to be negatively autocorrelated in terms of observable proxies for quality. However, our identifying assumption requires that autocorrelation in unobserved aspects of case quality is also not negative.

IV. L oan O fficers

Our second empirical setting examines loan officers making loan application decisions.

IV.A. Loan Officers: Data Description and Institutional Context

We use field experiment data collected by Cole et al. (2015) . ¹¹ The original intent of the experiment was to explore how various incentive schemes affect the quality of loan officers’ screening of loan applications. The framed field experiment was designed to closely match the underwriting process for unsecured small enterprise loans in India. Real loan officers were recruited for the experiment from the active staff of several commercial banks. These loan officers had an average of 10 years of experience in the banking sector. In the field experiment, the loan officers screen real, previously processed loan applications. Each loan file contained all the information available to the bank at the time the loan was first evaluated.

Each loan officer participated in at least one evaluation session. In each session, the loan officer screened six randomly ordered loan files and decided whether to approve or reject the loan application. Because the loan files corresponded to actual loans previously reviewed by banks in India, the files can be classified by the experimenter as performing or nonperforming. Performing loan files were approved and did not default during the actual life of the loan. Nonperforming loans were either rejected by the bank in the loan application process or were approved but defaulted during the actual life of the loan. Loan officers in the experiment were essentially paid based upon their ability to correctly classify the loans as performing (by approving them) or nonperforming (by rejecting them). In our sample, loan officers correctly classified loans approximately 65% of the time. The percentage of performing loans they approved is 78%, while the percentage of nonperforming loans they approved is 62%, which shows they exhibited some ability to sort loans. Overall, the tetrachoric correlation between the binary variables, loan approval, and loan performance (1 = performing, 0 = nonperforming) is 0.29 and significantly different from random chance.

Participants in each session were randomly assigned to one of three incentive schemes, which offered payouts of the form $[wP,wD,w¯]$ , where w _P is the payout in rupees for approving a performing loan, w _D is the payout for approving a nonperforming loan, and $w¯$ is the payout for rejecting a loan (regardless of actual loan performance). Beyond direct monetary compensation, participants may have also been motivated by reputational concerns. Loan officers were sent to the experiment by their home bank and the experiment was conducted at a loan officer training college. At the end of the experiment, loan officers received a completion certificate and a document summarizing their overall accuracy rate. The loan officers were told that this summary document would only report their overall accuracy without reporting the ordering of their specific decisions and associated accuracy. Thus, loan officers might have been concerned that their home bank would evaluate these documents and therefore were motivated by factors other than direct monetary compensation. Importantly, however, the approval rate and the ordering of decisions was never reported. Therefore, there was no incentive to negatively autocorrelate decisions for any reason.

In the “flat” incentive scheme, payoffs take the form $[20,20,0]$ , so loan officers had monetary incentives to approve loans regardless of loan quality. However, loan officers may have had reputational concerns that led them to exert effort and reject low quality loan files even within the flat incentive scheme. ¹² In the “stronger” incentive scheme, payouts take the form $[20,0,10]$ , so loan officers faced a monetary incentive to reject nonperforming loans. In the “strongest” incentive scheme, payouts take the form $[50,−100,0]$ , so approval of nonperforming loans was punished by deducting from an endowment given to the loan officers at the start of the experiment. The payouts across the incentive treatments were chosen to be approximately equal to 1.5 times the hourly wage of the median participant in the experiment.

The loan officers were informed of their incentive scheme. They were also made aware that their decision on the loans would affect their personal payout from the experiment but would not affect actual loan origination (because these were real loan applications that had already been evaluated in the past). Finally, the loan officers were told that the loan files were randomly ordered and that they were drawn from a large pool of loans, of which approximately two-thirds were performing loans. Because the loan officers reviewed loans in an electronic system, they could not review the loans in any order other than the order presented. They faced no time limits or quotas.

Table IV presents summary statistics for our data sample. The data contains information on loan officer background characteristics such as age, education, and the time spent by the loan officer evaluating each loan file. Observations are at the loan officer × loan file level. We consider an observation to correspond to a moderate loan officer if the average approval rate of loans by the loan officer in other sessions (not including the current session) within the same incentive scheme is between 0.3 and 0.7.

IV.B. Loan Officers: Empirical Specification Details

We define Y _it as an indicator for whether the loan is approved. Loans are ordered within a session. Our sample includes observations for which the lagged loan was viewed in the same session (so we exclude the first loan viewed in each session because we do not expect reliance on the previous decision to necessarily operate across sessions, which are often separated by multiple days). In some specifications, we split the sample by incentive scheme type, that is, flat, strong, or strongest.

We control for heterogeneity in mean approval rates at the loan officer × incentive scheme level using the mean loan officer approval rate within each incentive treatment (calculated excluding the six observations corresponding to the current session).We also include an indicator for whether the loan officer has ever approved all six loans in another session within the same incentive treatment, to control for the fact that these types of loan officers are likely to have particularly high approval rates in the current session. Finally, we include an indicator for whether the current session is the only session attended by the loan officer within the incentive treatment (if so, the first two control variables cannot be calculated and are set to 0). Because the loan officer field experiment data is limited in size and each session consists of only six loan decisions, we do not control for a moving average of each loan officer’s average decision rate over the past five decisions within the session (as we do in the asylum judge setting). In the Online Appendix , we also present results controlling for loan officer fixed effects.

IV.C. Loan Officers: Results

Table V , column (1) shows that loan officers are 8 percentage points less likely to approve the current loan if they approved the previous loan when facing flat incentives. This implies that 2.6% of decisions are reversed due to the sequencing of applications. These effects become much more muted and insignificantly different from zero in the other incentive schemes when loan officers face stronger monetary incentives for accuracy, as shown by the other interaction coefficients in column (1). A test for equality of the coefficients indicate significantly different effects across the three incentive schemes. In column (2), we control for the true quality of the current loan file. Therefore, all reported coefficients represent mistakes on the part of the loan officer. After including this control variable, we find quantitatively similar results, indicating that the negatively autocorrelated decision making results in decision errors.

In columns (3) and (4) of Table V , we repeat the analysis for loan officers with moderate approval rates (estimated using approval rates in other sessions excluding the current session). In the loan officers experimental setting, a potential additional reason why the effect sizes are much larger in the moderate loan officers sample is that some loan officers may have decided to shirk in the experiment and approve almost all loans. Removing these loan officers from the sample leads to much larger effect sizes. Comparing the coefficient estimates with those in the same row in columns (1) and (2), we find that within each incentive treatment, moderate decision makers display much stronger negative autocorrelation in decisions. Under flat incentives, moderate decision makers are 23 percentage points less likely to approve the current loan if they approved the previous loan, implying that 9% of decisions are reversed. Even within the stronger and strongest incentive treatments, loan officers are 5 percentage points less likely to approve the current loan if they approved the previous loan. Overall, these tests suggest that loan officers, particularly moderate ones, exhibit significant negative autocorrelation in decisions, which can be mitigated through the use of strong pay for performance.

Tables A.VII–A.X in the Online Appendix report robustness tests for the loan officer sample. Table A.VII further tests whether the loan officers in the experiment are exerting effort in making accurate decisions and how that effort varies with incentives. We assess whether the loan approval decision is correlated with the ex ante quality of the loan file, as proxied by the fraction of other loan officers who approved the loan file, and the average quality score/rating given by other loan officers for the loan file. We further explore how the correlation between decisions and ex ante loan quality interact with the three incentive schemes. The results show that loan officers are more likely to approve loans that other loan officers approve or rate highly, and that the consensus in decision making increases with stronger incentives.

Table A.VIII shows that the results are similar when using other binary regression models, such as logit and probit. Table A.IX reports results from a specification that includes loan officer fixed effects. The coefficients are directionally similar but more negative than those in Table V . This is expected because the inclusion of fixed effects, particularly in short panels such as in the loan officers’ experimental setting, biases the coefficients downward. Table A.X reports results from a bivariate probit model that adjusts for the bias that, when using a binary dependent variable, the moderate subsample may mechanically exhibit more negative autocorrelation. The results using the bivariate probit model confirm that the negative autocorrelation in decisions is stronger for moderates even after adjusting for this potential bias, and the negative autocorrelation decreases with incentives.

In the remaining analysis, we pool the sample across all three incentive treatments unless otherwise noted. Table VI shows that loan officers with graduate school education and who spend more time reviewing the current loan file display significantly reduced negative autocorrelation in decisions. ¹³ Older and more experienced loan officers also display significantly reduced negative autocorrelation. These results are similar to our previous findings on asylum judges, and suggest that education, experience, and effort can reduce behavioral biases. Again, because we focus on decisions rather than predictions, our results do not necessarily imply that more educated, experienced, or conscientious loan officers suffer less from cognitive biases. These loan officers may still suffer equally from the gambler’s fallacy but draw, or believe they draw, more precise signals regarding current loan quality, leading them to rely less on their (misinformed and based-on-case-sequence) priors regarding loan quality.

Table VII examines decisions following streaks of decisions. We find that after approving two applications in a row, loan officers are 7.5 percentage points less likely to approve the next application, relative to when the loan officer denied two applications in a row. The effects are larger and more significant when restricted to moderate loan officers (column (2)). We easily reject that β₁ = β₂ = β₃ and β₁ = β₃ for the past sequence of decisions. However, “Lag reject-approve” has a less negative coefficient than “Lag approve-reject,” even though a gambler’s fallacy model where recency matters would predict the opposite. The sample size is small, however, and the difference between these two coefficients is insignificant and small in the sample of moderates.

Last, we discuss why our results are robust to a unique feature of the design of the original field experiment. Within each session, the order of the loans viewed by the loan officers on the computer screen was randomized. However, the original experimenters implemented a balanced session design. Each session consisted of four performing loans and two nonperforming loans. ¹⁴ If the loan officers had realized that sessions were balanced, a rational response would have been to reject loans with a greater probability after approving loans within the same session (and vice versa). We believe there are several reasons it is unlikely that loan officers would react to the balanced session design.

First, loan officers were never informed that sessions were balanced. Instead, they were told that the six loans within each session were randomly selected from a large population of loans. Second, if loan officers had “figured out” that sessions were balanced, we would expect that loan officers would be more likely to use this information when subject to stronger pay for performance. In other words, there should have been greater negative autocorrelation within the incentive treatments with stronger pay-for-performance, but this is the opposite of what we find. Also, the better educated may be more likely to deduce that sessions are balanced, so they should display stronger negative autocorrelation, which is again the opposite of what we find.

In columns (1) and (2) of Table A.XI in the Online Appendix , we reproduce the baseline results showing that the negative autocorrelation in decisions is strongest in the flat incentive scheme treatment. In columns (3)–(6), we show that the true performance of the current loan is negatively related to both the lagged decision and the true quality of the lagged loan file, and the negative autocorrelation in true loan quality is approximately similar in magnitude across all three incentive treatments. The results in columns (1) and (2) are inconsistent with loan officers realizing that sessions were balanced. If loan officers had realized that sessions were balanced, we would have expected the opposite result, that is, that the negative autocorrelation in decisions would be equally or more strong under the stronger incentive schemes.

V. B aseball U mpires

Our final empirical setting uses data on called pitches by the home plate umpire in Major League Baseball.

V.A. Baseball Umpires: Data Description and Institutional Context

In MLB, one important job of the home plate umpire is to call a pitch as either a strike or ball, if a batter does not swing. The umpire has to determine if the location of the ball as it passes over home plate is within the strike zone as described and shown in Figure I . If the umpire decides the pitch is within the strike zone, he calls it a strike and otherwise calls it a ball. The boundaries of the strike zone are officially defined as in the caption for Figure I , and are not subject to individual umpire interpretation. However, each umpire is expected to use his “best judgment” when determining the location of the ball relative to the strike zone boundaries. Hence, umpire judgment matters.

Figure I

Open in new tabDownload slide

Baseball Umpires: The Strike Zone

According to Major League Baseball’s “Official Baseball Rules” 2014 Edition, Rule 2.00, “The STRIKE ZONE is that area over home plate the upper limit of which is a horizontal line at the midpoint between the top of the shoulders and the top of the uniform pants, and the lower level is a line at the hollow beneath the kneecap. The Strike Zone shall be determined from the batter’s stance as the batter is prepared to swing at a pitched ball.”

We test whether baseball umpires are more likely to call the current pitch a ball after calling the previous pitch a strike. Of course, pitch quality (e.g., location) is not randomly ordered. For example, a pitcher will adjust his strategy depending on game conditions. An advantage of the baseball umpire data is that it includes precise measures of the trajectory and location of each pitch. Thus, while pitch quality may not be randomly ordered over time, we can control for each pitch’s true location and measure whether mistakes in calls, conditional on a pitch’s true location, are negatively predicted by the previous call.

We use data on umpire calls of pitches from PITCHf/x, a system that tracks the trajectory and location of each pitch with respect to each batter’s strike zone as the pitch crosses in front of home plate. The location measures are accurate to within a square centimeter. The PITCHf/x system was installed in 2006 in every MLB stadium and implemented for part of the 2007 season. Our data cover approximately 3.5 million pitches over the 2008 to 2012 MLB seasons, when the system produced an entire season of pitch data. We restrict our analysis to called pitches, that is, pitches in which the batter does not swing (so the umpire must make a call), excluding the first called pitch in each inning. This sample restriction leaves us with approximately 1.5 million called pitches over 12,564 games by 127 different umpires. In some tests, we further restrict our sample to consecutive called pitches, where the current called pitch and the previous called pitch were not interrupted by another pitch in which the umpire did not make a call (e.g., because the batter took a swing). Consecutive called pitches account for just under 900,000 observations.

Baseball umpires in our sample do not receive immediate feedback regarding whether each call was correct (data on whether each pitch was within the strike zone according to the PITCHf/x system is available after the game). Nevertheless, the umpire likely receives some cues. At the very least, umpires can observe the extent to which others disagreed with the call. First, the umpire knows roughly where the pitch landed and how “close” the call was. A call made on a pitch near the edge of the strike zone is more ambiguous, for instance. More to the point, the umpire also receives cues from the batter’s reaction, the pitcher’s reaction, and the crowd’s reaction to the call. These parties voice their disagreement if they believe the umpire made a mistake. Making an unambiguous erroneous call will likely draw a stronger reaction from at least one of these parties.

Table VIII summarizes our data sample. Approximately 30% of all called pitches are called as strikes (rather than balls). Umpires make the correct call 86.6% of the time. We also categorize pitches by whether they were ambiguous (difficult to call) or obvious (easy to call). Ambiguous pitches fall within ±1.5 inches of the edge of the strike zone; 60% of ambiguous pitches are called correctly. Obvious pitches fall within three inches around the center of the strike zone, or six inches or more outside the edge of the strike zone; 99% of obvious pitches are called correctly.

V.B. Baseball Umpires: Empirical Specification

Our baseline tests explore whether umpires are less likely to call the current pitch a strike after calling the previous pitch a strike, controlling for pitch location, which should be the sole determinant of the call. The sample includes all called pitches except for the first in each game or inning. We define Y _it as an indicator for whether the current pitch is called a strike, while Y _i,t−1 is an indicator for whether the previous pitch was called a strike.

To attribute negative autocorrelation in decisions to cognitive biases, we assume that the underlying quality of the pitches (e.g., the location of the pitch relative to the strike zone), after conditioning on a set of controls, is not itself negatively autocorrelated. To address this potential concern, we include detailed controls for the characteristics of the current pitch. First we control for the pitch location relative to an absolute point on home plate using indicators for each 3 × 3 inch square. We also control explicitly for whether the current pitch was within the strike zone based on its location, which should be the only characteristic that matters for the call according to MLB rules. Finally, we control for the speed, acceleration, curvature, and spin in the x , y , and z directions of the pitch, which may affect an umpire’s perception. For a complete detailed list of all control variables, please see  Appendix D . Our control variables address the concern that pitch characteristics are not randomly ordered. In addition, the fact that we control for whether the current pitch is actually within the true strike zone for each batter implies that any nonzero coefficients on other variables represent mistakes on the part of the umpire. Nothing else, according to the rules, should matter for the call except the location of the pitch relative to the strike zone. Specifically, any significant coefficient on the lagged umpire decision is evidence of mistakes.

Of course, umpires may be biased in other ways. For example, Parsons et al. (2011) show evidence of discrimination in calls: umpires are less likely to call strikes if the umpire and pitcher differ in race and ethnicity. However, while biases against teams or specific types of players could affect the base rate of called pitches within innings or against certain pitchers, they should not generate high-frequency negative autocorrelation in calls, which is the bias we focus on in this article. ¹⁵ In addition, in the Online Appendix , we include umpire, batter, and pitcher fixed effects, which should account for these sorts of biases, and find similar effects. More relevant for our tests, Moskowitz and Wertheim (2014) show that umpires prefer to avoid making calls that result in terminal outcomes or that may determine game outcomes. To differentiate our finding from these other types of biases that may affect the probability of the umpire calling strike versus ball at different points in the game, we control for indicator variables for every possible count combination (number of balls and strikes called so far on the batter), ¹⁶ the leverage index (a measure developed by Tom Tango of how important a particular situation is in a baseball game depending on the inning, score, outs, and number of players on base), indicators for the score of the team at bat, indicators for the score of the team in the field, and an indicator for whether the batter belongs to the home team.

In our previous analysis of asylum judges and loan officers, we controlled for heterogeneity in each decision maker’s approval rate using the decision maker’s leave-out-mean approval rate, moving average of the past five decisions, and/or decision maker fixed effects. We also conducted subsample analysis limited to moderate decision makers. These control variables for decision maker heterogeneity are less relevant in the setting of baseball umpires because professional umpires tend to have very homogeneous mean rates of strike calls. ¹⁷ Therefore, we present our baseline regression results without including controls for individual heterogeneity (the lack of controls should be a bias against findings of negative autocorrelation), and show in the Online Appendix that the results are very similar if we control for umpire fixed effects or a moving average of the past five decisions.

Finally, we use the sample of called pitches, that is, pitches in which the batter chose not to swing. Whether the batter chooses to swing is unlikely to be random and may depend on various game conditions, which is partly why we add all of the controls above. However, endogenous sample selection of this form should also not bias our results toward finding spurious negative autocorrelation in umpire calls. We test, within the sample of called pitches, whether umpires tend to make mistakes in the opposite direction of the previous decision, after controlling for the true quality (location) of the current pitch. We also show that, insofar as pitch quality is not randomly ordered, it tends to be slightly positively autocorrelated within this sample, which is a bias against our findings of negative autocorrelation.

V.C. Baseball Umpires: Results

Table IX , column (1) shows that umpires are 0.9 percentage points less likely to call a pitch a strike if the most recent previously called pitch was called a strike. Column (2) shows that the negative autocorrelation is stronger following streaks. Umpires are 1.3 percentage points less likely to call a pitch a strike if the two most recently called pitches were also called strikes. Further, umpires are less likely to call the current pitch a strike if the most recent pitch was called a strike and the pitch before that was called a ball than if the ordering of the last two calls were reversed. In other words, extreme recency matters. We easily reject that β₁ = β₂ = β₃ in favor of β₁ < β₂ < β₃ < 0. These findings are consistent with the gambler’s fallacy and less consistent with a quotas explanation (in addition, umpires do not face explicit quotas). The results are also less consistent with a learning model about where to set a quality cutoff bar because there is an objective quality bar (the official strike zone) that, according to the rules, should not move depending on the quality of the previous pitch.

All analysis in this and subsequent tables includes detailed controls for the actual location, speed, and curvature of the pitch. In addition, because we control for an indicator for whether the current pitch actually fell within the strike zone, all reported non-zero coefficients reflect mistakes on the part of the umpires (if the umpire always made the correct call, all coefficients other than the coefficient on the indicator for whether the pitch fell within the strike zone should equal zero).

In columns (3) and (4) of Table IX , we repeat the analysis but restrict the sample to pitches that were called consecutively (so both the current and most recent pitch received umpire calls of strike or ball) without any interruption. In the consecutive sample, the umpire’s recent previous calls may be more salient because they are not separated by uncalled pitches. We find that the magnitude of the negative autocorrelation increases substantially in this sample. Umpires are 2.1 percentage points less likely to call the current pitch a strike if the previous two pitches were called strikes. This represents a 6.8% decline relative to the base rate of strike calls. We test whether the differences in magnitudes between the full sample and the consecutive called pitches sample are significant and find that they are, with p -values below .001. In all subsequent analysis, unless otherwise noted, we restrict the sample to consecutive called pitches.

Tables A.XII–A.XVI in the Online Appendix report robustness tests of these results. Table shows similar results with batter, pitcher, and umpire fixed effects. Table A.XII shows similar results with batter, pitcher, and umpire fixed effects. Table A.XIII reports similar results using the moving average of the umpire's past five calls as a control variable. Table A.XIV shows similar effects and economic magnitudes using logit and probit models, and Table A.XV shows similar results using a bivariate probit model. Table A.XVI shows similar results if we exclude control variables for the count (number of balls and strikes called so far on the batter).

Since in this setting we are particularly concerned that the “quality,” that is, the location, of the pitch will also react to the umpire’s previous call, we control for each pitch’s true location (plus the other controls described in  Appendix D ) and measure whether mistakes in calls conditional on a pitch’s true location are negatively predicted by the previous call. If our location and other controls are mismeasured or inadequate, however, then autocorrelation in the quality of pitches could still be an issue. To assess how concerning this issue might be, we also reestimate the regression by replacing the dependent variable of whether a pitch is called a strike with an indicator for whether the pitch is actually a true strike. We also estimate a version of the analysis where the dependent variable is replaced with the distance of the pitch from the center of the strike zone. We then test whether these proxies for the true location of the pitch depend on whether the lagged pitch was called a strike. In other words, how does the actual quality of the pitch respond to the previous call?

Table A.XVII in the Online Appendix shows that the negative autocorrelation in umpire calls is unlikely to be caused by changes in the actual location of the pitch. We continue to restrict the sample to consecutive called pitches and repeat the analysis using the current pitch’s true location as our dependent variable (to identify the effect of previous calls on the location of the current pitch, we exclude location controls). Columns (1) and (2), which use an indicator for whether the current pitch was within the strike zone as the dependent variable, show that pitchers are more likely to throw another strike after the previous pitch was called a strike, resulting in positive, rather than negative, coefficients on the previous call. Hence, autocorrelation in the quality of pitches biases us against our finding of negatively autocorrelated decision making. In columns (3) and (4), we use the distance of the pitch in inches from the center of the strike zone as the dependent variable. If pitchers are more likely to throw true balls (more distant from the center of the strike zone) after the previous pitch was called a strike, we should find significant positive coefficients on lagged strike calls; again, we find the opposite. In other words, endogenous changes in pitch location as a response to previous calls should lead to positive rather than negative autocorrelation in umpire calls because the quality of pitches is slightly positively autocorrelated. Finally, in columns (5) and (6), we include the same set of detailed pitch location controls (dummies for each 3 × 3 inch square) as in our baseline specifications, and find that all coefficients on lagged calls become small and insignificant, suggesting that our controls effectively remove any autocorrelation in the quality of pitches and account for the pitcher’s endogenous responses to previous calls.

Table X shows that the negative autocorrelation in decisions is reduced when umpires receive more informative signals about the quality of the current pitch. Columns (1) and (2) restrict the analysis to observations in which the current pitch is ambiguous—pitches located close to the boundary of the strike zone, where it is difficult to make a correct strike or ball call. Columns (3) and (4) restrict the analysis to observations in which the current pitch is likely to be obvious—pitches located close to the center of the strike zone (“obvious” strikes) or far from the edge of the strike zone (“obvious” balls). We find that the magnitude of negative autocorrelation coefficients are 10 to 15 times larger when the current pitch is ambiguous relative to when the current pitch is obvious. We can confidently reject equality of the estimates for ambiguous and obvious pitches in columns (1) and (3) with p -values well below .001. This is consistent with the gambler’s fallacy model that the decision maker’s prior beliefs about case quality will have less impact on the decision when the signal about current case quality is more informative.