Fairness in Educational and Psychological Testing: Examining Theoretical, Research, Practice, and Policy Implications of the 2014 Standards

INTRODUCTION

Conceptualizing and Contextualizing Fairness Standards, Issues, and Solutions Across Professional Fields in Education and Psychology

J

ESSICA

L. J

ONSON AND

K

URT

F. G

EISINGER

Buros Center for Testing, University of Nebraska–Lincoln

Asubstantive change introduced in the 2014 edition of Standards for Educational and Psychological Testing ¹ was the elevation of fairness as a foundational element of professional testing and assessment practice, along with validity and reliability (American Educational Research Association [AERA] et al., 2014; hereinafter, Standards). The 2014 edition of the Standards endorses the view that all steps in the testing process—including test design, development, administration, scoring, validation, and use—should be addressed in such a manner as to promote valid and fair score interpretations for the intended uses for all examinees in the intended population by minimizing extraneous factors that distort the meaning of scores (construct-irrelevant variance). Because the Standards are intended to be relevant for both test developers and test users, individual standards are carefully written to be appropriate and understandable to a variety of audiences beyond professionals in the field. Although the Standards provide general guidelines for testing practice, they do not cite scholarship, best practice methodologies, or examples of practice (Sinharay et al., 2014). Therefore, it is possible for the guidelines in the Standards to be further clarified and contextualized through identification and examination of how they are applied in research and practice. This book offers a review of this fairness research and practice to advance awareness and understanding of fairness issues encountered in educational and psychological testing. Chapters are authored by researchers and practitioners from the fields of educational measurement, school psychology, industrial and organizational psychology, and counseling and clinical psychology. The authors were asked to share fairness issues they have encountered in their work to identify gaps in theoretical, empirical, and applied research, and to discuss the implications for researchers, practitioners, policy makers, and, in some cases, the public.

Another goal of this book is to address the disparities that currently exist between research and practice and across disciplines when considering fairness guidelines in the Standards. The dissemination of the Standards and of effective methods, practices, and processes for gathering fairness evidence is of limited utility to practitioners if the primary mechanism is through discipline-specific conferences and journals to which practitioners have limited access. In addition, compartmentalization of the available knowledge and conceptualizations of fairness in testing in different disciplines can hamper dialogue between scholars and practitioners across professional fields. An intended outcome of this book is to share insights from various contexts and disciplines to prompt an examination of where the 2014 Standards appropriately address fairness issues in those contexts, as well as the limitations of those guidelines, and to identify future implications for research, practice, and policy. To accomplish these purposes, the book includes 16 chapters in three sections: Conceptualizing Fairness in Testing and Assessment, Encountering Fairness in Different Fields and Contexts, and Understanding Specific Subgroups When Considering Fairness. This diverse array of professional perspectives and the intersection of the discussed scholarship should prompt more careful consideration of fairness issues such as differences among test takers (e.g., individuals with disabilities, English learners, and individuals of different cultural backgrounds), differences across assessment settings (e.g., educational, clinical/counseling, and employment settings), and implications for all participants in the testing process (e.g., researchers, practitioners, policy makers).

Overview of Fairness in the 2014 Standards

Fairness in the 2014 Standards was elevated in status in part because multiple chapters in the 1999 Standards addressed fairness considerations for individuals with disabilities and individuals with diverse linguistic and cultural backgrounds. In the 2014 edition, these distinct topics were combined into Chapter 3, Fairness in Testing, and placed in the foundational section of the 2014 Standards, along with the chapters on validity and reliability. The intent was to represent fairness as an area with its own methods and processes for gathering evidence, separate from those used for reliability and validity. Nevertheless, the fairness chapter broadly addresses considerations throughout all stages of test development and use (p. 49).

A second reason the two chapters from the 1999 Standards were combined into a single chapter in the 2014 edition was to emphasize that fairness applies to all individuals in the intended population of test takers as a foundational concern. In addition, a single chapter integrating fairness topics not only emphasizes measurement bias as a threat to fairness in testing, but also highlights the importance of accessibility. Accessibility provides test takers an unobstructed opportunity to demonstrate standing on the construct being measured. This goal is facilitated by the use of universal test design as a purposeful, up-front approach to minimize construct-irrelevant features that might otherwise impede the performance of intended examinee groups, and to maximize, to the extent possible, access for as many examinees as possible in the intended population (p. 50).

As a framework for the guidelines that follow, Chapter 3 in the 2014 Standards outlines four general views of fairness: First, comparability of score interpretation should be prioritized over standardization of testing procedures to enable all test takers to receive fair and equitable treatment during the testing process. The other three views involve three different stages of the assessment process by addressing (a) lack of measurement bias in test development, (b) accessibility of measured constructs to all test takers in the test design, and (c) test score interpretation and use that reflect how the individual characteristics of a test taker interact with the test and the context in which the test is given. The chapter identifies several important points within these four general views of fairness:

•Tests should be designed to be sensitive to the diversity of individual characteristics and not just diversity at a broader subgroup level such as that defined by race, ethnicity, gender, culture, language, age, disability, or socioeconomic status.

•The applicability of different fairness considerations depends on context; thus, it is essential to identify the potential threats to valid interpretations for examinees as a result of differing testing contexts.

•Providing comparable contexts for test takers is essential to ensure fairness in the testing process. Although standardization of test administration conditions can support good testing practices, they may need adjustment to be responsive to the needs of specific groups or individuals to achieve comparability.

•Although lack of measurement bias continues to be a necessary component of fairness in testing, the usefulness of common statistical procedures is limited by how subgroups are defined and the size of those subgroups. Because sampling limitations have some bearing on the utility of those statistical procedures, alternate methods that do not require large and diverse samples should be considered. These alternate methods may include cognitive interviews, use of scholarship, and use of expert judgment to determine whether measured constructs have equivalent meaning across individuals and groups. These alternative methodologies can also be utilized to investigate evidence of potential measurement biases identified during test design and development.

•Accessibility addresses whether targeted knowledge, skills, and abilities are the only skills needed for an individual to demonstrate their standing on the measured construct. If knowledge, skills, and abilities unrelated to the measured construct become part of responding, they may hinder individuals’ performance. This situation would indicate a need either for accommodations, which would still allow scores to be comparable, or for modifications, which may well render scores incomparable.

•Fairness defined in terms of equality of testing outcomes for relevant test-taker subgroups is not a view held from a measurement perspective but is often considered an aspect of fairness for public (non-measurement) audiences. This factor contributes to disjointed understandings and arguments about fairness. For example, two groups could have different average scores or percentages of test takers passing when the test is still fair from a psychometric perspective.

Understanding and Minimizing Fairness Threats

Chapter 3 in the 2014 Standards also identifies aspects of tests and the testing process that can introduce score differences due to extraneous factors (construct-irrelevant variance): test content, test context, test response, and opportunity to learn. Test content threatens fairness in testing when the material used to represent a construct differentially favors or disadvantages one subgroup of test takers. This unfairness can occur if one subgroup has prior knowledge or experience unrelated to the construct, or if that subgroup views the material as more or less interesting than other test takers do, or finds the content to be emotionally disturbing or offensive. Test content can also become a fairness issue when schools differ in curricula offered or access to instructional resources, or when some students do not have access to the standard curriculum (e.g., special education students). Threats of test context are aspects of the test and the testing situation or environment that might affect performance, including interactions with a test administrator, lack of clear instructions, unnecessarily complex language demands of tasks, and various characteristics of test items. If these threats occur, individuals from diverse ethnic, cultural, and linguistic groups can be particularly impacted during the testing process; therefore, these barriers merit more research. Test response issues arise where test items lead to responses other than intended. For example, some response formats might be more difficult for some test takers than others or differentially valued. Technology-enhanced testing contexts are one example, where the mode of response may be relatively unfamiliar to individuals from a particular subgroup. Performance assessments scored with rubrics are vulnerable to test response threats and should be developed to ensure that credit is awarded only for response characteristics central to the construct being measured (p. 56). Finally, opportunity-to-learn issues exist if test takers have not been exposed to instruction or knowledge that affords them the opportunity to learn the content and skills targeted by the test (p. 56), and this deficit can be caused by measuring content that a test taker was never exposed to but should have been. However, opportunity to learn is difficult to define, identify, and quantify—making it another area of fairness that would benefit from additional guidelines, methodologies, or best practices.

The fairness chapter in the Standards addresses how these construct-irrelevant aspects of testing can be minimized with the use of adaptations. These adaptations can be accommodations, defined by the 2014 Standards as adaptations that result in comparable scores and often result in minor changes to the presentation or format of the test. On the other hand, the adaptations can be modifications, which are defined by the 2014 Standards as adaptations that do not result in comparable scores because they transform the construct being measured (p. 59). Modifications may seem counterproductive, but as the 2014 Standards points out, a modification may permit at least a partial measurement of the construct, which may not otherwise be possible (e.g., assessing map reading in a blind student). The chapters in the present book focus on attempts to minimize these threats.

Overview of the Book

The primary goal of this book is to identify scholarship, best practice methodologies, and examples of policy and practice from different professional fields in education and psychology to illuminate the fairness guidelines in the 2014 Standards. It is hoped that the discussions in this book will serve as a starting point for informing future scholarship regarding the theoretical, empirical, and applied aspects of fairness in testing.

The authors for each chapter were asked to attend to the following five questions to provide a common structure throughout the book:

1. What key statements or standards in the fairness chapter do you address?

2. How do the issues raised in your chapter attend to fairness considerations for diverse groups (e.g., race and ethnicity, gender, and disability)?

3. What theoretical and empirical research from your professional area of expertise can help readers better understand the intent of the fairness standards?

4. Which policies and practices elucidate the intent of the fairness standards and the importance of fairness considerations in testing?

5. What future scholarship would assist in furthering how the field thinks about and practices fairness?

The chapters are grouped into three sections, whose titles represent the three subthemes described next.

Subtheme 1: Conceptualizing Fairness in Testing and Assessment

The five chapters in this section focus on the past, present, and potential future influences on how fairness in testing and assessment is conceptualized within the testing profession and in society more broadly. Geisinger (Chapter 1) discusses how fairness considerations have evolved since World War I. He suggests that test fairness/bias received very little attention until federal legislation and court cases forced the profession to develop appropriate models and techniques for determining bias. He concludes his chapter by sharing a current framework for fairness in testing according to the 2014 Standards, with an orientation to further conceptual development. Herman and Cook (Chapter 2) focus on the relevance of the Standards for a broader set of stakeholders beyond the measurement professions, including researchers, teachers, and policy makers. They suggest that specialized versions adapted to different stakeholder needs may be necessary to increase the Standards’ reach, and they recommend collaborating with relevant professional organizations to achieve maximum utility and impact. Pullin (Chapter 3) uses four examples of fairness testing controversies to demonstrate marketplace shifts in how the public and policy makers view the purpose and legitimacy of testing. She argues it is vital for the Standards to take these perceptual and contextual factors into account. This perspective includes not only which aspects of testing are addressed by the Standards but also the impact of the Standards on the practices of professionals, the public, and policy makers. Dorans, Syp, and Walker (Chapter 4) consider how the context of testing plays a key role in considerations of fairness in testing. They provide an overview of the fairness perspectives, practices, challenges in statistical approaches, and alternative approaches to decisions when tests are viewed as contests by the test takers for purposes such as college admissions. Dorans et al. also argue that previous definitions of subgroups may be less relevant than they were 50 years ago, and that the field needs to reexamine which subgroups to study for fairness practices to better achieve the aim of enhancing fairness at an individual level. Camilli and Newton (Chapter 5) conclude the section by addressing the broader meaning of fairness as it relates to social justice. They argue that measurement experts must be sensitive to multiple frames of reference beyond the Standards to effectively communicate issues of test fairness to those outside the measurement community.

Subtheme 2: Encountering Fairness in Different Fields and Contexts

In the second section of the book, six chapters examine specific fairness issues that arise in different fields and contexts. Thurlow (Chapter 6) starts the deliberations by focusing on accessibility policies in educational contexts, to highlight the gaps between those policies and the 2014 Standards. Emerging definitions of accessibility in education go beyond accommodations and modifications for those with disabilities as defined in the 2014 Standards. This misalignment impacts the needs of individuals from multiple-characteristic groups (e.g., English learners with disabilities) as well as the differences in opportunity to learn for students with certain characteristics. Camara and Davis (Chapter 7) raise practical concerns of comparability of scores as addressed in the 2014 Standards, given the increased use of technology in educational testing that results from the use of innovations and different applications. They point out how assessment modes, forms, devices, and other factors can alter measurement of the intended constructs; they emphasize the importance of research and the Standards keeping up with the rapid changes in technology to inform fair testing practices and score use. Technology also raises issues in employment settings, as outlined by Ryan and Nye (Chapter 8). They examine the potential benefits as well as the fairness concerns that technology creates when assessments are delivered, enhanced, responded to, or scored with the use of technology. In many cases, the impact of these changes on the measurement of the construct has not yet been fully vetted in research, and the authors propose a future agenda for this work. Welch and Dunbar (Chapter 9) offer a review of professional best practices for fairness processes in the development and production of large-scale achievements tests. They share how often these best practices are used, how they can be improved, and what barriers may inhibit the sharing of results from such studies. They also look to the future by identifying fairness considerations for new areas of test development. Laitusis (Chapter 10) continues Welch and Dunbar’s discussion of fairness considerations for new areas of test development by providing insights on ensuring fairness in scoring the next frontier of assessments. These new assessments incorporate process data, automated scoring, and technology-enhanced items, including simulations, but introduce new fairness challenges for individuals with disabilities. Laitusis reviews fairness challenges for these innovations, how they might be mitigated, future research needs, and what future editions of the Standards might consider regarding these testing approaches. In Chapter 11, Kanaya, Wai, and Worrell analyze a very specific testing problem, the Flynn effect, which can impact the fairness of testing outcomes in psychoeducational tests and decisions about disability and gifted and talented classifications. Kanaya et al. focus on how the 2014 Standards attend to this phenomenon and what additional research is needed if we are to fully understand its potential effect on fair interpretation and use of scores.

Subtheme 3: Understanding Specific Subgroups When Considering Fairness

The third and final section of this book contains five chapters that address issues related to specific test-taker subgroups such as English learners, diverse ethnic and cultural groups, and individuals with disabilities. Solano-Flores (Chapter 12) bridges research and practice by providing a validity framework for designing, using, and evaluating test accommodations for English learners that can be applied by test developers and test users. This framework is responsive to the needs of individual students. It highlights the shortcomings in methodological rigor and identifies multiple ways that accommodations can be misused with English learners. Ortiz and Wong (Chapter 13) consider the needs of English learners by identifying key theoretical, empirical, and practical issues in the testing of English learners. They help to support and illustrate the diversity that exists within groups of English learners and how more informed theories and more nuanced empirical studies could improve testing practice. Their discussion includes how test development and norming studies can more effectively measure the learning of English learners. Kettler and Dembitzer (Chapter 14) advance key points from the 2014 Standards by prompting thinking about how to differentiate between an accommodation and a modification, and what evidence could be used to accurately identify an adaptation as one or the other—a judgment that has often been left to the opinion of a test developer or user. The authors also discuss alternative methodologies for gathering evidence that remedy the shortcomings of commonly used methods, and they offer suggestions for better definitions of subgroups that could assist in the provision of individualized adaptations. Santiago Rivera and Cahill (Chapter 15), using the clinical assessment of depression as a backdrop, provide a context for greater understanding of why cross-cultural studies of acculturation are critical to test fairness. They not only point out the dilemmas but also identify frameworks that have been proposed for identifying these cultural and language barriers, which is critical to understanding how to apply the fairness guidelines from the 2014 Standards in a practical way. Suzuki (Chapter 16) examines and illustrates how cultural competence is a significant fairness consideration in the use of psychological tests and interpretation of test performance. She identifies constructs related to culture that can impact the use of psychological tests and potential cultural variables that have been found to influence test performance. She then highlights areas in need of further investigation to address the cultural limitations of several well-known psychological measures to illustrate her key points.

We conclude this book by revisiting its purpose and identifying key themes that connect these chapters with possible considerations for future editions of the Standards, as well as for future research and practice. We encourage our audience to read all the chapters in this book for a broader and more applied understanding of various aspects of fairness in testing as addressed in Chapter 3 of the 2014 Standards. At the same time, we recognize that readers have specific interests, and we therefore provide Table 1 as a guide to the issues addressed in each chapter. The table also shows in detail which key elements from Chapter 3 in the 2014 Standards are examined in this book, under the general categories of steps in the testing process; efforts to minimize threats to construct-irrelevant variance; recommendations for research, practice, and policy; and subgroups.

Table 1 demonstrates how the chapters in this book provide contextual discussions about fairness that are wide-ranging. This kind of broad treatment is valuable for improving our understanding of what fairness in testing is and why it is important. The chapter authors have attempted to discuss the issues in a way that is thoughtful and engaging for measurement professionals and accessible and informative for individuals who are not measurement experts. In most cases the authors provide definitions of terms that may not be familiar to a broad set of readers. In addition, a reader of the entire volume will encounter a fair amount of redundancy in background information about how the 2014 Standards address fairness in testing. We decided to preserve the redundancy because the background information will be needed for readers who only read individual chapters.

Table 1. Topic Listings for Chapters

Geisinger’s chapter noted that for the first half of the 20th century, fairness was not, or at least did not appear to be, a topic of interest to test developers, test users, and testing scholars. Since at least the passage of the Civil Rights Act of 1964 and the development of the 1985 Standards, and perhaps the 1999 Standards, fairness has taken on an increasingly important role and will continue to do so. In the 2014 Standards, fairness becomes a truly central foundational pillar of testing and test evaluation. There are other references (e.g., Camilli, 2006; Dorans et al., this volume) where one can learn some of the methodological approaches to assessing the fairness of tests, test results or scores, or test score interpretation using psychometric, statistical, and even judgmental methods. The Standards, however, in all of their incarnations have addressed professional concerns, which include methodological approaches, but go much further. This volume attempts to amplify and explicate those professional issues as they concern test fairness.

Note

1. The 2014 edition of the Standards is now open access in English and Spanish (in PDF, PDF eBook, and ePub formats); the 1999 edition is also available, in English only (PDF format). They can be downloaded free of charge at https://www.testingstandards.net/open-access-files.html.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Camilli, G. (2006). Test fairness. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 221–256). Praeger.

Sinharay, S., Rijmen, F., Choi, S., & Dorans, N. J. (2014). The revised Standards and its role in research on educational measurement. Educational Measurement: Issues and Practice, 33(4), 36–38.

Section 1

Conceptualizing Fairness in Testing and Assessment

Chapter 1

The History of Bias Assessment in Educational and Psychological Testing

K

URT

F. G

EISINGER

Buros Center for Testing, University of Nebraska–Lincoln

Educational and psychological testing has long aided decision making for practitioners in many fields, including education, mental health, and industrial-organizational psychology. Decisions can be made in a reliable fashion, in which case we know that the nature of the decisions has a degree of stability; that is, the decisions are made repeatedly with the same or very similar results. Tests can produce valid scores, that is, they can be made in a fashion whereby evidence, often empirical evidence, suggests that the grounds for making an intended decision are well supported. Finally, decisions can be made fairly. In the context of the United States, this means that the scores from tests are interpretable regardless of whether members of different groups (based on ethnicity, gender, race, ability status, gender identity, and age, among other things) achieve the same interpretations whether they are members of different or even apparently favored groups. Indeed, some of the basis for using tests in the first place is that they are seemingly objective and, therefore, it is hoped that they are not influenced by such inappropriate factors. Fairness and bias and their history in educational and psychological testing are the focus of this brief chapter. This chapter traces the history of testing in a few pages, followed by a discussion of approaches that testing professionals have used to address the fairness of test results. It concludes with some generalizations about how the topic of fairness has evolved in various editions of the Standards for Educational and Psychological Testing (e.g., 2014; hereinafter, the Standards ).

The Advent of Modern Testing

Many scholars and historians of testing issues (e.g., DuBois, 1970; Geisinger & Usher-Tate, 2016; R. M. Thorndike & Lohman, 1990) have thought that the beginning of modern psychological and educational testing was based on the work of Alfred Binet, a French psychologist who was charged by the Parisian schools with identifying individual pupils who were not able to profit appropriately from the new public schools in Paris. Teachers and others complained that some pupils were simply not able to succeed at school. Previous attempts to assess pupils had been based on factors such as bumps on the head, reaction time, and various other physical and physiological variables, and had not been successful in making useful predictions. Binet and his colleagues, on the other hand, focused on age-appropriate problem-solving skills. They found that tests composed of a variety of these problem-solving skills differentiated students who were able to succeed in schools from those who did not. This work initiated and advanced the understanding of intelligence testing through the next hundred years.

Lewis Terman was the most successful of the Americans who translated and adapted the Binet intelligence tests for use with Americans. Given that Terman worked at Stanford University, Stanford-Binet was the name of the measure that Terman developed (DuBois, 1970). Working with Terman was another American psychologist, Arthur S. Otis, who was largely credited with developing the first group-administered intelligence test while working with Terman. They correlated intelligence test scores earned in group settings with test scores administered individually by a psychologist and found reasonably good agreement across the administrative modes. Otis’s group-administered intelligence test was the model for the first major use of a group-administered intelligence test in testing recruits for the U.S. military in World War I.

One of the first recognitions of some diversity elements in testing occurred just prior to World War I. The then president of the American Psychological Association, Robert Yerkes, organized psychologists into committees to help with the war effort. A committee was established to consider the psychological testing of recruits, and that committee included Terman. The committee debated how to test the recruits; Yerkes favored individual assessment of the recruits. However, primarily for reasons of efficiency, the committee decided to test recruits using group tests. This decision and its implementation represented a major step in testing, for two reasons. The first is simply that it was the first massive group testing of intelligence. The second is more relevant to issues of fairness. One test, the Army Alpha, was developed for the group testing of those who were literate in English. A second test, the Army Beta, was developed for those who were illiterate in English due to lack of schooling, immigrant status, or other reasons. The Army Alpha was heavily dependent on verbal skills. The Army Beta was dependent on pictures and diagrams, and contained no verbal responses (DuBois, 1970; R. M. Thorndike & Lohman, 1990). The acknowledgment that a different test was needed for testing individuals with different backgrounds in order to obtain a more representative assessment of those individuals for decisions making was a major early step for testing and for professionals in the field.

A Focus on Intelligence Testing

After the success of using intelligence testing to decide who was able to serve as a soldier in World War I and what role they should serve within the military, intelligence became an important variable in the assessment of individuals who wished to immigrate to the United States (Gould, 1981; Kamin, 1974). The Immigration Act of 1924 (The Johnson-Reed Act) restricted both the number of people permitted to enter the United States for purposes of immigration and the countries from which they came. It permitted U.S. officials to use measures to help to determine whether individuals from certain countries and of different racial types were allowed into the United States. Quotas were established for immigrants from certain countries. Because of these restrictions, policy makers and implementers began using tests to decide which immigrants to accept. One psychologist, Howard Andrew Knox, was known to observe potential immigrants as they walked off the planks from ships to Ellis Island to decide if they needed to be tested for entrance into the country (Richardson, 2011). Such individuals were often those who looked different from others, were of different races, were from countries perceived to be of less desirability, or had disabilities. They were often tested with rudimentary intelligence tests, some of which were dependent on English language skills, although many were also performance-based tests that required only relatively weak English language skills. The overall effect was that immigrants were more likely to be rejected for entry if they came from countries whose language and culture were relatively unlike those in America, such as those in Eastern Europe, Russia, and Asia. To say in retrospect that this use of testing has given the measurement community a black eye would be a most generous interpretation. Using tests to make immigration decisions, a use for which they were never intended, is now generally seen as both a discriminatory use and a misuse of tests. A few well-known scholars (Humphreys, 1983; Jensen, 1980) have nevertheless defended this use of tests for immigration purposes and noted that some of the facts and their interpretation enumerated by critics such as Kamin (1974) and Gould (1981) were certainly not as negative in consequence as suggested by these critics. The Immigration and Nationality Act of 1965 abolished the earlier quota system based on national origin and established new immigration policies.

Testing After the World Wars

There is a dearth of material available regarding the issues of testing and diversity broadly defined in the period between the beginning of the 20th century and the 1950s. During this period of more than fifty years, the field was relatively silent on issues of test fairness and test bias, whether deliberate or unintentional. The field, perhaps like many other aspects of society, was simply blind to such issues. To make this case, I note that testing and psychometric textbooks such as those by Lord and Novick (1968) and Magnusson (1966/1967) include nary a word about such issues. There were also several fine books of readings that were often used in conjunction with the training of psychometricians that were essentially devoid of such coverage. Jackson and Messick (1967) and Mehrens and Ebel (1967), for example, were essentially absent of such topics. Jackson and Messick had one chapter (Wolfe, 1967) that broadly covered the diversity of talent but did not address gender, racial, and other differences even in the slightest; they were certainly not a focus of the chapter. Mehrens and Ebel did not mention these issues. This absence is surprising given the passage of the Civil Rights Act of 1964. Wolfe approached the diversity of talent from an ability perspective with the viewpoint that society needed to both assess and develop talents generally, and that these talents were highly related to intelligence, as well as some special abilities that could move society forward. One book that related to similar topics was John W. Gardner’s (1961) Excellence: Can We Be Equal and Excellent Too? In it, Gardner, a psychologist trained at the doctoral level, who later served as secretary of health, education, and welfare under President John F. Kennedy, discussed the apparent conflict between some widely held American values. These factors related to the decline of hereditary privilege and, with it, the joint issues of wanting the best for all members of society through education and other social institutions while at the same time trying to raise up the underprivileged in society. Gardner sometimes mentions race and ethnicity, but his work is largely an inclusive attempt to state that we need to identify talent wherever it may be found and provide underserved individuals and all others with the kinds of training and education that will move society forward. The book was published in 1961, that is, even before the Civil Rights Act of 1964, and was clearly ahead of its time. It continues to be an important, forward-thinking book.

Dubois’s (1970) history of psychological testing devoted one page to criticisms of testing. The first topic mentioned related to the use of objective-type achievement tests as opposed to more authentic assessments of school learning. The second criticism covered cautions needed when clinical and other personality measures are used for personnel selection purposes. A final criticism related to fair employment practices, and it is clear from the brief 19-line description of this criticism that the topic was still in the birthing stage. Dubois describes a case study of an individual who believed that he was not hired based on the results of his aptitude test performance and who argued that the results were due to his race. DuBois writes:

The question of the fairness of psychological tests to individuals of different backgrounds is one that has evoked both discussion and action. In 1968 the U.S. Department of Labor issued instructions to government contractors requiring that when tests are used to determine qualifications for hire, transfer, or promotion in nonprofessional, technical, and managerial occupations definite empirical data demonstrating test validity be developed. It directed that as far as possible evidence of criterion-related validity be obtained and that results be examined for implication of possible discrimination against members of minority groups. . . . this official step put psychological testing procedures in the framework of a government regulation. (pp. 130–131)

Purported Culture-Fair and Culture-Free Testing

There have been attempts to assess intelligence without regard to culture. Initially, such measures were called culture-free tests (e.g., Cattell, 1940). It was believed that tests that used geometric figures and their spatial manipulation rather than language-based measures of intelligence would be culture-free. Such tests often demonstrated as much cultural influence as did more language-based measures. Anastasi and Urbina (1997) suggest the following conclusions regarding these measures:

We now recognize that hereditary and environmental factors operate jointly at all stages in the organism’s development and that their effects are intertwined inextricably in the resulting behavior. For humans, culture permeates nearly all environmental contacts. Since the cultural milieu in which the individual is reared affects all behavior and since psychological tests are but samples of behavior, cultural influences will and should be reflected in test performance. It is therefore futile to try to devise a test that is free from cultural influences. (p. 342)

This belief became relatively widely understood in psychological testing circles, and terms such as culture-fair and cross-cultural became more commonly used in describing certain tests, especially general ability measures. Measures of culturally universal human characteristics are referred to as etic measures. However, tests devised for members of a single culture (sometimes called emic measures) are likely to be somewhat unfair to individuals from other cultures, unless they have become thoroughly acculturated into the new (or the dominant) culture.

The Beginning of Test Fairness Considerations

Dubois’s brief description that articulates the beginning of test fairness considerations, of course, followed the Civil Rights Act of 1964. The Civil Rights Act was a landmark civil rights and labor law for the United States and demonstrated a change in societal orientation toward fairness for all members of society. It outlawed discrimination based on race, color, religion, gender, or national origin and prohibited racial segregation in schools, employment, and other public accommodations. The enactment of the Civil Rights Act followed that of the Equal Pay Act of 1963 (U.S. Department of Labor, 1963), which prohibited wage differentials based on gender, a clear indication and foreshadowing that the passage of a well-intentioned law does not necessarily bring about the proposed change. Shortly after the passage of the Civil Rights Act, a law banning age-based discrimination in employment was also passed (Age Discrimination in Employment Act of 1967). Certainly, it was a time when walls representing traditional and biased grounds for making decisions were crumbling. In fact, Nancy Cole (1993) stated, Test and item bias concerns in their modern form grew out of this era, were responses to it, were influenced by it, and took their role as a standard part of the enterprise because of it (p. 25). Indeed, since the 1960s, the testing field has reacted responsively to the need to assess whether tests and test items have bias. In fact, most testing agencies have built into their test development and evaluation routines a check to ensure that their tests and the items or other components constituting the tests are fair. Indeed, those testing practices were primarily influenced by laws and court cases related to testing.

At least in part in response to the Civil Rights Act of 1964, psychometricians and testing experts began working on psychometric approaches to fairness and bias. Some early books (e.g., Samuda, 1975) discussed the cultural bias of tests when used with ethnic minorities; others (e.g., Jensen, 1980) suggested that racial differences were genetic. Testing professionals sought psychometric models to determine if tests were fair. The early models were related to criterion-related validity models appropriate for some cases of job selection and promotion and higher education admissions (i.e., Berk, 1982; Reynolds & Brown, 1984). The first model was the Cleary model, developed in large measure by T. Anne Cleary (1968). It considers that a test is fair for two subgroups of a single population if there are no systematic over- or under-predictions for the members of either of those subgroups when using a single prediction equation. In other words, the proportions and extents of over- and under-predictions for each group must be essentially equal. Of course, until the 1960s, the traditional (some might read, liberal) belief was that members of ethnic and other underserved groups would be largely underpredicted. It was a common belief that standardized tests based on majority culture in our country would suppress the scores of such underserved and culturally different groups. The model attempted to ensure that all individuals were treated equivalently, regardless of their racial, ethnic, gender, or other protected group status. Others (e.g., Peterson & Novick, 1976) have referred to the Cleary model as the regression model because it checks that slopes, intercepts, and standard errors of estimation are the same for both groups if the test is fair. Hunter et al. (1984) found virtually all tests to be fair to minority groups using the Cleary approach.

However, there was a complication with the Cleary model. First, many ethnic and protected groups scored lower on cognitive educational and psychological measures. While they also often scored lower in higher education grade-point averages (the most common educational criterion in higher education admissions studies) and on indices of the quality of job performance, they scored better on these criteria, relatively speaking, than they did on the predictor tests. (This is the classic case of regression to the mean.) R. L. Thorndike’s (1971) approach to the determination of fairness was based on the above facts. According to Thorndike, the qualifying scores on a test should be set at levels that will qualify applicants in the two groups in proportion to the fraction of those two groups reaching a specified level of criterion performance (p. 63). That is, he pointed out that given the typical differences on the predictor variable while members of both subgroups would meet minimal expectations to be accepted for advanced education or hired, the proportion of such identified persons would typically be much lower in the lower performing group on the predictor than in the higher performing group. Moreover, the proportion of protected group members (that is, members of the lower scoring group) who met the standard on the criterion and who were accepted/hired is likely to be proportionally lower than the proportion of the higher-scoring group members who both meet the same standard and are accepted/hired. He proposed a different approach to assure that a standard or cut score for success on the predictor be identified and then the percentage of each group that meets that standard be proportional to the percentages. In Thorndike’s words, the qualifying scores on a test should be set at levels that will qualify applicants in the two groups in proportion to the fraction of the two groups reaching a specified level of criterion performance (p. 61). Thus, if 60% of the majority (higher scoring) group meet this standard on the predictor and 40% of the minority (lower scoring) group meet that same standard on the predictor, then 60% of the majority and 40% of the minority group would need to be hired or accepted. Thorndike argued that the Cleary model is fair to each individual, but not to individuals from lower scoring groups. Similarly, it might be argued that Thorndike’s approach is fair to the groups but not to all individuals. His model has been referred to as the Constant Ratio model.

A third approach, similar to R. L. Thorndike’s, was presented by Cole (1973). Cole proposed a model for culture-fair selection based on the conditional probability of being selected given satisfactory criterion performance; it has sometimes been referred to as the Conditional Probability model. Cole’s model used similar assumptions with, and the approach of, Thorndike’s model, by dichotomizing applicants into those ultimately successful and those ultimately unsuccessful. Thorndike and Cole were both concerned with the relative proportions of the two applicant groups (at that time invariably thought of as Black and White). Thorndike focused on that proportion of each group selected by the test, advocating that it should equal the proportion of the group who would succeed on the job or be accepted into a selective educational program. Cole, on the other hand, looked first at the part of each group that succeeds on the job or in the educational program and suggests that the probability of being selected in that group is the same as the proportion successful for both minority and majority groups. Thorndike and Cole both found that the Cleary model was fair to each individual but was unfair to minority and underrepresented groups, because a smaller percentage of minorities would typically be selected by that procedure than either of theirs. Thorndike’s model deviates moderately, and Cole’s more so, from the traditional one (Flaugher, 1974, p. 14). Thus, following Cole’s model, if in a college admissions situation, the critical value of grade-point average is a C average (2.00 on a 4.00 scale), then the proportion of the majority and the minority groups should be proportionate to the percentages of the two groups meeting that C standard. In such an instance, the score on the predictor variable may be different across the two groups. Research on both the Thorndike and Cole approaches demonstrates that a fair test according to the Cleary model may not be fair to members of underserved groups using these other approaches, and vice versa. The difference, again, is that Cole looks initially at who would succeed on the job or in the classroom, whereas Thorndike examines who would succeed on the predictor test and infers that they would then succeed on the job or in the classroom. Both follow the same assumptions (Flaugher, 1974); they just apply different approaches and, ultimately, Cole’s model differs more from Thorndike’s model than from the Cleary model, given that the Cleary and Thorndike models are ultimately based more on the predictor test than the criterion.

The model proposed by Darlington (1971) differs even more from the previous models. He begins with the assumption that there is no objective, commonly accepted definition of fairness; the above three psychometric models of fairness should make that point clear. He argued that fairness is a value decision, and one must be explicit in announcing one’s values. In a sense, he then corrects criterion values for the lower scoring group. This correction can vary in its size or impact, depending on the values of the individuals making the decisions. That is, if one does not value diversity in the workforce, one can give it no value at all. However, if one does value diversity, then one can use a sliding scale to decide how much to adjust a critical criterion value for higher education acceptance or hiring of individuals. This step makes the approach much like affirmative action. Thus, to diversify a student body, majority students would be required to score on a predictor test such that they would minimally earn a C+ average in college. On the other hand, the protected group members would be required only to score on the predictor test such that they would be predicted to earn a C average in college. While these two values, C+ and C, are purely hypothetical, they demonstrate the use of the Darlington procedure. Flaugher (1974) presents a second way to understand the Darlington approach. One can perceive it using a single standard of performance (e.g., grade-point average) but be willing to accept a higher risk in achieving criterion performance in the underserved group. Flaugher also points out that Darlington still does not provide a definition of fairness, he simply permits weighing factors differently for each group. It might be noted that some (perhaps many) people consider this approach as one that essentially gives points on the predictor test to members of underserved groups and these people consider this approach unfair. Others see it as an attempt to provide balance to a world that is inherently biased to those same benefiting groups. It was later declared illegal to use different passing scores for different racial, ethnic, and gender groups.

There are other models of test fairness that have not been presented in this chapter, including those that apply decision-theoretic models to fairness approaches, and others. In some cases, such as Darlington’s and the decision-theoretic models, those making decisions are told to set values for different types of decisions. Examples include it being more desirable for a minority candidate to be hired but fail on the job than hiring a majority group candidate who also fails. Similarly, hiring a minority candidate who succeeds might be valued twice as highly as hiring a majority candidate who succeeds. While these values demonstrate the goals of having a successful workforce as well as a diverse one, it is truly difficult to quantitatively decide how much more valuable one type of decision is over another. Legally, being explicit about such weightings could be highly problematic. For this reason, these values-based models have not been used frequently, and the goals of achieving successful students and workers as well as diverse workforces and student bodies are left undefined precisely. The Bakke decision (Regents of the University of California v. Bakke, 1978) held that minority or protected group status could be used as a factor in decisions, as long as quotas or explicit numbers or percentages of seats are not put aside for such individuals.

The discussion of models of test fairness was one of the primary topics of professional conversation in measurement journals and at conferences during the 1970s. While some of these models were quite elaborate psychometrically, the Equal Employment Opportunity Commission (EEOC; 1978) took a very different approach, as their goal was to ensure that discrimination in employment did not occur. Their model perhaps most closely approximates R. L. Thorndike’s model, I believe, and is described in the Uniform Guidelines on Employee Selection (EEOC, 1978; hereinafter, Uniform Guidelines), which appeared after and were to some extent based on the first major testing discrimination case that led to a Supreme Court decision (Griggs v. Duke Power, 1971). These guidelines begin by determining whether a test causes adverse impact on a legally protected group, a more neutral word than discrimination. The primary definition of adverse impact is based on what has been termed the 4/5ths rule (Arvey & Faley, 1988; Hartigan & Wigdor, 1989). That is, if less than 4/5ths of a particular protected group fails to achieve the cut score (e.g., the score above which one is hired or potentially hired) relative to the majority group, then adverse impact against that protected group has occurred. An example sometimes makes this rule of thumb clearer. Assume that 50% of the majority group passes an employment test. If less than 40% (4/5ths of 50%) of any protected group fails to pass the test, then the 4/5ths rule is established for that group. While the 4/5ths rule has continued to be used to this day, other definitions have sometimes been used in litigation. Examples of these rules of thumb include statistically significant differences across the means, and significant differences in the proportions of groups found acceptable, as well as less statistical approaches. A finding of adverse impact against an employer places the burden of proof on the employer, who must demonstrate subsequently that the test is job related. Some members of the psychometric profession do not like the 4/5ths rule, believing that it is not consistent with the fairness literature, and have suggested the use of differential item functioning, as described later in this chapter. The so-called Golden Rule procedure of excluding items that demonstrate group differences on tests has been similarly rejected by most psychometricians (Geisinger, 1988; Zieky, 2011). The Golden Rule was a decision made by the then president of Educational Testing Service based on potential litigation when Blacks in a state disproportionately failed a licensure test for insurance sales. The decision was to drop all items that lead to substantial differences, whether or not they test valid and necessary knowledge.

Once adverse impact is found in a litigious situation, then the defendant must demonstrate that the test is valid (job related). The Uniform Guidelines lay out specifically how to defend tests by demonstrating validity. The Uniform Guidelines have separate approaches described in terms of the types of validity that were consistent with earlier versions of the Standards (1974, 1985). That is, the usefulness of the test in question can be validated using a criterion-related strategy, a content-related strategy, or a construct-related strategy; at the time the Uniform Guidelines were written, validity was understood as occurring in three forms: construct validity, content validity, and criterion-related validity. While these terms have faded into obscurity in the more current Standards (1999, 2014), as data evidence strategies they still represent approaches to justifying the use of a test. While the current (2014) Standards present a unitary view of validity, the Uniform Guidelines rightly note that different kinds of evidence are needed to justify different types of decisions that are made in part on the basis of test scores and their interpretations. Thus, it would be generally inappropriate to justify the use of an achievement test to determine whether students are learning material in a statewide assessment of student progress with a criterion-related validation strategy. On the other hand, it is appropriate to justify a college admissions test based on criterion-related validity evidence.

Internal Evidence of Bias

Factor Analytic Results

One of the most important questions regarding evidence that a test is measuring equivalent constructs across racial, ethnic, gender, and other groups is answered using factor analysis. The test responses for different groups are factored separately, and their results can be compared using either exploratory or confirmatory factor analysis methods. According to Scheuneman and Oakland (1998), this work has been performed primarily using tests of intelligence. Reynolds (1982) argued that such a result is strong evidence that the constructs being assessed are highly similar across groups. That is, the constructs would appear to be similarly meaningful across groups. Examples where intelligence factor-analytic results were found to be reasonably consistent across groups, but with some differences, may be found in Kaufman (1975) and Sandoval (1982). Such analyses have been performed for many personality tests too, but the results are generally less consequential. Geisinger (2005), for example, concluded that the Big Five personality traits (Costa & McCrae, 1992) have been replicated quite well internationally in many Western countries and across subgroups in the United States, but that in many Asian countries there is a sixth trait related to Chinese Traditions/Communal Behaviors (Cheung, 1996; Cheung & Leung, 1998).

Differential Item Functioning

With approaches to estimating overall test bias occurring immediately after the advent of the Civil Rights Act of 1964, the study and application of differential item functioning (DIF) techniques emerged shortly thereafter, around the late 1970s and early 1980s (Cole, 1993; Dorans, 1989; Osterlind & Everson, 2009; Zieky, 2011). There were probably two reasons why these procedures emerged. First, the test fairness models that were already presented (e.g., the Cleary model), generally presumed that the test was a selection measure being evaluated using criterion-referenced evidence of validity, and the study of fairness was actually the study of the fairness of predictions. Such attempts are especially true in the case of tests justified on the basis of content evidence of validity, such as achievement tests used in the schools where there is no clear criterion. Second, there is a need to remove items on tests that may be the inappropriate cause of differences across groups in overall scores. The 1999 Standards offered the following justification for the need for differential item functioning techniques: Differential item functioning exists when examinees of equal ability differ, on average, according to their group membership in their responses to a particular item (p. 25). Since the beginning of the study of DIF techniques, there has been a huge proliferation of research studies and different DIF techniques (Rogers & Swaminathan, 2016). A good number of statistical procedures have been developed to identify DIF in test items; a comprehensive description of these methods is beyond what this chapter can establish, but a brief discussion of some of the most common methods follows (Osterlind & Everson, 2009; Penfield & Camilli, 2006).

Research techniques to identify problematic items (where the performance by different groups did not parallel their overall performance on the measure or their independent estimates of skill) were initially termed item bias detection techniques. The subsequent term for the same techniques, differential item functioning, appears to have been coined by Holland and Thayer (1988); it removes the implication of bias, especially intentional bias. The study of differential item functioning techniques is due, in part, to the inability of expert panels on minority group education and similar disciplines to identify items that ultimately demonstrate performance differences between groups (Title, 1982). (It was hoped that panels of diverse experts in test subject matter and testing per se could identify items that led to differential group performance, but unfortunately, Title found that the evidence of such diagnostic efforts simply did not accrue. Much research was performed looking at gender differences in item performance as well. However, panels such as these are indeed able to identify stereotyping and use of inappropriate examples and, therefore, such panels continue to exist.) Examples of the types of items that have been found to show consistent differences between groups includes items that use negatives in the stem of the question (which often adversely affect African Americans) and items that include Spanish cognates that mean something different in English (which often adversely affect Hispanic or Latino test takers). Perhaps a dozen or so DIF methods or techniques have been identified, and some of these methods have been associated with specific psychometric models underlying the tests.

One common method is known as the Mantel-Haenszel method, after the medical researchers who developed it for disease-related purposes (Mantel & Haenszel, 1959). Their technique is based on a modified chi-square analysis, wherein each test item is analyzed according to (a) demographic membership in one of two groups (usually called the reference or majority group and the focal or minority group) and (b) whether a question is answered correctly or not. The goal of this procedure is to identify items that differentially impact one group or the other. This procedure is performed separately for a number of overall test score intervals on the test as a whole for each item, where test takers within an overall score integral group might be considered as having the same overall level of ability. The groups are compared against each other item by item in this fashion. If there are average differences across groups, this procedure handles that situation by separating groups into score intervals, such as 95–100, 90–94, 85–89, 80–84, 76–80, and so on. When two groups do not perform equally well on an item, then the item exhibits differential impact with respect to the two groups. The differences in ability distributions are accounted for by matching the groups on a relevant characteristic, almost always the total test score. This logic is based on the assumption that all of the items and the test as a whole are intended to be measures of the same underlying construct. Matching on overall ability level, and then comparing the two groups’ performance on the item, is the central concern in the study of DIF. If the overall test score is used, then once items are found to lead to DIF, those items are removed from the overall score and the analyses are repeated, at least if the DIF is extensive. It is crucial in making the distinction between differences in item p-values attributable to differences in item functioning versus differences in group ability (Dorans & Schmitt, 1991) (Michaelides, 2008, p. 10). Dividing the two groups (majority and minority) using a criterion such as the overall test score is common to most DIF methods. Today, given that many tests are scored using one-, two-, and three-parameter item response theory models, many DIF procedures have been developed that compare the item characteristic curves, often using the differences of area under the curves. Many of these procedures can be found and reviewed in Camilli (2006) or Osterlind and Everson (2009). One concern, however, about DIF procedures is that they identify items with a differential impact on either group, that is, focal or reference group. Also, to the extent that differential functioning across groups is highly uniform across items, few or no items are identified. Another concern relates to the common finding that DIF techniques identify items that differentially impact one group or the other. It is a common finding that an approximately equal number of items benefit the minority group and a highly similar number favor the majority group. Should one eliminate or change both sets of items? Should one eliminate items favoring the majority group even if they still score better on the item than the minority group? These are policy decisions that need serious consideration (Zieky, 2011).

Individuals With Disabilities

Many psychologists and testing experts believe that the first legislation affecting the testing of individuals with disabilities was the Americans with Disabilities Act of 1990. However, Section 504 of the 1973 Rehabilitation Act was actually the first disability civil rights law enacted in the United