Fairness Evidence Expectations for Educational Tests

Catherine Welch, PhD Author

04/24/2018 Added

117 Plays

Description

Webinar - Fairness in testing series

Searchable Transcript

Search:

[00:00:00.120]Good afternoon to everyone.
[00:00:01.550]I hope you're enjoying a lovely day where you are.
[00:00:04.360]It's lovely here in Iowa.
[00:00:06.000]I very much appreciate the opportunity
[00:00:08.010]to share some thoughts about fairness in testing
[00:00:10.580]and based on some experiences, both from research
[00:00:14.355]perspective as well as from a practitioner perspective.
[00:00:20.620]So it's exciting to sort of blend those two perspectives
[00:00:24.010]and share my thoughts.
[00:00:26.130]As we go forward, I want to focus the conversation today
[00:00:29.870]around three different topics.
[00:00:31.650]One, really the role of fairness,
[00:00:34.670]and that it is this fundamental issue.
[00:00:36.890]It really cuts across everything.
[00:00:38.690]It cuts across everything we do on the front end
[00:00:41.160]from a design and development,
[00:00:42.430]all the way to the use and interpretation,
[00:00:45.390]and really putting into place what we want our
[00:00:47.860]assessments to do.
[00:00:49.020]So we'll spend some time on that.
[00:00:50.599]The bulk of today will really be on practice,
[00:00:52.803]and procedures.
[00:00:54.018]Sort of what's current in the field,
[00:00:56.390]what are the standards that Jessica referenced.
[00:00:58.670]What are they telling us, what should we be doing
[00:01:00.545]and what the field is really doing that some of it
[00:01:05.453]is appropriate, but some of it falls a little short,
[00:01:09.220]and then we'll end today with more specific strategies
[00:01:11.973]on where can we go as we move forward
[00:01:14.945]as the whole field of ed measurement moves forward
[00:01:17.104]with respect to new technologies,
[00:01:20.710]new abilities to combine learning with assessment.
[00:01:23.734]What can we do from a fairness perspective
[00:01:25.789]to ensure that we continue to have good faith
[00:01:31.790]that our assessments are valid, fair, and reliable
[00:01:35.890]for our test takers.
[00:01:36.904]So that very first one is really about fairness
[00:01:40.480]as a more global concept.
[00:01:42.168]How does it cut across everything we're doing,
[00:01:44.735]what is it really that we mean when we say fairness.
[00:01:47.389]And fairness is sort of one of those words,
[00:01:49.687]it has many interpretations for different people
[00:01:52.789]as you listen to people's interpretation,
[00:01:57.692]they really, they loosely use the word fairness
[00:02:01.388]to mean validity or they use the word fairness
[00:02:04.870]to mean constructs, and the standards as just a reference
[00:02:09.530]are really a way to help us see fairness as a foundational
[00:02:13.891]essential component of testing.
[00:02:17.588]And it's really there to intersect with validity,
[00:02:21.589]to intersect with use and interpretation,
[00:02:24.336]to allows us to protect test takers,
[00:02:26.410]test users, across the whole continuum.
[00:02:30.590]From the experience of taking the test
[00:02:32.917]and the test items that they have in front of them,
[00:02:34.730]all the way though to when they get the scores
[00:02:37.388]and the use and interpretation that those scores are put to.
[00:02:40.126]That's what fairness means.
[00:02:42.330]And it really ties to validity.
[00:02:44.587]It is, as we all know, validity is allowing us
[00:02:48.140]to use evidence we have about an assessment,
[00:02:50.706]to use it in a way that we feel good.
[00:02:53.768]That the validity of the evidence is supporting
[00:02:56.734]that particular use or that particular claim
[00:02:59.085]about an assessment.
[00:03:01.054]Well fairness is an integral part of that validity argument.
[00:03:03.530]It has to be fair in order to be valid.
[00:03:07.222]So many of us maybe learned as we were measuring
[00:03:11.900]students to think about well you have to be reliable
[00:03:16.355]in order to also be valid,
[00:03:19.098]but you can be reliable by itself but not be valid.
[00:03:22.420]And I think about fairness in that same way.
[00:03:24.590]In order to say something is valid,
[00:03:27.770]it has to be fair as well.
[00:03:30.430]And overall, good assessments and the design
[00:03:32.333]and development of good assessments is really to
[00:03:34.804]create an opportunity for that test taker
[00:03:37.323]to really demonstrate their performance,
[00:03:40.069]whether it be through a multiple choice test,
[00:03:44.121]whether it be an essay, whether it be a performance
[00:03:46.418]assessment, whatever it is, really we're trying to create
[00:03:49.074]an opportunity for the test taker to really demonstrate
[00:03:54.023]what they know or can do about a particular subject.
[00:03:57.801]And so by taking fairness into account,
[00:03:59.961]we are enabling that to happen in a better,
[00:04:02.796]more complete way.
[00:04:05.541]It's also I think fairness is a shared responsibility.
[00:04:11.256]Too often I think in my world we have people
[00:04:14.910]that focus on content and then we have people
[00:04:16.790]that focus on psychometrics, then we have people
[00:04:18.952]that focus on the delivery of the test.
[00:04:21.380]And often times these three entities on the front end
[00:04:25.119]have very independent and sort of siloed approaches,
[00:04:29.890]but what makes it really critical is when they integrate
[00:04:32.870]with one another, and they talk to one another
[00:04:35.106]and knowing that the psychometrics and the decisions
[00:04:38.010]I'm making there are affecting the content
[00:04:40.774]and how that's represented.
[00:04:42.870]It's also affecting what I'm doing on the back end.
[00:04:45.520]I would say that same principle applies to fairness.
[00:04:48.606]It's really a shared responsibility.
[00:04:50.455]The test developer along with the group or the
[00:04:54.367]test deliverer, along with that user,
[00:04:57.673]all three of those people or groups or entities
[00:05:00.305]have to share responsibility for what is fair.
[00:05:03.768]So I could create a test that's fair in one situation,
[00:05:07.374]but if that test user picks it up
[00:05:10.118]and uses it in some very different situation,
[00:05:13.043]then fairness of it no longer applies.
[00:05:16.610]Maybe I've done my due diligence on the front end,
[00:05:18.847]but the test user's taken it in a different way.
[00:05:21.366]And that's where we really have to think about fairness
[00:05:23.526]as an ongoing iterative process between developer,
[00:05:28.435]deliverer, and test user.
[00:05:31.410]And so it's no one particular groups responsibility,
[00:05:34.820]rather it's a shared responsibility.
[00:05:38.330]So Jessica referenced, the 2014 Standards
[00:05:40.176]for Educational and Psychological Testing,
[00:05:42.247]they really did a nice job of highlighting fairness.
[00:05:44.857]The standards that predated this were finished
[00:05:50.330]and published in 1999.
[00:05:52.220]And fairness was very much part of those standards
[00:05:55.000]as well.
[00:05:55.882]It was more in there as part of an accommodations piece,
[00:05:59.109]as far as treating special groups of population.
[00:06:03.230]Coming at different accessibility, however the joint
[00:06:07.130]committee that really created the 2014,
[00:06:10.150]listened to a lot of people between 1999 and 2014,
[00:06:15.240]listened to various voices from both the professional side,
[00:06:18.540]the credentialing licensing side, from the education side,
[00:06:21.940]from the clinical side,
[00:06:23.760]and really saw fairness as more of a foundational
[00:06:26.793]concept of an assessment than how it was
[00:06:29.970]presented in the 1999 standards.
[00:06:32.660]So therefore as the joint committee shaped the 2014
[00:06:35.990]standards, they moved it into one of these three
[00:06:38.229]foundational chapters in the beginning of the standards.
[00:06:40.885]So as you sit down, sort of better understand
[00:06:43.985]what the standards are saying,
[00:06:46.017]they're really saying again these three concepts
[00:06:49.070]of validity, reliability and fairness,
[00:06:50.920]really go hand in hand and they set the stage
[00:06:54.050]for everything that comes after that.
[00:06:56.004]I think there's some critical messages
[00:06:58.072]in particular in chapter three of the Standards,
[00:07:03.660]where they're emphasizing the idea of fairness.
[00:07:06.167]And some of these again are shared beliefs,
[00:07:08.287]and some sort of take on different interpretations
[00:07:11.529]depending on what field or what aspect
[00:07:15.038]of assessment you're coming to the conversation with.
[00:07:18.770]So first and foremost, fairness is not the same as bias.
[00:07:22.690]Fairness is really about accessibility.
[00:07:25.660]It's about creating an opportunity in an assessment
[00:07:28.754]that allows that opportunity not to advantage
[00:07:33.526]or disadvantage a certain group of individuals,
[00:07:37.797]whether they be students that are taking it online
[00:07:40.228]versus those that are taking it paper and pencil.
[00:07:43.000]Whether it be females or males,
[00:07:44.760]whether it be english language learners or
[00:07:46.500]not english language learners.
[00:07:48.461]It is about accessibility for those students
[00:07:51.460]and trying to minimize anything that may
[00:07:53.818]advantage or disadvantage a certain group.
[00:07:57.580]So but that is different than bias.
[00:07:59.984]The other part is accessibility to the construct,
[00:08:03.907]and this bullet, the construct meaning
[00:08:06.103]that I have designed a test where what I am measuring
[00:08:10.435]is really what I intend to measure.
[00:08:12.980]So my definition of the construct,
[00:08:14.695]my definition of what it is I want to be able
[00:08:18.060]to say about this test taker
[00:08:19.890]after he or she completes this test
[00:08:25.330]that ties to the construct and what I have
[00:08:28.197]intended to measure through my construct.
[00:08:31.660]So giving students equal access to that construct
[00:08:35.082]is fairness.
[00:08:37.360]Fairness is also about the validity of individual
[00:08:39.808]test scores, and the interpretations.
[00:08:42.831]And we know that once we give a test,
[00:08:45.590]that test score sort of go and they migrate
[00:08:48.490]into a lot of different,
[00:08:50.180]or with a lot of different audiences.
[00:08:51.554]I often times as a person who works
[00:08:55.430]with educational testing, I know that an audience
[00:08:57.497]can be that individual test taker,
[00:09:00.780]who might be a kindergartner, also the parent
[00:09:03.560]of that individual test taker,
[00:09:05.130]I know it's also the teacher, the policy maker,
[00:09:09.220]all the way up to perhaps the state's department
[00:09:13.590]of education, so all of those are different
[00:09:16.367]audiences for me,
[00:09:18.508]and all of them bring with them different interpretations
[00:09:21.166]and uses.
[00:09:22.785]So I want to have messages that are valid all the way from
[00:09:23.784]the kindergartner all the way through the
[00:09:26.616]state policy maker.
[00:09:28.903]And fairness again, transcends all of those various
[00:09:33.450]audiences.
[00:09:34.620]I want what I say and the metrics and the messages
[00:09:37.760]that I provide to these various audiences
[00:09:39.793]to all have been addressed for issues of fairness,
[00:09:43.756]as well as other related validity issues,
[00:09:47.174]and other interpretations, to be technically sound
[00:09:49.874]and so forth, from a psychometric perspective.
[00:09:52.934]But really, to think about fairness as just as important
[00:09:55.998]as the integrity of the data.
[00:09:59.193]And the last one here is really a critical point,
[00:10:02.610]that fairness is not really saying there's no
[00:10:05.720]variability in the groups.
[00:10:07.820]Fairness is not the same as all groups,
[00:10:10.307]all test takers walk away with the same average score
[00:10:13.141]or the same results.
[00:10:15.210]Otherwise there would be sort of no point in assessment
[00:10:18.094]if we knew what the end result was going to be.
[00:10:20.743]Rather fairness is about leaving in the appropriate
[00:10:24.498]types of variability.
[00:10:27.410]I'm looking for variability on the construct.
[00:10:29.745]So if I'm giving a mathematics assessment
[00:10:32.620]to fourth grade students, and some students really
[00:10:35.330]have mastered it and other students are really
[00:10:37.889]struggling with it, and those groups are defined
[00:10:42.364]in different ways, that's okay,
[00:10:45.320]because the construct is measuring the math standards
[00:10:48.332]in that state.
[00:10:49.990]I have not put something into the measurement
[00:10:52.370]of that construct that would differentiate
[00:10:55.845]those groups on group membership
[00:10:57.690]or some other group associated variable
[00:11:00.930]that is not of interest to me when I'm measuring
[00:11:03.676]fourth grade mathematics.
[00:11:05.745]So again, one of these issues, one of these concepts
[00:11:07.680]that people sometimes put together is
[00:11:10.920]well if it was really fair, wouldn't we all be doing
[00:11:13.441]the same, and that's not true.
[00:11:15.555]What we're saying is we're trying to eliminate
[00:11:17.629]any variables that contribute in ways
[00:11:21.230]that we don't want them to contribute.
[00:11:23.690]We want to get to the core of the construct
[00:11:26.222]that we're trying to measure and really hold that
[00:11:28.833]as fixed, and is constant across the entire
[00:11:31.757]test taking population as we possibly can.
[00:11:35.671]The other thing about fairness is it's going to vary.
[00:11:38.956]How I define and how I articulate and the evidence
[00:11:43.419]I bring to fairness is really going to be defined
[00:11:46.564]by my purpose.
[00:11:47.911]So if I'm assembling and developing an assessment
[00:11:52.330]for an end of course assessment in a district
[00:11:53.918]for high school biology students,
[00:11:58.037]how I go about defining fairness is going to be
[00:12:02.852]directly related to the standards and the curriculum
[00:12:06.271]and the instruction that those students received
[00:12:08.881]as part of their biology course in that district.
[00:12:14.440]That might, what I do there might be a different way
[00:12:16.264]to look at fairness than if I was looking
[00:12:20.340]at the state's 10th grade assessment that cuts across
[00:12:23.600]all content areas.
[00:12:25.950]So purpose helps us define the type of audience,
[00:12:29.465]or sorry, the type of evidence that we really want
[00:12:32.590]to bring to this.
[00:12:33.970]Okay so purpose is really sort of that first segue
[00:12:37.380]in as I articulate what I'm trying to do,
[00:12:39.710]just like you would when you define a test
[00:12:42.140]and the validity of the test,
[00:12:43.480]and you outline all the steps you're going to put
[00:12:45.689]into place to validate that assessment.
[00:12:48.540]Fairness has to be part of that conversation.
[00:12:50.840]What am I going to do from a fairness perspective
[00:12:53.700]to help me guarantee that the end result is valid
[00:12:57.960]across all test takers?
[00:13:00.960]The end from a measurement perspective,
[00:13:03.320]sort of I have two little competing priorities.
[00:13:07.743]I'm trying to maximize access,
[00:13:08.890]and so much of what the new technologies
[00:13:12.180]and the new sort of artificial intelligence,
[00:13:14.970]the new learning integrated assessment,
[00:13:17.434]all of those things, everything our field is pushing forward
[00:13:21.820]on is about maximizing accessibility.
[00:13:23.765]And those are all positive things.
[00:13:26.012]I want to be able to get in front of the student,
[00:13:29.664]the test taker, with the appropriate assessment.
[00:13:32.312]At the same time I'm trying to minimize,
[00:13:34.878]what we'll talk a little bit more that term
[00:13:37.173]construct irrelevance.
[00:13:39.153]The things that are no longer relevant.
[00:13:41.640]The things that are not tied to my fourth grade math test.
[00:13:45.710]The things that are not an important indicator
[00:13:48.602]of whether that student has mastered fourth grade
[00:13:50.901]mathematics or not.
[00:13:52.750]All of those other things that are not relevant
[00:13:54.635]ar those that I'm trying to get rid of,
[00:13:57.340]I'm trying to minimize, and I do that again
[00:13:59.900]through a system of checks and balances,
[00:14:02.509]practices, processes, and documentation
[00:14:06.066]that will allow me to then say with some confidence,
[00:14:09.033]I have minimized those that I'm aware of
[00:14:12.902]and built a test with that in mind.
[00:14:17.750]Now the standards sort of present a great,
[00:14:20.059]in my mind, global approach.
[00:14:25.010]They approach various audiences,
[00:14:26.808]they come at it very much from both the practitioner
[00:14:30.110]and a researcher perspective.
[00:14:31.623]I wanted to add just a couple of other sources here
[00:14:34.504]to help us put a little context.
[00:14:36.870]Because fairness is being addressed in other ways as well.
[00:14:39.800]So the standards are out there as sort of our
[00:14:42.450]guiding set of processes,
[00:14:44.811]but they also get translated into more specific things,
[00:14:48.740]again, given the purpose of an assessment.
[00:14:50.931]So if you're familiar with the Council of Chief
[00:14:53.410]State School Officers,
[00:14:56.090]and recently they have created sort of a high quality
[00:14:59.438]assessment document that goes through the steps
[00:15:02.583]that they are saying that districts and states
[00:15:06.003]should put into place to be sure that they have
[00:15:09.820]an assessment that is aligned to
[00:15:11.136]college and career readiness.
[00:15:12.895]And so this is a very specific audience.
[00:15:14.959]This is ties most generally to assessments
[00:15:19.324]that are being used for Every Student Succeeds Act,
[00:15:22.690]or Precursor No Child Left Behind,
[00:15:25.110]but really coming at the college and career
[00:15:28.200]readiness is the construct I'm trying to measure.
[00:15:31.700]So whether I'm in sixth grade or eighth grade
[00:15:33.780]or tenth grade here, it's really about defining
[00:15:36.780]and measuring that skillset that allows me
[00:15:39.126]by the end of high school to say I am prepared
[00:15:41.737]to go on and successfully take college level coursework.
[00:15:45.696]So as part of this commitment to high quality assessment,
[00:15:49.710]CCSSO in their documentation and in their guidelines
[00:15:53.170]have highlighted fairness as what is essential
[00:15:56.633]for this high quality assessment.
[00:15:59.590]And I just pulled up here point number four
[00:16:03.120]from their larger document and references for these
[00:16:05.495]will be in the slides that you receive from Jessica,
[00:16:09.100]but again, following something we'll talk about
[00:16:11.730]in greater detail, but the principles of universal design,
[00:16:14.315]and those assessments being developed in accordance
[00:16:17.620]with these principles of universal design
[00:16:20.970]and sound testing practice,
[00:16:22.770]so that the test, regardless of whether it's online,
[00:16:25.340]regardless of whether it's a paper and pencil,
[00:16:28.270]just not somehow interfere with how that student
[00:16:32.290]is taking that test.
[00:16:34.140]The second point being the appropriate accommodations
[00:16:36.905]and modifications are offered to students,
[00:16:39.650]so defining what the allowable accommodations
[00:16:43.826]are that again maintain the construct,
[00:16:45.770]I'm not talking about changing the construct.
[00:16:47.217]I'm talking about allowing students to have the
[00:16:50.085]appropriate accommodations so that they
[00:16:52.560]can access the construct.
[00:16:54.490]And again, the construct in this case is college
[00:16:56.349]and career readiness.
[00:16:58.030]So as states and districts go about building assessments
[00:17:00.366]that are meant to measure college and career readiness,
[00:17:06.487]really they are being tasked with accessibility
[00:17:10.807]through universal design and through
[00:17:12.960]accommodations and modifications.
[00:17:14.951]Second sort of user and again showing up in their
[00:17:20.450]documentation, if you work within a state,
[00:17:23.890]knowing that there's a peer review process
[00:17:26.094]that every state must go through in order to begin
[00:17:31.621]sort of continue No Child Left Behind,
[00:17:34.031]until it's replaced with Every Student Succeeds Act,
[00:17:36.952]but as states put together plans that define
[00:17:38.747]their assessment system, and not just their summative
[00:17:40.940]assessment system, but cutting across all of their
[00:17:43.671]interim assessment systems as well,
[00:17:46.204]again, the peer review process is pulling out
[00:17:49.332]and saying to states that you must have critical evidence
[00:17:52.723]as far as fairness and accessibility.
[00:17:56.040]So in this case, 4.2 of their document,
[00:17:59.093]the state has taken reasonable and appropriate steps
[00:18:01.731]to ensure that it's assessment
[00:18:04.024]are accessible to all students
[00:18:06.160]and fair across student groups in the design,
[00:18:09.430]develop, and analysis of these assessments.
[00:18:12.470]So again the federal government in this case for that
[00:18:15.730]funding is saying you have to show me evidence
[00:18:18.317]that these assessments are fair and accessible.
[00:18:21.693]And it's not just enough to say yes,
[00:18:24.290]I built them to be fair and accessible,
[00:18:26.540]rather the federal government peer review
[00:18:29.400]is asking for specific evidence in this case,
[00:18:32.684]just following through with that same as a state
[00:18:35.586]sort of creates their response to peer review,
[00:18:39.748]including things like descriptions of processes,
[00:18:42.880]including things like samples of the items,
[00:18:45.470]including things like the instruction,
[00:18:47.650]the directions, the accommodations that are allowable
[00:18:49.795]online.
[00:18:51.620]Showing this evidence to the peer review committees
[00:18:55.140]is essential in that peer review committee
[00:18:58.210]then saying yes, you have evidence here of that very
[00:19:01.360]critical element of fairness and accessibility.
[00:19:04.850]Because void of that, you're not going to receive
[00:19:07.190]the peer review sort of that you met the requirements
[00:19:11.576]of peer review for that assessment to be given
[00:19:14.710]in your state.
[00:19:15.930]And the same sort of level of rigor,
[00:19:18.230]that same sort of accountability
[00:19:20.089]should be what cuts across our entire profession.
[00:19:23.585]It shouldn't just be the statewide assessments.
[00:19:25.796]It should be the same evidence, the same documentation,
[00:19:29.620]the same rigor is required whether I'm creating
[00:19:34.340]a district level assessment, whether I'm creating
[00:19:36.730]an interim assessment for diagnostic purposes
[00:19:39.174]for prediction purposes.
[00:19:41.170]Regardless of what I'm doing, the same principles
[00:19:44.484]of fairness and the same best practice should apply.
[00:19:48.540]Meaning I should be able to go in and look for
[00:19:50.930]those descriptions, that documentation,
[00:19:53.784]show me those exemplary test items,
[00:19:56.270]show me those directions, show me those accommodations,
[00:19:58.867]that help convince me I have created a test
[00:20:03.020]that is both fair and accessible.
[00:20:05.977]Now it all ties together because really at the end of
[00:20:10.820]the day, we're about assessment, right?
[00:20:13.586]We're trying to create student achievement indicators
[00:20:16.088]that are reliable, valid, and fair.
[00:20:20.020]And this information, it has become more and more,
[00:20:22.717]I've been in the field about 40 years now,
[00:20:26.220]it's become more and more a fundamental part
[00:20:28.536]of how we evaluate.
[00:20:30.477]It's often times used as a significant source
[00:20:35.069]as we argue for funding, it's a significant source
[00:20:38.500]of information for parents,
[00:20:40.210]it's a significant source of information for
[00:20:42.284]residents within a community, states within the
[00:20:45.323]United States, it's a significant source of information
[00:20:48.355]as we compare the United States for international
[00:20:50.559]comparisons.
[00:20:54.150]So it continues to be a very significant source
[00:20:57.292]of information in our society.
[00:20:59.277]And so with that, you create significance
[00:21:01.594]and really that visibility, I think even more so now,
[00:21:06.231]pushes us as test developers and test users
[00:21:08.345]that we have to define and guide everything we do
[00:21:13.530]towards the appropriate use and interpretation.
[00:21:16.820]'Cause without that, I think we have a lot of different
[00:21:20.100]sort of things going on that we don't have confidence in,
[00:21:23.830]we don't have faith in, we cannot compare across.
[00:21:26.450]And yet at the end of the day, we want to use this
[00:21:29.050]information in a way that helps inform,
[00:21:31.475]helps shape, help sort of impact where we go next.
[00:21:35.890]And so we must balance the complexities of all the
[00:21:39.295]validation work that we all know so well,
[00:21:42.800]and know that we have metrics and information coming
[00:21:45.211]out of assessments,
[00:21:46.566]but we must value that all of that work
[00:21:49.113]including the fairness and the communication
[00:21:51.898]that we have to bring with it to students,
[00:21:54.481]educators, and policy makers.
[00:21:56.850]We must have that information, that communication down
[00:22:00.630]so that we have convinced in a very real way,
[00:22:03.444]policy makers and educators that what we're doing here
[00:22:06.649]is contributing to the greater picture.
[00:22:10.320]We are helping to inform a very complicated landscape
[00:22:14.282]of student achievement, but we are a valid
[00:22:17.104]and fair contribution to that.
[00:22:19.840]And I think that's important.
[00:22:21.440]So that assessment of course relies on our desire
[00:22:23.888]and our sort of requirement to measure,
[00:22:29.080]and that measurement is really where we make
[00:22:31.650]all these decisions about what it is we're going to
[00:22:34.610]put into this assessment.
[00:22:36.210]What is the information that we will get out
[00:22:37.875]of that assessment, so everything we do from design,
[00:22:41.941]development, review, forms assembly,
[00:22:45.193]until we get it in the hands of the person that's using it
[00:22:48.175]is really that measurement component.
[00:22:51.280]And every aspect of that needs to be checked for fairness.
[00:22:54.663]It's not just I want to look at the items
[00:22:56.584]and then if the items look good, I'm good.
[00:22:59.196]No we must check every potential step along the way
[00:23:02.931]to ensure that we have again, faith in the results,
[00:23:07.550]because we know these results are going to be used
[00:23:09.800]in significant ways.
[00:23:12.860]And what does that mean every step along the way.
[00:23:15.070]Well when we get in the second part here,
[00:23:17.100]we'll talk in greater detail.
[00:23:18.570]But I need the conception of the test itself
[00:23:21.603]through the definition of the test specification
[00:23:25.385]knowing those test specifications are pulling from
[00:23:28.045]that target domain, what it is that I'm trying to say
[00:23:32.481]about a student that reflect those content standards
[00:23:36.978]or curriculum standards, or the other essential skills
[00:23:40.824]that are important for the decisions being made,
[00:23:43.477]that are then trying to generalize to if you do well here,
[00:23:45.668]you will be successful there, all the way to the
[00:23:50.180]construct itself.
[00:23:51.520]So knowing my test is really trying to make an inference
[00:23:54.238]back about that construct.
[00:23:56.540]And we know testing is not perfect.
[00:23:58.640]We know that testing is as best sort of one more
[00:24:02.010]piece of this complicated puzzle,
[00:24:05.354]but that shouldn't stop us from striving to make
[00:24:06.980]that test, and every step along the way,
[00:24:09.527]as fair and as transparent as possible.
[00:24:14.100]We don't want to just sort of wave our hands
[00:24:15.782]over things and say don't worry, I've got it,
[00:24:18.070]I'm the test developer.
[00:24:19.080]Rather we need to convince people with real data
[00:24:21.885]and real process that what we've done was appropriate
[00:24:24.779]and evidence based.
[00:24:27.965]And then the research design part of everything we do
[00:24:31.382]of course, it has to include the appropriate elements,
[00:24:37.270]the appropriate design components,
[00:24:39.783]the appropriate students test takers,
[00:24:42.386]so that I can design my research to have an answer
[00:24:46.047]to my questions, my inquiries about fairness.
[00:24:51.865]That needs to affect everything I do as well.
[00:24:54.710]So the structure of that design is critical to me as well.
[00:24:59.930]Okay, I'm going to stop right there and ask
[00:25:02.590]if there's any questions just sort of to that intro piece.
[00:25:06.010]Yeah thank you Dr. Welch.
[00:25:08.290]So if you have any questions at this time,
[00:25:10.500]we'll give you a few moments to enter those
[00:25:12.865]in the chat box before Dr. Welch moves on
[00:25:16.558]to talking about more test, more details in test
[00:25:19.810]development practices and fairness strategies.
[00:25:24.000]Okay, looks like we do have a question.
[00:25:26.130]Can you give an example of a testing issue
[00:25:28.320]that might have occurred in the past
[00:25:29.924]that is now thought of not to be fair?
[00:25:33.340]That is now thought not to be fair.
[00:25:42.578]I couldn't like provide a citation right now,
[00:25:46.392]but I think the field as it has rushed into
[00:25:50.405]technology enhanced items, for the last 15 years or so,
[00:25:56.270]the field has tried to create technology enhanced items,
[00:25:59.760]and what I mean is just more of an innovative
[00:26:02.210]approach to online items that capture student responses
[00:26:05.760]that can then be scored through artificial intelligence,
[00:26:08.880]and I think what that has caused.
[00:26:12.439]There was such a rush because they were felt
[00:26:14.261]to be better measures of the construct.
[00:26:17.522]And so by putting these out in front of students,
[00:26:21.650]they felt like the measurement was better
[00:26:23.372]on the front end, from a content domain perspective.
[00:26:27.090]But what we're seeing now is more and more group
[00:26:30.630]differences on those items.
[00:26:32.420]And so I think now the literature as we step back,
[00:26:35.193]and it cuts across personal devices,
[00:26:39.532]how the items are presented,
[00:26:41.882]the structure of how much text versus how much graphic
[00:26:45.130]is presented to the students,
[00:26:47.720]those are all showing differences from a differential
[00:26:50.146]item functioning that's now putting a little bit more
[00:26:53.630]of a hold on what we thought was good from a content
[00:26:56.908]perspective.
[00:26:59.142]So I think that is part of our field
[00:27:02.822]that is now sort of stopping and trying to catch up
[00:27:05.588]with itself from the fairness perspective
[00:27:07.985]to catch up to the technology.
[00:27:10.188]And everybody rushed forward thinking technology
[00:27:12.486]in and of itself is good, but now that the fairness checks
[00:27:14.745]have caught up, we're seeing differences that are probably
[00:27:19.364]unfortunately looking at things that are not
[00:27:23.474]related to the construct itself,
[00:27:25.628]related more the student's ability to maneuver
[00:27:29.210]through online delivery systems,
[00:27:32.050]and we're seeing more of a cut on things
[00:27:34.810]like access to computers, socioeconomic status,
[00:27:37.608]and so forth.
[00:27:39.120]I think that's something that has transitioned
[00:27:40.746]just in the last few years.
[00:27:44.480]Okay, and then the next question,
[00:27:46.163]I think I'm actually going to hold off on
[00:27:48.911]because I think you're going to get to,
[00:27:50.470]but the next question was please elaborate
[00:27:52.250]more on incorporating fairness in the research
[00:27:54.188]design, and you will be talking more about that.
[00:27:56.309]There was another question.
[00:27:58.166]This might have been in reference to your response
[00:28:00.619]to the first question about being more specific
[00:28:04.512]regarding the type of item or test
[00:28:06.576]you're speaking about?
[00:28:08.938]And that is one of the difficulties of doing
[00:28:12.711]something like this is we're trying to cut across
[00:28:15.488]sort of a wide audience.
[00:28:17.276]What I'm really talking about though,
[00:28:19.734]and we'll look at a couple items here in the next section.
[00:28:23.243]That introduce both an open ended sort of question
[00:28:27.454]and a multiple choice question,
[00:28:29.168]and trying to look at how even those two item formats
[00:28:33.888]can introduce different sources of fairness issues
[00:28:37.821]in the scoring and reporting,
[00:28:40.140]in the subjectivity of that evaluation.
[00:28:43.510]And so decisions that are made in the item design
[00:28:46.667]and format at the very front end of this
[00:28:49.772]really have longer term implications
[00:28:51.897]for how that plays out.
[00:28:53.811]So I think there's a whole body of research that
[00:28:57.499]sort of follows different item formats
[00:28:59.904]through these fairness issues.
[00:29:02.017]Yeah, we probably won't get into great detail today
[00:29:05.952]just because of the nature of this.
[00:29:09.080]Yeah, sounds like a follow up webinar.
[00:29:10.959]Alright, thank you Dr. Welch.
[00:29:13.200]You can continue with your presentation.
[00:29:17.940]Thanks Jessica.
[00:29:19.750]So here in the second part, the second of there,
[00:29:23.483]I really want to focus more on those procedures
[00:29:25.363]and what we can do.
[00:29:26.807]And I want to reference as we go through here
[00:29:29.166]the actual language from the standards.
[00:29:31.112]So when I pull up like this standard 3.1,
[00:29:34.851]if you go to 2014 standards, this is the first standard
[00:29:38.800]that appears in the fairness chapter.
[00:29:43.292]And it is saying in putting responsibility for test
[00:29:47.590]development, administration, across all steps of
[00:29:50.920]the testing process should be held accountable for
[00:29:54.892]the promotion of valid scoring interpretations
[00:29:57.190]and for the widest possible range of individuals in
[00:30:01.130]relevant subgroups in the intended population.
[00:30:04.100]So when we say test development revision
[00:30:07.550]and administration here,
[00:30:09.920]and it references designing of all steps,
[00:30:12.800]I think this a part of our field that most people think
[00:30:15.377]well we've done a lot of work, a lot of research in
[00:30:21.530]different and particular steps in the test development
[00:30:24.300]process.
[00:30:25.210]So here I have a bulleted list of what you might see
[00:30:28.230]in a test development process,
[00:30:29.630]all the way from starting with the articulation
[00:30:32.500]of what it is you want to measure,
[00:30:34.570]and then designing that through the specifications
[00:30:37.390]and blueprint stages, and then my items,
[00:30:39.849]the development and the review of those items,
[00:30:42.158]and then the collection of empirical data
[00:30:44.870]on those items, whether it be through pilot testing
[00:30:47.660]or field testing, or try out whatever you want to call that,
[00:30:51.610]but the empirical collection of data,
[00:30:53.513]and then moving into whether I'm putting together
[00:30:56.912]a form or whether I'm putting together a pool
[00:30:59.385]for a computer adaptive test.
[00:31:02.473]Whatever I'm doing there, finalizing what that is
[00:31:05.407]and then administering.
[00:31:07.590]So when I think about test development,
[00:31:09.210]I think about all of these various steps.
[00:31:11.840]And this is very much a iterative process
[00:31:14.070]and different people would have different steps
[00:31:16.130]in their process, but the idea here is I would have
[00:31:20.172]at least gone through all of these in my thinking
[00:31:23.406]about the front end to the back end of the design
[00:31:28.850]of any particular test.
[00:31:30.480]So when I try to take my standard 3.1 and apply it,
[00:31:33.977]people tend to think well fairness comes up in
[00:31:37.520]item development, fairness comes up in item review,
[00:31:41.695]and again in review of the forms itself.
[00:31:43.377]And people tend to think that well I have people,
[00:31:44.873]I have experts, I have fairness reviewers that I bring
[00:31:49.376]to the process.
[00:31:50.970]They might be part of the developers,
[00:31:52.224]they might be part of the people that look at
[00:31:55.070]and critique the items both at the front end
[00:31:57.421]before I have empirical data and then again
[00:31:59.694]after I have DIFF data, or differential item
[00:32:03.730]functioning data associated with those items
[00:32:05.746]and I'm ready to put them into a form.
[00:32:07.860]So I think this is sort of the classic how standard
[00:32:10.522]3.1 plays out from a development perspective.
[00:32:15.169]And yet what I'm trying to convince you is that
[00:32:19.110]it really should be at every step of this process,
[00:32:21.416]that there are critical elements along every step
[00:32:24.981]of this process that we should be looking to
[00:32:29.709]and for with respect to fairness.
[00:32:30.660]So if we walk through a few of these,
[00:32:32.600]I hope they become a little bit more obvious.
[00:32:34.221]And again knowing that whether we're assigning
[00:32:38.110]a state assessment or a district,
[00:32:40.590]or a very very specific assessment.
[00:32:42.430]Maybe I'm trying to design an assessment in my district
[00:32:45.860]for use in identification for gifted and talented,
[00:32:48.564]or maybe I'm trying to design a test that's very
[00:32:51.203]diagnostic in purpose.
[00:32:53.180]We still all start with what it is I'm trying to do.
[00:32:56.470]What is it I'm trying to measure,
[00:32:57.940]the purpose of what I'm trying to do.
[00:33:00.080]And regardless of where that construct is being
[00:33:03.020]defined, whether it be in that curriculum,
[00:33:04.305]whether it be in prerequisite coursework,
[00:33:07.720]whether it be in content standards,
[00:33:09.444]we really need to look at that documentation itself
[00:33:12.463]as a source, but we also have to view that with
[00:33:17.022]a fairness lens.
[00:33:18.850]Was that curriculum, was that set of standards,
[00:33:20.937]was fairness taken into account in the original
[00:33:25.282]sort of source documentation that I'm going to use
[00:33:28.887]and that's going to guide everything that I'm going to do
[00:33:32.210]going forward.
[00:33:33.172]So I need to be sure that what I'm turning to
[00:33:35.384]as my source for my content validation
[00:33:39.279]is really in in of itself, also fair and accessible.
[00:33:44.180]Now many states take this on and districts
[00:33:46.410]take this one as they define their curriculum
[00:33:48.191]and so forth, but again it's just sort of a specific
[00:33:53.015]stop and take stock of that before moving forward.
[00:33:57.170]When I get into design and specification,
[00:33:59.222]I'm making real decisions.
[00:34:01.280]I'm making decisions about how much of a certain content.
[00:34:04.072]I'm making decisions about item types.
[00:34:07.460]There I'm meaning multiple choice or essay or
[00:34:10.909]innovative items.
[00:34:12.719]I'm making decisions about the balance of those.
[00:34:15.331]I'm making decisions about how long the test
[00:34:17.399]is going to be.
[00:34:18.918]I'm making decisions about the alignment.
[00:34:22.060]Am I aligning it at grade level,
[00:34:23.413]am I aligning at a standard level,
[00:34:25.664]and am I aligning at a domain level?
[00:34:29.585]I'm also making decisions about whether it will be online
[00:34:33.116]or whether it will be paper and pencil.
[00:34:36.179]Whether it will be both and I want to use them
[00:34:38.909]interchangeably, or whether it will be both
[00:34:41.013]and I want to use them separately.
[00:34:43.195]I'm making decisions about how big my pool is,
[00:34:46.944]how many different forms, and all of these decisions
[00:34:52.250]are part of my test specification step.
[00:34:54.880]And so in each of these regardless of the decisions
[00:34:58.687]I'm making, I should be thinking about it from
[00:35:01.000]a fairness perspective.
[00:35:02.490]So if my standards have articulated,
[00:35:04.957]certain standards at certain grades,
[00:35:07.618]and I could measure them both through multiple choice
[00:35:11.598]and open ended items,
[00:35:13.243]I need to make decisions about what is the right balance
[00:35:16.912]of that item type given my test taking population.
[00:35:21.618]Given my test takers, if I'm thinking about
[00:35:24.592]different characteristics of my test taking population
[00:35:27.471]for the test I'm attempting to measure,
[00:35:31.515]or attempting to build, then I need to be very aware
[00:35:34.705]and match and balance things so that again,
[00:35:39.710]I'm not bringing into question oh I've created a test
[00:35:42.151]that's way too short for instance.
[00:35:45.380]I've created a highly speeded test for my
[00:35:48.641]english language learners, or I've created a highly
[00:35:51.723]verbal based test for my english language learners,
[00:35:56.240]although the test is in mathematics.
[00:35:58.250]Have I inadvertently created something in my specifications
[00:36:01.153]by just aligning to standards without thinking
[00:36:05.327]more broadly about fairness issues.
[00:36:07.650]And so I need to be able to document exactly
[00:36:11.614]my decisions of how I went from my underlying validation
[00:36:17.200]documents or content validation documents
[00:36:20.070]to my specifications.
[00:36:22.140]And it's at this point I need to include experts.
[00:36:25.240]I need to start talking to people about length
[00:36:28.113]and mix of items, and accessibility of different
[00:36:32.903]technologies and understanding how
[00:36:35.963]every student in my test taking population
[00:36:38.690]will come to that and how they will access
[00:36:41.781]with or without difficulties in those decisions I've made.
[00:36:46.150]So as I go through this process,
[00:36:48.300]and we've all seen different sort of development processes
[00:36:51.703]written out, where I might go out and identify
[00:36:55.455]item writers.
[00:36:57.580]I'm going to provide those item writers with some training,
[00:37:00.980]with some guidance about how I want them to write.
[00:37:04.650]I'm going to then review those items by people
[00:37:08.050]that are measurement experts, content experts.
[00:37:10.774]I'm going to review them for editorial,
[00:37:14.041]for consistency with the standards.
[00:37:17.610]I'm going to then field test those items
[00:37:21.387]and hopefully have a carefully selected group
[00:37:24.235]of students to participate in that field test
[00:37:27.199]that mirror and are representative of the group
[00:37:30.840]that I'm ultimately trying to generalize to.
[00:37:33.255]I'm going to then create an item analysis and look at
[00:37:36.462]item level data to determine whether those items
[00:37:39.411]are working, and are they working appropriately
[00:37:43.800]and equally well for different groups of my test
[00:37:46.174]taking population.
[00:37:47.940]And then of course I go through reviews again
[00:37:50.610]and bring it back to the items are ultimately used
[00:37:52.360]or not used.
[00:37:55.340]So within every one of these steps,
[00:37:57.350]I can think of ways to help myself maximize fairness issues.
[00:38:02.570]I can think about my item writers,
[00:38:04.330]I can think about my reviewers,
[00:38:05.980]I can think about the characteristics
[00:38:07.916]and the breadth of what I want those individuals
[00:38:11.160]to represent in this discussion.
[00:38:13.560]So we often times tend to think of those
[00:38:15.560]as item writers that have both the access, I'm sorry,
[00:38:21.550]the knowledge of the standards and access to
[00:38:23.791]the student test taking population,
[00:38:26.410]but I might also broaden that to be sure
[00:38:29.340]that they are well versed in what I mean by fairness.
[00:38:33.000]How am I guiding these people?
[00:38:35.050]What procedures do I have in place to help these
[00:38:36.978]individuals understand the issues and the concepts
[00:38:41.500]that I'm trying to address from the item development
[00:38:43.670]process.
[00:38:44.790]So there's just many opportunities throughout that.
[00:38:47.245]In the item review then,
[00:38:49.665]we often times think of it sort of as different
[00:38:53.123]buckets of review content, where I'm looking for
[00:38:56.710]whether those items are aligned.
[00:38:58.370]Did I sample appropriately?
[00:38:59.817]From a fairness, I'm looking for topics that I should
[00:39:04.630]be avoiding, topics I should be treating with caution.
[00:39:08.089]Whether I'm doing things like avoiding stereotypes,
[00:39:10.935]whether I'm avoiding topics that are best taken
[00:39:14.209]care of in a classroom but not appropriate in a large
[00:39:18.260]scale assessment and so forth.
[00:39:19.940]And then from the physical presentation,
[00:39:22.264]universal design principles.
[00:39:23.860]Focus on students across the whole population,
[00:39:28.620]but also a particular focus
[00:39:30.200]on test takers with disabilities.
[00:39:32.211]I want to look and be sure that the principles
[00:39:35.850]of universal design that we'll touch upon in a minute
[00:39:38.036]have been applied throughout my entire development.
[00:39:42.298]Construction of the items as well as the review
[00:39:45.304]of the items.
[00:39:47.327]Alignment continues to be an issue in the review process,
[00:39:50.751]and again, alignment is that opportunity for me to say
[00:39:54.581]my item is measuring what that standard is asking me
[00:39:58.526]to measure.
[00:40:00.000]So when that standard says a student in fourth grade
[00:40:02.960]should be able to do X, Y, and Z,
[00:40:05.421]my item is measuring that.
[00:40:08.282]But I want to be sure my item hasn't gone beyond that
[00:40:10.933]or perhaps I incorporated too many other things
[00:40:15.352]from my item that it takes me off course
[00:40:18.960]from what the original standard was.
[00:40:20.900]So being sure that alignment is there
[00:40:23.680]from a content validity perspective is another key point
[00:40:27.580]to fairness.
[00:40:28.905]The principles of universal design
[00:40:31.190]that I was just referencing.
[00:40:33.038]It is again an attempt to make as relevant as possible
[00:40:36.560]what that test taker is going to be engaged with.
[00:40:41.320]So from the very get go what the very first thing
[00:40:43.930]that the test taker sees is those directions.
[00:40:46.410]Are they clear, concise, precise, things like
[00:40:50.661]the layout, the legibility, the materials,
[00:40:54.259]the presentation of the materials.
[00:40:56.700]Thing like visual materials.
[00:40:58.147]The use of graphics, the use of figures, shading,
[00:41:01.220]color, all of those things.
[00:41:03.990]Taking into account whether certain items can be
[00:41:06.082]accommodated after the fact.
[00:41:09.220]If I know I'm going to move a test from an online test
[00:41:13.090]but I also have to create a large print version
[00:41:15.511]or a braille version of that, thinking of items
[00:41:18.469]that ultimately have to be accommodated in that world,
[00:41:21.980]and understanding that I want to minimize
[00:41:25.560]complexity of language, to make it no more difficult
[00:41:28.540]than what is being required in the standards.
[00:41:31.550]So things like that fall into the larger umbrella
[00:41:34.467]of universal design, and again, captured in the development
[00:41:38.720]process, also captured again in the review process.
[00:41:43.670]Okay, standard 3.2 then says that those test developers
[00:41:47.783]that are responsible for developing tests must measure
[00:41:50.765]the intended construct, obviously,
[00:41:53.170]but also trying to minimize
[00:41:55.240]these non relevant characteristics.
[00:41:57.890]And those non relevant characteristics can cut
[00:42:00.360]across a lot of different things,
[00:42:01.740]linguistic communicative, cognitive, cultural,
[00:42:04.920]physical, other types of things that again
[00:42:07.121]have an interaction with given test takers
[00:42:10.283]and the item that I did not intend
[00:42:12.503]to be as part of that test that I've just assembled.
[00:42:19.073]When I say this, and the standards call this
[00:42:21.070]the variance in test takers scores that's
[00:42:23.090]attributed to extraneous factors.
[00:42:26.120]And those extraneous factors distort the meaning
[00:42:28.218]of the scores, and therefore have a detrimental impact
[00:42:32.600]on the validity.
[00:42:33.800]They somehow contributed in different ways
[00:42:36.618]for different groups of the test taking population.
[00:42:39.827]However if I look at it in a different way,
[00:42:43.120]in a recent publication by NCME that was edited
[00:42:47.310]by Linda Cook, Zieky defines construct irrelevance
[00:42:53.127]in three different sort of categories,
[00:42:54.860]and I think this is a nice way to think about it,
[00:42:56.916]the structure.
[00:42:58.500]He thinks about them as cognitive sources,
[00:43:00.174]those things that are not required.
[00:43:02.860]Those things that all of a sudden I put into my item,
[00:43:05.695]but they were not a requirement from what I was trying
[00:43:09.343]to measure in the first place.
[00:43:11.626]The second category, physical sources,
[00:43:13.717]things like universal design we call visual displays,
[00:43:18.711]test formatting, other things and often times
[00:43:22.370]we'll see this in the online environment,
[00:43:24.712]those things that again sort of interact differently
[00:43:28.979]for different test takers.
[00:43:30.782]And then he lists a third source, effective sources.
[00:43:33.436]Materials, topics, pieces, written materials
[00:43:38.781]that might elicit different reactions.
[00:43:41.184]Sensitive reactions, for instance, politics
[00:43:45.130]I would put into this category.
[00:43:46.970]Things that we want to stay away from again,
[00:43:48.681]from a large scale testing perspective
[00:43:50.450]we want to stay away from that.
[00:43:51.754]Handled in the classroom where we have the resource
[00:43:54.851]of a teacher there to deal with different perspectives,
[00:43:57.737]that's fine.
[00:43:59.300]But when we're dealing with large scale assessment,
[00:44:01.232]we don't know how different students will interact
[00:44:04.010]with these materials, best to avoid those things.
[00:44:07.510]So the goal here is to avoid.
[00:44:09.212]And I have just a couple examples here of
[00:44:11.678]what Zieky's calling those different sources
[00:44:14.248]of variability.
[00:44:18.069]Language for instance.
[00:44:20.070]If I'm trying to create a test here,
[00:44:22.114]and my purpose is to define proficiency,
[00:44:24.993]and I want to say whether a child has mastered
[00:44:28.838]the requirements and the standards,
[00:44:32.950]and it may be that I want something that I know
[00:44:36.030]students in my class are from the high end
[00:44:37.981]to the low end, so I don't mean to say I'm trying
[00:44:41.090]to create an easy test or a difficult test.
[00:44:43.760]So when I say it may include across the continuum of
[00:44:45.648]performance, I understand that, because my standards
[00:44:49.118]really go from there.
[00:44:51.283]They have a hierarchy to them as well.
[00:44:54.180]So but at the same time, I need to check every piece
[00:44:57.359]of information that I put into the directions
[00:44:59.787]into the item stems and into the test questions themselves
[00:45:05.237]and the stimulus materials.
[00:45:07.060]Is the sentence structure appropriate for the grade level
[00:45:09.942]and for what I'm trying to determine.
[00:45:12.270]Is the vocabulary level appropriate?
[00:45:15.212]Have I looked at the stimulus materials?
[00:45:17.590]And when I say that, I mean if I've set up a passage
[00:45:21.330]from a reading task, if I've set up a series of data
[00:45:25.330]or tables and graphics in a science test.
[00:45:28.800]Have I aligned that stimulus material back to the standards,
[00:45:32.540]to be sure that they have a readability level
[00:45:35.759]that is appropriate given the level I'm trying to assess at.
[00:45:40.040]Have I included other qualitative and quantitative
[00:45:43.730]indices of text structure and complexity,
[00:45:46.790]so that I know, I'm not creating a fourth grade
[00:45:49.410]math test that has a eighth grade reading level.
[00:45:51.552]That's a issue of fairness.
[00:45:53.257]It might work for good readers,
[00:45:55.050]but at the same time, I know in my fourth grade math class
[00:45:57.201]I have students that struggled with reading.
[00:46:00.855]So I introduced a component there that is interfering
[00:46:04.083]with my ability to assess that mathematics
[00:46:08.191]by including a high level of reading in that test.
[00:46:12.130]Okay, so the evidence might remain the same if
[00:46:15.670]I've changed my purpose.
[00:46:16.810]If I'm creating a test that's more for credentialing
[00:46:19.270]for instance, where I'm just trying to determine
[00:46:21.480]whether a person has gotten over a threshold.
[00:46:24.980]Have they been put into the category that they
[00:46:28.390]have been credentialed or not credentialed.
[00:46:31.930]In tests when I'm building there,
[00:46:33.960]then my source documentation really might be the demands
[00:46:36.790]of the occupation, the demands of the field
[00:46:39.950]I'm trying to credential within.
[00:46:42.400]I never want to add language that's beyond
[00:46:45.053]what I'm trying to credential in,
[00:46:48.180]and it seems like something that's so common sense,
[00:46:50.610]why would I ever do that?
[00:46:51.930]And yet often times when we're writing items,
[00:46:54.497]we're very concerned about qualifying things,
[00:46:58.830]being sure the key is the absolute key,
[00:47:01.520]being sure the distractors
[00:47:03.210]are good evidence based distractors.
[00:47:06.480]Understanding all of those things,
[00:47:08.450]and often times the language of test questions
[00:47:10.830]and the language and rigor of those start getting
[00:47:14.056]elevated, because I want to be sure this key is
[00:47:16.523]absolutely the correct key, and how do I do that?
[00:47:21.258]I qualify it more.
[00:47:22.091]I add more language about it,
[00:47:24.634]and what have I done there?
[00:47:25.467]I've introduced things that go beyond the construct
[00:47:28.740]I'm really trying to measure in the first place.
[00:47:31.450]So complexity of language is critical.
[00:47:33.897]Making sure that what I'm trying to measure is one
[00:47:37.118]in the same with the level where I've started.
[00:47:40.526]And I'm talking here a lot of times,
[00:47:43.890]but the complexity of language in tests
[00:47:46.330]where I'm not really, the construct is not language itself.
[00:47:49.261]It's in mathematics, it's in science,
[00:47:51.930]it's in social studies, it's in math computation,
[00:47:54.720]it's other things.
[00:47:56.750]It's in the ability to listen.
[00:47:58.350]It's other constructs, not the language itself
[00:48:01.950]where I have complicated things by adding
[00:48:04.858]more high level language to it,
[00:48:08.083]that I did not intend nor did I need
[00:48:10.166]for the purpose of my assessment.
[00:48:12.998]This is a mathematics example.
[00:48:17.530]Seeing sort of here we have a couple different issues
[00:48:20.343]going on with respect to fairness.
[00:48:22.395]This is ...
[00:48:27.890]this is an item that was written for fourth grade
[00:48:31.190]students in mathematics, and at the top of the screen
[00:48:33.706]you can see that the standard itself
[00:48:35.656]says understand a multiple of a over b
[00:48:38.940]as a multiple of one over b,
[00:48:40.690]and use this understanding to multiply a fraction
[00:48:43.830]by a whole number.
[00:48:44.990]So that's the standard.
[00:48:46.860]And as I bring together item writers to say
[00:48:49.311]okay now I need to create test questions that measure
[00:48:53.560]that standard. One of the very first things I do
[00:48:56.415]is worry about well, should I go about this
[00:49:00.453]in an open ended item, where I'm asking a student to,
[00:49:03.983]for the sake of the word, understand in this standard,
[00:49:08.413]is that the same as identify in a multiple choice,
[00:49:11.062]or do I need an open ended where I'm asking them
[00:49:13.809]to explain something?
[00:49:15.284]If I made that decision that that standard warrants
[00:49:17.895]something that's more open ended,
[00:49:20.410]I might create an item, and this is a draft of an item
[00:49:23.492]that was written last year here at Iowa that we
[00:49:27.745]field tested in the state, and it was an open ended item.
[00:49:31.362]And so we were getting at the a over b and so forth,
[00:49:36.460]we feel good from an alignment perspective.
[00:49:37.527]We made a decision about the item format.
[00:49:42.914]Okay, and we feel good about both of those things.
[00:49:45.839]Now we have to worry about the scoring side of things,
[00:49:49.330]and we start to think about scoring from
[00:49:51.197]the student identifies and explains,
[00:49:53.795]and so forth, and coming up with a two point
[00:49:57.162]or three point rubric here that in response to this item,
[00:50:03.170]the student has to not only accurately figure it out,
[00:50:08.170]but also explain.
[00:50:09.720]So by doing that, I'm introduced into this sort
[00:50:12.480]of a language component.
[00:50:14.314]So if this was a fourth grade math item
[00:50:17.020]and I wanted to use it in the entire state of Iowa
[00:50:19.601]where I know about 12% of my students are english
[00:50:23.648]language learners in fourth grade in the state of Iowa,
[00:50:27.800]have I for those students, complicated something here
[00:50:31.049]by asking them to explain.
[00:50:33.390]Could I have gotten to the measurement of that standard
[00:50:35.328]through a multiple choice item without the explanation?
[00:50:40.620]And so it's issues like that that are tied
[00:50:43.593]to this design and item development part of the process,
[00:50:47.435]that every decision along the way is going to have
[00:50:50.484]implications.
[00:50:51.810]The minute I make this decision,
[00:50:53.412]I've created for myself another potential
[00:50:57.680]issue for fairness.
[00:50:59.210]Now you might argue well, if you didn't have to explain,
[00:51:01.524]you have a different issue.
[00:51:02.970]You have an alignment issue in that understand
[00:51:05.640]is not best measured through multiple choice.
[00:51:08.330]Understand is really best measured through the articulation
[00:51:11.450]or the explanation of what the student understands
[00:51:14.114]about this item.
[00:51:15.100]So I have sort of competing priorities there again.
[00:51:17.886]But I need to be aware of it,
[00:51:20.450]and I need to be cognizant of what decision I'm making
[00:51:23.216]going forward.
[00:51:25.610]I've lost my cursor there, okay.
[00:51:27.713]So if this because my evaluation of my response
[00:51:30.424]where I'm taking into account communication skills,
[00:51:33.970]sentence structure, organization, I've all of a sudden
[00:51:37.419]gotten a long ways from the conciseness of the item itself.
[00:51:44.610]So I've introduced another variable
[00:51:46.790]that I need to control for.
[00:51:49.164]So this complexity of language in the math example
[00:51:50.894]that tends to be something we struggle with.
[00:51:54.567]And it's not just mathematics, it's again as I said,
[00:51:57.630]other types of content where the construct I'm trying
[00:52:01.258]to measure is not a language based construct
[00:52:04.356]to begin with, okay.
[00:52:06.660]So if I think about english language learners
[00:52:09.880]versus native speakers of english language
[00:52:12.091]becomes a barrier in one or the other.
[00:52:14.490]And I have to decide do I need my fourth grade students
[00:52:18.402]in this case to show evidence of good sentence structure
[00:52:21.687]and organization and so forth.
[00:52:23.097]Here's a second item, again same sort of issue.
[00:52:25.881]We decided to go out with this as an open ended item
[00:52:30.360]again, so Steve was writing a report,
[00:52:32.361]found that the record low temps.
[00:52:34.713]We've got things like International Falls,
[00:52:36.758]high record temps in Phoenix, blah blah blah.
[00:52:42.517]Okay, so simple math concept,
[00:52:45.648]but what have I done here?
[00:52:47.330]I've thrown in language, language like International Falls,
[00:52:50.599]Minnesota, Phoenix, Arizona, I've thrown in words
[00:52:54.780]like Fahrenheit, I've thrown in temperatures.
[00:52:58.552]So I've thrown in a vocabulary in my stem
[00:53:02.347]that is beyond the level of where the mathematics is.
[00:53:06.528]So I've introduced another potential fairness issue
[00:53:10.164]in the stem itself, by having these words in there.
[00:53:13.496]I could of just as easily said they found this in
[00:53:18.018]city one and in city two,
[00:53:20.640]taken all that vocabulary out of there.
[00:53:22.901]So I've not only introduced a potential for fairness
[00:53:27.884]from a language complexity perspective in the stem,
[00:53:31.240]I again have that same problem with the response.
[00:53:34.140]Again, explain why Steve's conclusion was correct
[00:53:37.020]or incorrect.
[00:53:38.190]Again, asking students to do that, I have to be aware
[00:53:40.804]of that response, again that has to be evaluated
[00:53:44.633]for things that I find of value for the fourth grade
[00:53:50.614]mathematics test.
[00:53:53.135]Zieky's second source was physical sources.
[00:53:57.924]I just drew up a couple of really extreme sources
[00:54:01.210]before universal design became an ongoing
[00:54:03.927]part of where we are today.
[00:54:05.551]You would see tests like this.
[00:54:07.672]This is a page out of a 1977 book.
[00:54:10.588]This is a page out of a 1982 test booklet,
[00:54:12.915]where you could not braille these very well,
[00:54:18.220]you could not turn these into audio versions
[00:54:20.165]of tests for students that have visual issues.
[00:54:22.529]These items become a huge challenge,
[00:54:26.800]and I have introduced a physical component that again,
[00:54:29.199]is going to advantage or disadvantage students
[00:54:32.700]with visual issues.
[00:54:35.850]And I have created this just in the presentation
[00:54:38.002]of my items themselves.
[00:54:39.970]Okay, in today's world, you might struggle more
[00:54:42.148]from an online versus paper and pencil here.
[00:54:45.480]PPT meaning a paper and pencil version
[00:54:48.330]versus a computer based version where on a screen
[00:54:51.407]for the computer based version I have a single item,
[00:54:53.863]versus paper and pencil I have three items on a page.
[00:54:57.685]Have I introduced again an issue of fairness,
[00:55:01.164]and for little young students, some of the research
[00:55:05.280]suggests that yes, students are much better
[00:55:07.160]in the computer based because I can't get off,
[00:55:10.170]I can't move without answering that question,
[00:55:12.668]and I can't inadvertently go to others.
[00:55:15.150]So things that Zieky again call physical,
[00:55:17.970]are thing we're trying to take control of.
[00:55:20.360]And then the last one he talks about are affective sources,
[00:55:23.230]and we're eliciting sort of different reactions to items,
[00:55:27.680]and this is probably the most difficult thing to do.
[00:55:30.870]This is sort of the classic thing that we ask
[00:55:33.270]external reviewers to come in and look for.
[00:55:35.750]What are basic issues of things, fairness,
[00:55:38.890]that have been there
[00:55:39.880]and what are things we should get rid of?
[00:55:43.450]So any kind of topic that might elicit emotional responses,
[00:55:47.805]anything political, religious, holidays,
[00:55:50.415]those things are eliminated.
[00:55:52.439]I think again this has been something our
[00:55:55.130]field has really taken care of in the last 20 years or so.
[00:55:58.790]But this is where a lot of the fairness standards
[00:56:00.448]have come out of.
[00:56:02.360]Okay, I'll come back to that.
[00:56:04.830]As I go into one my final steps of test assembly,
[00:56:07.300]then alignment, reflecting my test taking population,
[00:56:10.705]understanding the balance of individuals within that,
[00:56:13.996]and then the final review of those items
[00:56:17.282]is critical in that test assembly process.
[00:56:20.510]Alright, standard 3.3 is the other sort of standard
[00:56:23.143]that I think we all think about with respect to fairness,
[00:56:26.267]and this is the one where I've collected information
[00:56:28.171]on subgroups, relevant subgroups as the standards say,
[00:56:32.806]with respect to validity and reliability that
[00:56:35.530]I'm going to look at with respect to fairness.
[00:56:37.663]And so critical to fairness in this issue
[00:56:40.634]is finding samples for my research that really look
[00:56:44.235]like my overall test taking population,
[00:56:46.620]that are adequately sized to allow me to do
[00:56:49.904]some of the statical analysis that I need to be done,
[00:56:52.200]and that are collected at the appropriate points
[00:56:55.843]in time and using the appropriate,
[00:56:58.413]whether it be online or paper and pencil.
[00:57:02.010]I can't go out and collect things paper and pencil
[00:57:04.530]and then assume that they're going to translate
[00:57:05.881]into an online environment.
[00:57:07.863]So really identifying what I need in my research samples
[00:57:11.377]so that I can make some valid
[00:57:13.980]and sort of accurate statements about differential
[00:57:17.670]item functions.
[00:57:19.180]And we do this analysis in sort of two different levels,
[00:57:22.280]the item level and again at that test assembly level.
[00:57:26.040]The most sort of classic approach at the item level,
[00:57:28.111]differential item functioning, that if I have
[00:57:33.419]adequate samples in front of me,
[00:57:34.845]I can evaluate whether items are performing
[00:57:36.600]differently for different groups of population
[00:57:39.286]when I'm controlling for the underlying ability.
[00:57:42.334]Again this is not just did this item do better
[00:57:45.203]for females than it did for males.
[00:57:47.510]It's not that.
[00:57:48.600]It's about when I condition on students
[00:57:51.810]in that female population that are at the same
[00:57:53.770]ability level as that male population.
[00:57:55.129]When I condition there, now how is that item
[00:57:59.410]performing for those two different groups.
[00:58:01.400]And I need to be able to collect as a researcher,
[00:58:03.929]information statistical like this that I can then
[00:58:07.530]analyze and make part of my review process.
[00:58:10.310]Other things that I look at the item level
[00:58:12.123]are plots of difficulty and discrimination,
[00:58:14.869]being sure that the relationships there between
[00:58:19.038]difficulty and discrimination are consistent
[00:58:21.826]across different groups, as well as looking
[00:58:24.690]at item characteristic curves and item information
[00:58:27.328]functions if I have the appropriate samples
[00:58:29.720]to generate item response theory.
[00:58:33.545]At the test level, I might look at things like
[00:58:36.631]confirmatory factor analysis.
[00:58:38.388]To be sure the structure is the same for different
[00:58:40.933]and looking the same and behaving the same
[00:58:43.580]for different groups of interest.
[00:58:45.550]I'm also looking at the relationship between
[00:58:46.971]different groups and variables of interest.
[00:58:50.294]For instance in college and career readiness,
[00:58:52.510]we started with CCSSO.
[00:58:53.761]Are these tests predicting in the same way
[00:58:56.646]for different groups of interest?
[00:58:58.629]As well as looking at again an IRT way whether
[00:59:00.990]the test characteristic curves,
[00:59:04.070]the test information functions and so forth.
[00:59:05.958]Am I getting the efficiencies for a test
[00:59:08.199]for different groups of the population.
[00:59:10.870]Okay, as standard 3.4, 3.5, they're the last two
[00:59:15.166]standards in chapter three, and they're talking about,
[00:59:18.364]and I think you will have additional webinars on this,
[00:59:21.544]but the test administration and scoring processes
[00:59:24.988]and then the documentation side,
[00:59:27.257]and being sure that I have documented my procedures
[00:59:30.094]to be sure I have in place that transparency I need.
[00:59:33.706]Okay, and that documentation not only transparent,
[00:59:37.375]but it has to be accessible.
[00:59:39.080]It has to be accessible to everyone,
[00:59:41.472]from parents to students to evaluators, to users.
[00:59:46.490]And it has to be descriptive enough,
[00:59:48.670]with enough detail that I have confidence
[00:59:50.372]that what I'm saying based on this sample
[00:59:52.535]is indeed generalizable to what I want to do with that test.
[00:59:58.660]Okay, the third section, and I know
[01:00:00.599]that took longer than I thought, so I'm sorry.
[01:00:02.450]So we're gonna jump into strategies here,
[01:00:04.887]and then we'll open it up for more questions.
[01:00:07.500]Okay, I think fairness is one of these areas of research
[01:00:11.700]that people tend to again say yes, we did this,
[01:00:14.480]we went through this process.
[01:00:16.150]And what I'm suggesting is it has to be well defined.
[01:00:18.991]You have to be able to as a user go find the documentation.
[01:00:23.470]As a test developer I have to define it,
[01:00:25.680]then I have to go about putting it in place
[01:00:27.695]and then evaluating it.
[01:00:29.390]I can't just put it into place and leave it there
[01:00:31.304]and have the same group of people come in year after year
[01:00:34.110]and look at my item pool and just have it be you know,
[01:00:37.430]sort of a check in my process.
[01:00:39.720]I have to evaluate is that helping me,
[01:00:41.730]am I identifying items, what is it telling you about
[01:00:44.160]the items, what am I doing in my process
[01:00:46.170]to change those items, and I have to revise things
[01:00:48.406]in order to make them better in the long run.
[01:00:51.720]So if we're not learning from this process,
[01:00:53.244]then it's really not of any value.
[01:00:56.530]It's just becomes sort of something I have to do.
[01:00:59.690]For the sake of the integrity of this,
[01:01:02.210]we really have to learn from this and change in
[01:01:04.740]order to move forward.
[01:01:06.501]So what I'm talking about evidence,
[01:01:08.605]I'm talking about evidence that's looking at the
[01:01:12.203]threats to validity, okay?
[01:01:14.520]Telling me about universal deigns,
[01:01:16.120]telling me about accommodations,
[01:01:17.900]telling me about other features that are there,
[01:01:19.740]telling me about item development,
[01:01:21.345]and tell me about forms development.
[01:01:23.663]So what do I, just like that peer review slide showed,
[01:01:26.165]I'm going to turn to you and show you my item specs.
[01:01:30.423]I'm going to tell you how I instructed item writers
[01:01:34.025]to write items.
[01:01:35.860]What to avoid, what to include,
[01:01:37.550]how to align to my standards.
[01:01:39.250]And then I'm going to tell you what checklist,
[01:01:41.780]what protocols, what guidelines, what training
[01:01:44.070]did I provide to my item reviewers as they went through
[01:01:46.648]and looked at these items through the different items,
[01:01:51.820]and then what did I do with that empirical evidence
[01:01:54.560]and how did I then review all of this
[01:01:57.025]in a fact or change.
[01:01:59.946]And if I had to change something significantly,
[01:02:02.145]what did I do with all of the other sort of dependencies
[01:02:04.991]in that process to allow for me to make those changes?
[01:02:10.230]And my audience, is variable.
[01:02:13.820]Course we've been talking about that,
[01:02:15.110]but for the sake of this documentation,
[01:02:17.040]I think there's three primary audiences.
[01:02:19.510]The test developers, the users,
[01:02:22.144]and then people like the Buros people.
[01:02:23.180]The test evaluators.
[01:02:24.481]What are you looking for?
[01:02:26.535]When I want a third party external reviewer
[01:02:30.017]to come in and evaluate my assessment,
[01:02:33.060]what is the documentation I have in front of me.
[01:02:35.581]So taken from the code which was the code
[01:02:39.600]of fair testing practice, which actually came between
[01:02:43.040]the most recent version came between the 1999 standards
[01:02:45.836]and the 2014 standards, it divided this into those
[01:02:49.319]two audiences of users and developers.
[01:02:51.838]And I think again, consistent here with what we've
[01:02:54.560]been talking about, are all the different things
[01:02:56.480]that I as a developer should be documenting.
[01:02:58.909]And we won't go through these again,
[01:03:00.640]but I think the original code which came out in the late
[01:03:05.795]80's and then was revised, is a nice way to look at it
[01:03:09.291]by developer versus user,
[01:03:12.480]that they tried to talk about the shared
[01:03:14.562]sort of responsibility, but then they articulate into
[01:03:17.956]these two different users.
[01:03:20.550]So I just include those here as a reference for you,
[01:03:23.270]but I think the unique one here for today's conversation,
[01:03:27.240]those test evaluators, those individuals are going
[01:03:30.238]to come in and evaluate whether those procedures
[01:03:33.398]that I have documented, were they enough?
[01:03:36.810]Did I go far enough?
[01:03:38.100]When the peer review evaluators look at my documentation,
[01:03:40.915]they're going to rate me.
[01:03:43.490]Did I show enough along the way to say yes,
[01:03:46.372]you get a check mark on that, or not,
[01:03:48.830]and if I have a deficit in that,
[01:03:51.160]what can I do to improve?
[01:03:52.820]They're also going to, from a evaluation perspective,
[01:03:55.280]review the performance of those test takers,
[01:03:57.526]not only look at the statistics,
[01:04:00.050]but look at the action that was taken by the developer.
[01:04:02.833]What happened?
[01:04:04.680]What happened to those items that were flagged?
[01:04:07.530]What happened to those procedures and how did it
[01:04:10.262]affect things moving forward?
[01:04:12.750]And so they will also look at other types
[01:04:15.347]of performance differences that may have been a result
[01:04:19.407]of something in the test.
[01:04:20.890]So I need to have not only things documented,
[01:04:23.770]but I need to again go on to revise,
[01:04:27.010]evaluate and revise myself as a test developer.
[01:04:30.997]I think I'm gonna just end on a few things about
[01:04:34.610]new strategies and looking forward.
[01:04:36.833]I think there's been a certain amount of research,
[01:04:40.530]and I think NCME, AERA, the standards have been great,
[01:04:44.350]different users have been great,
[01:04:46.031]but I think in the test development world,
[01:04:48.490]there's also some sort of current practice
[01:04:50.880]that's become a little too entrenched and
[01:04:53.808]it's become part of a process without really
[01:04:55.746]evaluating it.
[01:04:57.169]I think a lot of the research that's been done
[01:05:00.475]and I reference technology enhanced items earlier,
[01:05:03.608]has been more proprietary and tied to a certain contract
[01:05:07.674]or a certain testing vendor and it's not necessarily
[01:05:11.252]as public as we would like.
[01:05:13.470]So I would encourage people to be as open
[01:05:15.564]and as transparent as possible in this process
[01:05:21.390]and in the research as we go forward.
[01:05:24.260]So with the current item review process,
[01:05:25.994]I would say that the traditional approach is great.
[01:05:29.010]It's in process, but it's often times a little too
[01:05:32.960]far along when we looked at that,
[01:05:34.726]the process from the articulation of purpose
[01:05:37.818]all the way to the end, you can see design
[01:05:40.304]and all those things at the top,
[01:05:42.609]they're not really part of that review,
[01:05:44.066]and what I'm saying is that they really should be.
[01:05:47.440]We need to reexamine those procedures,
[01:05:49.024]knowing that I've got new technologies,
[01:05:51.011]new delivery platforms.
[01:05:52.911]I have changed things along the way,
[01:05:54.716]and I need to involve experts, technology experts,
[01:05:58.663]fairness experts.
[01:06:00.400]I need to involve all kinds of people early on
[01:06:02.304]in that design and development process.
[01:06:05.587]And so that I can again sort of go into item writing
[01:06:08.864]with that in mind.
[01:06:10.558]Second one, the collection of empirical data.
[01:06:13.881]The standards, and I know they again are trying
[01:06:17.390]to be broad, but they use language such as if
[01:06:19.795]the sample sizes exist, or permit,
[01:06:22.316]or to the extent possible and where feasible.
[01:06:26.230]And they open up the suggestion that if there's not
[01:06:31.416]a sample size, they don't have to do this.
[01:06:33.810]And they allow sort of this tolerance for maybe
[01:06:37.640]the collection of empirical data
[01:06:39.440]at the pilot test or earlier in the field test stages
[01:06:42.460]and so forth is not a possibility.
[01:06:44.490]And I think again this language has contributed to
[01:06:47.162]okay well I only field tested on 300 students,
[01:06:50.990]and the only valid group I can come up with is male female.
[01:06:55.550]I have nothing else in that 300 group that will
[01:06:58.110]allow me to generate stable statistics,
[01:07:00.950]so I'm just not going to do it.
[01:07:02.630]So I think that is an issue with our current procedures
[01:07:05.230]and that collection of empirical data.
[01:07:07.265]There's just too many ways to avoid it.
[01:07:10.500]So I'm suggesting that state agencies,
[01:07:13.860]other testing organizations that are,
[01:07:16.396]other testing entities and so forth that are seeking
[01:07:20.077]tests to be developed by these testing organizations,
[01:07:23.030]they need to require that.
[01:07:24.600]They need to require the collection of empirical data,
[01:07:27.420]and that means upping the sample sizes.
[01:07:29.377]And that means upping the empirical, I'm sorry,
[01:07:32.990]the representation of those empirical sample sizes
[01:07:35.412]so that I can do all of the analysis that I need to do.
[01:07:39.241]So I need to build in timelines and I need
[01:07:42.252]to build in budgets that will allow me to structure
[01:07:45.587]and support this work.
[01:07:46.734]And so if a testing company is trying to get something
[01:07:50.890]to market in six months, I'm suggesting
[01:07:52.821]that we as an industry should say no,
[01:07:56.010]you need the appropriate evaluation in order
[01:07:59.295]for you to be accepted out there,
[01:08:01.249]you need to show me evidence, and that evidence
[01:08:04.130]comes from again, increased cost,
[01:08:06.310]but those increased costs in the front end
[01:08:08.567]have huge implications for the interpretation
[01:08:10.748]and fairness of interpretation on the back end.
[01:08:13.700]And I would put this with respect to all the
[01:08:17.649]new technologies, and everything we're doing in our field,
[01:08:20.184]automated item generation,
[01:08:21.777]technology enhanced items, the automated scoring
[01:08:25.706]using artificial intelligence,
[01:08:27.910]all of those things need to be pulled apart,
[01:08:30.276]and some people have started this process
[01:08:32.211]but again, it's not sort of out there
[01:08:34.465]and publicly known.
[01:08:35.899]All of those procedures need to be pulled apart
[01:08:37.599]and fairness needs to be part of that.
[01:08:39.611]Just because an item is automatically generated somewhere
[01:08:43.069]doesn't make it fair.
[01:08:45.144]I need to still evaluate what went into that generation
[01:08:48.176]and need the ultimate result,
[01:08:52.060]and I need to articulate these.
[01:08:55.300]And then my final recommendation is we need to
[01:08:57.983]monitor the success of these procedures.
[01:08:59.984]We have them in place,
[01:09:01.054]I've been working for a long time in this industry,
[01:09:03.917]and we have these documents.
[01:09:06.066]They're there.
[01:09:07.733]But I also want to be sure that we go forward
[01:09:11.819]with research and the public sharing of this information,
[01:09:15.676]because again, it shouldn't be seen as a proprietary
[01:09:18.564]piece, it should be something that we all share
[01:09:20.857]what's good about fairness research and using that
[01:09:25.111]in different testing organizations.
[01:09:28.500]And coming up other alternatives, smaller sample
[01:09:31.036]research and analysis, other ways that we can get to the
[01:09:33.950]end results that again still help us from a fairness
[01:09:38.020]perspective, but might sort of create a path
[01:09:41.850]that is more realistic for us.
[01:09:44.515]So okay, in that case, of course as we evaluate,
[01:09:50.042]looking and what are my success indicators?
[01:09:53.149]Just to have these documents isn't probably enough,
[01:09:55.510]so I want to know how these documents,
[01:09:59.080]how these procedures have affected things.
[01:10:02.926]Have I increased in my fairness indicators?
[01:10:05.090]Have I created on the back end,
[01:10:06.660]better information?
[01:10:08.330]Have I had more sort of success from the validation
[01:10:12.170]of the test itself by the very implementation
[01:10:13.366]of these procedures on the front end?
[01:10:17.780]Alright, I wanted to end with a quote
[01:10:19.900]from a gentleman that founded Iowa Testing Programs here,
[01:10:22.455]and a lot of his huge contribution to the field
[01:10:25.654]of measurement, but he was big about the front end.
[01:10:27.991]And this gentleman wrote this in like 1952 or something,
[01:10:33.530]but I think it's still relevant today
[01:10:35.310]that the decisions that are really preliminary,
[01:10:38.750]they're in the front end,
[01:10:39.781]they're so important.
[01:10:41.560]And we I think our field gets so lost
[01:10:43.426]and sometimes in all the psychometrics
[01:10:45.092]and the true statistical analysis,
[01:10:48.240]we forget everything on the front end
[01:10:49.423]that we really should be controlling for from the get go.
[01:10:52.966]Alright, Jessica.
[01:10:55.770]Alright, thank you Dr. Welch.
[01:10:57.782]So if you have questions, we have, let's see here.
[01:11:01.060]We've got about 10, 15 minutes for questions,
[01:11:03.320]and we do have someone who's already submitted one
[01:11:05.657]as we wait for others.
[01:11:07.818]The question is what are your thoughts about tests
[01:11:10.878]such as the Woodcock Johnson for cognitive
[01:11:13.754]or achievement testing administered via telepractice.
[01:11:19.530]The stimulus material has been transferred to the computer
[01:11:22.470]so the student sees the same information
[01:11:24.500]in the same ways it'll be typically presented
[01:11:26.524]on the stimulus book, but now it is on the screen.
[01:11:29.539]And then this student and examiner can see,
[01:11:33.003]hear each other, the same manner as we're using today
[01:11:36.157]via video.
[01:11:37.910]Do you feel this interferes with fairness in any way?
[01:11:42.457]Well I don't know about that test in particular,
[01:11:44.620]but I would hope that the publisher of that test
[01:11:47.092]went through the research that showed that
[01:11:51.322]that environment is the same,
[01:11:53.570]that we're not getting different results
[01:11:55.320]when we put the proctor online versus
[01:11:59.110]when the proctor is sitting there administering
[01:12:01.070]that test to that test taker.
[01:12:03.420]That by putting the materials in front of that student
[01:12:06.799]and removing the person, that in the case
[01:12:11.390]of the Woodcock Johnson or other tests like that,
[01:12:13.840]I think some of the interpretation is tied
[01:12:16.470]directly to norms, and whether they be aged based norms
[01:12:19.670]or grade based norms,
[01:12:21.427]but to be sure that interpretation piece holds up
[01:12:24.260]is something that I would want to evaluate
[01:12:26.694]with respect to what the publisher did
[01:12:31.103]before they just went that way.
[01:12:34.040]I think that's sort of a common practice right now
[01:12:37.765]is to try to take everything we have on paper
[01:12:40.300]and put it online, and maybe if the stakes
[01:12:44.016]are very very low, it doesn't matter if it's a practice
[01:12:46.579]sort of thing, but when I'm starting to make decisions
[01:12:49.009]about an individual child and their next steps,
[01:12:53.260]then I want to see the research that suggests that's still
[01:12:55.444]a fair interpretation, that those norms
[01:13:01.213]are just as appropriate in that online environment
[01:13:02.610]as they were in the original paper and pencil environment.
[01:13:07.600]Okay, thank you.
[01:13:09.132]And while we wait to see if there are some other questions,
[01:13:11.050]I will mention that we will receive a request
[01:13:14.354]to complete an online evaluation of this presentation
[01:13:17.830]today, it should take you about 10 minutes to complete.
[01:13:21.510]And your input would be very helpful to the future
[01:13:22.858]program planning.
[01:13:25.660]And if you have requested continuing education credit,
[01:13:27.987]you will need to complete that evaluation
[01:13:30.375]before we can send you your documentation of attendance.
[01:13:34.334]So the next question that we have is
[01:13:36.763]how can, or should fairness be articulated for the CAT
[01:13:40.588]measure, a student computer adaptive testing?
[01:13:44.143]Yeah, well there, because I think everything we talked
[01:13:47.789]about with respect to items, it is very much in place.
[01:13:51.929]Right?
[01:13:53.570]All that front end, everything I'm doing right up
[01:13:55.978]to the assembly of my item pools themselves,
[01:13:59.050]is still relevant.
[01:14:00.070]I'm still looking at the CAT items
[01:14:03.260]that pool, I'm still going through all of those
[01:14:05.500]individual item reviews on the front end.
[01:14:08.042]All of that is in place.
[01:14:09.490]When I get to the test itself,
[01:14:11.320]and a given test is given to a student,
[01:14:13.990]what we're making the inference there is
[01:14:16.260]because every item in the pool has been reviewed
[01:14:19.474]and approved and is eligible for inclusion,
[01:14:23.023]then by the very selection of items that I'm
[01:14:26.020]putting together, if I get 30 items
[01:14:26.954]and Jessica gets a different set of 30 items,
[01:14:31.530]that by it's very nature, coming from this already
[01:14:35.530]approved pool, that that should be a fair interpretation
[01:14:37.920]that those items are fair and the collection
[01:14:40.490]of items for a given test taker are fair.
[01:14:43.870]We don't have the same you know, ability to obviously
[01:14:47.460]look at test characteristics, last week just go
[01:14:50.310]in and create or simulate and so forth,
[01:14:53.370]and a lot of large testing programs do that as well.
[01:14:55.423]Come up with the many different iterations of the CAT
[01:14:59.640]and look at that for fairness as well,
[01:15:02.350]which I think is part of the sort of the procedures
[01:15:04.424]that should be in place for CAT.
[01:15:07.430]Okay, very good.
[01:15:08.760]And the next question we have is how do we know
[01:15:11.220]for efforts to make more tests more fair actually work?
[01:15:13.960]Is there an analysis tool you can recommend?
[01:15:17.130]No, I don't think there is an analysis tool.
[01:15:19.370]And I'm sorry, I think we have to take fairness
[01:15:21.886]and assume it's what we're striving for,
[01:15:25.230]and then weave it in to our validity research studies.
[01:15:28.725]So when we think about why we designed the test
[01:15:32.190]in the first place, and we put into that procedure
[01:15:35.345]all of our fairness stops,
[01:15:37.290]we're hoping at the end that we've enhanced the validity
[01:15:40.580]of that assessment.
[01:15:41.760]So in the same way that we would validate
[01:15:44.220]whether it be on the content side through alignment,
[01:15:47.130]whether it be on a prediction side,
[01:15:49.450]whether it be on a you know, relate to another
[01:15:52.920]sort of criterion, that those very I guess
[01:15:56.312]indicators of validity on that back end
[01:16:00.790]will be seen in a more fair way,
[01:16:02.630]that that message is clear and can be interpreted
[01:16:06.527]with fairness in mind.
[01:16:09.600]But I don't think there's an analysis itself.
[01:16:12.014]I think what these different groups are doing
[01:16:15.260]that like peer review, they're coming down to
[01:16:18.225]just sort of the documentation side of things
[01:16:21.100]and not really the after the fact,
[01:16:24.550]after I give the test
[01:16:26.050]and analyze it that way, unfortunately.
[01:16:30.750]Alright, and Cathy, if you would
[01:16:33.480]go on to the next slide,
[01:16:35.010]I'll talk a little bit about some of our upcoming ones
[01:16:37.090]while we wait to see if there's other questions.
[01:16:40.260]So we do, as I mentioned, this was a first webinar
[01:16:43.245]in a five part series, and we do have four additional
[01:16:48.247]presentations coming up,
[01:16:49.890]and actually the two upcoming ones in April
[01:16:52.580]and May are notable school psychologists,
[01:16:54.718]Dr. Samuel Ortiz and Dr. Ryan Kettler,
[01:16:59.430]and they will be talking about some of their research
[01:17:01.878]and theory and research and what they've learned
[01:17:05.780]from that, and how that's been put into practice
[01:17:07.950]to make tests fair for english language learners
[01:17:11.957]and individuals with disabilities.
[01:17:14.710]And so I encourage you or you have colleagues
[01:17:17.560]who might be interested in attending those.
[01:17:20.191]I also will mention if I haven't already
[01:17:23.990]is that we are recording these presentations,
[01:17:26.400]and they will be available on our video library.
[01:17:29.326]And in fact our video library has an archive
[01:17:34.620]of many of the programs like this that we've done.
[01:17:37.030]We've done a series on clinical assessment,
[01:17:39.240]decoding the standards for educational and
[01:17:41.310]psychological testing which featured members
[01:17:43.530]of the committee that actually helped draft
[01:17:47.731]or guide the drafting of the 2014 standards,
[01:17:50.746]and then evaluating tests for quality that featured
[01:17:55.270]some of our mental measurement yearbook reviewers,
[01:17:57.420]long term distinguished reviewers.
[01:17:59.640]And so those might be of interest to you as well.
[01:18:03.257]It does not appear at this point that we're receiving
[01:18:06.870]any other questions, although please submit
[01:18:08.973]that if you are, and we might be able to catch it
[01:18:11.240]before we're done.
[01:18:12.580]But I do want to thank you Dr. Welsh for your
[01:18:14.670]informative presentation, I think you provided
[01:18:16.443]some information that could be very helpful
[01:18:18.244]and thinking more carefully about fairness.
[01:18:22.789]Whether we're developing assessments,
[01:18:24.318]selecting assessments, using assessments,
[01:18:27.378]and so I think those are the kinds of things
[01:18:30.169]we need to do as a field to make sure
[01:18:32.327]that we have scores.
[01:18:36.750]We are obtaining and interpreting and using
[01:18:39.123]that are fair and valid for all participants.
[01:18:43.082]So if there aren't any other questions today,
[01:18:45.878]we do appreciate your attendance.
[01:18:48.398]Thank you for support of our programming.
[01:18:50.643]Thank you Dr. Welch, we appreciate your input
[01:18:54.200]and your expertise, and as I mention I have provided
[01:18:59.149]our email addresses, both myself and Dr. Welch
[01:19:02.540]in case you have any follow up questions.
[01:19:04.540]Otherwise thank you and enjoy the rest of your day.
[01:19:07.248]Thank you Jessica.
[01:19:08.890]Yeah, thank you.

The screen size you are trying to search captions on is too small!

You can always jump over to MediaHub and check it out there.

Comments

0 Comments

Fairness Evidence Expectations for Educational Tests

Description

Searchable Transcript

Comments icon comment

Related Channels

Comments