Fairness and English Learners

Samuel Ortiz, PhD Author

05/01/2018 Added

42 Plays

Description

Webinar

Searchable Transcript

Search:

[00:00:00.170]Introduction and hello everyone
[00:00:01.740]and thank you for choosing to spend an afternoon with me.
[00:00:06.890]That's wonderful.
[00:00:07.723]I hope that you'll find the information informative,
[00:00:09.750]useful in some way.
[00:00:11.540]And, as Jessica mentioned,
[00:00:12.930]if there are any questions that you have,
[00:00:13.763]I'll be happy to try and address them
[00:00:16.520]as best I can at the end of the presentation.
[00:00:20.160]So, I'm going to move right into things,
[00:00:23.530]because I think it's important to have a chance
[00:00:27.100]to discuss issues of fairness with English learners,
[00:00:30.250]because I know this is a big problem.
[00:00:31.920]I was talking recently to the National Center for,
[00:00:35.940]National Joint Center for Learning Disabilities,
[00:00:38.930]I forget the actual title,
[00:00:40.200]and they were concerned about some of the wording in IDEA
[00:00:44.120]and how it expresses particular approaches
[00:00:48.370]or conditions and exclusionary variables and so on.
[00:00:51.860]And that's something I've been wrestling with
[00:00:53.590]for at least 25 years.
[00:00:55.930]And I know that the issue of fairness for English learners
[00:00:59.870]is something that is so important,
[00:01:01.340]because without it, and without some attention to it,
[00:01:04.610]what happens is that English learners become highly at risk
[00:01:08.610]for misidentification of disabilities,
[00:01:11.050]placement in special education,
[00:01:12.490]incorrect decisional making, diagnostic mistakes,
[00:01:16.750]all kinds of things that lead to negative outcomes.
[00:01:19.890]So, I think it's important
[00:01:21.080]that we understand why this occurs
[00:01:23.860]and what we can do about it to be able to say,
[00:01:26.280]hey, there is a way that we can understand what fairness is
[00:01:30.720]and how we can achieve it with English learners,
[00:01:33.180]even if it's not necessarily the ways
[00:01:35.110]that we have thought about doing so in the past.
[00:01:37.450]And I think that's something that comes up quite a bit,
[00:01:39.540]is that we maybe are reliant
[00:01:42.190]a little bit too much on inertia,
[00:01:43.520]and not understanding whether or not the things
[00:01:45.750]that we've been doing have actually been successful or not.
[00:01:48.940]So, I'd like to start with just a quick review
[00:01:51.240]of some of the things that people have done
[00:01:53.250]to try and address the issue of score validity,
[00:01:56.400]test score validity in particular,
[00:01:58.320]with English language learners.
[00:02:00.410]And all of these methods are things
[00:02:02.610]that you probably are very familiar with
[00:02:04.450]and have done and are continuing to do even today,
[00:02:07.680]such as the use of a translator.
[00:02:10.240]In some cases, there's just no alternative
[00:02:12.820]to using a translator.
[00:02:14.040]If you cannot communicate with the individual,
[00:02:16.190]you cannot evaluate, or test, or assess the individual.
[00:02:19.070]Communication is an essential part of that process,
[00:02:22.430]so having somebody that can communicate with a child
[00:02:24.840]is sometimes required.
[00:02:25.920]If you can't do it, you need help.
[00:02:27.570]But the problem with it, as noble as the idea is
[00:02:31.170]that we want to be helpful to the child,
[00:02:34.870]we want to give them every opportunity
[00:02:36.310]to be able to demonstrate what they can and cannot do,
[00:02:39.310]the problem is that using a translator/interpreter
[00:02:41.390]is a non-standardized procedure.
[00:02:43.970]There are no norms available that say
[00:02:47.700]here's what you can expect
[00:02:49.210]when you use a translator or interpreter.
[00:02:52.370]You suffer from things like third-party observer effects.
[00:02:56.070]You have content that may not even translate properly,
[00:02:59.280]so how do you move that over.
[00:03:01.000]For example, phonological processing,
[00:03:02.370]you can't translate phonology.
[00:03:04.240]There are sounds that don't exist in other languages
[00:03:06.680]and sounds in other languages that doesn't exist in English.
[00:03:10.640]So, when you violate the standardization,
[00:03:13.130]when you introduce these other variables,
[00:03:14.820]it's basically creating error and to an unknown degree.
[00:03:18.350]Without norms, you have no idea of knowing
[00:03:20.590]what the psychometric properties of the test are anymore.
[00:03:23.370]What is an average performance score?
[00:03:26.160]What is the standard deviation?
[00:03:28.020]You have no sense of this.
[00:03:29.890]You don't know the standard error of the measure.
[00:03:31.860]So, while it may be necessary
[00:03:34.150]to use translators and interpreters,
[00:03:35.680]and I'm not against doing so,
[00:03:37.410]it really is going to depend on
[00:03:39.260]if and when that's necessary and appropriate.
[00:03:41.900]But in terms of generating valid test scores by using them,
[00:03:45.910]we're really at a loss here,
[00:03:47.190]so it's not going to be the answer that we would like.
[00:03:50.190]The same thing applies to modified or altered testing.
[00:03:53.830]And this, by the way,
[00:03:55.210]should be distinguished from accommodated testing.
[00:03:57.830]An accommodation is generally defined,
[00:03:59.950]according to the standards,
[00:04:00.950]as something that does not alter the construct
[00:04:04.140]that is being measured by the test.
[00:04:06.080]So, something that you can change that allows the individual
[00:04:08.940]to be able to do what they need to do.
[00:04:11.530]So, it's important to recognize
[00:04:14.590]that a modified or altered kind of testing situation
[00:04:18.769]is one in which there is a change
[00:04:21.720]in the testing process itself,
[00:04:23.740]which then can modify the construct that might be measured.
[00:04:27.180]So, once again, if you do all the kinds of things
[00:04:30.140]that usually fall under the rubric
[00:04:31.580]of modified or altered testing,
[00:04:32.960]including something like testing of the limits
[00:04:34.640]where you eliminate or you extend the time constraints,
[00:04:38.050]you repeat the questions, even if you're not supposed to,
[00:04:40.630]you accept responses in an alternate language,
[00:04:43.500]you mediate the concept before you move ahead,
[00:04:47.010]if you do any of these kinds of things, then, once again,
[00:04:49.870]you're violating the standardization protocol
[00:04:51.780]unless the publisher provides you with norms
[00:04:54.410]based on that type of administration,
[00:04:56.770]which is unlikely that they're going to do that
[00:04:58.760]given that a lot of these modifications
[00:05:00.820]are random and haphazard at best.
[00:05:02.760]They're not consistent across people or testing situations.
[00:05:06.120]So, in doing this, we simply lack the norms again
[00:05:09.450]to tell us what the psychometric properties of the test are.
[00:05:12.780]We don't know the mean,
[00:05:13.870]we don't know the standard deviation,
[00:05:15.201]we don't know the standard error of measure,
[00:05:16.290]we don't know, then, what to make of our results,
[00:05:18.740]even if the intent is to increase the child's performance.
[00:05:22.550]Now, English learners have long been thought,
[00:05:24.940]well, we can just move to nonverbal assessment
[00:05:27.970]and that will take care of everything.
[00:05:29.770]Now, in a way, it kind of does,
[00:05:31.640]but in another way, it doesn't really at all.
[00:05:34.430]Nonverbal testing is nonverbal, but it's non-language.
[00:05:39.550]Or I should say it's language reduced,
[00:05:41.650]because it's not non-language.
[00:05:43.640]What that means is that, again,
[00:05:45.190]you cannot give a test to an individual
[00:05:47.630]without communicating with them in some way.
[00:05:50.130]Now, you may choose to not use an oral language,
[00:05:52.800]but if you're using anything,
[00:05:54.790]such as gestural communication, sign language,
[00:05:57.860]then you are using a language to communicate with the child.
[00:06:01.540]So, you see that nonverbal doesn't mean
[00:06:03.570]that there's no language involved.
[00:06:04.820]It just simply means that oral language is eliminated
[00:06:07.700]and the language effect, as much as possible,
[00:06:10.530]is generally reduced.
[00:06:12.520]Part of that, though, comes from limiting the amount
[00:06:14.560]of constructs that are being measured.
[00:06:16.630]Nonverbal skills are things like visual processing,
[00:06:19.270]easy to measure nonverbally,
[00:06:20.810]some nonverbal fluid reasoning, perhaps,
[00:06:23.790]some short-term memory if it's visually-oriented.
[00:06:26.290]So, you see that the constructs that are being measured
[00:06:29.440]are defined by trying to remove as much reliance on language
[00:06:32.890]as well as some reliance on cultural kinds of issues.
[00:06:36.137]But the problem is that will limit
[00:06:37.620]the types of abilities that can be measured.
[00:06:39.800]You can't measure auditory processing in a nonverbal manner.
[00:06:42.590]Likewise, you can't measure comprehension
[00:06:45.720]or crystallized intelligence, GC, in a non-language manner,
[00:06:50.050]because GC is language.
[00:06:52.330]And I would say, more importantly than anything,
[00:06:55.660]nonverbal runs away from the measurement of language
[00:06:58.910]because that's the central issue in working within ELL.
[00:07:01.760]It's a language issue, so let's just avoid that.
[00:07:04.280]But language is too important to be avoided.
[00:07:06.910]It correlates too highly with many other indexes of success,
[00:07:10.150]academic outcomes, general ability, language development,
[00:07:13.640]to be able to say, well, it's not important,
[00:07:15.590]it's okay if we don't measure it.
[00:07:16.900]In fact, in the measurement of English language learners,
[00:07:19.830]there's probably nothing more important
[00:07:21.530]than the measurement of language.
[00:07:23.070]So, nonverbal tests aren't going to give us
[00:07:24.820]a complete or satisfactory solution either.
[00:07:27.210]Now, another approach that's long been used
[00:07:29.070]has been dominance evaluation, where people say
[00:07:31.700]let me evaluate in which language the child is dominate.
[00:07:35.050]Now, while that's useful, because what it would tell you is
[00:07:39.690]if I then test the child in the dominant language,
[00:07:42.330]I'm very likely to get higher scores.
[00:07:45.280]The question is
[00:07:46.260]are you getting developmentally appropriate scores,
[00:07:48.150]and that answer is no, not necessarily.
[00:07:50.490]And the reason for that is
[00:07:52.390]that dominance only tells you
[00:07:53.660]which language is better developed.
[00:07:55.530]It doesn't tell you that the child's level of development
[00:07:58.660]is commensurate with same-age peers.
[00:08:01.330]And this is a big problem for English learners,
[00:08:03.250]because they generally will be learning a second language,
[00:08:07.450]let's say that's English,
[00:08:08.630]at a point other than from birth, like myself.
[00:08:11.530]I'm an English learner
[00:08:12.690]and I didn't start learning English until the age of five.
[00:08:15.460]So, when I'm six-years-old, you can't, then, at that point,
[00:08:19.160]say, "Well, okay, let's see, you've learned enough English.
[00:08:21.647]"Now you're becoming more dominant.
[00:08:22.877]"Maybe we should just test you in English."
[00:08:24.580]Now, maybe that doesn't happen by six,
[00:08:25.870]but by seven, by eight, it surely did.
[00:08:27.840]In fact, by seven years of age,
[00:08:29.490]I was already telling my parents,
[00:08:30.957]"I'm not speaking Spanish anymore."
[00:08:32.770]So, I was clearly preferential
[00:08:34.740]and dominant in English by that point,
[00:08:36.630]but I only had two years of learning English
[00:08:38.520]compared to my peers who had seven years' worth
[00:08:40.950]of learning English.
[00:08:42.000]So, it isn't a fair assessment,
[00:08:44.670]and I think it's a too simplistic approach
[00:08:47.890]to be able to say, "We can generate valid results this way."
[00:08:51.110]I don't think it really works at all, again,
[00:08:53.530]other than to tell you whichever language is dominant.
[00:08:55.830]You're going to get higher scores,
[00:08:56.990]but necessarily average or better scores.
[00:08:59.800]And then, finally, native language testing.
[00:09:01.670]And this, I think, has been probably the area
[00:09:04.690]that there's been more interest of as of late,
[00:09:07.860]but there are some tremendous limitations.
[00:09:10.330]Number one, when we do native language testing,
[00:09:12.780]we often validate those types of tests on native speakers,
[00:09:16.280]monolingual speakers of that language.
[00:09:18.650]But the individuals that we work with in the U.S.,
[00:09:21.380]invariably, are not monolingual.
[00:09:23.560]Unless you're testing them on the very day they arrive
[00:09:26.190]from another country, for some reason,
[00:09:29.170]they will be learning English
[00:09:30.880]from the moment they start their residence here in the U.S.,
[00:09:33.527]and that means, especially if they're in school,
[00:09:35.500]that means that they are bilingual,
[00:09:37.330]no longer monolingual speakers of their language.
[00:09:40.550]So, you can't use monolingual norms.
[00:09:43.300]That's not going to provide
[00:09:44.330]the comparability that is necessary.
[00:09:46.280]Moreover, it doesn't control for differences
[00:09:50.070]in native language experience
[00:09:51.740]as well as education in the native language.
[00:09:55.090]In the U.S., some children may get
[00:09:56.730]some native language instruction, others may not.
[00:09:59.040]I had none.
[00:09:59.970]If you tested me by the time I mentioned,
[00:10:02.810]by the time I was seven-years-old in the second grade,
[00:10:05.020]my Spanish skills probably would've shown
[00:10:06.680]significant deterioration, because I did not get
[00:10:09.740]any education in my native language.
[00:10:12.630]So, none of these approaches
[00:10:14.690]are really going to be satisfactory.
[00:10:16.070]And, with respect to the last native language testing,
[00:10:18.540]even if that were a solution,
[00:10:20.440]there are 350 different languages
[00:10:22.320]spoken in the United States,
[00:10:23.500]according to the American Community Survey of 2015.
[00:10:25.780]That means no one's going to develop
[00:10:28.170]350 different versions of the same language test.
[00:10:31.580]And it may not even be as important as we think.
[00:10:34.200]The specific language that the person speaks
[00:10:36.550]may not be the issue,
[00:10:37.710]and I'm going to show you some evidence about this
[00:10:38.627]a little bit later.
[00:10:40.630]So, this is where we have been.
[00:10:42.750]This is what has created kind of the panic
[00:10:45.490]among practitioners in the field
[00:10:46.710]working with English learners.
[00:10:47.800]What do you do?
[00:10:48.890]Everything that you may have been taught,
[00:10:50.580]everything that has been tried
[00:10:52.110]hasn't exactly led to something that has been useful
[00:10:55.810]or which you could say actually demonstrates fairness
[00:11:00.420]particularly when it comes to measurement validity.
[00:11:02.920]And so, it was in this spirit
[00:11:04.490]that the standards were really revised significantly
[00:11:08.140]to include fairness as a foundational issue
[00:11:11.260]and a foundational chapter.
[00:11:13.170]Plus, the issues of fairness were spread out
[00:11:15.260]across the other chapters as well so that it was clear
[00:11:18.900]that fairness is basically an issue of validity
[00:11:22.420]and that it must be something
[00:11:24.090]that is attended to at all stages
[00:11:26.840]of both test development, construction, and use.
[00:11:31.410]So, it isn't something that we just have to think about
[00:11:33.870]a little bit on the side.
[00:11:34.990]It's actually an integral part of what you're required to do
[00:11:37.610]when you're working with English learners.
[00:11:39.520]In previous versions of the standards,
[00:11:42.350]most of the emphasis and much of it was really related to
[00:11:45.130]either specific subgroups that were language-based,
[00:11:48.260]or were maybe disability based,
[00:11:50.590]or had different racial or ethnic backgrounds,
[00:11:53.610]which sometimes got equated
[00:11:55.590]and conflated with cultural backgrounds,
[00:11:57.260]which is really not the same thing.
[00:11:59.130]And, again, that's part of why there were some difficulties.
[00:12:02.280]The current version really has taken that apart
[00:12:04.530]and disentangled it to a great extent to be able to say,
[00:12:08.947]"Look, here is what fairness actually is
[00:12:12.107]"and what you must do."
[00:12:13.830]And I highlighted it for you here in the quote.
[00:12:16.190]This is all taken from page 49
[00:12:18.830]of the standards itself, the latest revision.
[00:12:21.434]And that is that, basically, when you're working with ELLs,
[00:12:26.170]there must be some kind of response
[00:12:27.930]to the test taker characteristics that could interfere
[00:12:30.760]with the validity of test score interpretation.
[00:12:33.460]And this is something that has been readily misunderstood,
[00:12:35.830]because we've taken it for granted
[00:12:37.330]that what interferes with it is the fact
[00:12:39.480]that there's merely a different language,
[00:12:41.890]but it's not that simple.
[00:12:43.380]Or we've taken and accepted the fact that it must be race
[00:12:47.984]or ethnicity that is creating the difference,
[00:12:50.990]and that's not necessarily true either.
[00:12:53.260]And I'm going to show you how that works and why
[00:12:55.820]in light of the research that has been laid out.
[00:12:58.960]Now, when I first started learning about these issues
[00:13:02.300]many, many decades ago, I was surprised,
[00:13:05.380]because I read and read and read
[00:13:09.100]that there seem to be no bias in testing.
[00:13:12.030]And I'm thinking, "But how can that be?"
[00:13:13.860]If English learners aren't scoring as well
[00:13:16.090]as they should be,
[00:13:17.410]then either people who were saying a century ago
[00:13:21.570]that English learners must be,
[00:13:23.420]or bilingual individuals, must be less intelligent
[00:13:26.720]than monolingual English speakers would be right.
[00:13:30.060]But I didn't think that was true.
[00:13:32.030]And, yet, when you look for bias, you didn't find it.
[00:13:34.600]Now, part of the reason you didn't find it, though,
[00:13:36.890]was that, as traditionally defined,
[00:13:39.150]bias relates primarily to issues of reliability,
[00:13:42.910]so the items themselves, the content or the novelty of them,
[00:13:45.880]the structure, the measurement of the reliability
[00:13:49.460]in terms of its accuracy or the error,
[00:13:51.730]none of these things actually demonstrate any kind of bias.
[00:13:54.770]Neither does the fact or structure,
[00:13:56.360]the correlation between the items and the constructs
[00:14:00.030]that are being measured.
[00:14:01.100]Even predictive validity,
[00:14:02.350]which I know is an asterisk of validity,
[00:14:03.950]but, to me, it's a lot more like reliability anyway,
[00:14:06.100]but it doesn't change the fundamental meaning
[00:14:09.950]of the measurement in any way.
[00:14:11.930]If you lack the ability to speak English,
[00:14:15.670]then you perform poorly on a test that requires English.
[00:14:19.400]And if requiring English skills is part and parcel
[00:14:22.610]of what it takes for you to succeed in your job,
[00:14:24.890]succeed in your education,
[00:14:26.550]then we will predict you will not succeed,
[00:14:29.050]because you don't have those skills.
[00:14:30.660]So, there's nothing wrong with the predictive validity.
[00:14:33.520]The question is why don't you have that ability,
[00:14:36.430]and this is where things change.
[00:14:38.680]Even differential item functioning,
[00:14:40.390]which is designed to look at
[00:14:43.940]the task and the item characteristics
[00:14:46.610]independent of who's taking the test, all right?
[00:14:49.400]It still doesn't show any difficulty
[00:14:51.910]when we look at it traditionally,
[00:14:53.580]partly because what we're looking at
[00:14:55.730]interacts with the test in a developmental manner,
[00:14:58.710]not in a content or measurement manner.
[00:15:02.020]So, on the other side, what you see here is
[00:15:03.610]the potential bias in working with ELLs
[00:15:05.460]comes from the idea and the notion of construct validity.
[00:15:08.600]It's very simple.
[00:15:09.900]When you give the test,
[00:15:11.090]are you measuring what you think you're measuring?
[00:15:12.960]So, for example, if I give you a test
[00:15:16.640]that requires you to listen to my instructions,
[00:15:19.170]and they're lengthy,
[00:15:20.336]I may not be measuring what I think I'm measuring.
[00:15:24.640]I might be, instead, measuring your ability
[00:15:26.490]to just understand what I said.
[00:15:28.163]So, that would be an aspect of construct validity.
[00:15:30.027]On the other hand, there are times
[00:15:32.090]when I am measuring what I think I'm measuring,
[00:15:34.780]and yet, it still may not be valid
[00:15:37.010]when it comes to interpretation.
[00:15:38.900]This is where diagnostic validity is very, very important,
[00:15:42.210]because interpretive invalidity
[00:15:44.380]follows from not understanding why
[00:15:48.240]the performance is low.
[00:15:50.730]And if the performance is low
[00:15:52.270]and attributed to a disability,
[00:15:54.620]then that's interpretive invalidity.
[00:15:56.640]But if the interpretation is
[00:15:58.210]that although the performance is low,
[00:15:59.820]the reason for being low can be explained
[00:16:02.400]by external and extrinsic factors,
[00:16:04.850]then that cannot be a disability.
[00:16:07.280]And I'm going to give you an example of what I mean by this,
[00:16:09.430]because it probably sounds a lot more complicated
[00:16:11.450]than it really is.
[00:16:12.417]And here's a simple one.
[00:16:14.130]I think you're all familiar with this.
[00:16:15.230]So, let's say we have a block construction kind of task
[00:16:18.260]on the left-hand side here.
[00:16:19.830]And this is typical of what you might find
[00:16:21.290]on many different kinds of tests.
[00:16:22.990]But let's say that,
[00:16:24.310]although the intent of this particular task
[00:16:26.710]is to measure visual processing,
[00:16:28.380]such as visualization as a narrow ability,
[00:16:31.250]what if the instructions
[00:16:32.380]given to a five-year-old child go like this?
[00:16:35.400]Assemble these blocks together in the correct manner
[00:16:37.840]so they are identical to this illustration.
[00:16:40.770]Well, that's pretty fancy language
[00:16:42.800]to be asking a five-year-old,
[00:16:44.140]and obviously, nobody constructing a test would do that.
[00:16:46.690]But if it were, and then the individual taking the test,
[00:16:50.090]the child, has trouble with the block construction task,
[00:16:52.977]who do you blame?
[00:16:54.160]Is it the child's fault that they actually lack the ability
[00:16:57.830]to engage in visual processing?
[00:16:59.300]They have deficits in this area.
[00:17:01.030]Or is it merely that they didn't understand
[00:17:03.520]exactly what you were asking them to do?
[00:17:05.800]They didn't understand the wording,
[00:17:07.660]so they don't know what they were supposed to do
[00:17:10.329]in quite the manner you expected them to do it.
[00:17:13.430]So, that would be evidence of construct invalidity.
[00:17:16.170]The presence of language factors, in this case,
[00:17:19.510]basically confounded the measurement of visual processes.
[00:17:23.410]So, that would be an example of basic construct invalidity,
[00:17:26.930]which can happen.
[00:17:28.220]But there's another type.
[00:17:29.760]And the other type is on the right-hand side here.
[00:17:31.900]Now, in this case, you have a listening comprehension task.
[00:17:35.350]So, basically, if I were to ask,
[00:17:38.450]after putting a blue block on top of a purple one,
[00:17:41.107]"Put the green block on the blue one."
[00:17:43.610]Now, that wording is more simple.
[00:17:45.320]Maybe a five or six-year-old should be able to get it.
[00:17:47.570]Relies on a couple of prepositions, but simple ones,
[00:17:51.180]and the names of the colors,
[00:17:52.880]so not very difficult linguistically.
[00:17:56.500]If a child who's an English learner
[00:17:58.210]then performs on this task poorly,
[00:18:00.880]what is the interpretation?
[00:18:02.780]Well, the interpretation would be,
[00:18:04.670]obviously, their listening comprehension is poor.
[00:18:07.300]Yes, but the reality is
[00:18:09.570]the listening comprehension is poor only in English
[00:18:12.510]so far as you can tell.
[00:18:14.250]And that's really key, because if you then say,
[00:18:16.630]well, the listening comprehension is poor,
[00:18:18.820]then that must mean they have
[00:18:19.683]a listening comprehension problem,
[00:18:21.010]some type of speech/language deficit, that is incorrect.
[00:18:24.280]That is interpretive invalidity.
[00:18:25.970]You're ascribing an intrinsic,
[00:18:29.660]let's say, failure
[00:18:30.830]to what might be an extrinsic kind of concern or variable.
[00:18:35.857]And so, the only way you would know
[00:18:37.610]whether listening comprehension is a problem
[00:18:39.370]is to follow up with some evaluation in the native language
[00:18:42.300]to see if that type of listening comprehension
[00:18:44.990]is also present in the native language.
[00:18:47.227]And this gets a bit tricky
[00:18:48.450]because of the issues that I mentioned before.
[00:18:50.900]The types of tools that we have
[00:18:52.140]may not necessarily provide you a fair comparison either.
[00:18:55.200]But it is arguable, and you can say that,
[00:18:58.020]well, it is a valid measure
[00:19:00.480]of the child's listening comprehension abilities,
[00:19:02.760]let's say GC, in English.
[00:19:05.270]That you can say, but what you cannot say is
[00:19:08.290]that that must then represent a disability.
[00:19:10.850]To say that would be incorrect and would be wrong.
[00:19:14.620]So,
[00:19:16.930]we're generally giving tests
[00:19:18.640]and we want to discuss the concept of comparability.
[00:19:21.980]And the assumption of comparability
[00:19:23.350]is something that underlies all testing that we do,
[00:19:26.170]particularly standardized norm reference testing.
[00:19:29.160]And there's a wonderful quote
[00:19:30.230]that I've long used from Salvia and Ysseldyke
[00:19:33.952]that they wrote in their book on assessment back in 1991,
[00:19:36.480]and it really clarifies what this point is.
[00:19:39.750]Okay?
[00:19:40.583]And what they're saying is that, essentially,
[00:19:43.690]each child comes to school or grows up to become an adult
[00:19:48.510]with a particular set of experiences.
[00:19:51.590]And these experiences are in terms of education,
[00:19:54.310]social, environment, language, culture,
[00:19:57.740]related to ethnicity, and all kinds of things.
[00:19:59.740]And then when we use a standardized test
[00:20:01.920]to measure their abilities in various aspects,
[00:20:05.030]basically, we have to assume
[00:20:06.950]that their experiences are similar.
[00:20:09.050]They don't have to be identical,
[00:20:10.210]but similar to the children on whom
[00:20:12.780]that test was normed and standardized.
[00:20:15.690]So that's what we call it.
[00:20:17.310]Now, we often think of this as a pattern of acculturation,
[00:20:21.280]but it's probably better to think of it as
[00:20:22.990]a pattern of acculturative knowledge acquisition.
[00:20:25.960]What have you learned during that time?
[00:20:27.860]And the reason I say
[00:20:28.895]acculturative knowledge acquisition versus acculturation,
[00:20:30.510]acculturation has a connotation related to identity,
[00:20:34.850]and people think, "Well, is it that I'm identifying
[00:20:37.667]"with the mainstream or not identifying?"
[00:20:39.567]"How much?"
[00:20:40.497]"Can't I identify to both of them?"
[00:20:42.210]Yes, you can, so it's not about that.
[00:20:44.170]It's about the acculturative knowledge acquisition.
[00:20:46.120]And how do I know this and how do I know that's correct?
[00:20:48.650]Well, when you continue and you examine the quote,
[00:20:50.600]they are talking about
[00:20:51.660]the child's general background experiences
[00:20:54.150]differing from the children on the test,
[00:20:57.110]so these background experiences.
[00:20:59.840]Okay?
[00:21:00.673]And they say if you don't have children that are comparable,
[00:21:03.810]then incorrect educational decisions
[00:21:05.450]or diagnostic decisions are likely to be made.
[00:21:08.110]And it's clear, with the very last line there,
[00:21:10.920]that when they say
[00:21:11.820]or when they're talking about a child's acculturation,
[00:21:14.200]they literally mean that they're talking about
[00:21:17.400]experiential background that differs,
[00:21:20.200]not simply that the child is of a different ethnic origin
[00:21:23.810]or necessarily identity.
[00:21:25.500]In other words, you have the same type of experiences.
[00:21:29.550]When I was seven-years-old, sitting in second grade,
[00:21:32.220]my linguistic experiences were vastly different
[00:21:35.820]than other seven-year-old monolingual English speakers.
[00:21:39.090]I'd only been learning English for two years.
[00:21:41.270]I was instructed in a language
[00:21:42.690]that was not my native language.
[00:21:44.330]Neither of those things happened to children
[00:21:47.370]who were native English speakers.
[00:21:48.850]When I was at home,
[00:21:49.890]I was learning about Puerto Rican culture,
[00:21:51.610]because when you have Puerto Rican parents,
[00:21:53.700]that's what they're going to teach you.
[00:21:54.980]They aren't going to teach you about American culture,
[00:21:56.700]because, frankly, they don't know American culture
[00:21:59.470]any more than they know English.
[00:22:00.960]So, they were teaching me the language and the heritage
[00:22:03.780]and the culture that they knew,
[00:22:05.030]and that was Puerto Rican culture.
[00:22:06.620]So, by the time I'm seven, even learning about the culture,
[00:22:09.670]I only knew two years' worth
[00:22:11.390]of cultural knowledge by that point,
[00:22:13.323]not the seven years that my peers were getting.
[00:22:15.920]And so to compare us on tests that require that knowledge,
[00:22:19.490]on tests that are based on that type of development,
[00:22:22.830]would've been erroneous
[00:22:24.390]and would've led to problems like this.
[00:22:26.820]Here's an example of a distribution.
[00:22:29.350]Let's say this is a distribution
[00:22:30.520]for native English speakers,
[00:22:31.580]and then we have two children who are English learners,
[00:22:35.006]Chaseito and Panchito, and neither of them scored very well.
[00:22:38.040]Chaseito's score is approximately ninth percentile.
[00:22:40.790]Panchito's score is at the first percentile.
[00:22:42.490]So, invariably, they look like they have
[00:22:44.700]some intrinsic deficits, something is really wrong,
[00:22:47.290]but this is not a true peer comparison group.
[00:22:49.390]This group does not meet the assumption of comparability
[00:22:52.200]that is necessary to derive a true diagnostic inference
[00:22:56.880]from their performance.
[00:22:58.620]So, what we need is this,
[00:23:02.180]a different distribution, a distribution of English learners
[00:23:06.010]who share the same type of cultural/linguistic background,
[00:23:09.437]as well as education and other factors
[00:23:11.710]like Chaseito and like Panchito, to be able to compare them.
[00:23:15.160]And when we do that, what you see,
[00:23:17.300]in terms of what happens, is that, suddenly,
[00:23:19.310]Chaseito scores about the 46th percentile, approximately,
[00:23:24.029]so that's about a 95/96 standard score,
[00:23:26.570]but Panchito still remains
[00:23:28.050]around the eighth or ninth percentile,
[00:23:29.390]so he's still way down there
[00:23:32.615]with a score that would indicate a significant deficit.
[00:23:36.370]So, in this case, only Panchito might truly be considered
[00:23:40.260]to have some type of deficit or disability here,
[00:23:42.770]but Chaseito would not.
[00:23:44.410]Nobody would look at that score and say,
[00:23:45.927]"Well, there's a problem,"
[00:23:46.980]because the comparison here is fair.
[00:23:50.020]It provides the comparability
[00:23:52.300]that establishes a true peer comparison
[00:23:55.600]so that there is diagnostic validity
[00:23:58.290]to the decision that might be made.
[00:24:00.840]And we move on.
[00:24:02.710]I think it's important to recognize
[00:24:04.360]that race and ethnicity, as I mentioned early,
[00:24:06.660]does not necessarily establish comparability.
[00:24:10.149]I have a nice quote here
[00:24:11.490]from an article by Lohman and colleagues,
[00:24:14.510]and they talk about how most studies,
[00:24:17.170]prior to maybe even five to ten years ago,
[00:24:19.890]didn't even really classify children as English learners.
[00:24:24.050]Rather, they classified them by ethnic groups,
[00:24:27.360]so they were Hispanic children,
[00:24:28.970]they were African American children,
[00:24:30.370]they were Asian Pacific Islanders.
[00:24:32.130]They weren't categorized by the fact
[00:24:34.920]that they spoke a different language.
[00:24:37.040]And notice that when you categorize individuals
[00:24:39.610]as English learners, they become a broader group,
[00:24:42.630]because that encompasses all types of children
[00:24:45.130]from all manner and form of ethnic and cultural backgrounds.
[00:24:48.930]It's not just a circumscribed group.
[00:24:51.470]But it also, then, widens the issue
[00:24:53.760]with respect to what an English learner is,
[00:24:56.410]because it implies somebody who is not
[00:24:58.890]a native English speaker.
[00:25:00.600]That is what becomes important,
[00:25:02.620]rather than the race or ethnicity.
[00:25:04.800]And so, they were calling for the need
[00:25:06.800]to, basically, separate the influence of ethnicity
[00:25:08.770]or race and ELL status on observed score differences.
[00:25:12.770]And this is what must be done
[00:25:14.320]when it comes to true peer relationships.
[00:25:16.300]If you're not taking this into account,
[00:25:18.620]then there is no way that you're able to construct
[00:25:22.930]a true peer comparison that provides you
[00:25:25.440]with the interpretive validity that you're looking for.
[00:25:29.960]And a good example of this is, again, myself.
[00:25:31.740]By the time I was ten-years-old and in fifth grade,
[00:25:35.900]I had been learning English for approximately five years,
[00:25:38.540]you could say, from kindergarten.
[00:25:40.330]But there was another ten-year-old Puerto Rican kid
[00:25:43.560]in my classroom.
[00:25:44.910]He was the same age, ten.
[00:25:46.450]He was in the same grade, five.
[00:25:47.283]He was in the same school,
[00:25:49.180]which means he was in the same geographic location.
[00:25:51.320]He had the same SESs I did.
[00:25:53.320]His father was in the military just like my family was.
[00:25:55.900]He was Puerto Rican, same race/ethnicity as I was.
[00:25:58.900]So, virtually, on every, single stratification variable
[00:26:03.310]you can think of, we were matched.
[00:26:05.790]So, does that mean
[00:26:06.730]that evaluating us provides comparability?
[00:26:09.790]That we are the same, so therefore,
[00:26:11.830]how I perform is how he should perform?
[00:26:13.580]Or how he performs is how I should perform?
[00:26:15.640]Especially when I say
[00:26:16.473]that I had been learning English for five years?
[00:26:18.910]But he actually had just gotten here
[00:26:20.970]from Puerto Rico last year,
[00:26:22.830]so that meant he had been learning English one year
[00:26:25.340]and had received his prior education all in Spanish.
[00:26:28.740]I had no Spanish education.
[00:26:30.740]That means if you test us in English,
[00:26:32.880]I'm likely to do far better than he would do.
[00:26:35.230]But if you test us in the native language,
[00:26:36.850]he's likely to do far better than I would do,
[00:26:39.330]because of my attrition and lack of education
[00:26:41.920]due to lack of education in my native language.
[00:26:44.230]So, we are not comparable even though we are matched
[00:26:47.600]on every, single possible variable
[00:26:49.490]that people could think of,
[00:26:50.630]because the one variable we need to be matched on,
[00:26:52.860]which is language exposure,
[00:26:54.790]either in English or even in a native language,
[00:26:56.920]is not controlled in the norm sample.
[00:27:00.330]So, the other thing that I think is important
[00:27:02.650]is to understand that the native language itself
[00:27:07.040]isn't going to establish comparability either.
[00:27:09.440]And in the example that I just gave,
[00:27:11.420]if we are evaluating in Spanish,
[00:27:13.080]you're more likely to get scores closer to true ability
[00:27:17.210]for the other child than you were for me,
[00:27:19.580]because I may have been growing up speaking Spanish,
[00:27:23.620]but I didn't get educated in Spanish.
[00:27:25.630]There was no formal teaching going on.
[00:27:27.970]My parents are also not very highly educated,
[00:27:29.990]so why would my development in Spanish language,
[00:27:33.050]particularly academic skills like reading and writing,
[00:27:35.750]why would they develop in Spanish
[00:27:37.520]without any instruction or without any models
[00:27:39.480]from my parents to be able to provide that?
[00:27:41.290]That's not what was going on.
[00:27:43.100]And so, we, again, have to understand
[00:27:44.840]that it isn't the language
[00:27:47.040]or it isn't even the specific language
[00:27:51.010]that matches children.
[00:27:52.670]It is the degree of difference between
[00:27:56.940]that child and that child's peers based on that development.
[00:28:01.840]So, as I just explained,
[00:28:03.700]a child who is ten-years-old in my class
[00:28:06.590]when I'm ten-years-old, we both have same age, grade, SES,
[00:28:10.440]gender, geographic location, race, ethnicity,
[00:28:14.670]we have everything the same, but we're not the same,
[00:28:17.430]because our language exposure is different,
[00:28:21.150]and it's different in both of those languages.
[00:28:24.100]So, merely testing both of us in Spanish
[00:28:27.800]isn't going to really solve the issue.
[00:28:29.670]Merely testing both of us in English
[00:28:31.420]isn't necessarily going to solve the issue
[00:28:33.100]unless we can establish true peer comparison.
[00:28:36.300]Because I, in my experience, cannot be equated
[00:28:39.990]or stand as a, let's say, normative basis
[00:28:43.480]for comparing the other child.
[00:28:45.330]We must have different peer groups to be able to do that.
[00:28:49.037]And so, that's where, I think,
[00:28:54.300]the development of fairness in testing
[00:28:57.340]needed to go with English learners,
[00:28:59.400]that the concept of differential language exposure
[00:29:02.580]was more important than people had either understood
[00:29:06.060]or given it credit for.
[00:29:07.840]And
[00:29:10.070]it's a variable that I think is difficult to understand.
[00:29:13.090]I'm going to try to explain it the best I can,
[00:29:15.090]but a lot of people, I don't think, really grasp it,
[00:29:18.050]because we see it in a more simplistic manner
[00:29:21.130]by either dichotomizing it and saying,
[00:29:22.857]"Well, either it's verbal or nonverbal,"
[00:29:25.940]which doesn't actually work,
[00:29:27.330]or it's, "Well, if you speak Spanish
[00:29:29.437]"and that's your native language,
[00:29:30.397]"then we test you in Spanish," and that doesn't work either.
[00:29:33.170]It is the differential language exposure
[00:29:35.370]that is most important.
[00:29:37.350]So, English learners must be understood
[00:29:39.560]as not being a monolithic group.
[00:29:42.060]I mentioned that EL broadens the category
[00:29:45.190]when we talk about English learners,
[00:29:46.463]because it can include individuals
[00:29:48.630]who are simply nonnative English speakers.
[00:29:51.950]The category of ELLs contains everyone
[00:29:55.380]who is not a monolingual native English speaker.
[00:29:59.550]Everybody else is an English learner,
[00:30:01.510]and that means that there's such variability
[00:30:04.880]in terms of all of the factors
[00:30:06.460]that we're really talking about,
[00:30:07.970]but the one that makes the difference is the exposure
[00:30:12.010]to the language in which you're evaluating the individual
[00:30:14.810]and their opportunity for learning that particular language
[00:30:18.010]and developing the acculturative knowledge
[00:30:20.500]within that language or within the culture of that language.
[00:30:23.700]And so, this is a wonderful quote here from Fisher and Frey,
[00:30:27.440]and I wish I had written this, because it's beautiful,
[00:30:30.840]and they get to the heart of the matter
[00:30:32.730]that I'm talking about right here very clearly,
[00:30:34.760]and I'd like to read it for you here.
[00:30:35.980]It says, "It is unlikely that a second-grade English learner
[00:30:38.677]"at the early intermediate phase of language development
[00:30:41.127]"is going to have the same achievement profile
[00:30:43.017]"as the native English-speaking classmate
[00:30:44.667]"sitting next to her."
[00:30:45.900]And I think it's important to understand what this means.
[00:30:48.510]It means that
[00:30:51.330]you can't expect a child in second grade, third grade,
[00:30:56.270]to have the same abilities in things like reading,
[00:31:00.940]writing, and language development,
[00:31:03.610]right, as compared to their native English-speaking peers
[00:31:06.860]in that same second or third grade,
[00:31:08.980]because they have not had that much time
[00:31:12.120]learning in English.
[00:31:13.780]And it's clear, it says,
[00:31:14.857]"The norms established to measure fluency, for instance,"
[00:31:17.550]they used fluency as an example,
[00:31:18.737]"are not able to account
[00:31:19.687]"for language development differences
[00:31:21.787]"between the two girls."
[00:31:23.060]To me, they are the key words,
[00:31:24.960]language development differences.
[00:31:27.460]And what they're talking about,
[00:31:28.293]and they used fluency as an example,
[00:31:29.680]and I'll give you a fluency example.
[00:31:31.370]They didn't elaborate on that.
[00:31:32.700]But you're probably very fluent
[00:31:36.540]with things like the alphabet,
[00:31:38.170]things you over-learned when you were a child
[00:31:40.170]in elementary school.
[00:31:41.230]They made you say the alphabet so many times
[00:31:43.630]that if I say, Please say the alphabet quickly,"
[00:31:45.710]you have no trouble going,
[00:31:46.727]"A-B-C-D-E-F-G-H-I-J-K-L-M-N-O-P."
[00:31:48.947]All right?
[00:31:49.780]Now, you think that means you're very fluent.
[00:31:51.810]Well, if I didn't have time to practice as much as you did,
[00:31:56.250]if I didn't even learn the alphabet
[00:31:58.110]until years after you did,
[00:32:00.550]then did I develop that same amount of fluency?
[00:32:03.480]Probably not, because by the time I'm in fourth grade,
[00:32:06.380]fifth grade, and now I've mastered the alphabet,
[00:32:08.960]we're not doing rehearsal and practice anymore.
[00:32:11.540]I'm not likely to have that.
[00:32:13.110]So, for me, when you say, "Please say the alphabet quickly,"
[00:32:16.920]I might do it as well as you.
[00:32:19.140]In fact, for me, it would be as if I were to ask you
[00:32:22.540]to say the alphabet quickly, but do it backwards.
[00:32:26.050]Go ahead.
[00:32:27.810]Z-Y-X-W-T-U-V I'll give you to start off.
[00:32:30.360]S-R-Q-P-O-N-M,
[00:32:32.340]okay, I'm waiting for the rest of you
[00:32:34.020]to catch up with me, right.
[00:32:35.180]You can't do it.
[00:32:36.020]Why can't you do it?
[00:32:37.190]It's not that you lack fluency.
[00:32:38.900]Well, it is that you lack fluency,
[00:32:40.550]but you have fluency in a direction
[00:32:42.640]and in only the direction
[00:32:43.730]which you rehearsed it and practiced it,
[00:32:45.130]not in the direction that you don't have,
[00:32:46.720]and so that's a problem.
[00:32:48.150]If I didn't get a chance to even practice it forward,
[00:32:50.840]why should I have it?
[00:32:51.960]And it's not just about the alphabet.
[00:32:54.090]What about numbers and counting?
[00:32:55.920]Counting to ten quickly, counting by twos,
[00:32:57.780]counting by fives, counting by tens,
[00:32:59.130]things you can do quite readily, quite easily.
[00:33:01.900]But if I say, "Count backwards from 100 by sevens, go,"
[00:33:05.940]93, I'll give you the first one, 86,
[00:33:08.600]you see how slow you suddenly become?
[00:33:10.250]It's not that you have all this automatic fluency.
[00:33:13.590]You have the fluency only in the ways
[00:33:15.530]that you practiced and rehearsed what you did,
[00:33:17.840]but an English learner doesn't have always
[00:33:21.130]that same level of opportunity
[00:33:23.310]to be able to master the fluency like you did,
[00:33:26.280]and that creates the difference.
[00:33:27.443]And, of course, in here, in the quote, they end it with,
[00:33:29.857]"A second analysis of the student's progress
[00:33:31.460]compared to linguistically similar students is warranted."
[00:33:35.350]And that is what must be done.
[00:33:37.620]If we're going to get a valid measure
[00:33:39.410]of whether or not you truly fluent in a particular language,
[00:33:43.330]then I must compare you to others
[00:33:45.620]who have the same level or similar level of development
[00:33:49.440]in that language as you have.
[00:33:51.760]And you see how it's tricky?
[00:33:52.900]Because, ordinarily, age takes care of that for us.
[00:33:55.990]For a monolingual English speaker,
[00:33:57.920]age is a perfect, perfect control variable
[00:34:00.970]for language development differences,
[00:34:02.433]because we assume, don't we, that a child growing up
[00:34:06.270]has some adult that is raising them
[00:34:09.540]and that that adult is speaking to them in some language,
[00:34:12.830]so there's some development that is occurring.
[00:34:15.290]We're not talking about feral children,
[00:34:17.050]or children raised by wolves, or in caves,
[00:34:19.489]or by themselves, and in closets.
[00:34:21.949]We're talking about children who have social interaction
[00:34:25.500]and language as a fundamental part of that.
[00:34:27.710]So, if you're seven-years-old
[00:34:29.179]and you have a parent and that parent speaks English,
[00:34:31.210]you've been learning English for seven years.
[00:34:33.230]Now, some people say,
[00:34:34.063]"Well, what about if you have
[00:34:35.357]"one parent who speaks one language,
[00:34:36.617]"the other one speaks the other language?"
[00:34:38.000]Well, then you're splitting time between the two languages.
[00:34:43.210]Are you a monolingual speaker at that point?
[00:34:45.040]No, so, in essence, you're more like an English learner,
[00:34:48.130]however, you have an advantage
[00:34:49.300]of having heard at least some English
[00:34:52.210]and some other language from birth,
[00:34:54.030]which means you've split your time between the two.
[00:34:56.219]So, by the time you're five-years-old,
[00:34:57.730]you have about two-and-a-half years of development
[00:35:00.290]in each one.
[00:35:01.320]The nice thing, though, is that as you get older,
[00:35:03.370]you begin to connect those two and transfer between them,
[00:35:05.840]something that a bilingual or an English learner
[00:35:08.220]who doesn't learn from birth can't do,
[00:35:10.090]because they're not getting sufficient early exposure
[00:35:13.560]to be able to make that connection.
[00:35:15.380]They don't develop
[00:35:16.213]the kind of cognitive academic language proficiency required
[00:35:19.100]to make that transfer.
[00:35:20.250]So, learning two languages simultaneously
[00:35:22.290]from the beginning, wonderful.
[00:35:23.640]No problem at all.
[00:35:24.840]But that's not the typical ELL that we run across.
[00:35:27.260]And, again, it's still important to make sure
[00:35:29.670]that we compare individuals
[00:35:31.200]with the same level of language exposure.
[00:35:34.200]So, unfortunately,
[00:35:37.170]this concept has either been misunderstood,
[00:35:39.760]ignored, not identified, I don't know,
[00:35:42.550]but the historical literature is really replete
[00:35:46.630]with studies that look at things
[00:35:48.430]from a verbal versus nonverbal kind of approach,
[00:35:51.840]and that really is a false dichotomy.
[00:35:54.330]Not all tests require the same amount of language
[00:35:57.420]and not all tests remove the same amount
[00:35:59.980]of language or culture.
[00:36:01.320]They're variable.
[00:36:02.370]It's more of a linear proportional kind of effect.
[00:36:05.080]Likewise, ELLs don't perform at the same level
[00:36:07.900]on all nonverbal tests,
[00:36:09.370]so it's not like their scores should be exactly the same
[00:36:11.640]on all of those
[00:36:12.473]and then perform the same as they do on all verbal tests.
[00:36:15.470]There's variability within their performance
[00:36:17.540]even among those groups.
[00:36:18.980]And, finally, excuse me, we should really then look at
[00:36:22.190]test performance for English learners
[00:36:23.770]as a continuum formed by a proportional attenuation
[00:36:28.440]of performance related to how much language
[00:36:31.870]and acculturative knowledge is required
[00:36:34.210]in a particular sub-test or test
[00:36:36.360]that we're giving to the individual.
[00:36:38.200]And this is what I think has been missing,
[00:36:40.950]but it's always been there.
[00:36:42.120]It's not that this is something that should be new.
[00:36:44.410]It's always been in the information.
[00:36:46.990]In fact, you can see it quite readily.
[00:36:48.940]If I just create a simple linear pattern from low to high
[00:36:53.820]and I look at the test performance of English learners,
[00:36:57.650]okay, then what I see is essentially this.
[00:37:00.070]Tests that require little age-based knowledge
[00:37:02.710]or cultural acquisition and language development
[00:37:07.520]are those tests in which English learners perform
[00:37:09.720]close to the mean, near 100.
[00:37:11.570]But on those tests that require a great deal of language,
[00:37:15.380]age-expected levels of development in language
[00:37:17.997]and age-expected levels of cultural knowledge,
[00:37:21.390]those performances for ELLs
[00:37:23.770]on those types of tests are much lower,
[00:37:26.120]frequently going to about a standard deviation
[00:37:29.360]to a standard deviation and a third lower, up to 20 points.
[00:37:32.880]So, the other tests kind of fall in between.
[00:37:35.990]And how do we know that they actually do fall in between?
[00:37:38.610]Well, it's actually fairly easy to examine,
[00:37:41.330]because we can look at various studies that show us
[00:37:44.720]whether or not there are differences in performance.
[00:37:46.350]And this is a great example of a contemporary study
[00:37:50.860]that really emphasizes how important
[00:37:53.720]possession of age-appropriate language ability truly is
[00:37:57.560]in the performance of a given test.
[00:37:59.310]And this is a study done by Kevin McGrew,
[00:38:04.350]Cormier, and Ysseldyke as well.
[00:38:06.400]And what they did here was
[00:38:08.050]to independently operationalize language
[00:38:10.750]and then use that to predict how much variance was explained
[00:38:14.740]on each of the sub-tests of the WJ4.
[00:38:18.620]Is it the WJ4?
[00:38:19.660]Or it might WJ3, I don't recall.
[00:38:22.480]Anyway, here are the sub-tests listed
[00:38:24.890]and here's the amount of variance listed for you
[00:38:26.900]in terms of how much was explained
[00:38:29.030]by language operationalized independently.
[00:38:31.670]And the pattern is not random.
[00:38:34.790]If you look at it,
[00:38:35.910]verbal comprehension gets explained the most by language
[00:38:40.600]in terms of almost 79 to 80% of the variance.
[00:38:44.500]That is quite high.
[00:38:45.530]That is remarkable, so that means that verbal comprehension,
[00:38:48.310]not surprisingly, requires a lot of language ability,
[00:38:50.860]age-appropriate language ability.
[00:38:52.550]But as you go down the list, general information,
[00:38:55.360]that's a acculturative knowledge.
[00:38:56.750]Concept formation, that's more fluid reasoning test,
[00:38:59.270]except that the instructions for that one
[00:39:01.360]are very linguistically demanding,
[00:39:03.170]requiring the child to use conjunctions
[00:39:05.370]like and and or correctly,
[00:39:06.850]so there's a lot of language there.
[00:39:08.200]Visual-auditory learning, and you go down the list,
[00:39:10.560]you start to see that they become less language-oriented.
[00:39:13.730]By the time you get to the very bottom,
[00:39:15.970]you can see spacial relations, planning, picture recall,
[00:39:18.610]tests that really don't require much language at all.
[00:39:21.460]And so, what that means is that this list,
[00:39:23.880]in terms of the order of the variance explained,
[00:39:26.393]goes from highest language demands
[00:39:28.360]to the lowest language demands,
[00:39:30.390]and that language accounts for more and more of the variance
[00:39:34.780]the higher the test demands in terms of language,
[00:39:37.390]in terms of age-expected development in that language.
[00:39:40.470]But it can be virtually zero
[00:39:42.220]by the time you get down to tests that don't require that.
[00:39:45.040]So, it isn't a simple dichotomy.
[00:39:47.070]You can't draw a line through this and say,
[00:39:48.787]"Hey, here you go.
[00:39:49.747]"This is verbal and this is nonverbal."
[00:39:51.580]You really have to say it is a linear kind of effect,
[00:39:54.510]and you see it for seven to ten-year-olds,
[00:39:56.280]11 to 14, 15 to 18,
[00:39:57.860]although the 11-14 pattern is a little bit different
[00:40:00.960]than the seven to ten, 15 to 18,
[00:40:02.500]and they did not discuss why that might've occurred.
[00:40:04.740]I'm not really sure myself.
[00:40:06.200]The only thing I ever usually say is maybe that's hormones.
[00:40:08.540]I don't know, teenagers, but the nice thing is
[00:40:11.020]that by the time teenagers get a little bit older,
[00:40:12.840]they seem to come back to their senses
[00:40:14.150]and they look like they did when they were seven to ten,
[00:40:16.310]so much more closely there.
[00:40:17.850]But the pattern is very, very consistent
[00:40:21.490]across the age ranges and from top to bottom
[00:40:24.680]that different levels of language provide different levels
[00:40:28.430]of explanation and accounting for variance
[00:40:32.520]depending upon the nature of each of the tests.
[00:40:35.530]So, you could easily, if you wanted to,
[00:40:37.450]just create sections here and say here's very high language,
[00:40:41.010]here's moderate, here's low language,
[00:40:42.407]here's very low language, and so on.
[00:40:44.440]It's really quite simple.
[00:40:45.880]Now,
[00:40:47.200]I did some work earlier where I looked at the actual values
[00:40:51.680]in the sub-tests and the performance
[00:40:53.760]and said, "Well, do we see
[00:40:55.847]"a similar kind of pattern of decline?"
[00:40:57.800]So, this was preceding the study that I just showed you,
[00:41:00.580]but it really is reinforcing what we have seen already.
[00:41:04.450]If you look on the left side here,
[00:41:06.140]the sub-test names are all in order.
[00:41:08.140]Information, vocabulary, similarities, comprehension,
[00:41:10.320]in a particular order, but that order is not random either.
[00:41:13.380]High language from the very top
[00:41:15.540]down to low language, like coding, picture completion,
[00:41:17.840]object assembly, and so on.
[00:41:19.120]And then here are four different studies
[00:41:21.430]starting with Jane Mercer's work
[00:41:22.610]with the SOMPA back in 1972,
[00:41:24.835]Dyana Vukovich and Richard Figueroa's follow up
[00:41:27.190]to that study.
[00:41:28.023]They took a subsample of about 400 kids
[00:41:30.620]from the 900 that Jane had in her sample,
[00:41:33.390]followed them for ten years,
[00:41:35.250]and here are the scores they got.
[00:41:36.730]And what's interesting about it is you notice
[00:41:38.270]that because their English proficiency went up, obviously,
[00:41:41.130]over the ten-year period, their performance, right, went up,
[00:41:47.220]but the pattern stayed much the same.
[00:41:49.770]It didn't really change it a whole lot.
[00:41:52.280]And then when you look at the third group here, this is ESL.
[00:41:55.580]Actually, I believe it was primarily
[00:41:58.950]French/English emersion students in Canada,
[00:42:01.190]so they're not even in the U.S.
[00:42:02.310]They lack some cultural knowledge,
[00:42:03.540]so their scores tended to be lower.
[00:42:05.000]But look at the pattern again.
[00:42:07.130]All right, and then, finally,
[00:42:08.180]one of my former students, for her dissertation,
[00:42:09.730]did work on Long Island
[00:42:11.070]with a group of Spanish-speaking bilinguals,
[00:42:14.060]and again, look at the pattern.
[00:42:15.240]Very consistent with like Jane's work.
[00:42:17.230]And so, what you see is that when tests
[00:42:19.680]like information, vocabulary, similarities,
[00:42:21.970]rely on cultural knowledge and language,
[00:42:25.060]the performance tends to be the lowest.
[00:42:26.670]But when you move to tests like picture completion, coding,
[00:42:29.850]or object assembly, which have less language demands,
[00:42:31.920]less cultural knowledge built in,
[00:42:33.820]then the performance is much higher.
[00:42:36.110]And what I did, also, to give you an idea
[00:42:37.790]of how this could be used
[00:42:39.170]at least to guide what we're talking about
[00:42:41.560]in a very straightforward way,
[00:42:43.060]is notice that if you convert all of these scale scores
[00:42:45.810]into deviation IQ metric, and then take a grand mean,
[00:42:48.760]you see that the mean for information
[00:42:51.460]is approximately one standard deviation below the mean,
[00:42:53.393]a score of 85.
[00:42:54.780]Whereas the mean for coding is closest
[00:42:57.970]to the normative mean of 100.
[00:43:01.110]So, there you have it in terms of being able to say
[00:43:04.240]this is what we see, this is what we expect
[00:43:07.500]when we look at tests that are arranged
[00:43:10.450]from high language expectations and demands
[00:43:13.100]in terms of development
[00:43:14.040]to low language expectations in development.
[00:43:16.550]The pattern of performance is not merely dichotomous.
[00:43:19.760]It actually varies from very, very low
[00:43:22.520]standard deviation or more away from the mean
[00:43:24.770]to being right at the mean.
[00:43:26.780]And this, I think, is a nice way of looking at it,
[00:43:29.190]because instead of just looking at the values,
[00:43:31.220]you actually can look at the decline of the scores
[00:43:35.840]from left to right in the same order
[00:43:38.300]and how it really displays
[00:43:41.780]a pretty ubiquitous and robust kind of slope.
[00:43:45.940]It doesn't change a whole lot.
[00:43:47.720]This is typically what you see.
[00:43:49.240]Again, if you review the entire literature,
[00:43:51.420]you look at the sub-tests that are reported,
[00:43:54.230]particularly from studies where individuals are not referred
[00:43:58.660]and are not disabled.
[00:43:59.900]Obviously, we don't want to look at performance
[00:44:02.060]based on groups of children who might have disabilities,
[00:44:04.490]because that could change the normative standard anyway.
[00:44:07.890]So, when you have normal ELLs, this is what you get.
[00:44:12.010]And it's very consistent, it's very robust.
[00:44:15.260]It is something that really begins to speak
[00:44:17.330]toward the differential pattern,
[00:44:20.350]the proportional attenuation of performance
[00:44:23.710]based on differences in language exposure
[00:44:25.857]and language development.
[00:44:27.320]Now, there were two additional studies
[00:44:28.860]that really took this to task and said,
[00:44:31.560]okay, if language proficiency is really that important,
[00:44:34.540]then why don't we just look at the language
[00:44:36.650]and its effect directly on test performance.
[00:44:40.270]And this is an example of that.
[00:44:42.760]And this was published by a colleague of mine.
[00:44:46.270]I actually mentored dissertation.
[00:44:47.530]We published this later on.
[00:44:48.910]And it shows the performance on seven of the sub-tests
[00:44:53.980]from the WJ3 starting with, on the left, spacial relations,
[00:44:57.970]then visual matching, numbers reversed, sound blending,
[00:45:02.530]visual/auditory learning, concept information,
[00:45:03.840]the one I told you with the language,
[00:45:05.270]and verbal comprehension.
[00:45:06.280]So, they are arranged, primarily,
[00:45:08.280]in order of how much language is really required
[00:45:11.360]in each one of them.
[00:45:12.450]So, on the left side, low language.
[00:45:14.090]On the right side, high language.
[00:45:15.710]Well, if you're at the proficient level,
[00:45:17.320]according to the New York State
[00:45:18.770]English as a Second Language Achievement Test,
[00:45:20.450]this a test like the WIDA, the SELT, the ELPAC
[00:45:24.650]that people in states generally use
[00:45:26.410]for Title III compliance to measure general proficiency.
[00:45:29.170]So what do you get?
[00:45:30.090]Well, with the kids scoring
[00:45:31.680]at the highest proficiency group,
[00:45:33.960]you get very little decline.
[00:45:36.180]They only really drop toward the very end
[00:45:37.870]when language is very high,
[00:45:39.040]and even then, they're in the average range,
[00:45:40.710]sot here's not much of a decline
[00:45:42.290]once they've reached that level of proficiency.
[00:45:44.550]But what about if they're only at the advanced level,
[00:45:47.240]which is a misnomer, because, at that point,
[00:45:49.250]they really begin to drop significantly,
[00:45:51.110]almost down to about 82 in GC.
[00:45:54.330]The intermediate drops even further,
[00:45:57.430]and the beginning group drops even further than that.
[00:46:00.700]There's a widespread difference in their abilities,
[00:46:03.370]and I think this begins to indicate
[00:46:05.420]that this is language that's happening here,
[00:46:07.970]because we are measuring language proficiency
[00:46:10.240]and allowing it to vary, and that's precisely what we see
[00:46:12.910]in terms of what's going on here.
[00:46:14.830]Now, here's another example.
[00:46:16.700]This one was with the Woodcock-Munoz Language Survey Revised
[00:46:19.340]and the Wechsler Abbreviated Scale of Intelligence.
[00:46:21.800]So, using the Woodcock-Munoz Language Survey
[00:46:23.640]to group children into low, intermediate,
[00:46:25.810]and high proficiency, here's what you get.
[00:46:27.900]And when you measure their abilities
[00:46:29.830]according to, again, less language, on the right-hand side,
[00:46:33.630]more language, I meant on the left-hand side,
[00:46:35.670]more language on the right-hand side,
[00:46:36.960]so matrix reasoning, block design, letter/word ID
[00:46:40.550]all the way up to similarities, vocabulary,
[00:46:42.420]picture vocabulary, and so on, what do you get?
[00:46:44.620]Well, again, very little decline
[00:46:46.250]for the children with the highest proficiency levels,
[00:46:48.210]more of a decline for children
[00:46:49.760]with the intermediate proficiency level,
[00:46:51.760]and a very stark decline
[00:46:53.520]for children who have a very low proficiency level.
[00:46:56.690]So, clearly, language exposure,
[00:46:59.560]language proficiency in this case,
[00:47:01.490]but as a proxy for language exposure,
[00:47:03.690]is playing a role in how ELLs perform,
[00:47:06.550]which means that it would be, then, inappropriate
[00:47:08.960]to group all ELLs together without consideration
[00:47:12.080]for how much exposure and development they have in language.
[00:47:14.700]Otherwise, we would get a discriminatory kind of result.
[00:47:19.264]So, it was on this basis that recently embarked on an effort
[00:47:24.480]to try and see if this could actually be something
[00:47:27.260]that could be incorporated
[00:47:28.530]into test design and construction.
[00:47:31.150]And so, rather than use the old kinds of ideas
[00:47:35.120]where we're going to create a nonverbal test
[00:47:36.960]or we're going to do a native language test,
[00:47:39.000]there was some new types of decisions made
[00:47:41.970]and considerations that were put out there,
[00:47:43.940]particularly in light of
[00:47:45.070]what the standards were sort of addressing.
[00:47:47.760]One of the concepts in the standards is universal design.
[00:47:50.900]Can we create tests that are
[00:47:52.870]as universally applicable as much as possible?
[00:47:56.100]Now, that's not always going to be possible,
[00:47:58.000]but we can do that.
[00:47:59.480]And in light of that and these considerations,
[00:48:02.750]it is, perhaps, attainable.
[00:48:05.540]It is something you can do.
[00:48:06.900]It is, the concept of fairness
[00:48:08.087]and being able to achieve it is something that can be done.
[00:48:10.040]So, for example, think these considerations.
[00:48:12.610]In the U.S., we know that the focus
[00:48:14.150]is primarily on English language acquisition
[00:48:16.140]and academic skills, particularly for school-aged children.
[00:48:18.960]Anyone who comes to the U.S. and is in school is required
[00:48:23.910]to learn English, and I don't have a problem
[00:48:25.450]with them having to learn English.
[00:48:26.600]I have a problem, sometimes,
[00:48:27.433]with the way we help them learn English,
[00:48:29.110]but I don't have a problem
[00:48:30.080]with them having to learn English.
[00:48:31.530]So, that means nobody in the U.S. remains monolingual,
[00:48:35.040]so to speak, okay.
[00:48:36.410]And English is the accountability standard that all states,
[00:48:40.830]districts, agencies have to really comply with.
[00:48:43.670]Now, another thing,
[00:48:44.740]we know that English language acquisition,
[00:48:46.270]unlike a number of abilities,
[00:48:47.390]are really highly related to academic success.
[00:48:50.150]Highly related.
[00:48:51.590]Okay?
[00:48:52.423]Nothing more important than GC
[00:48:54.490]or language and cultural knowledge
[00:48:56.010]in the development of academic success,
[00:48:57.590]particularly areas like reading and writing,
[00:48:59.080]which are symbolic aspects of language to begin with.
[00:49:01.810]Number two, English can serve, if we think about it,
[00:49:04.960]as a common metric for measurement,
[00:49:08.080]because one thing that all English learners
[00:49:10.550]in the Unites States, not necessarily in other countries,
[00:49:13.170]but in the Unites States, that they have in common is
[00:49:15.910]that they're all learning English.
[00:49:17.440]So, irrespective of their native language,
[00:49:19.650]English is a process that they begin
[00:49:23.230]the moment they arrive here
[00:49:24.610]and the moment they begin to learn English.
[00:49:27.090]Number four here, receptive vocabulary, then, is something
[00:49:30.073]that we can measure immediately in the acquisition process.
[00:49:32.650]That's something ideal, because we don't have to wait.
[00:49:35.550]I can determine
[00:49:36.640]that you have absolutely no receptive vocabulary in English.
[00:49:39.560]I could determine that you have
[00:49:40.630]a very, very, very limited receptive vocabulary in English.
[00:49:44.391]But if I want to measure your expressive vocabulary,
[00:49:46.395]that's going to take more time,
[00:49:47.494]because I have to let you develop it first.
[00:49:49.510]You might not have any words until four months or so.
[00:49:52.680]You might not have much expressive language
[00:49:54.850]from anywhere between four months to a year.
[00:49:57.200]And so, the silent period, the pre-production phase,
[00:49:59.800]could last up to a year,
[00:50:01.660]so I would have to wait a much longer period of time
[00:50:04.210]to be able to evaluate you in that case.
[00:50:06.120]And then how much development you've acquired
[00:50:08.830]during that period of time would be related
[00:50:10.620]to other kinds of factors.
[00:50:11.500]But it doesn't make it as amenable to measurement
[00:50:14.550]as does receptive vocabulary.
[00:50:16.420]Number five, exposure to English is quantifiable.
[00:50:18.370]That's something we can do,
[00:50:19.630]and it addresses opportunity to learn.
[00:50:21.230]We know and can determine, to a certain extent,
[00:50:23.970]when a person began to formally learn
[00:50:26.520]and be exposed to English.
[00:50:28.460]Number six, testing in English eliminates the need
[00:50:30.220]for multiple language-specific versions of a test.
[00:50:32.280]This is a very important practical consideration,
[00:50:34.100]because there are never going to be
[00:50:35.750]350 different versions of the same test.
[00:50:38.298]It's just not going to happen.
[00:50:39.131]Once you get beyond Spanish,
[00:50:40.520]there's limited availability already there,
[00:50:42.960]even though we have many, many other languages
[00:50:45.050]for which there's tremendous need.
[00:50:46.610]And finally, testing in English eliminates the need
[00:50:48.410]for the evaluator to actually be bilingual.
[00:50:50.703]It's an important consideration,
[00:50:52.160]because there are very few bilingual professionals
[00:50:55.450]who are competent and qualified to conduct evaluations
[00:50:58.100]in the native language.
[00:50:59.580]So, that means that the majority
[00:51:00.950]of English learners in the U.S. will invariably be evaluated
[00:51:04.100]by people who do not speak their language,
[00:51:05.850]who do not understand their language,
[00:51:07.670]who are not bilingual themselves,
[00:51:09.380]and cannot communicate with them in that manner
[00:51:12.060]or provide any evaluation in that language.
[00:51:14.950]So, that's problematic.
[00:51:16.960]But based on that, that means we can begin a process
[00:51:20.010]by thinking if we test in English,
[00:51:22.430]we might be able to use that as the common standard
[00:51:24.490]and eliminate all of the difficulties that we've had before.
[00:51:27.050]However, there are some assumptions.
[00:51:28.550]One, we'd have to make sure
[00:51:30.130]that language does explain variance
[00:51:32.360]above and beyond race and ethnicity,
[00:51:34.270]so that would have to be demonstrated.
[00:51:35.430]And number two, that English language acquisition
[00:51:37.930]remains invariant regardless of the child's first language.
[00:51:41.770]So, we tried to accomplish that task
[00:51:44.080]in a project that was recently completed
[00:51:45.878]in the development of a test called the Ortiz PVAT.
[00:51:48.770]Here are the two norm samples from that particular test.
[00:51:50.927]On the left, the typical one for English speakers.
[00:51:53.480]On the right, English learners.
[00:51:55.060]And everything looks the same
[00:51:56.580]except for when you get down to race and ethnicity.
[00:51:59.990]For English speakers,
[00:52:00.980]that was a controlled stratification variable, it's there.
[00:52:04.430]But for English learners, it's not there.
[00:52:06.620]And the question is does it need to be there.
[00:52:08.190]And the answer is, as you're going to see, is no.
[00:52:10.220]What needs to bet here is what's listed there
[00:52:12.690]that is not listed in the other one,
[00:52:14.700]and that is language spoken at home.
[00:52:17.380]All right?
[00:52:18.213]There were 53 different languages spoken and sampled
[00:52:20.890]in the English learner sample.
[00:52:22.110]And proportion of lifetime exposure to English.
[00:52:25.390]There were 11 categories developed
[00:52:26.960]from zero to six months up to 16-plus years.
[00:52:30.120]There were ways of measuring it
[00:52:31.180]all the way down to one month
[00:52:33.280]if the person's only been learning English
[00:52:35.060]for less than a year as well.
[00:52:36.890]Now, what were the results of creating norm samples
[00:52:39.920]based on school-aged children like this
[00:52:41.800]with these variables?
[00:52:42.850]Well, here's part of what you see.
[00:52:45.330]The development was based on continuous norms,
[00:52:48.040]and this graphic doesn't provide
[00:52:50.140]continuous norm kind of idea.
[00:52:51.710]It really just shows groups.
[00:52:53.740]So, if you were to take a group of native English speakers
[00:52:57.930]and plot them, well, that you can do,
[00:52:59.705]because they don't vary in terms of exposure.
[00:53:02.800]They vary in terms of age, which controls for exposure.
[00:53:05.210]But for English learners, you have to vary exposure,
[00:53:08.037]and so that's why there are lines
[00:53:09.360]that are going up and down.
[00:53:10.600]But the problem is you can't do it continuously
[00:53:12.940]from 0% exposure to 99% exposure.
[00:53:15.910]So, instead, we kind of grouped them.
[00:53:17.560]We grouped them into high exposure
[00:53:19.670]from 51 to 100% lifetime exposure,
[00:53:22.060]moderate from 11 to 50, and then low from zero to 10%,
[00:53:25.830]and then created an average line for those groups,
[00:53:29.000]and this is what you see.
[00:53:30.530]Nice, beautiful, parallel lines
[00:53:32.510]that move from left to right in terms of age,
[00:53:36.550]so that as age increases, English language development,
[00:53:40.500]receptive vocabulary that is, actually increases,
[00:53:43.120]but also increases along the y-axis from bottom to top,
[00:53:47.500]that the higher your exposure is all the way up to native,
[00:53:50.620]the better your performance is as well.
[00:53:53.510]And that begins to show that age and exposure must be used
[00:53:57.670]in conjunction to develop true peer comparison groups.
[00:54:00.877]And how well does it actually represent that?
[00:54:03.570]Here's a good example here.
[00:54:05.630]If you compare monolingual English speakers
[00:54:07.840]to monolingual English speakers,
[00:54:09.550]the average monolingual English speaker,
[00:54:12.350]that's the blue line right here,
[00:54:14.270]the blue bar, I should say, on the left-hand side.
[00:54:16.590]You get a score of 100.
[00:54:18.110]Well, that's average.
[00:54:18.950]We all know that's what it is.
[00:54:20.520]But if you compare the children
[00:54:22.410]who were in the high exposure group
[00:54:24.290]to native English speakers, their score is only 94.
[00:54:27.830]It's already dropped by a third of the standard deviation.
[00:54:30.080]If you compare the medium exposure group, it's 90.
[00:54:33.610]It's two-thirds of the standard deviation below the mean,
[00:54:35.670]and if you compare the low exposure group,
[00:54:37.790]you see that they're at 86.
[00:54:39.100]They're almost one full standard deviation below the mean
[00:54:41.930]compared to monolingual English speakers,
[00:54:43.860]and that creates a problem,
[00:54:45.210]because these scores, then, do not have diagnostic validity
[00:54:49.190]and using them would be inappropriate
[00:54:51.090]other than for the purposes
[00:54:52.310]of being able to determine instructional level and need.
[00:54:56.040]But in terms of reporting them and using them
[00:54:58.290]for diagnostic purposes, that would be a mistake.
[00:55:00.760]The numbers that should be provided are the ones
[00:55:03.210]over on the right-hand side.
[00:55:04.550]When you compare children or the average child
[00:55:07.180]in the high exposure group, their score is 100.
[00:55:11.170]The average performance
[00:55:12.250]for the medium exposure group is 102.
[00:55:14.900]The average performance
[00:55:15.760]for the low exposure group is 100.2.
[00:55:18.350]Those are the scores that retain
[00:55:20.200]the diagnostic validity necessary to be able to say
[00:55:23.180]there's a deficit or there is no deficit.
[00:55:25.850]And you can see that not only can you create
[00:55:29.660]a norm sample that provides this,
[00:55:32.090]you can also demonstrate that race and ethnicity
[00:55:35.090]really aren't the driving influences on a particular test.
[00:55:38.450]For the English-speaking group,
[00:55:41.440]examination of the differences between the performance
[00:55:44.320]among African American, Hispanics, Whites,
[00:55:46.750]Caucasians, and other groups,
[00:55:48.910]pairwise comparisons show, basically,
[00:55:51.050]no significant difference between their performance.
[00:55:53.998]Even the effect size is .005.
[00:55:56.160]It's so small.
[00:55:57.974]Even an effect that was significant would be so small
[00:56:02.380]as to be probably meaningless.
[00:56:04.140]And this is on a language-based test.
[00:56:07.670]You see how different this is from what has been done before
[00:56:10.110]where we used nonverbal tests
[00:56:11.930]to try to reduce these racial differences.
[00:56:14.680]It doesn't have to be a nonverbal test to reduce them.
[00:56:17.040]We can do it with language as well.
[00:56:18.910]And beyond that, what about the other assumption
[00:56:20.940]of the invariance of learning English?
[00:56:23.040]What we find there as well is
[00:56:24.943]that no matter the first language
[00:56:26.900]that an individual is learning,
[00:56:28.530]the process of learning English is invariant
[00:56:31.170]and stays the same.
[00:56:32.840]And I think that's remarkable,
[00:56:34.010]because it means, then, that English learners
[00:56:36.860]don't need to be treated
[00:56:39.010]as if the language that they speak,
[00:56:40.710]specifically, is the problem.
[00:56:42.840]So, it's not about sampling all of the languages,
[00:56:45.960]although we did our best
[00:56:47.120]to be representative as much as possible.
[00:56:51.650]The actual language that a person speaks first,
[00:56:54.120]and then is now learning English,
[00:56:55.710]does not affect their acquisition of English.
[00:56:58.150]It is an invariant process.
[00:56:59.760]It does operate the same regardless of the language
[00:57:03.150]that the person tends to speak.
[00:57:04.740]And both of these, then,
[00:57:06.190]are really powerful indicators of fairness,
[00:57:09.890]the types of fairness that can set a standard
[00:57:12.680]for where testing can go.
[00:57:15.510]Because, as I've explained,
[00:57:17.040]where we've been before hasn't really produced
[00:57:19.900]the types of results or the attention
[00:57:22.070]to the issue of validity that we would expect
[00:57:24.950]or which is now currently specified in this standard.
[00:57:28.280]I believe that this type of direction is something
[00:57:30.830]that can really elevate what is being done
[00:57:33.610]when it comes to working with English learners
[00:57:36.730]in ways that we've not been able to accomplish before.
[00:57:40.070]Fairness isn't something
[00:57:41.530]that needs to be terribly complicated,
[00:57:43.850]but it is something that requires
[00:57:45.930]a great deal of attention and a great deal of time.
[00:57:48.340]These issues, by the way, were resolved over the course
[00:57:51.010]of about seven-and-a-half years of development.
[00:57:53.220]It wasn't something that just happened
[00:57:55.527]and just said, "Here's how easy it is to do it."
[00:57:58.633]I recognize the complexity of it,
[00:58:00.640]but I think it is what offers us
[00:58:02.490]the ability to be able to move forward.
[00:58:05.010]So, in summary, true peer group comparison is the only way
[00:58:09.780]that we're going to provide any kind of foundation
[00:58:12.120]for fairness and interpretive validity
[00:58:15.200]when we're doing diagnostic evaluations of English learners.
[00:58:18.150]It isn't going to happen by using any specific procedure,
[00:58:21.700]any specific tool, any specific kind of approach or idea.
[00:58:26.640]It has to be done in a way that we say that the individual,
[00:58:31.050]against whom we're comparing the examinee,
[00:58:34.980]the individuals in that norm sample
[00:58:37.877]and that group are true peers
[00:58:41.120]based on the amount of exposure
[00:58:43.830]that an individual has in that language.
[00:58:46.930]And this, again, applicable to English learners, obviously.
[00:58:49.600]But that's what's important when we're working with ELs.
[00:58:52.760]And construct validity must be understood, then,
[00:58:55.280]to be something that does not guarantee
[00:58:58.210]diagnostic or interpretive validity.
[00:59:00.490]Just because we say that the test
[00:59:03.210]is supposed to measure something,
[00:59:04.560]doesn't mean it always will, and even when it does,
[00:59:07.500]does not extend into diagnostic validity
[00:59:09.970]unless we can demonstrate that it's a true peer comparison.
[00:59:13.830]The developmental differences that I've been speaking about
[00:59:17.470]are not controlled by things like race or ethnicity.
[00:59:21.230]Not well or at all possibly
[00:59:23.640]as I've just kind of demonstrated.
[00:59:25.450]Cultural knowledge and language development
[00:59:27.580]are related to how much time you've been able to learn them,
[00:59:30.560]how much time you've had
[00:59:32.010]to be able to acquire that information.
[00:59:35.130]And if there's differences in the opportunity for learning
[00:59:38.280]and the amount of exposure,
[00:59:40.100]then those differences must be controlled.
[00:59:41.950]Otherwise, comparisons will be inherently unfair.
[00:59:48.050]What else? I think, finally,
[00:59:50.257]it is the beginning of something
[00:59:52.250]that maybe is unfamiliar to everyone,
[00:59:54.260]but if we don't use norms, if we don't have tests
[00:59:58.330]that have these types of true peer comparisons,
[01:00:02.530]then we have very little way of being able to determine
[01:00:06.310]whether our results are fair or not.
[01:00:09.240]It's simply something that must be attended to
[01:00:12.180]in order to control for those kinds of differences,
[01:00:15.200]because as we move toward aspiring to
[01:00:18.820]what the standards expect, I should say, or require,
[01:00:24.130]we have to, then, be able to say
[01:00:25.910]that the comparison meets those particular standards
[01:00:30.100]by demonstrating that the validity has not been compromised.
[01:00:34.500]Not in terms of the construct being measured
[01:00:36.420]and not in terms of the interpretation
[01:00:39.170]that we are providing.
[01:00:40.080]And if we do that,
[01:00:41.490]I think there's a whole new world out there
[01:00:43.450]that we're probably going to move toward,
[01:00:45.190]and I don't think it's impossible or difficult to do,
[01:00:47.360]and I've given you just one example of how it can be done.
[01:00:50.900]So, I believe I'm pretty much done here.
[01:00:54.160]I know that Jessica will probably pop back in
[01:00:56.550]and will start feeding me some questions.
[01:00:58.700]So, I'd be happy to use the remaining time
[01:01:01.060]to answer whatever questions I can.
[01:01:03.300]Thank you very much, Dr. Ortiz.
[01:01:05.980]So, if you have questions for Dr. Ortiz,
[01:01:08.520]you may enter those or copy those into the chat box
[01:01:12.400]while we wait a moment
[01:01:13.890]to see what kinds of questions we have.
[01:01:15.990]I will mention that if you requested to receive
[01:01:20.910]continuing education credit for this webinar,
[01:01:24.830]I will be sending to you a link
[01:01:26.740]for an evaluation at the conclusion
[01:01:30.800]and I will need you to complete that evaluation
[01:01:32.960]in order to send you your certificate.
[01:01:37.710]So, please be looking out for that.
[01:01:40.580]If you haven't requested continuing education credit,
[01:01:44.170]we still would like to have your feedback
[01:01:47.040]on our programming.
[01:01:49.070]It's very helpful for us as we continue to move forward
[01:01:51.930]and try to improve the professional development programming
[01:01:55.740]that we provide here from Buros.
[01:01:59.890]So, we'll give that some time.
[01:02:01.810]It's one of the things that it's difficult to tell from this
[01:02:05.790]if someone's typing or entering something,
[01:02:08.000]so we'll give just a little bit of time.
[01:02:10.760]Well, Jessica, I might've been just so completely thorough
[01:02:13.620]that nobody has any questions now.
[01:02:14.980]That's possible.
[01:02:16.869]It happens to me all the time.
[01:02:19.790]Just kidding.
[01:02:20.623]I'd be surprised if that was the case.
[01:02:22.926]I would be, too.
[01:02:23.759]I think it's compelling enough,
[01:02:24.610]I would be surprised if there wouldn't be someone
[01:02:26.510]who'd want to ask about it.
[01:02:28.580]I think I tend to raise more questions
[01:02:32.118]than I provide answers sometimes,
[01:02:33.500]so I'm sure there's much out there.
[01:02:39.157]I will also mention at this time,
[01:02:41.890]as I mentioned, this is the second in a series of webinars
[01:02:45.230]that we are hosting.
[01:02:46.850]And our next one will be also featuring
[01:02:49.850]a school psychologist on May 11th,
[01:02:53.250]Friday May 11th.
[01:02:54.083]That's Ryan Kettler from Ruckers University
[01:02:56.580]and he will be talking about
[01:02:58.370]the testing of individuals with disabilities
[01:03:01.430]and some of his research and practice in that area.
[01:03:04.310]So, if you have not registered for that, please join us.
[01:03:07.930]Or if you know of a colleague who would be interested,
[01:03:11.680]please ask them to do that as well.
[01:03:15.490]So, I still don't have anything at this point,
[01:03:18.890]but I do want to give, no, we do have one.
[01:03:22.170]So, how many school districts currently use the Ortiz PVAT?
[01:03:29.710]That's an interesting question.
[01:03:32.550]Let me explain that it, actually,
[01:03:34.590]only became available in January, so it's been out,
[01:03:38.390]what, February, March, April, three months.
[01:03:41.540]But from what I understand,
[01:03:43.560]it has been quite popular across the U.S.
[01:03:47.740]In fact, to give you an example,
[01:03:49.420]I've been traveling once a week, twice a week
[01:03:52.160]for probably the last seven months
[01:03:55.290]doing presentations and so on
[01:03:57.340]and I'll be going to California next week.
[01:03:59.190]I'll do four presentations there
[01:04:01.400]in southern California alone.
[01:04:03.430]So, I know it has been adopted pretty widely
[01:04:06.640]for a brand-new test,
[01:04:07.900]but primarily because it addresses big needs for ELLs.
[01:04:12.320]And one of the nice things,
[01:04:13.300]I was only talking about diagnostic validity here,
[01:04:16.400]that the test itself actually does address issues
[01:04:20.040]related to things like progress monitoring, growth,
[01:04:23.750]intervention, and instructional level and need.
[01:04:26.670]So, ELLs don't create just a problem
[01:04:30.120]when it comes to testing.
[01:04:31.010]They create a problem that's more broad-based,
[01:04:33.430]and so the test responds to that,
[01:04:34.497]and I think there's been
[01:04:36.650]really a lot of excitement and current adoption.
[01:04:39.200]So, I don't know the specific numbers,
[01:04:42.240]but I know that it's doing very well
[01:04:44.810]in terms of what people need,
[01:04:45.820]because it addresses, obviously, a big need here.
[01:04:50.230]Yeah, and I'll also mention
[01:04:51.560]that at Buros here, we also published
[01:04:54.150]the Mental Measurement Yearbook series
[01:04:55.760]and we have received the test for reviewing for that series,
[01:05:00.130]so we also will be having some reviews for that
[01:05:02.450]in the near future.
[01:05:04.050]So, a second question is,
[01:05:05.900]the medium exposure group has a higher mean score
[01:05:09.620]than the high exposure group
[01:05:11.100]for the test developed by Dr. Ortiz.
[01:05:13.570]I'm just wondering why.
[01:05:14.830]And you could maybe go back to that screen
[01:05:17.050]if that'd be helpful.
[01:05:17.883]Okay, let me backspace.
[01:05:21.430]Which one are you talking about?
[01:05:22.690]I'm sorry, you're probably talking about this one here.
[01:05:25.840]Right?
[01:05:26.890]Yeah, I bet that is it. I think
[01:05:27.723]they're referring to the green one right there.
[01:05:29.620]Right.
[01:05:34.510]That's a good question.
[01:05:36.830]It's a sampling issue here,
[01:05:38.100]because we grouped the individuals from,
[01:05:41.592]as you can see, from 11 to 50%,
[01:05:44.710]so in that group, okay,
[01:05:48.090]the average score isn't really the average
[01:05:52.090]of what that group should get.
[01:05:53.240]So, in other words, they shouldn't equate to 100
[01:05:54.990]for each of them.
[01:05:56.060]It's just that there were higher scores within that group
[01:05:58.990]than there were within the other groups
[01:06:01.240]in terms of what their average was.
[01:06:02.870]So, it's a way that it's partitioned.
[01:06:04.610]And this, actually, would never happen,
[01:06:06.920]because the test doesn't use group.
[01:06:08.580]In other words, you would never be comparing
[01:06:10.890]an individual to a group with medium exposure.
[01:06:13.160]You would only compare that individual
[01:06:15.140]to other individuals with the same amount of exposure
[01:06:17.790]as determined by the question.
[01:06:19.480]So, if it was determined like I said in my case,
[01:06:21.870]so I had approximately 50% lifetime exposure
[01:06:25.360]by the time I was ten-years-old,
[01:06:27.440]then I would only be compared to other individuals
[01:06:29.750]who have 50% lifetime exposure.
[01:06:32.240]And in that continuous norming process,
[01:06:34.440]the average score for any one of those percentages
[01:06:37.830]would actually be anchored at 100.
[01:06:40.140]So, this is is just a reflection
[01:06:41.750]of the way that the groupings were made,
[01:06:43.500]and so that's why they ended up
[01:06:44.580]just a little bit higher there.
[01:06:47.043]All right, very good.
[01:06:48.330]And we do have a request for your contact information
[01:06:51.820]that they want to pass along
[01:06:53.010]to their special ed and ELL directors in their district,
[01:06:56.260]which we do have an email address for Dr. Ortiz
[01:06:59.510]at the end of the webinar that I will provide you.
[01:07:02.610]And, yes, please free to reach out to him
[01:07:05.380]for any kind of needs or kind of consultation
[01:07:08.490]that you might have.
[01:07:10.500]And I will preface that by saying
[01:07:14.650]I will do my absolute best to respond
[01:07:16.820]to any of your questions, but I have had three,
[01:07:21.160]just three emails, in my life, okay,
[01:07:24.440]and they've all, the first two piled into this one.
[01:07:28.380]I receive hundreds of emails a day
[01:07:30.440]that I can't possibly keep up with,
[01:07:32.320]but I do have some assistants.
[01:07:34.130]I do try and filter them, I do go through them.
[01:07:36.570]So, if it's a legit email
[01:07:37.840]from someone asking about a question or having some concern,
[01:07:41.840]needing more information, generally, I will find it
[01:07:44.300]and, generally, I will get back to you.
[01:07:46.100]But if you don't hear back from me after a week or two,
[01:07:50.610]please feel free to stalk me and go ahead and repeat.
[01:07:54.466]I will not mind,
[01:07:55.660]because I know that I can be difficult to get ahold of,
[01:07:57.770]so feel free to repeat that email if you need to.
[01:08:00.780]Put it in bold caps or something,
[01:08:02.980]and I'm sure it will get my attention.
[01:08:05.440]Yeah, and same,
[01:08:06.273]if you would like to advance the slides,
[01:08:08.230]we'll go ahead and go to that last one,
[01:08:09.703]because I also speak a bit about some of the other,
[01:08:15.710]I think we did add that to it where, yeah, there it is.
[01:08:19.500]I just passed it there.
[01:08:21.104]= [Jessica] Yeah, so also, just as we wait
[01:08:22.680]to make sure there's not some additional questions,
[01:08:25.200]I wanted to point out that we do,
[01:08:27.270]actually, originally, we had a five-part series.
[01:08:29.939]We are going to add a sixth presentation to the series,
[01:08:33.910]and it will probably be sometime this fall.
[01:08:36.510]But as I mentioned,
[01:08:37.510]we have Ryan Kettler coming up on Friday, May 11th,
[01:08:40.250]but then we also will be having
[01:08:41.920]two clinical counseling psychologists talk about
[01:08:45.960]some ethnic and cultural considerations with testing
[01:08:50.140]that are very insightful as well.
[01:08:52.380]And then, what we'll add in the fall is
[01:08:54.850]from Willy Solano Flores, also a language specialist
[01:08:58.200]in educational measurement.
[01:08:59.810]He will talk about some methodological alternatives
[01:09:02.180]for looking at test fairness on tests.
[01:09:04.750]And as you can see on this slide,
[01:09:06.080]we do have the contact information for myself and Dr. Ortiz.
[01:09:10.439]So, if you would like to follow up with either one of us,
[01:09:13.670]and as I mentioned,
[01:09:14.510]I will be sending copies of his PowerPoint slides to you.
[01:09:19.830]And we are recording this,
[01:09:21.899]and it will be available on our video library
[01:09:23.979]if there are people that you think would be interested
[01:09:26.779]and weren't able to attend at the time today,
[01:09:29.840]but still would like to know about it.
[01:09:31.310]I know I've had several requests already
[01:09:33.180]of people who couldn't attend,
[01:09:34.350]but they do want to know more about the topic.
[01:09:37.740]All right, well, it does not appear
[01:09:40.560]that we have any more questions at this point,
[01:09:42.800]so Dr. Ortiz, thank you so much
[01:09:45.029]for your informative presentation today.
[01:09:47.350]I think it probably was very insightful.
[01:09:48.477]I know it's insightful for me.
[01:09:50.410]I've heard you several times, and every time,
[01:09:52.109]I hear something new that's very helpful for me
[01:09:55.690]as I work as a testing measurement specialist.
[01:09:59.520]So, thank you for your time, and thank you, too,
[01:10:02.460]for the audience and supporting our programming.
[01:10:06.090]I hope you all have a very good day.
[01:10:08.080]Thank you very much, Jessica, and thank you everybody.
[01:10:11.080]Thanks.

The screen size you are trying to search captions on is too small!

You can always jump over to MediaHub and check it out there.

Comments

0 Comments

Fairness and English Learners

Description

Searchable Transcript

Comments icon comment

Related Channels

Comments