Modeling Students' Understanding of Light Matter Interactions using an Ordered Multiple-Choice Assessment

Christopher (Bud) Jenkins Author

04/04/2021 Added

15 Plays

Description

This is a video outlining the second pilot assessment created by the Moon group and our analysis of student responses

Searchable Transcript

Search:

[00:00:00.840]Hello everyone.
[00:00:02.100]My name is Bud Jenkins and
[00:00:03.860]this is Modeling Students' Understanding
[00:00:06.780]of Light Matter Interactions using an
[00:00:09.440]Ordered Multiple-Choice Assessment.
[00:00:12.190]First, I want to take you through
[00:00:14.730]why students' understanding of light matter interactions
[00:00:17.800]is so important.
[00:00:20.060]Light and matter interact with each other,
[00:00:22.350]and a tool we could use to describe those interactions
[00:00:25.180]is wave particle duality.
[00:00:27.560]Wave particle duality simply put is the idea
[00:00:30.650]that both light and matter exhibit properties
[00:00:33.770]of waves and particles.
[00:00:36.940]This topic is foundational to science.
[00:00:40.060]It's taught here at UNL as early
[00:00:41.780]as the general chemistry courses.
[00:00:44.200]It's also used in nearly all aspects of science.
[00:00:47.870]Everything from biochemistry to physics
[00:00:50.040]has a role for wave particle duality
[00:00:52.490]because of light matter interactions.
[00:00:55.440]So because of these reasons
[00:00:57.820]we sought to create a model that shows
[00:00:59.710]how students understand wave particle dualities.
[00:01:03.740]And to that end, we created our assessment.
[00:01:07.850]Our assessment was given out in fall of 2020.
[00:01:11.810]It was comprised of 18 ordered multiple choice questions,
[00:01:16.140]which we refer to as items.
[00:01:19.500]The assessment is divided into four different categories,
[00:01:23.850]each probing and different aspect of wave particle duality.
[00:01:28.310]Those categories are particle progress variable,
[00:01:31.640]general wave progress, double slit progress,
[00:01:34.970]and wave particle duality progress.
[00:01:37.820]This is not a standard multiple choice
[00:01:39.930]question assessment, though.
[00:01:43.580]This assessment was built using
[00:01:45.190]a partial credit model and construct maps.
[00:01:47.910]What does that mean?
[00:01:49.210]First, the items in this assessment are not dichotomous.
[00:01:54.050]Rather, they were constructed using a partial credit model.
[00:01:57.800]Each response has a partial point value
[00:02:00.170]associated with the response.
[00:02:03.610]If an item had three potential responses
[00:02:06.230]the values for the responses would be zero, half, and one.
[00:02:11.350]They are then organized into a construct map
[00:02:13.880]where they are ordered from least to greatest
[00:02:15.710]and grouped into a series of levels
[00:02:17.450]that are based on the complexity of the response.
[00:02:20.380]Students can then be assigned two different levels
[00:02:23.130]of understanding based on their responses.
[00:02:26.060]These levels can give us a sense
[00:02:27.570]of where a student's abilities lay,
[00:02:29.874]and we can also use them to help
[00:02:32.240]model student understanding.
[00:02:34.570]From this pilot assessment we gave in the fall,
[00:02:37.240]we received responses from almost 600 students.
[00:02:42.580]We used several tools to help us interpret our data,
[00:02:46.150]the first of which being item characteristic curves.
[00:02:50.120]Item characteristic curves model students' ability
[00:02:53.270]versus the probability of their response to an item.
[00:02:56.940]Characteristic curves were generated for each item.
[00:03:00.200]And these are two examples of those curves.
[00:03:02.890]On the left we have the characteristic curve
[00:03:04.860]for item number three.
[00:03:06.660]The characteristic curve for item three
[00:03:08.730]is the opposite of what we wish to see.
[00:03:11.350]Students with a higher ability are more likely
[00:03:13.900]to get full credit on this item,
[00:03:15.600]while students with a lower ability are most likely
[00:03:18.780]to get zero credit on this item.
[00:03:21.200]Students with an intermediate ability, however,
[00:03:23.940]are never most likely to answer
[00:03:26.100]with a more intermediate option.
[00:03:28.600]This means that the construct map
[00:03:30.660]for item three needs a bit more work
[00:03:33.290]in terms of where we organize student ability.
[00:03:36.530]The characteristic curve for item 16, however,
[00:03:39.580]is what we like to see.
[00:03:41.530]Students with a more intermediate ability
[00:03:43.860]are more likely to answer this item
[00:03:46.840]in such a way that earns them one-third
[00:03:49.270]or even two thirds of the credit of the question.
[00:03:52.400]The construct map for item 16 better organizes
[00:03:55.730]student understanding based on ability than item three.
[00:04:00.960]The next tool we use to assess students' understanding
[00:04:04.960]is a wright map.
[00:04:09.630]A wright map allows us to model students latent ability
[00:04:12.950]versus the difficulty thresholds of the items responses.
[00:04:16.910]The thresholds exist between each response for an item,
[00:04:20.230]such as a threshold between one and two,
[00:04:23.070]two and three, and so on.
[00:04:26.260]We can use these thresholds to determine
[00:04:28.420]what a student's ability needs to be
[00:04:30.450]in order to break through to the next response.
[00:04:33.340]For example, a student would need an ability
[00:04:36.380]of approximately negative 1.8 logits
[00:04:39.580]away from the mean student ability
[00:04:41.870]in order to change their answer from response one
[00:04:45.860]to response two, as indicated by the cat one
[00:04:50.520]under question one.
[00:04:52.100]The bar graph on the side shows us the distribution
[00:04:55.040]of students abilities that our model predicts
[00:04:57.960]with the majority of students
[00:04:59.500]having an ability around or at zero logits.
[00:05:03.360]The final tools that we used to evaluate our assessment were
[00:05:06.738]infit and outfit statistic.
[00:05:09.800]Infit and outfit statistics can help us to determine
[00:05:11.922]if an item is problematic and its fit,
[00:05:15.130]or if the observed variance is as expected.
[00:05:18.148]Infit and outfit statistics are evaluated
[00:05:21.240]on how close they are to one.
[00:05:24.120]If an item have both infit and outfit statistics
[00:05:28.850]that are less than 0.75 or greater than 1.33,
[00:05:33.360]we consider those items to be problems.
[00:05:35.950]With all of our items we observed infit and outfit numbers
[00:05:39.950]that were close to or practically at one,
[00:05:43.070]so none of our items actually ended up being problematic
[00:05:46.460]because they were all within the normal bell.
[00:05:49.090]So with all these different evaluations of our items,
[00:05:52.260]where do we go from here?
[00:05:54.460]Well, this assessment is actually part
[00:05:56.560]of an iterative process.
[00:05:58.510]The data compiled here is actually
[00:06:00.170]from the second pilot assessment,
[00:06:01.850]or the first revision of the initial assessment.
[00:06:04.940]All of these different analyses will help us
[00:06:06.990]to continue to refine our items
[00:06:09.360]in this assessment to better model
[00:06:11.630]and predict student ability.
[00:06:13.970]With a better model we can use it to have a better idea
[00:06:18.040]of students' understanding of light matter interactions
[00:06:20.900]and reach our final goal,
[00:06:22.570]to use this assessment in a classroom setting
[00:06:25.560]as a learning tool to help instructors
[00:06:27.743]get a better idea of their students' abilities.
[00:06:32.200]I would like to thank Dr. Alena Moon,
[00:06:34.340]Dr. Morgan Balabanoff, Archer Harold
[00:06:37.300]and the rest of The Moon Group
[00:06:38.390]for all their help and support with this endeavor,
[00:06:40.640]because without which this project
[00:06:42.430]would not have been possible.
[00:06:44.240]Thank you.

The screen size you are trying to search captions on is too small!

You can always jump over to MediaHub and check it out there.

Comments

0 Comments

Modeling Students' Understanding of Light Matter Interactions using an Ordered Multiple-Choice Assessment

Description

Searchable Transcript

Comments icon comment

Related Channels

Comments