Brown Bag: ACE Outcomes (Jan 26, 2017)

Chad Brassil Author

01/30/2017 Added

297 Plays

Description

Using ACE Outcomes in Canvas

Searchable Transcript

Search:

[00:00:00.000]Okay, in this presentation we're gonna talk about
[00:00:03.149]using Canvas to assess ACE outcomes.
[00:00:07.175]And there will be some general lessons out of that,
[00:00:11.238]about using outcomes in general, in courses,
[00:00:14.409]and at department levels and college levels,
[00:00:18.025]to assess outcomes in a similar way
[00:00:20.914]that we're doing for ACE outcomes.
[00:00:23.192]All right, so ACE has been around for a while now.
[00:00:25.691]Ace.unl.edu will get you to the main page of ACE.
[00:00:28.536]There's a series of 10 learning outcomes that a course is,
[00:00:34.758]to a degree, a part of that course
[00:00:36.425]is supposed to be aligned with.
[00:00:38.215]About a year and a half ago, some of these,
[00:00:41.274]the wording on some of these was tweaked a little bit
[00:00:44.283]to clarify it, and at that same time,
[00:00:47.733]we've started to rethink about how
[00:00:51.925]assessment is going to happen,
[00:00:53.917]and recertification is gonna happen.
[00:00:56.273]So down here on the ACE webpage,
[00:00:57.979]there is these stacks of Ace cards,
[00:01:01.784]and this one lists the process for ACE Recertification.
[00:01:06.769]That's the process that takes place every five years,
[00:01:09.580]in which a department or the instructors of a class,
[00:01:12.794]reflect back on what changes have been done on a class,
[00:01:17.388]and how the class is structured and aligned with the course,
[00:01:20.234]would be a ACE outcome.
[00:01:22.595]This stack of Ace cards next to it, if you click on that,
[00:01:26.494]Department ACE Reporting,
[00:01:28.850]that is the new, the shift,
[00:01:32.911]or the new way in which we'll collect data on
[00:01:38.768]courses on a regular basis, will be having
[00:01:42.752]the instructor identify a signature assignment,
[00:01:45.724]and that signature assignment will be assessed
[00:01:49.180]for every student, every semester.
[00:01:51.576]In essence, the signature assignment and the assessment
[00:01:55.243]of that on a regular basis, becomes the data.
[00:01:58.660]It's a continuous assessment,
[00:02:00.324]and that continuous piece of data
[00:02:02.197]that's being collected every semester,
[00:02:05.165]gathers over time, and then that leads to
[00:02:07.646]a larger self-reflection process at that five year.
[00:02:10.975]So most of what I'm gonna be talking about
[00:02:12.884]from this point forward, is focused on this last thing,
[00:02:15.239]on how do we actually make the assessment
[00:02:18.531]on this signature assignment happen
[00:02:20.602]in a smooth and regular way, and specifically,
[00:02:23.038]how does Canvas facilitate that.
[00:02:26.222]All right, so part of this process
[00:02:28.486]is utilizing the ACE rubrics that have been created.
[00:02:30.460]They've been out for about a year.
[00:02:32.617]These take that outcome for each ACE
[00:02:36.680]and break it into a series of rows in the rubric,
[00:02:41.229]and then for each row, there are four levels
[00:02:43.463]with descriptions as to what would
[00:02:46.281]place a student's work at those four levels.
[00:02:49.625]Four being highest.
[00:02:51.130]Three being the sort of mastery level,
[00:02:53.935]the achievement level at which
[00:02:55.152]we'd like to see students, minimally.
[00:02:57.066]Two is they've got some work to do,
[00:02:58.609]and one, they have a lot of work to do.
[00:03:04.141]Okay, if you go to the ACE website,
[00:03:06.141]to the Assessment part of the ACE website,
[00:03:08.417]about halfway down there is a heading
[00:03:11.146]called Assessment Through Canvas.
[00:03:12.544]That's where your key links are
[00:03:15.023]that you can go back and refer to,
[00:03:16.445]you can share with colleagues,
[00:03:18.436]that describes the details of what I'm about
[00:03:20.343]to talk about, and gives more detail.
[00:03:22.458]So this key link, the first one,
[00:03:23.800]is the ACE Assessment Instructor Guide.
[00:03:25.831]That currently has kinda everything we know
[00:03:29.492]about how to do this and all the exceptions
[00:03:32.784]and the workarounds to make it work for
[00:03:35.062]every course that we have come across at this point.
[00:03:38.074]And as more issues arise, that will be revised over time.
[00:03:41.539]There's also three videos on there.
[00:03:43.045]The first two videos, the first video is a minute,
[00:03:45.180]a minute and a half, one minute, 20 seconds.
[00:03:47.060]And it's really everything an instructor needs to know,
[00:03:49.532]minimally, to implement ACE assessment.
[00:03:52.094]So it's really quite simple to do this.
[00:03:54.087]And those first two videos are essentially
[00:03:55.875]going to be encapsulated in this presentation.
[00:03:59.045]The third video goes into some more advanced stuff,
[00:04:01.037]we may or may not get to in this presentation.
[00:04:03.560]So revisit this site at ace.unl.edu/assessment
[00:04:07.299]part way through for the Assessment Through Canvas
[00:04:09.817]to follow up with more details.
[00:04:11.969]Okay, the feature in Canvas that we're gonna
[00:04:13.636]take advantage of is two features, the Outcomes,
[00:04:17.338]and closely tied to Outcomes, are Rubrics.
[00:04:20.349]The Outcomes sits in all the Canvas classes,
[00:04:23.131]over here on the left-hand side.
[00:04:24.898]It's typically grayed-out,
[00:04:26.561]and you typically just wanna leave it grayed-out.
[00:04:28.230]Grayed-out means the students are not gonna access it.
[00:04:32.041]That's the Outcomes that we're talking about,
[00:04:33.995]and Rubrics doesn't sit over there,
[00:04:35.989]but you can get to Rubrics through Outcomes,
[00:04:38.916]through the Manage Rubrics format.
[00:04:41.438]So Outcomes and Rubrics,
[00:04:42.822]they have very specific meanings in Canvas,
[00:04:44.773]and meanings that make some intuitive sense.
[00:04:47.946]So here's how you do it.
[00:04:49.126]If you're an instructor and you wanna do
[00:04:51.082]ACE assessment in your class,
[00:04:54.697]you have to have your course set up in Canvas,
[00:04:57.174]you have to do the normative stuff
[00:04:58.312]in Canvas like make an assignment.
[00:04:59.815]So you have an assignment in Canvas.
[00:05:01.191]Let's say this is an assignment in which
[00:05:02.898]you're expecting the students to upload their work,
[00:05:05.715]like an essay or a paper,
[00:05:07.507]and that's gonna be your signature assignment.
[00:05:10.437]Once that's created, you'd click on that assignment
[00:05:12.881]in the Assignments list, and it would give you this.
[00:05:15.280]It's a summary page of the details on that assignment.
[00:05:18.235]I'm calling this Insightful Assignment
[00:05:20.104]and is worth 10 points.
[00:05:21.607]If you notice down here at the bottom,
[00:05:23.487]there's a little box that says plus Rubric.
[00:05:26.860]So you're gonna click on that.
[00:05:29.091]That pops up another box, which is where you would normally
[00:05:32.225]build a rubric if you were customizing it.
[00:05:34.468]In this case, we just wanna go and pull
[00:05:36.504]the existing rubric for the ACE course,
[00:05:39.147]that your course is aligned with.
[00:05:40.733]So you click on Find a Rubric, that brings up a box.
[00:05:43.780]You click on University of Nebraska-Lincoln,
[00:05:46.463]that will show the whole list of ACEs.
[00:05:48.822]In this case I'm gonna align with ACE 4,
[00:05:50.456]so I choose ACE 4, and then the rubric appears
[00:05:53.340]over here on the right-hand side.
[00:05:54.758]If I scroll down there, the full rubric is in there,
[00:05:56.629]with all the boxes, but I would just click that top,
[00:05:59.506]ACE 4, and we'll take that rubric,
[00:06:01.742]and pull it into my assignment.
[00:06:03.691]So my assignment now, at the bottom
[00:06:05.398]instead of a plus Rubric link, says ACE 4 Rubric.
[00:06:09.353]And the whole rubric is down there
[00:06:11.467]with the four rows and all the columns in there,
[00:06:14.679]sorry, all the rows in there.
[00:06:17.486]Those rows are actually the outcomes.
[00:06:21.598]In Canvas talk, each row is an outcome.
[00:06:24.810]And since I pulled this rubric from that
[00:06:28.348]University of Nebraska set of ACE-hosted rubrics,
[00:06:32.734]those admin-level, those university-level rubrics,
[00:06:37.212]are collections of outcomes which are also
[00:06:39.573]hosted up at the highest level,
[00:06:41.891]and so any information that you feed in here
[00:06:43.919]will be fed up and collected at that highest admin level.
[00:06:48.474]That's indicated by, it's kind of at
[00:06:50.466]the bottom of the screen and hard to see,
[00:06:51.848]but there's this little yellowish tab in front of it.
[00:06:55.551]That tells you that that rubric
[00:06:57.340]is not just one sitting in your class,
[00:06:58.888]but it's tied to a higher administration level.
[00:07:01.325]Okay, that's a bit of an aside.
[00:07:02.992]So you added the rubric, all you did,
[00:07:05.392]and now maybe you waited,
[00:07:06.859]and students go and they submit their assignment,
[00:07:09.700]or maybe this was the end of the semester.
[00:07:11.163]The assignments are already in there
[00:07:12.300]and you remembered, oh I need to do that ACE assessment.
[00:07:14.695]You go ahead, add your rubric.
[00:07:16.930]When you're ready to assess it,
[00:07:20.427]to score the assignments, you click on SpeedGrader.
[00:07:24.453]All right, and it brings up SpeedGrader.
[00:07:26.362]If this in this case, represents a scenario in which
[00:07:29.250]a student has submitted an assignment,
[00:07:31.523]so the assignment is over here on the left-hand side.
[00:07:35.135]It'll show as say a PDF or a Word document,
[00:07:37.394]and you can interact with this,
[00:07:39.387]you can scribble on it, you can make little comments.
[00:07:41.784]I could put in little notes.
[00:07:43.697]There's lots of electronic ways to give feedback
[00:07:46.716]to the students and the students would see that.
[00:07:49.399]I can go over here and in this case,
[00:07:51.559]this assignment has been set up to work with Turnitin,
[00:07:55.110]which is the plagiarism check system
[00:07:57.349]that integrates with Canvas.
[00:08:01.006]So I could click on that and get the full report,
[00:08:03.197]or here's the summary of the plagiarism potential.
[00:08:06.614]Here's where I would give it a grade.
[00:08:08.281]All that's the normal ways in which you would
[00:08:10.088]interact with this assignment in Canvas.
[00:08:13.135]What I have here now though is this View Rubric,
[00:08:15.167]since I've attached a rubric.
[00:08:16.387]Click on View Rubric and that will open up the full rubric.
[00:08:20.006]So here's all my rows in my rubric,
[00:08:22.324]and all the one through four boxes.
[00:08:25.973]Now in this particular case,
[00:08:27.561]it's blank over here on the left-hand side.
[00:08:29.843]This little middle bar here I can drag
[00:08:32.111]bigger or smaller, so I drag it over bigger
[00:08:34.310]so we can see the full rubric, and in this case,
[00:08:36.552]there is no student assignment submitted.
[00:08:38.951]Maybe it's because you didn't require it.
[00:08:41.108]You can as well as do this, by just say make an assignment,
[00:08:44.449]not require the students to submit anything,
[00:08:46.425]they turn in a piece of paper,
[00:08:47.640]and you pull up those pieces of paper,
[00:08:49.265]and then you open up this SpeedGrader
[00:08:51.253]so that you can score it through the rubric.
[00:08:54.585]And you do that by just simply clicking, right,
[00:08:56.454]click, click, click, and these green boxes is the rubric
[00:09:01.613]boxes, right, the rubric, the assessment
[00:09:03.463]that's been assigned to this particular student.
[00:09:07.406]Then you click Save and then you're done.
[00:09:10.619]And a full stop here to say
[00:09:12.446]that's all that you need to do.
[00:09:15.318]That's the minimum to take a course,
[00:09:18.964]identify your signature assignment.
[00:09:20.746]You just have to click plus Rubric, find your rubric,
[00:09:22.774]and then go through and do it for each of the students.
[00:09:26.150]It's that simple.
[00:09:27.248]And sometimes when I talk to faculty
[00:09:29.516]and they're worried about the burden
[00:09:31.421]that ACE assessment is gonna present to their class,
[00:09:34.683]they're worried about trying to do this
[00:09:35.949]for all the students, all the time,
[00:09:37.901]I say wait a minute, this is actually really simple.
[00:09:40.315]And when you see how simple it is,
[00:09:41.871]you realize the whole thing can be done
[00:09:43.423]in that one-minute-and-20-second video,
[00:09:45.505]if you just want the punch, it's all there.
[00:09:47.589]So that's all that's required of an individual instructor.
[00:09:50.120]From here I'm gonna talk about some of the extra stuff.
[00:09:52.776]Like once you put in that effort,
[00:09:54.042]maybe you wanna see what's happening in your class,
[00:09:56.374]or maybe you wanna do some customization of it,
[00:09:58.373]or eventually maybe use it for
[00:10:01.411]outcomes that you write for your own class,
[00:10:04.098]or outcomes that sit at your departmental
[00:10:05.604]college level for certification.
[00:10:07.401]So we'll go on to talk about some of those things.
[00:10:09.038]But I wanted to stop here to point out
[00:10:11.503]how little it is that one really needs to know
[00:10:14.022]to use outcomes to assess an ACE class
[00:10:16.853]in Canvas, and how smooth it is.
[00:10:19.737]All right, so you've done this,
[00:10:21.247]and now you actually wanna review the results of your class.
[00:10:25.089]To do that you have to enable two things.
[00:10:28.704]First you have to go to the settings within your class,
[00:10:32.581]Feature Options, on the right-hand side here,
[00:10:35.766]and go to Learning Mastery Gradebook and turn that on.
[00:10:38.792]By default this is gonna be off, so I've checked that.
[00:10:40.917]I say no, I wanna see the Learning Mastery Gradebook.
[00:10:44.180]So you turn that on.
[00:10:45.076]There's another one below here,
[00:10:45.909]Student Learning Mastery Gradebook,
[00:10:47.195]we can leave that one off.
[00:10:48.549]That enables the students to see
[00:10:51.735]how they've been assessed on the various outcomes.
[00:10:55.546]That's optional.
[00:10:56.659]Really, we just wanna be able to see what's happening,
[00:10:58.349]so we're gonna turn on Learning Mastery Gradebook.
[00:11:01.414]Step two is we have to go to that Outcomes thing.
[00:11:04.871]Over here in your list of sections within your course,
[00:11:10.233]there's that grayed-out outcome,
[00:11:11.882]so you click on that and it brings up this,
[00:11:13.989]where you could add your own customizable outcomes,
[00:11:16.092]but in this case, we just wanna find those
[00:11:18.980]ACE Outcomes for the ACE thing that we're interested in,
[00:11:22.364]that we wanna, that this course is aligned with,
[00:11:24.553]so we click on Find, that brings up a dialog box
[00:11:27.688]in which we click through to get to the ACE.
[00:11:30.205]You click on Account Standards,
[00:11:31.897]and that will bring up your college, your department,
[00:11:34.829]but also University of Nebraska-Lincoln,
[00:11:36.568]so we click on University of Nebraska, click on ACE,
[00:11:40.288]that will bring up a list of all the ACEs,
[00:11:42.517]scroll down to the ACE that you're interested in,
[00:11:44.662]click on that, that will bring up a list
[00:11:47.748]of the rows of the rubric for that relevant ACE.
[00:11:56.635]Those are outcomes in Canvas terminology.
[00:11:59.858]That's the ways it's collecting
[00:12:01.509]and aggregating the data for those outcomes,
[00:12:04.899]and the rubric that we looked at before
[00:12:06.964]was just an easy way that had bundled all those,
[00:12:09.691]but now we're saying we actually wanna see
[00:12:12.667]what's happening with those scores,
[00:12:15.184]so we have to go and import the outcomes
[00:12:19.353]that are being held at the highest level here
[00:12:22.203]of the admin level, at the University of Nebraska,
[00:12:24.560]we wanna pull those into our course,
[00:12:26.122]so that within our course, we can see what's happening.
[00:12:27.939]So you click Import and it'll say,
[00:12:30.005]do you wanna copy these in, and yes, you pull them in.
[00:12:32.973]Now, back when you went in and enabled that
[00:12:35.414]Learning Mastery in your Gradebook,
[00:12:37.977]that added a tab, so you go to your course
[00:12:40.291]and you click Grades, and there's this new tab.
[00:12:42.895]You click on that new tab, and instead of
[00:12:46.750]Grades and Assignments, you see the summary information
[00:12:51.091]for the rows in your rubric,
[00:12:56.666]for the outcomes that you just imported
[00:12:58.727]in the previous step, they're now there.
[00:13:00.917]It will show you the levels for each individual student.
[00:13:05.962]They're colored here, there's also numbers,
[00:13:08.153]and the colors match up to colors over here.
[00:13:10.635]And then at the top, it's showing you
[00:13:13.153]what your averages are across that class.
[00:13:16.293]There's an Export button so you can export that
[00:13:18.566]into a CSU file if that's useful for you.
[00:13:21.750]If you mouse over some of these learning outcomes,
[00:13:27.784]a little popup will show you the distribution
[00:13:30.426]of the students in all categories.
[00:13:33.525]I find this distributional information really useful.
[00:13:38.062]So that gives you the information as the instructor,
[00:13:41.033]as to what's happening within your class,
[00:13:43.263]'cause you put that time in, those,
[00:13:44.913]that minimal 10 seconds per student,
[00:13:46.769]after you graded that assignment.
[00:13:48.505]Put that extra time in, you go click, click, click,
[00:13:50.605]and now here's the summary information.
[00:13:54.576]All right, let me step back for a minute.
[00:13:56.392]If you were to go to that ACE instructor guide
[00:13:59.120]on the ACE assessment website,
[00:14:02.262]the ACE instructor guide for Canvas,
[00:14:05.071]you would see these four options.
[00:14:07.380]Options A, B, C and D.
[00:14:09.073]And actually if you've noticed,
[00:14:10.137]at the top of some of the slides,
[00:14:12.112]everything we've just talked about
[00:14:14.426]is option A and option B.
[00:14:16.373]That is, if students are electronically submitting
[00:14:19.762]an assignment, and you're gonna look at each assignment
[00:14:22.652]and then grade it through a rubric, that's option A.
[00:14:26.412]Option B would be if a student is
[00:14:29.714]just gonna give you something on paper,
[00:14:32.483]or it's an oral presentation
[00:14:34.630]and you need to assess each student,
[00:14:36.653]you really do it the same way, you just choose that
[00:14:39.093]there's not gonna be an online submission,
[00:14:41.324]but you do the same thing, you go to SpeedGrader,
[00:14:43.470]you click, click, click through on the rubric.
[00:14:45.412]And what I haven't mentioned here,
[00:14:47.765]but it came up in a question,
[00:14:50.080]and it's the obvious question.
[00:14:51.278]You can take those rubrics,
[00:14:53.258]and you can integrate into those rubrics
[00:14:56.643]your own columns that you are gonna use for scoring.
[00:14:59.619]So maybe you have five ACE outcomes,
[00:15:04.662]and you can take that same rubric and modify it
[00:15:07.266]and add seven more rows, and those are the rows
[00:15:10.816]that you're gonna use for your grading.
[00:15:12.678]So you can pull up one integrated rubric,
[00:15:15.327]and you can go click, click, click, click, click,
[00:15:17.679]and some of that information gets pulled up
[00:15:20.699]and used for ACE assessment over time.
[00:15:23.589]Some of that information gets turned into the score
[00:15:26.647]that the students see for that assignment.
[00:15:29.251]And that can happen with option A or option B.
[00:15:31.524]Really, those things in which you're gonna
[00:15:33.716]go click, click, click, per student.
[00:15:37.062]All right, but there's option C and option D.
[00:15:39.745]Option C is what if you have 250 students,
[00:15:44.782]and there's no assignment where you're
[00:15:47.093]individually grading one-by-one,
[00:15:49.154]and everything's done through quizzes, exams,
[00:15:53.122]things that are automatically graded
[00:15:54.650]like multiple choice questions,
[00:15:56.140]or numerical questions, et cetera.
[00:15:57.587]So in Canvas terminology, that's called a quiz.
[00:16:00.355]A quiz means, what we would call a quiz,
[00:16:03.000]something you do before class or after class,
[00:16:06.190]but also if you do an electronic exam,
[00:16:10.159]in the exam commons in the proctored space,
[00:16:12.803]it's still called a quiz in Canvas terminology,
[00:16:15.319]you're just having it done in a proctored space.
[00:16:17.957]So any of those things, quizzes or electronic exams,
[00:16:21.519]i.e, quizzes in the exam commons,
[00:16:23.501]can also be connected to outcomes.
[00:16:27.882]And in there they can be based on
[00:16:29.649]automatically graded questions.
[00:16:31.551]So here's how you do this, what I call option C.
[00:16:35.315]You go to your quizzes,
[00:16:37.997]and up here in the right-hand corner is the gear.
[00:16:42.536]Clicking on the gear brings up the Manage Question Banks.
[00:16:45.674]So that is the key thing to remember,
[00:16:47.902]that if you wanna connect a quiz to an outcome,
[00:16:52.604]you have to do it through question banks.
[00:16:55.121]You basically align question banks with outcomes,
[00:16:58.510]or an individual outcome or multiple outcomes.
[00:17:00.573]You cannot align individual questions,
[00:17:03.925]but note, asterisk, big asterisk,
[00:17:06.648]a question bank can have a minimum of one question in it.
[00:17:10.405]Canvas is trying to guide you to say, well,
[00:17:12.752]we think it makes most sense if you're doing alignment
[00:17:15.594]on sets of questions and question banks.
[00:17:18.028]So they won't let you align
[00:17:19.556]an individual question in a quiz,
[00:17:21.170]but they will let you align a question bank.
[00:17:23.270]The question bank can have as little
[00:17:24.762]as a single question in it, okay.
[00:17:26.516]So if we click on Manage Question Banks,
[00:17:28.586]we would get a list and we'd say
[00:17:29.826]okay, I wanna make a new bank.
[00:17:31.969]You'd make a bank, in this case I called my bank
[00:17:33.956]Interpret Data Bank, 'cause I'm gonna align it with
[00:17:36.517]the Interpret row in my rubric.
[00:17:39.243]And I put a question in there.
[00:17:40.563]What's the capital of Nebraska?
[00:17:41.950]So in this case, I just have a single question,
[00:17:44.158]and the key is, down here on the bottom right,
[00:17:47.236]there's a button that says, Align Outcome.
[00:17:49.921]You click on that and it would give you a list
[00:17:52.014]of the outcomes that sit in your class.
[00:17:54.746]So previous to this, you would have had to go
[00:17:56.545]and pull the outcomes for the aligned ACE,
[00:18:00.064]in this case, ACE 4, so they're sitting in your class,
[00:18:02.273]and then I click on Align Outcome,
[00:18:03.577]and I choose and I say that's the one
[00:18:05.364]that I want for this question bank,
[00:18:07.124]and I've already done that here and you see Interpret Data,
[00:18:09.534]so Interpret Data is the outcome
[00:18:13.702]that I have aligned with this question bank.
[00:18:17.071]You could align two outcomes if you wanted.
[00:18:20.057]That might make a lot of sense if say,
[00:18:22.185]this question is going to be used for
[00:18:24.266]Interpret Data in ACE 4, but maybe
[00:18:28.236]it's also gonna be used for Interpret Graphs
[00:18:32.228]in a college-level, accreditation outcome
[00:18:36.891]that's gonna be used for a college-level accreditation.
[00:18:39.714]All of that stuff could dovetail here
[00:18:41.796]in terms of question banks.
[00:18:45.515]Okay, so you create question bank,
[00:18:47.597]have one or more questions within that,
[00:18:50.400]and then align that question bank,
[00:18:52.357]and then when a student goes and answers that,
[00:18:54.397]the score that they get on that question
[00:18:57.703]will be parsed by Canvas and that will determine
[00:19:01.581]the score that they receive in the,
[00:19:08.643]for that particular outcome, right?
[00:19:10.155]So you'll see that in your summary,
[00:19:11.462]in your Learning Mastery Gradebook,
[00:19:13.423]and that's the information that will go up
[00:19:15.339]to the higher levels when we gather data
[00:19:18.072]across courses, across semesters.
[00:19:22.333]Okay, so all that's sitting there.
[00:19:24.832]You've done it through form A, B or C,
[00:19:28.513]and now this next step is really things
[00:19:31.208]that are beyond what an instructor sees.
[00:19:33.293]It's what sits in the system.
[00:19:34.928]If you have an admin-level account or a sub-account,
[00:19:38.191]sub-account is the accounts that can be given to
[00:19:42.076]department level or college level,
[00:19:44.781]so this could be done at those levels,
[00:19:46.999]at that level, you can download
[00:19:49.526]what's called an Outcome Results report.
[00:19:51.362]And it's a big ugly report that has
[00:19:53.077]a bunch of rows on it, basically.
[00:19:54.743]Like the first row here is for this particular student,
[00:19:58.288]and in this particular class.
[00:20:00.086]For the first rubric row, they scored a three.
[00:20:03.722]And then this, same student in the same class.
[00:20:06.832]For the second rubric row, they scored a four.
[00:20:09.654]And on and on and on.
[00:20:10.592]So this might be 100,000 rows, a big dump of data.
[00:20:15.102]So then it becomes a challenge to form that data
[00:20:18.007]into something that's meaningful and interpretable,
[00:20:20.207]and that's where we currently have plans.
[00:20:22.563]They're being enacted here over the spring semester of 2017.
[00:20:29.400]Arts and Sciences is donating some programming time
[00:20:34.341]to this effort to try to get something
[00:20:37.241]that will work and ideally work elegantly,
[00:20:39.777]and work a way that will work for all of us.
[00:20:42.809]The short end of it, is that we'll pull those sums
[00:20:48.000]for the rubric cells per class.
[00:20:51.109]Like how many students scored a four on the first row?
[00:20:55.915]How many students scored a three on the first row?
[00:20:57.992]How many students scored a two on the first row, et cetera.
[00:21:00.484]Store all that into some kind of SQL database over time,
[00:21:04.650]and then have a website that can pull from that database,
[00:21:07.093]and assemble displays in ways that we can interpret.
[00:21:10.514]That's the vision.
[00:21:11.659]We've got some sort of prototypes and we're working on it.
[00:21:13.618]Let me show you a couple mock-ups.
[00:21:14.923]So this isn't real information here,
[00:21:17.285]but this is what I'm envisioning,
[00:21:19.626]is that for example here, in BIOS 101L,
[00:21:23.062]which would be our course at the bottom,
[00:21:24.774]we have our four rubric rows,
[00:21:26.451]and there's some colored bars there
[00:21:29.022]indicating the number of students
[00:21:31.436]that scored the four, three, two, one.
[00:21:33.561]And then the far right-hand side is
[00:21:36.855]basically just turning that graph on its side
[00:21:38.748]and showing that graph over time, so you can say,
[00:21:40.707]oh look, over time I'm seeing more fours
[00:21:43.795]or fewer ones, or whatever sort of patterns
[00:21:45.756]you wanna describe, you can describe that.
[00:21:49.454]And that is nested below in the structure of the university.
[00:21:55.458]So up at the top I have Biological Sciences.
[00:21:57.945]Maybe that represents the average distribution
[00:22:01.088]for all the courses in Biological Sciences.
[00:22:04.598]So to illustrate that, you could have this hierarchy.
[00:22:07.375]So within ACE 4, we have the University
[00:22:08.941]of Nebraska-Lincoln, various colleges.
[00:22:11.078]Within those colleges, there's various departments.
[00:22:13.241]Within those department, courses,
[00:22:15.243]and then sections within there.
[00:22:17.408]So for example, I could look at the top row,
[00:22:20.518]and I could see what's happening across
[00:22:23.836]all ACE 4 classes at the University of Nebraska.
[00:22:26.940]Or I could look at down here at this bottom row,
[00:22:30.001]and look at an individual course and see what's happening.
[00:22:33.596]But let's say 101L has a TA,
[00:22:37.351]and there's always a different TA each semester,
[00:22:39.146]so there's a lot of noise in there.
[00:22:40.410]I don't really care what happens
[00:22:41.837]in a particular section over time.
[00:22:43.716]Maybe I care what happens across the sections.
[00:22:46.205]The idea is this row up, the BIOS 101L,
[00:22:49.845]would give me the aggregate information
[00:22:52.590]across all the sections,
[00:22:54.270]'cause maybe that's the relevant level
[00:22:56.363]at which I find it informative.
[00:22:59.581]And the whole notion with this Canvas
[00:23:01.665]outcomes and ACE assessment is that
[00:23:03.666]we're providing simple-to-use tools
[00:23:07.579]that ideally give some meaningful data,
[00:23:11.256]and that data can be used by instructors,
[00:23:15.550]over time to sort of self-reflect or adjust
[00:23:18.127]or do what one wants with it.
[00:23:20.894]And minimally, it's easy to collect that data,
[00:23:25.061]but ideally, the data that's collected
[00:23:27.269]is going to be useful at some level.
[00:23:30.575]Maybe at a department level to have
[00:23:32.337]conversations across departments.
[00:23:34.419]Maybe for an instructor who's overseeing
[00:23:37.246]a number of TAs, to talk about what's happening
[00:23:39.411]in different TA sections, et cetera.
[00:23:43.210]All right, so here's some,
[00:23:45.377]this is an actual data from a class.
[00:23:46.971]This is last semester, and this is a little bit more
[00:23:49.084]as we're working on the mock-up here
[00:23:50.712]where I have color matched this now,
[00:23:52.390]so the color matching on this is the same that you see
[00:23:55.861]when you're looking at in Canvas, they're familiar colors.
[00:24:00.484]Here's the actual numbers, here they are
[00:24:02.206]in percentages, or in fractions.
[00:24:04.089]I have added a dash line here to track
[00:24:06.210]the average score in these over time,
[00:24:09.266]so you can see okay, well,
[00:24:10.949]the lowest one here is Interpret Data.
[00:24:13.608]So this is what we're envisioning as we move to this,
[00:24:16.959]to be added here would then be, as semesters are gathered,
[00:24:20.018]to flip these graphs and have them
[00:24:21.649]over say five years of information,
[00:24:23.816]so you could look across and see what's happening.
[00:24:26.795]All right, if you're paying attention at home,
[00:24:28.386]you've realized we haven't talked yet
[00:24:29.775]about option D, the Matrix entry option.
[00:24:33.613]So in this and in dialogue with faculty,
[00:24:35.344]we've realized there's one kind of class,
[00:24:38.612]or a couple kinds of classes that just
[00:24:40.613]still don't work in the option A, B and C.
[00:24:44.861]For example in the Math department,
[00:24:47.347]they might have a whole bunch of sections,
[00:24:49.966]and they're giving a math exam,
[00:24:52.914]in which somebody's taking it on paper,
[00:24:55.078]they're writing down answers on that,
[00:24:57.281]and then they are gathering all of those
[00:25:00.596]across all the sections and maybe one instructor
[00:25:02.801]is grading everybody's question one across all the sections,
[00:25:05.985]and another instructor is grading all of question two.
[00:25:09.215]So they've got these things all, it's all very paper-based.
[00:25:12.285]Nothing's happening in Canvas.
[00:25:14.088]They don't really wanna flip through SpeedGrader,
[00:25:18.166]and click, click, click for every student.
[00:25:19.926]That would really be inefficient.
[00:25:22.672]But with a couple modifications of what they're doing,
[00:25:25.373]they could gather this kind of data
[00:25:27.533]and enter it directly in, i.e.,
[00:25:29.532]how many students scored a four on that first row?
[00:25:33.617]How many students scored a nine on that first row?
[00:25:35.742]That's really the core of the data over time,
[00:25:37.949]so the Matrix entry would bring up the vision,
[00:25:40.693]and this is sort of layered too here to the vision,
[00:25:43.146]would bring up something like this,
[00:25:44.654]where you would just directly enter 14, nine, et cetera.
[00:25:48.211]All the way down, fill in these boxes,
[00:25:50.292]based on the information,
[00:25:51.882]and then you would just have to have it tracked,
[00:25:55.026]well, how that is being collected,
[00:25:57.232]and in your five-year recertification you would say,
[00:26:00.417]well, here's what we did to collect this,
[00:26:02.420]but you still would have all the data.
[00:26:04.583]All this data could generate these kinds of graphs
[00:26:07.977]over time that we talked about.
[00:26:11.948]So we think that this, oops.
[00:26:22.244]Okay, sorry.
[00:26:23.308]We think that this data matrix,
[00:26:25.718]this direct-entry format will fully facilitate,
[00:26:30.050]I can have all the courses we can imagine.
[00:26:33.692]And there's some other things that we've talked about
[00:26:36.148]kind of in the pipe dream, maybe a Scantron system.
[00:26:40.567]Scantrons have these nice little reports now
[00:26:42.823]where we can specify outcomes, and you can identify,
[00:26:46.626]you know, question two, five and seven
[00:26:49.079]on my Scantron exam relates to using scientific knowledge.
[00:26:54.676]And the Scantron system could
[00:26:56.883]pull those out and give you those numbers,
[00:26:59.215]and those could potentially be added here,
[00:27:01.138]or we might even be able to have
[00:27:03.056]an upload system where that Scantron
[00:27:04.767]gets uploaded and automatically processed,
[00:27:06.893]and you check it against this.
[00:27:08.402]This Matrix entry opens up a lot of integration features,
[00:27:12.125]if we can get the programming working.
[00:27:15.346]All right, so let's give some sort of summary points here.
[00:27:18.815]The benefits of this Canvas and ACE,
[00:27:23.262]the Canvas Outcome system to assess ACE,
[00:27:26.947]is that you can roll up the data to the various course,
[00:27:29.488]department, college, university levels,
[00:27:30.934]if our coding works, like all the data's there,
[00:27:33.858]it's just aggregating it in appropriate ways,
[00:27:35.975]and coming up with nice figures.
[00:27:37.809]It's fully integrated into the LMS, into Canvas.
[00:27:41.265]It's what I call butter smooth, right?
[00:27:43.054]Those first steps of just adding a rubric, clicking through.
[00:27:47.645]It's really, really easy to do,
[00:27:50.886]and it can be integrated with the grading,
[00:27:53.510]like you can have some of the rows that apply to the grade
[00:27:55.674]while other rows apply to the ACE assessments.
[00:27:58.261]Really simple to use.
[00:27:59.984]And the students, from the student perspective,
[00:28:02.026]it's they don't even see it.
[00:28:03.630]They're just posting, or posting assignments
[00:28:05.894]into Canvas like they always have,
[00:28:07.834]and then for some of those assignments,
[00:28:09.835]it's just pulled off and they get assessed by ACE,
[00:28:12.052]so the same interface that they always see.
[00:28:14.143]Turnitin can be integrated for plagiarism checks
[00:28:17.174]very simply right now.
[00:28:18.648]It might get a little more complicated in the future,
[00:28:20.202]but it's integrated in.
[00:28:22.582]You can use all the tools that you would use in the LMS
[00:28:25.891]and integrate that in with the ACE rubrics.
[00:28:31.876]And critically, there's very low
[00:28:34.428]mid-level administration burden.
[00:28:35.745]Meaning, you don't have to tell anybody
[00:28:38.056]what course you're gonna do or when you're gonna do it.
[00:28:41.264]All that needed to happen, which has already happened,
[00:28:43.449]is those rubrics need to be loaded up
[00:28:45.968]in the admin level for the University of Nebraska,
[00:28:48.816]then you as an instructor decide
[00:28:51.090]what assignment you're gonna assess,
[00:28:54.338]you just click Rubrics, you're in full control.
[00:28:56.853]You can do it in the middle of the night,
[00:28:58.253]you can do it after the class is done.
[00:29:01.130]It's very easy, and really nobody needs to touch anything
[00:29:04.266]to make it happen, except the instructor.
[00:29:06.946]All right, limitations, it's not perfect.
[00:29:09.547]It's a little different than some of the other systems.
[00:29:11.649]So I haven't mentioned tk20,
[00:29:13.135]but tk20 is the parallel system.
[00:29:15.237]It's really the system we were starting on
[00:29:17.300]for facilitating this ACE assessment.
[00:29:19.389]In about November, December last,
[00:29:21.461]in 2016, we realized, wait,
[00:29:23.872]Canvas has some great tools that may be quite elegant.
[00:29:27.211]And in my opinion, they're really elegant,
[00:29:29.227]and I think many people end up using Canvas for this.
[00:29:32.606]But tk20 is still available
[00:29:34.249]as a parallel system at this point.
[00:29:36.517]And tk20 archives actual student artifacts,
[00:29:39.942]meaning the student papers.
[00:29:41.216]So the student submits a paper,
[00:29:44.145]those individual papers can be assessed in later reports,
[00:29:48.307]if a committee wanted to drill down and pulls those up,
[00:29:52.392]and that's more connected to really
[00:29:54.129]how we had done assessment up to this point,
[00:29:55.876]where we'd asked instructors to submit
[00:29:58.922]a great example, an okay example,
[00:30:01.024]and a bad example of a paper.
[00:30:02.963]But there was some limitations
[00:30:05.890]to what that was actually useful for,
[00:30:09.522]and the burden that that was taking on archiving,
[00:30:12.321]hanging on to all those copies of papers,
[00:30:14.630]and so we moved to this idea of assessing
[00:30:17.587]via a rubric, every student along the way.
[00:30:21.438]So it's sort of we'll try it, it's okay.
[00:30:23.947]We don't need those artifacts.
[00:30:25.449]It doesn't seem like they have that much utility.
[00:30:28.876]But it is a limitation, Canvas does not
[00:30:31.194]hang on to those outside of the instructor.
[00:30:33.337]The instructor can access those papers, obviously,
[00:30:35.736]but three years later, nobody can.
[00:30:40.279]There's no tracking of students, no portfolio tracking.
[00:30:45.258]CEHS uses another system called LiveText,
[00:30:49.837]and LiveText is like the full thing.
[00:30:52.432]It hangs on to artifacts.
[00:30:54.530]If you're assessing a particular student
[00:30:56.386]through some outcomes, and then two years later
[00:30:59.227]that student gets assessed by another instructor
[00:31:01.495]in another class for that same outcome,
[00:31:02.970]it's gonna track all that.
[00:31:04.161]So if you want a fully integrated, over-time tracking,
[00:31:07.548]you wanna move to something like LiveText.
[00:31:09.607]You pay for it, it's a great system,
[00:31:12.203]but it does a lot of stuff that Canvas just can't do,
[00:31:16.037]is no way built to do that.
[00:31:19.706]The Turnitin integration, right now is quite elegant.
[00:31:22.513]When you wanna add the plagiarism check,
[00:31:25.522]or the Turnitin, there's a box you check.
[00:31:27.376]It's like, oh, use Turnitin.
[00:31:28.839]It's that simple, it's really easy.
[00:31:33.130]2018, the plan is to move to
[00:31:35.290]an LTI-only integration of Turnitin,
[00:31:38.840]which is sort of fancy computer speak
[00:31:41.967]for a protocol that Canvas is using
[00:31:45.845]to talk to outside programs, which is what Turnitin is,
[00:31:48.938]and they wanna push to the LTI-only interface.
[00:31:51.746]In January 2017, they realized there was some bugs,
[00:31:55.291]and so they delayed moving to the LTI-only for another year.
[00:31:59.170]There is some push that they keep some of
[00:32:01.420]the elegant features of the current system, we're not sure.
[00:32:05.008]It can still be done if they move
[00:32:07.272]to what they currently have set up.
[00:32:08.751]It'll be a little clunkier.
[00:32:10.154]So the short hand is it'll be,
[00:32:12.103]you can still do plagiarism checks in 2018.
[00:32:15.196]It may be a little bit clunkier,
[00:32:16.791]it depends on what they develop.
[00:32:18.192]We have the details in the step-by-step
[00:32:20.575]guide, there on the website.
[00:32:23.405]Okay, we're also hoping there's some broader utility.
[00:32:27.287]As instructors use Canvas to assess their ACE classes,
[00:32:33.447]they're using rubrics, they're using outcomes,
[00:32:36.168]they become familiar with it.
[00:32:38.349]If that is a good fit for your class,
[00:32:40.942]the thought is that more people will start to say,
[00:32:42.818]hey, this really works, and they'll start to use
[00:32:45.190]outcomes for their own course.
[00:32:47.099]Maybe there's a particular thing
[00:32:48.670]that they would like to assess
[00:32:50.004]across seven different assignments and track,
[00:32:52.228]and so you as an instructor, once you figure this out,
[00:32:54.578]it's quite easy to add an outcome.
[00:32:56.509]Build them outcomes into your rubrics.
[00:32:57.952]You can integrate them into the very same rubric
[00:32:59.676]that you're using for the ACE assessment,
[00:33:02.767]and track your student outcomes over time.
[00:33:06.768]So it's a tool.
[00:33:08.542]It's a tool that we think some instructors
[00:33:10.606]will be really excited to use for their courses.
[00:33:13.192]And at the same level, it's a tool that I think
[00:33:14.979]many departmental arts or colleges,
[00:33:16.995]especially those that have external
[00:33:19.131]accreditation processes, may really like as well.
[00:33:23.535]'Cause when everybody's using Canvas
[00:33:25.427]in your department or college,
[00:33:27.034]and it's this easy to do outcome assessment,
[00:33:29.508]and you're already having to do it for ACE,
[00:33:31.406]it's easy to dovetail in and have a college-level rubric
[00:33:37.046]that's being hosted in college-level outcomes
[00:33:39.559]and then tell your instructors to
[00:33:41.005]pull from here and pull from there
[00:33:42.281]depending on what that course is aligned with.
[00:33:47.951]So that's it, if this was an in person,
[00:33:51.057]I'd open it for questions and discussion.
[00:33:52.959]Given that this is online recorded,
[00:33:56.182]you'll have to find other ways to answer your questions.
[00:33:59.989]I know undergraduate programs is always interested
[00:34:02.918]to hear things that are working and not working.
[00:34:06.585]If there are major snafus you can contact me.
[00:34:10.494]My name is Chad Brassil, in Biological Sciences,
[00:34:14.321]and I'm helping with this transition,
[00:34:20.028]and so fire away.
[00:34:22.545]Ask me question and hopefully I'll be able to answer them,
[00:34:25.385]or point you to where those answers might be.

The screen size you are trying to search captions on is too small!

You can always jump over to MediaHub and check it out there.

Comments

0 Comments

Brown Bag: ACE Outcomes (Jan 26, 2017)

Description

Searchable Transcript

Comments icon comment

Related Channels

Comments