Jared Stevens: “Collecting Your Own Data”

Nebraska Center for Research on Children, Families and Schools Author

03/05/2019 Added

71 Plays

Description

2018-19 Methodology Applications Series Jared Stevens, Graduate Research Assistant, Department of Educational Psychology This presentation will provide an introduction to Qualtrics, a data collection platform widely used by businesses, schools and individuals. It will demonstrate how to build different types of questions and surveys to address components of Total Survey Error (TSE) to improve data quality.

Searchable Transcript

Search:

[00:00:00.473](upbeat music)
[00:00:06.700]Awesome.
[00:00:07.533]Well thank you, Natalie.
[00:00:08.570]Again, my name is Jared Stevens.
[00:00:09.910]I'm in the qualitative quantitative
[00:00:12.357]psychometric methods program and again
[00:00:13.723]with the Nebraska Center of Research
[00:00:16.158]on children, youth, families, and schools.
[00:00:17.110]We really like our acronyms over there.
[00:00:19.339](laughing)
[00:00:20.501]Alright, the title of my presentation is called
[00:00:22.190]collecting your own data, improving data quality
[00:00:24.630]through Qualtrics survey design.
[00:00:27.690]Just a short overview of the presentation,
[00:00:29.960]I'll first be talking a little bit about
[00:00:31.230]what Qualtrics is and I'll give a short walkthrough
[00:00:33.630]and orientation actually on the website
[00:00:35.990]and then I'll introduce data quality
[00:00:37.560]as well as total survey error
[00:00:39.850]including measurement error due to respondents
[00:00:41.740]as well as measurement error due to questionnaire design
[00:00:44.410]and ultimately we're trying to answer the question
[00:00:46.300]can Qualtrics help lead to better data quality?
[00:00:49.990]Before we get started, I just wanna say
[00:00:51.370]that I have given several presentations
[00:00:53.040]about simply just how to use Qualtrics
[00:00:54.850]and how Qualtrics works like how to design and edit surveys.
[00:00:58.840]This presentation has a slightly different flavor.
[00:01:01.350]It's more focused on how Qualtrics can be used
[00:01:03.730]to help minimize forms of survey error,
[00:01:05.650]hopefully leading to better data quality.
[00:01:07.930]At the end, I will provide if you guys want the slides
[00:01:09.890]for the other presentation, just simply how to use Qualtrics
[00:01:12.270]you can reach out to me and I can provide those for you.
[00:01:15.950]Alright, so what is Qualtrics?
[00:01:18.556]Qualtrics is a powerful software
[00:01:19.580]for collecting and analyzing data.
[00:01:21.820]It does allow users to build and distribute surveys,
[00:01:24.400]analyze responses and create reports
[00:01:26.640]all within a web browser.
[00:01:28.670]It is point and click interface
[00:01:29.930]so it's really easy to just kind of get in there
[00:01:31.660]and play around and kind of figure it out for yourself
[00:01:34.330]and you don't have to install any software
[00:01:36.260]so it all does happen on that web browser.
[00:01:38.430]Obviously there are some caveats to that
[00:01:40.230]like you have to have internet to be able to use it.
[00:01:43.480]It is most common in the business world.
[00:01:45.480]For example, for market research, customer satisfaction,
[00:01:48.190]as well as product testing but its use is increasing
[00:01:51.510]in education research as well as social science research.
[00:01:56.130]Qualtrics is kind of viewed as this fancy software.
[00:01:59.100]It can create some pretty fancy surveys
[00:02:01.210]with some pretty cool features but is it actually
[00:02:04.190]useful for improving data quality?
[00:02:06.770]Hopefully this presentation will show that Qualtrics
[00:02:08.910]is one mechanism or option for collecting better data
[00:02:11.830]and improving data quality.
[00:02:13.550]There are obviously other options for collecting data
[00:02:16.140]including paper and pencil measures and again our center,
[00:02:19.587]we still do a lot of paper and pencil measures
[00:02:20.880]but these are just some other options here.
[00:02:23.160]We also have OMR software.
[00:02:24.470]OMR is optical mark recognition software.
[00:02:26.950]If you guys ever done something like Scantrons,
[00:02:28.700]the bubble forms and then you can scan them
[00:02:30.480]into the software so it is a traditional
[00:02:32.470]paper and pencil but it just makes
[00:02:33.850]the data entry a little easier
[00:02:35.600]and then there are plenty of other online survey platforms,
[00:02:38.760]Survey Monkey, Google Forms, and Red Cap
[00:02:40.870]are among the most popular.
[00:02:44.375]Obtaining a Qualtrics account.
[00:02:46.260]There are a couple different options here.
[00:02:48.130]There is simply just a free trial account
[00:02:49.950]and this is for if you kind of just wanna
[00:02:51.790]get in to see what Qualtrics is all about
[00:02:53.760]and play around with it.
[00:02:55.040]You can just go to Qualtrics.com
[00:02:56.570]and you click on that free trial account button.
[00:02:59.670]There is no time limit on that free account
[00:03:01.940]but you are limited to just one active survey
[00:03:04.240]and 100 responses total across all of the surveys
[00:03:06.700]that you have so it does really limit you
[00:03:08.830]on what you're able to do.
[00:03:11.210]You also have the ability to get a paid account
[00:03:13.540]or a regular account.
[00:03:14.920]You can pay for an individual Qualtrics account yourself.
[00:03:17.300]However, this is very pricey
[00:03:18.980]and there are some different levels
[00:03:20.200]or types of accounts.
[00:03:21.570]And then finally I'm assuming most of you here
[00:03:23.550]are affiliated with the university,
[00:03:25.559]so you do have the opportunity to get a university
[00:03:27.820]or also a business affiliated account
[00:03:29.910]and these accounts do have all the functionality
[00:03:32.250]of a paid account and sometimes even more so,
[00:03:35.380]even better functions than those paid accounts.
[00:03:37.970]If you want the link to do so,
[00:03:39.320]I have provided those below
[00:03:41.490]so you can just go to these websites
[00:03:42.590]in order to obtain your Qualtrics account
[00:03:44.070]through the university.
[00:03:45.940]There is a quick, I just wanted to provide
[00:03:48.853]the distinction between that free account
[00:03:50.293]and the university account.
[00:03:51.126]That free account, you are limited again,
[00:03:53.055]it's just one active survey as well as 100 responses allowed
[00:03:55.600]and then it really limits you what,
[00:03:59.012]limits what you're able to do actually
[00:04:00.375]within Qualtrics itself so you can only send
[00:04:01.210]10 outgoing emails.
[00:04:02.650]You also don't have access to some of the
[00:04:04.170]specialty functions in Qualtrics,
[00:04:07.050]for example things like the custom code,
[00:04:09.260]specialty question types,
[00:04:10.410]as well as exporting your data
[00:04:11.700]and sharing the products.
[00:04:13.655]The university and paid account does allow you to have
[00:04:15.480]unlimited active surveys, unlimited responses,
[00:04:18.410]unlimited outgoing emails, and then you get
[00:04:20.620]access to all those specialty features in Qualtrics.
[00:04:25.140]Now I'm gonna orient you guys a little bit
[00:04:27.040]on what Qualtrics is so I'm gonna go to the website here.
[00:04:30.890]Our map UNL Qualtrics account was not working this morning,
[00:04:34.490]so unfortunately I'm logged into somebody else's here
[00:04:37.780]but I can still kind of show you the basics here
[00:04:40.230]so first this is the home screen.
[00:04:42.850]After you log in this is kind of what you see
[00:04:44.650]within Qualtrics or this is what I like to by default
[00:04:47.670]see when I log in.
[00:04:49.575]You can kind of change the options here
[00:04:50.408]so if you want a slightly different view
[00:04:51.510]or if you don't wanna see the folders and such
[00:04:53.980]you can control that.
[00:04:55.710]We are in the projects tab here
[00:04:57.130]so these are the surveys that we have within our library.
[00:05:00.180]Along the left hand side you can set up different folders
[00:05:02.570]if you wanna organize them
[00:05:03.590]and then all of your shared projects will happen here
[00:05:05.600]so if somebody shares a Qualtrics survey with you,
[00:05:07.840]it will appear right there.
[00:05:10.250]Some other options across the top here.
[00:05:11.780]We do have a context tab so if you're trying
[00:05:15.150]to distribute your survey to particular contact lists,
[00:05:18.760]you can set up those within Qualtrics itself
[00:05:20.650]and it does make it really easy to then
[00:05:22.390]distribute your surveys.
[00:05:24.150]This library function is great if you want to
[00:05:26.760]make it really easy to create surveys in the future.
[00:05:29.720]You can save particular questions,
[00:05:31.250]you can save entire surveys,
[00:05:32.970]you can also save graphics, files, and messages.
[00:05:35.310]And this is just an easy deposit for all of them
[00:05:37.597]and you can access them all there.
[00:05:39.810]You can also access the help one.
[00:05:41.800]The help, if you're struggling with any functions
[00:05:45.378]in Qualtrics, there's a support site
[00:05:47.058]and then there's also a phone number.
[00:05:48.210]I have actually used that so many times
[00:05:50.270]on some of the surveys.
[00:05:51.590]Their support team is fantastic
[00:05:53.450]and they really work with you.
[00:05:55.630]Actually one time I had no idea how to do something.
[00:05:58.370]I called them and they're like, oh, that's tough.
[00:06:00.370]They're like, okay, let me hang up,
[00:06:01.620]see if I can figure it out.
[00:06:02.670]And then about an hour later
[00:06:04.679]they called me back and they walked me through how to do it
[00:06:06.557]so they're fantastic.
[00:06:07.430]I really really like their help feature there.
[00:06:10.510]And then finally if you just wanna check out
[00:06:12.320]your account settings, you can hit this little icon there.
[00:06:15.010]We'll go back to the projects and I'll kind of
[00:06:16.790]show you once you click on a survey.
[00:06:23.975]Alright, so we clicked on a survey.
[00:06:24.880]This is what appears when you actually click on a survey
[00:06:27.975]so you have all the questions listed within here.
[00:06:32.071]You can separate your survey into different blocks.
[00:06:35.580]Blocks are basically just a way to separate questions
[00:06:38.070]so oftentimes I'll put all the demographics in one block
[00:06:42.175]and then maybe the substance survey questions
[00:06:43.040]in another block.
[00:06:44.434]It's just a way to easily organize your survey.
[00:06:46.050]There are some different block options.
[00:06:47.410]If you just hit this down arrow,
[00:06:48.580]you can randomize the questions within the block,
[00:06:51.280]you can change the next and previous button text,
[00:06:53.800]you can copy blocks and you can also copy blocks
[00:06:56.090]to the library to make them easier to find later.
[00:07:00.700]Next I'll show you guys the look and feel tab
[00:07:02.590]and I'll kind of explain this in a little more detail
[00:07:04.610]at the end of the presentation but I just kind of
[00:07:06.160]wanted to show you guys where it is.
[00:07:08.120]Look and feel is basically where you control
[00:07:09.990]what the survey looks like
[00:07:11.180]including things like the theme, the style,
[00:07:14.172]the font, and the navigation of the survey.
[00:07:15.720]There's kind of some different options there.
[00:07:18.450]Next we have the survey flow.
[00:07:22.050]Survey flow is basically a block level view
[00:07:24.780]of the survey and you can add fancy stuff,
[00:07:27.480]for example things like randomization, branching,
[00:07:30.055]embedded data, tons of different options.
[00:07:32.796]This feature is kind of for more advanced Qualtrics users.
[00:07:36.890]A lot of my surveys, I don't even use this survey flow
[00:07:39.230]so I won't get into it in this presentation
[00:07:41.420]but if you do have some questions
[00:07:42.820]about how to use this, I'd be more than happy
[00:07:45.080]to help you with that.
[00:07:48.670]Next we have the survey options and this is exactly
[00:07:50.990]what it sounds like,
[00:07:52.030]some different options you have for your surveys.
[00:07:54.720]Some of the functions that I use frequently in here
[00:07:57.047]are the back button.
[00:07:58.390]I like to allow participants to sometimes go back
[00:08:00.740]on certain surveys.
[00:08:02.360]I also by default this save and continue button
[00:08:05.060]is always checked and this just allows participants,
[00:08:08.250]they don't have to complete that survey within one sitting.
[00:08:10.320]They can stop, maybe go do something and come back
[00:08:13.570]couple hours, next day later
[00:08:14.990]and still be able to finish the survey.
[00:08:17.540]Some of the other survey options that I use frequently
[00:08:19.430]are adding question numbers, adding a password to the survey
[00:08:22.390]so if you want to control who accesses your survey
[00:08:25.850]and then you can also customize
[00:08:26.920]the end of the survey messages
[00:08:29.028]so if you wanna send a particular message
[00:08:30.350]after they hit submit, you can change that there.
[00:08:35.440]And then finally we have this tools option here.
[00:08:38.691]So tools again are some different tools you have
[00:08:41.090]to kind of help with your survey so you can auto number
[00:08:44.500]questions, you can set up triggers
[00:08:46.500]and triggers are great if like,
[00:08:48.350]say you wanna set up a newsletter, an email newsletter.
[00:08:51.190]Basically once they put in their email
[00:08:53.130]to a particular box then you can set a trigger
[00:08:55.571]that you add it to a contact list
[00:08:56.730]and then it just makes it easy to download that
[00:08:59.815]and then you can send them emails,
[00:09:01.530]send that newsletter.
[00:09:04.000]Some of the other functionalities and tools
[00:09:06.551]are translation of the survey, scoring of the survey,
[00:09:10.060]and then you can also export and import the survey
[00:09:12.130]so sometimes I use this export function.
[00:09:15.095]I just create the survey within Qualtrics
[00:09:17.267]and then I just export it so then for IRB,
[00:09:19.400]it looks exactly like it does within Qualtrics
[00:09:22.830]so that's pretty helpful.
[00:09:24.320]Some of the other tabs across the top here.
[00:09:26.050]Right now we're in that survey tab.
[00:09:27.410]There are some other tabs across the top here.
[00:09:30.380]This is the actions tab.
[00:09:32.460]This is where you can set particular actions
[00:09:34.080]for your survey including things like
[00:09:35.480]sending emails, scoring surveys, and creating reports.
[00:09:39.100]However, these are mainly used within the business world.
[00:09:41.440]I actually haven't used the actions to be honest
[00:09:44.120]in any of the surveys I've created.
[00:09:46.540]You also have the ability to distribute within Qualtrics
[00:09:48.720]so that's when you're ready to collect your data.
[00:09:52.450]These to the left are the different options
[00:09:55.372]for collecting data so we do have the anonymous link,
[00:09:57.240]we have the ability to send emails,
[00:09:59.150]we can do personal links, you can also post to social media
[00:10:01.880]sites to gain kind of a further reach.
[00:10:04.380]There also is the offline app.
[00:10:05.740]This does cost extra money.
[00:10:07.370]It's a special feature in Qualtrics
[00:10:08.970]but we used to have access to it and it was really great
[00:10:12.034]for collecting in rural communities and such
[00:10:14.980]and then you also have the QR code
[00:10:16.770]so maybe somebody doesn't have an email or something
[00:10:19.340]and you're trying to collect data on a tablet,
[00:10:22.430]this QR code is pretty helpful
[00:10:24.530]and then finally you do have the ability
[00:10:25.880]to purchase respondents through Qualtrics.
[00:10:28.940]Next is the data analysis tab.
[00:10:31.010]There actually is some built in analysis through Qualtrics.
[00:10:34.210]It does let you filter, classify, merge,
[00:10:35.812]clean and statistically analyze your data
[00:10:39.050]so there are lots of built in options.
[00:10:41.610]I usually just export it and analyze it
[00:10:43.960]in SPSS or something but you do have the ability
[00:10:46.100]to do some analyzing here within Qualtrics.
[00:10:49.790]And then finally we do have the reporting function
[00:10:52.090]and this is, you can actually create some reports
[00:10:55.710]right here in Qualtrics.
[00:10:57.350]Visualizations and you can do things like add filters
[00:10:59.610]and you can share the report.
[00:11:01.290]I think they advertise this as providing like 90%
[00:11:03.800]of what you want in a report with only doing
[00:11:05.800]10% of the work so they like to really talk about
[00:11:08.623]their reporting feature in Qualtrics
[00:11:10.567]and it has been pretty helpful
[00:11:11.920]for some of the surveys that I've worked with.
[00:11:16.000]Alright, we're gonna go back into the presentation.
[00:11:24.628]Alright, so now that I've oriented you guys
[00:11:27.340]on what Qualtrics and a little bit about how it works,
[00:11:30.470]now we're gonna shift gears a little bit
[00:11:32.330]and talk about data quality as well as total survey error
[00:11:35.490]because in order to determine if Qualtrics is useful,
[00:11:37.500]we need to determine exactly what data quality is.
[00:11:41.330]So what is data quality?
[00:11:43.250]It's an umbrella concept that covers
[00:11:44.840]the three main sources affecting the validity
[00:11:46.970]and reliability of survey data and these include
[00:11:50.280]respondent behaviors, so things like the response quality
[00:11:53.530]including their verbal skills, their ability to
[00:11:55.580]retrieve the information requested,
[00:11:57.300]as well as satisficing behaviors.
[00:11:59.630]It also includes the study architecture,
[00:12:01.240]so elements of the survey design including the mode,
[00:12:03.750]the length, the number and format of response options
[00:12:06.220]as well as the complexity of language
[00:12:08.470]and finally the institutional practices
[00:12:10.450]of the data collection agencies.
[00:12:13.030]That includes things like the adequacy
[00:12:14.440]of the interviewer training if you're performing
[00:12:16.260]interviews for your surveys,
[00:12:18.067]appropriateness of the sampling design,
[00:12:20.300]as well as data entry monitoring procedures.
[00:12:23.690]Just a quick thing, you will see a common theme
[00:12:25.740]throughout my slides.
[00:12:26.573]Anything that is in red, so in this case
[00:12:29.510]the respondent behaviors as well as study architecture
[00:12:31.930]are things that Qualtrics do form,
[00:12:34.200]does provide some form of help with.
[00:12:38.130]Here Qualtrics surveys can help with respondent behaviors
[00:12:41.160]as well as the study architecture but unfortunately,
[00:12:43.830]it does not offer much help with
[00:12:45.050]the institutional practices of the data collection agencies.
[00:12:50.510]The ability to draw correct conclusions or insights
[00:12:52.830]from survey data does depend on the quality
[00:12:55.100]of the data you're collecting.
[00:12:57.400]We kind of have two goals when we're creating surveys.
[00:13:00.000]First is that the survey needs to be easy
[00:13:01.980]for respondents to provide valid, reliable,
[00:13:04.160]and accurate answers to each question
[00:13:06.520]and then our secondary goal is to minimize the difficulty
[00:13:09.300]of administering the survey.
[00:13:10.810]We want to make our survey and make the questions
[00:13:13.140]on the survey as easy as possible to complete
[00:13:15.800]and so Qualtrics is just one mechanism or option
[00:13:18.410]for creating and collecting survey data
[00:13:20.160]and its use can help improve
[00:13:22.740]the quality of the data collected.
[00:13:26.600]Total survey error is one example of a framework
[00:13:29.200]to improve the data quality of a survey.
[00:13:33.310]Total survey error is the accumulation of all errors
[00:13:35.700]that may arise in the design, collection,
[00:13:37.620]processing, and analysis of survey data.
[00:13:40.660]A survey error is defined as the deviation
[00:13:42.820]of a survey response from its underlying true value.
[00:13:46.370]Our goal when we create surveys is to try to minimize
[00:13:49.090]the total survey error and this includes
[00:13:50.950]making the correct design decisions
[00:13:52.980]which simultaneously requires considering many quality
[00:13:55.660]and cost factors as well as choosing the combination
[00:13:58.230]of design features and parameters
[00:13:59.690]that minimize the total survey error
[00:14:01.280]within all the specified constraints.
[00:14:03.470]Oftentimes we have a lot of different options
[00:14:05.430]for when we're collecting data
[00:14:07.683]and sometimes it does take some trade offs.
[00:14:08.810]You have to think about the differences between
[00:14:10.670]collecting paper and pencil data
[00:14:12.100]as opposed to collecting data
[00:14:14.010]on a survey platform like Qualtrics.
[00:14:18.730]Survey errors can be classified into three broad categories.
[00:14:22.410]First are errors of non observation and these are errors
[00:14:24.790]related to selecting respondents for the survey
[00:14:27.430]and these include errors of coverage,
[00:14:29.260]sampling, and non response.
[00:14:32.540]Survey error also includes errors of observation
[00:14:34.650]or measurement and these are response accuracy issues
[00:14:37.144]and errors related to the survey instrument,
[00:14:39.740]the respondent, the interview,
[00:14:41.520]as well as the mode of collection.
[00:14:43.890]And then finally we have errors of processing
[00:14:45.800]and these are errors that occur in processing
[00:14:47.600]and analyzing of survey data and include errors of coding,
[00:14:50.620]editing, and adjusting.
[00:14:53.610]We're gonna unpack errors of non observation
[00:14:55.610]as well as errors of observation and measurement.
[00:14:58.170]We do need to know that Qualtrics is limited
[00:14:59.940]in helping with some survey errors
[00:15:01.430]like errors of processing.
[00:15:05.230]This slide is just to help orient us
[00:15:06.730]so first I'm gonna talk about errors of non observation
[00:15:09.260]and with a particular focus on non response error.
[00:15:15.518]Errors of non observation are errors related
[00:15:18.620]to selecting respondents for a survey.
[00:15:21.630]First we have coverage errors
[00:15:22.680]and these are errors that occur when some members
[00:15:24.800]of a population are excluded from the sample framed
[00:15:27.190]used for the study.
[00:15:28.540]So a particular group have no opportunity
[00:15:31.410]to be selected for the study.
[00:15:33.550]Next are sampling errors and this is the degree
[00:15:35.460]to which a survey statistic differs from its true value
[00:15:38.600]due to the fact the survey was conducted
[00:15:40.460]among one of many possible survey samples.
[00:15:43.020]Just because you collect data on a survey sample
[00:15:45.180]it doesn't mean that that sample is necessarily
[00:15:46.770]gonna generalize to the public and then finally
[00:15:49.990]we have non response errors and this is when data
[00:15:52.160]are not collected on either entire respondents
[00:15:54.260]or individual survey questions.
[00:15:56.860]As you can see here, non response errors
[00:15:58.477]are the only one highlighted in red so technically speaking
[00:16:02.660]Qualtrics does offer some limited help
[00:16:04.840]with coverage and sampling error via that distributions tab
[00:16:08.140]but these are all non probability methods.
[00:16:10.010]They're convenience sampling so you do have
[00:16:11.840]to keep that in mind.
[00:16:13.510]You can set up things like email notifications.
[00:16:16.120]You can also do survey questions
[00:16:17.410]coming directly in the email
[00:16:18.620]which is shown to increase response rates.
[00:16:21.130]You can do things like send postcard invitations
[00:16:23.820]with a link or QR code for maybe individuals
[00:16:25.950]who don't have an email or you can't
[00:16:27.420]reach them via email.
[00:16:28.980]You can also post to social media sites if you wanna gain
[00:16:31.169]maybe a further reach and then again,
[00:16:33.990]Qualtrics does have the ability to purchase
[00:16:35.760]panel respondents so for example if you're trying to
[00:16:39.571]collect data from a bunch of nurses
[00:16:40.560]or a bunch of businesses,
[00:16:42.050]the purchase panel will actually do all the work
[00:16:44.430]in recruiting them and getting them to complete
[00:16:47.147]your survey and it's actually from what I heard
[00:16:49.267]fairly reasonably priced.
[00:16:50.100]We never used it at the center
[00:16:51.880]but from what I've heard it's actually not bad
[00:16:54.160]as opposed to all the time and effort
[00:16:56.771]you often spend trying to collect your own data.
[00:16:58.280]There are a few caveats you got to keep in mind.
[00:17:00.350]Qualtrics apparently does pay their panel respondents
[00:17:03.350]and so that can lead to some effects
[00:17:05.110]because they might be more motivated
[00:17:06.580]or those kind of things so that's something to keep in mind.
[00:17:11.640]We talked about coverage and sampling errors
[00:17:13.230]so how about errors of non response?
[00:17:15.770]Non response error is the general source of error
[00:17:17.770]encompassing both unit and item non response
[00:17:20.800]so unit non response is when a particular sample unit
[00:17:23.990]does not respond to any part of a questionnaire
[00:17:26.530]so for example maybe a household refuses
[00:17:28.530]to participate in a face to face survey
[00:17:30.710]or a mailed questionnaire survey is never returned.
[00:17:34.990]The other form of non response is item non response
[00:17:38.145]and that's when the survey is only partially completed
[00:17:40.245]because the interview was prematurely terminated
[00:17:43.170]or some items that should have been answered on that survey
[00:17:45.270]were left blank.
[00:17:49.700]There are several safeguards in Qualtrics
[00:17:51.440]that can help guard against forms of non response error
[00:17:53.830]and so I'm gonna unpack each one of these.
[00:17:57.190]First up is question validation and this is actually
[00:17:59.610]the ability to request or force a response.
[00:18:03.040]This feature is found in Qualtrics.
[00:18:04.870]I know it's kind of hard to see
[00:18:06.180]but you might be able to see it
[00:18:07.210]along the right hand side here.
[00:18:09.410]You can add it, it's just a validation option
[00:18:11.270]so once you click on a question it's gonna
[00:18:13.590]pop you down there and then you can select either force
[00:18:16.160]or request response.
[00:18:17.840]Request response, basically this is the screenshot
[00:18:20.020]to the left here.
[00:18:21.320]It adds a little prompt box pops up
[00:18:24.500]and it says there's on unanswered question on this page.
[00:18:26.810]Would you like to continue and then the respondent
[00:18:29.780]has the ability to actually continue without saving
[00:18:31.978]or they can go back and answer the question.
[00:18:33.970]This is really helpful.
[00:18:34.803]Maybe a respondent just missed a question on the survey
[00:18:37.839]and this just kind of prompts them to do it
[00:18:38.900]and it does lead to a little bit less
[00:18:41.170]item non response and then the screenshot to the right
[00:18:44.231]is forcing a response.
[00:18:47.488]If a respondent does not provide an answer to this survey,
[00:18:51.030]a red error bar will appear above that says
[00:18:54.960]please answer this question and it actually
[00:18:57.100]doesn't let them submit that page of the survey
[00:18:59.330]until they select an option.
[00:19:02.170]There is a quick caveat with that.
[00:19:03.860]Forcing respondents to answer questions
[00:19:06.940]is sometimes flagged by IRB and it's also not recommended
[00:19:10.080]for sensitive questions.
[00:19:11.680]There is some research showing that if you
[00:19:14.120]force a response for sensitive questions,
[00:19:15.890]it actually leads to greater non response
[00:19:17.930]so you got to kind of have a balance there.
[00:19:20.070]I usually just use forced response
[00:19:21.810]for extremely necessary information
[00:19:24.050]for example like their consent to participate in the survey
[00:19:27.040]or maybe their unique identifying code
[00:19:28.980]in order to analyze the data but for the most part
[00:19:31.690]I just use that request response
[00:19:33.140]as it just gives them another opportunity
[00:19:35.878]to maybe select an answer on something they have missed.
[00:19:39.140]You also can do content validation for text entry boxes
[00:19:42.320]and I'll get into that a little later as well.
[00:19:45.540]Next we have the don't know response options.
[00:19:47.830]While don't know response option is not exactly
[00:19:51.150]a non response error, it's often treated pretty similarly
[00:19:54.560]so in most cases the don't know response option
[00:19:56.570]should be used judiciously since it becomes
[00:19:59.100]an easy out for respondents who are unwilling
[00:20:00.980]to think about or commit to an answer
[00:20:02.930]and it is a form of missing data.
[00:20:05.100]To counter the possible missing data
[00:20:06.550]and ability of respondents to take the easy way out,
[00:20:09.090]contingency items or questions can be used
[00:20:11.070]for those who select that don't know response
[00:20:13.220]and these can be set up in Qualtrics
[00:20:14.850]by using skip or display logic
[00:20:16.750]and so I'll show each of these here.
[00:20:20.385]It happens in a survey basically once you click question.
[00:20:23.530]It's just down here under the actions.
[00:20:25.130]You can add display logic or add skip logic.
[00:20:27.950]I did say here if there's a lot of logic involved
[00:20:30.200]again you can use that survey flow to better control
[00:20:32.027]a participation's see but if you're just doing
[00:20:34.490]some pretty simple stuff,
[00:20:36.200]these add display logic and skip logic are pretty helpful.
[00:20:40.510]So first skip logic.
[00:20:42.290]Skip logic isn't necessarily super helpful
[00:20:44.150]with don't know response options.
[00:20:45.240]I did just wanna show you guys how to do it.
[00:20:47.890]Basically you just hit skip logic
[00:20:49.930]and then this little logic will appear
[00:20:52.010]and then you kind of enter the conditions
[00:20:53.480]so I have some setup here where if they select Modern Family
[00:20:57.140]for their favorite comedy TV show,
[00:20:58.810]that person is obviously wrong
[00:21:00.940](laughing) so we want them to
[00:21:02.090]end the survey immediately.
[00:21:03.830]We don't wanna know anything else
[00:21:05.286](laughing) from that respondent.
[00:21:09.160]We also have the ability to add display logic
[00:21:12.260]and this is basically it only displays a question
[00:21:14.670]if certain conditions are met
[00:21:16.400]and you can set up display logic
[00:21:17.810]based on answer choices, the device type,
[00:21:20.010]as well as things like quotas and so in this one
[00:21:22.690]I have some display logic.
[00:21:24.320]A question appears, please explain why you don't know
[00:21:26.800]what your favorite comedy TV show
[00:21:28.510]and it's only gonna appear if they select
[00:21:30.380]that don't know response.
[00:21:34.500]Alright, so we've covered question validation
[00:21:36.390]as well as probing don't know responses
[00:21:38.410]so now we'll get into automated emails.
[00:21:42.140]Again this is found in the distributions tab there.
[00:21:48.470]Once you hit the email so it's the one
[00:21:50.280]that's the arrow there.
[00:21:52.360]Once you hit the email distributions
[00:21:54.545]basically you can set up automated emails
[00:21:56.860]to particular contact lists and then the important thing
[00:22:00.210]is you can set email reminders.
[00:22:01.913]This screenshot to the bottom here,
[00:22:03.530]I sent out a survey a little over a week ago
[00:22:07.610]and then one week later it says set a reminder
[00:22:10.710]to all the unfinished participants
[00:22:12.750]requesting them hey, we'd really like you
[00:22:14.280]to try to complete this survey if you can.
[00:22:16.283]The more contacts you made obviously
[00:22:19.650]you hit a certain point where you don't wanna send
[00:22:21.340]them anymore, you don't wanna hound them
[00:22:23.707]but the more contacts you made usually leads to
[00:22:25.560]less item non response.
[00:22:29.830]You can also set up things like thank you messages
[00:22:32.070]so if you wanna thank them for participating in your survey.
[00:22:36.190]Alright, so we talked about errors of non observation,
[00:22:38.920]now let's move into errors of observation
[00:22:41.820]and errors due to the survey instrument
[00:22:43.340]as well as the respondents and this is really
[00:22:45.870]where the real meat of the presentation is.
[00:22:49.640]Errors of observation are also called errors of measurement
[00:22:52.470]and they're typically defined as the difference
[00:22:55.025]between what respondents report when they answer
[00:22:55.960]a survey question and the true underlying value
[00:22:58.410]of that attribute being measured.
[00:23:00.690]The American Association for Public Opinion Research
[00:23:02.812]defines errors of observation as errors that may arise
[00:23:06.260]due to question wording, the order of questions
[00:23:08.480]and categories, the behaviors of interviewers
[00:23:10.820]and respondents, data entry,
[00:23:12.460]and the mode of administration of a survey.
[00:23:16.320]Online surveys do strongly affect errors of observation.
[00:23:19.520]These errors are most often attributed to the social
[00:23:22.000]and cognitive process a respondent engages in
[00:23:24.870]while answering survey questions
[00:23:27.030]so for example webs respondents might be more prone
[00:23:29.160]to take cognitive short cuts on web surveys
[00:23:31.450]which can lead to more measurement error
[00:23:33.240]or they might be more willing to disclose
[00:23:35.010]personal information which leads to less measurement error.
[00:23:37.980]Again, this is going back to the trade offs
[00:23:39.630]of the mode you use to collect data.
[00:23:41.970]We do have to consider all of those
[00:23:43.820]and our job really as the survey creator
[00:23:46.000]is to attempt to reduce errors of measurement
[00:23:48.030]through sound survey design and so that's what
[00:23:50.350]the rest of the presentation will hopefully demonstrate.
[00:23:55.300]Observation errors include errors arising from respondents,
[00:23:58.230]interviewers, the survey questionnaire,
[00:24:00.210]as well as the mode of the interview and the survey
[00:24:02.817]and so I'll unpack each of these in red here
[00:24:05.870]and a lot of the recommendations come from
[00:24:08.540]Tourangeau, Conrad, and Cooper's 2013 book
[00:24:10.518]The Science of Web Surveys which kind of explains
[00:24:13.670]errors of observations for web surveys
[00:24:15.340]and how we can try to minimize them.
[00:24:18.040]First we'll get into errors of observation
[00:24:20.970]due to the respondents.
[00:24:24.433]Respondents may provide incorrect information
[00:24:26.230]in response to questions and this happens both deliberately
[00:24:30.480]as well as unintentionally so sometimes
[00:24:32.370]they don't even realize they're doing it.
[00:24:34.420]Errors do often occur during the cognitive response process
[00:24:37.480]that respondents engage in when completing a survey.
[00:24:40.930]The cognitive response process
[00:24:42.380]and this was formulated by Tourangeau in 1984.
[00:24:45.690]First the respondent comprehends the question
[00:24:48.055]and the instructions then they try to retrieve
[00:24:52.024]specific memories or information related to that question.
[00:24:55.761]Next they make judgements regarding the matching
[00:24:56.600]of the information to the question
[00:24:58.000]and the completeness of that information
[00:25:00.150]and then finally they formulate a response
[00:25:02.620]and hopefully they select a response option on the survey.
[00:25:06.570]Measurement errors can occur in any of these steps
[00:25:08.570]throughout the cognitive response process.
[00:25:10.910]For example, question wording and the look and feel
[00:25:13.050]of a survey can affect the ability of respondents
[00:25:15.240]to comprehend the question.
[00:25:17.140]Correct specification in the response options
[00:25:20.139]can affect the respondent's ability to retrieve information
[00:25:21.610]or make judgements and the response options
[00:25:23.820]as well as the respondent behaviors like satisficing
[00:25:26.150]can affect the ability of respondents
[00:25:27.780]to formulate a response.
[00:25:32.850]Respondent interaction with the survey instrument
[00:25:34.770]can result in several measurement errors.
[00:25:37.140]For example, certain things like demographic characteristics
[00:25:39.900]can result in measurement error
[00:25:42.500]so there's actually research showing that older
[00:25:45.120]and male respondents, sorry guys, actually lead to
[00:25:49.330]greater misreport on surveys than younger
[00:25:51.280]and female respondents, so yeah.
[00:25:54.570]Additionally, the respondent's characteristics
[00:25:56.450]may interact with the topic of the survey
[00:25:58.240]to produce response effects.
[00:26:00.890]Say for example you're collecting data about
[00:26:03.390]supporting the military and you have
[00:26:05.180]a couple individuals who are in the military
[00:26:07.330]who are completing those questions.
[00:26:08.770]They're probably gonna provide different answers
[00:26:10.810]than just the general public would.
[00:26:12.840]There are also things like context effects,
[00:26:14.590]respondent's motivation as well as fatigue and boredom.
[00:26:18.640]Unfortunately these are measurement errors
[00:26:20.673]due to the respondent that Qualtrics does not
[00:26:21.730]offer much help with.
[00:26:23.080]However, there are some respondent errors
[00:26:26.233]that Qualtrics does offer some help with
[00:26:28.407]and so I'm gonna unpack each one of these here.
[00:26:34.600]The first one is respondents answering too quickly.
[00:26:37.660]Sometimes survey respondents may perform
[00:26:39.540]inattentive responding and speed through the questions
[00:26:42.140]without actually reading the question.
[00:26:44.790]This behavior can actually usually be detected in Qualtrics
[00:26:47.800]by adding a timing question
[00:26:50.417]and this is one way that Qualtrics,
[00:26:52.170]you obviously don't have the ability
[00:26:53.490]to do this on paper and pencil measures
[00:26:55.310]and so this is one way that Qualtrics really helps
[00:26:58.400]with these kinds of behaviors.
[00:27:00.390]This timing question does detect how long respondents
[00:27:02.580]spend on a particular page and it can keep track
[00:27:05.220]of the first and last click,
[00:27:07.310]the total number of clicks,
[00:27:08.490]as well as how long it takes to submit to page
[00:27:11.090]and the timing question can also require participants
[00:27:13.430]to spend a certain amount of time on each page.
[00:27:16.240]How do we set this up on Qualtrics?
[00:27:18.110]I included a couple screenshots here.
[00:27:20.430]Basically it's actually just one of the question types.
[00:27:22.350]If you just hit change question type,
[00:27:24.510]you go down to the timing question.
[00:27:26.650]This little box will pop up and you have
[00:27:29.510]some different options here.
[00:27:30.790]You can enable submit after a certain amount of seconds
[00:27:33.300]so maybe if your respondents to spend like five minutes
[00:27:35.940]on a page the next button actually won't appear
[00:27:39.390]on that survey until that time has elapsed.
[00:27:41.810]You also have the ability to auto advance
[00:27:43.560]after a certain amount of seconds.
[00:27:45.250]We have used this auto advance feature
[00:27:47.150]on knowledge quizzes where they're taking a thing
[00:27:50.640]and we only wanna give them 10 minutes
[00:27:52.320]and so it basically just moves them on
[00:27:54.080]after that 10 minutes.
[00:27:55.680]You do have the ability to show the timer to participants.
[00:27:58.800]In most cases, it's probably not advisable to do so
[00:28:01.432]but in some cases it might be necessary
[00:28:04.490]and then finally the screenshot to the far right
[00:28:06.770]does show what the data export looks like
[00:28:08.480]for those timing questions.
[00:28:11.410]That one that's highlighted in red here,
[00:28:14.960]it may be an indication that this person
[00:28:16.600]is performing inattentive responding.
[00:28:18.680]Their first click was under a second,
[00:28:20.650]their last click was just over a second
[00:28:22.770]and they submitted the page in about a second and a half
[00:28:25.280]when everybody else was submitting the page
[00:28:26.940]in about 13 or so seconds so this might be
[00:28:29.920]an indication that this person is performing
[00:28:31.570]some form of inattentive responding
[00:28:33.860]and we might choose to leave them out
[00:28:35.410]when analyzing our data.
[00:28:39.553]Next are threats of satisficing.
[00:28:42.100]Satisficing is when behaviors devote less than optimal
[00:28:44.830]effort to answering survey questions
[00:28:47.120]and this includes acquiescence bias
[00:28:48.810]and social desirability bias, straight lining,
[00:28:51.130]as well as primacy and recency effects.
[00:28:53.710]There are several factors that do affect satisficing
[00:28:56.240]and these include the difficulty of the tasks,
[00:28:58.990]the respondent ability.
[00:29:00.580]There's research showing that low ability respondents
[00:29:02.400]are more likely to engage in satisficing behaviors,
[00:29:05.570]as well as a respondent's motivation.
[00:29:07.930]Oftentimes we kind of hear that satisficing
[00:29:10.010]are the respondent's problem but actually
[00:29:12.620]poor survey design can result in more satisficing behaviors
[00:29:16.300]and so some features in Qualtrics can help combat
[00:29:18.930]these types of things.
[00:29:21.770]First up we have acquiescence bias.
[00:29:23.700]This is when respondents tend to agree with suggestions
[00:29:26.390]or questions.
[00:29:27.810]This is most common with Likert scales
[00:29:29.630]as well as yes no and true false questions
[00:29:32.190]and there are several things that you can set up
[00:29:33.840]in Qualtrics that can help with this
[00:29:35.660]and these include avoid using generic response scales
[00:29:38.930]and use scales that are specific
[00:29:40.350]to the subject of the question.
[00:29:42.280]You can also try using both positively
[00:29:43.970]and negatively worded items which just kind of
[00:29:46.510]makes them slow down a little bit hopefully
[00:29:48.520]and you also want to avoid using matrices.
[00:29:51.090]It's a lot easier to just go down and hit
[00:29:53.400]all the strongly agrees if its easily laid out in a matrice.
[00:29:57.590]The above suggestions and the suggestions
[00:29:59.450]on the following few pages come from Vannette's
[00:30:01.340]The Qualtrics Handbook of Question Design.
[00:30:03.995]It's a really helpful handbook that this guy
[00:30:05.210]put together who works for Qualtrics
[00:30:07.580]and so that's where these recommendations come from.
[00:30:11.380]Next we have social desirability bias
[00:30:13.580]and this is the tendency of survey respondents
[00:30:15.550]to answer questions in a manner that will be viewed
[00:30:17.870]favorably by others.
[00:30:19.730]It often results in over reporting good behavior.
[00:30:23.280]For example things like voting records
[00:30:25.670]but under reporting bad or undesirable behavior
[00:30:27.910]like drug usage.
[00:30:29.460]There are some suggestions, some things in Qualtrics
[00:30:32.310]you can do to help combat this and these include again
[00:30:35.430]avoid using matrices.
[00:30:37.467]You could also try using both positively
[00:30:38.730]and negatively worded items.
[00:30:40.780]Research has also shown that beginning your survey
[00:30:43.000]with a confidentiality statement does lead to
[00:30:45.830]less social desirability bias and then finally
[00:30:49.670]if you're overly concerned about it,
[00:30:51.734]you can implement a social desirability scale
[00:30:53.480]and this is a special scale that's set up
[00:30:55.080]to detect social desirability bias.
[00:31:00.150]Next we have straight lining and this is when respondents
[00:31:02.390]provide the same answer for a number of survey questions.
[00:31:05.860]It's often the result of respondents not reading
[00:31:07.800]the question or statement and just simply
[00:31:09.440]clicking and filling in answer choices in a straight line
[00:31:12.140]and this is very related to the inattentive
[00:31:14.350]or speeded responding.
[00:31:16.320]There are some suggestions.
[00:31:18.134]There are some things you can do in Qualtrics
[00:31:19.050]to help with this.
[00:31:20.310]Again, we want to avoid using matrices.
[00:31:23.170]You also can try using that timing question
[00:31:25.120]because if they are performing straight lining behaviors,
[00:31:27.760]it's probably gonna take them less time.
[00:31:30.160]They're just gonna do it in about a couple seconds
[00:31:32.810]as opposed to actually how long it would take
[00:31:34.970]for them to actually read the questions.
[00:31:36.580]It's also important to use construct specific scales
[00:31:38.890]and kind of change up your scale
[00:31:40.560]rather than having the same scale throughout
[00:31:43.150]and then if you're overly concerned about it,
[00:31:45.000]you can actually ask one question per page.
[00:31:46.897]There is some research showing that
[00:31:49.230]if the participant has to actually submit the page
[00:31:52.190]after each question, it does lead to less
[00:31:54.940]straight lining behaviors just 'cause
[00:31:56.300]it's more work that they have to do.
[00:31:58.390]It is found in the look and feel general tab.
[00:32:03.240]And then finally the final satisficing behaviors
[00:32:05.000]are primacy and recency effects.
[00:32:06.870]A primacy effect is the tendency for respondents
[00:32:09.130]to select options at the beginning of a set of categories
[00:32:12.060]while the recency effect is the tendency
[00:32:13.810]for respondents to select options at the end of the scale.
[00:32:17.050]There are several things you can do to Qualtrics
[00:32:18.620]to help with this.
[00:32:20.490]For example you can randomize the response options
[00:32:23.760]which is one way to help with primacy and recency effects
[00:32:27.130]and if that response option is a scale,
[00:32:29.070]you obviously don't wanna randomize that scale
[00:32:31.120]but you can randomize the order in which
[00:32:32.680]the scale is presented.
[00:32:35.120]How do you do response option randomization?
[00:32:37.500]Once you click on the question that you're trying
[00:32:39.060]to randomize the response options for
[00:32:41.370]if you just hit this little settings tab here
[00:32:43.780]you go down to randomization and then you can
[00:32:45.340]add your randomization.
[00:32:46.890]You can choose to randomize the order of all your statements
[00:32:49.120]or you can set up some advanced randomization
[00:32:51.490]so for example if you have an other box at the end
[00:32:53.580]for them to put whatever, type whatever they want in,
[00:32:56.430]you always wanna put that at the end
[00:32:57.820]and so you can control that
[00:32:59.360]with this advanced randomization.
[00:33:04.350]We've now covered errors of observation due to respondents
[00:33:06.690]so now let's get into measurement error
[00:33:08.250]due to the questionnaire.
[00:33:09.850]The focus of the rest of the presentation
[00:33:11.280]is just to demonstrate how we can use Qualtrics
[00:33:13.920]to enhance our questionnaires
[00:33:15.620]and improve data quality.
[00:33:20.010]Alright, so the visual features of web surveys
[00:33:22.810]are likely to have more impact on the respondents
[00:33:25.220]and their answers than the visual features
[00:33:26.900]of traditional paper questionnaires.
[00:33:29.340]The survey or questionnaire can be a major
[00:33:31.090]source of error if it is poorly designed.
[00:33:35.280]This slide is basically just a list
[00:33:36.670]of common measurement errors which are due
[00:33:38.910]to the questionnaire and so we're gonna unpack
[00:33:41.280]each one of them that are highlighted in red.
[00:33:43.150]Again in red are the ones that Qualtrics can help with.
[00:33:47.810]First up we have the format of the question.
[00:33:50.730]It is important to utilize the correct question type
[00:33:53.140]for the construct that you're trying to measure.
[00:33:55.580]Qualtrics does allow for a ton of different question types
[00:33:58.250]in both open and closed question formats.
[00:34:01.280]The standards question types include things like
[00:34:02.913]multiple choice, matrix tables, text entry,
[00:34:05.573]as well as slider and rank order questions
[00:34:08.871]and so I'll kind of unpack each one of these here.
[00:34:11.250]First we have multiple choice questions.
[00:34:13.570]You can do single or multiple answer questions
[00:34:16.770]so oftentimes it's preferred to use single answer
[00:34:19.830]but if you do wanna do a select all that apply question,
[00:34:22.210]you do have the ability to do so.
[00:34:24.150]You could also implement a drop down list or a select box
[00:34:27.610]and these are just different visual options
[00:34:29.360]for presenting multiple choice questions.
[00:34:31.840]Additionally you can edit
[00:34:33.150]the positioning of response options
[00:34:34.660]so if you want them vertically aligned
[00:34:36.230]or horizontally aligned and you can add columns
[00:34:39.040]so maybe you have 20 different response options
[00:34:41.760]you can order them in columns
[00:34:43.560]and just make it a little easier
[00:34:44.760]than having them to scroll down every like,
[00:34:46.810]down the entire page.
[00:34:48.340]And then additionally multiple choice questions
[00:34:50.560]do allow for text entry so again if you want to add
[00:34:53.550]an other box and then an area for them to type in,
[00:34:56.210]you just hit this little blue arrow
[00:34:57.660]to the right of the response option
[00:35:00.060]and then you just hit allow text entry
[00:35:01.860]and then it'll just basically put this little box
[00:35:03.700]in next to it for them to type their own answer in
[00:35:05.800]and again you can use content validation for that as well.
[00:35:09.800]Next we have matrix tables.
[00:35:11.490]Matrix tables are grids used to display data
[00:35:14.070]in a structured format.
[00:35:16.170]They are helpful if you have similar questions
[00:35:17.900]with the same response scale
[00:35:19.840]but there is a lot of research showing that
[00:35:21.760]using them in web surveys actually leads to
[00:35:23.830]an inattentive responding so in most cases
[00:35:25.873]we don't want to use matrices
[00:35:28.200]but there are some cases as I'll demonstrate later
[00:35:30.310]where maybe matrices are appropriate.
[00:35:35.140]Next we have text entry questions
[00:35:36.750]and this is for open ended questions.
[00:35:38.990]You do have the ability to control the size of the box
[00:35:41.320]by the single line, multi line,
[00:35:43.110]or the SA text box.
[00:35:45.336]There is a lot of research showing that
[00:35:47.770]simply the size of the box actually influences
[00:35:51.110]how big a response the individual actually types in that box
[00:35:55.340]so for most of the time when I'm using text entry
[00:35:59.970]I just by default hit the SA text box
[00:36:03.046]'cause it's gonna encourage them to provide
[00:36:04.620]more rich data as opposed to them seeing
[00:36:06.440]this tiny little single line
[00:36:07.820]and then they feel like they're very limited
[00:36:09.470]in what they can actually type in.
[00:36:11.510]You do have the ability to set up some forms of validation
[00:36:13.870]with text entry.
[00:36:15.130]For example you can set a minimum or maximum
[00:36:17.040]length so maybe you want them to type at least 10 words
[00:36:20.550]or maybe for some forms so they're also,
[00:36:22.670]sorry, there's also form and passwords
[00:36:24.440]that you can set up.
[00:36:25.460]Forms are often used for contact information.
[00:36:28.210]So say you want to control a date of birth,
[00:36:31.440]so when they're entering their date of birth
[00:36:32.860]you can control it to a specific format
[00:36:35.120]to only be numbers and so that's really really helpful
[00:36:37.530]just to make sure that you're getting
[00:36:39.520]pretty good data for those text entry questions
[00:36:42.790]and you don't have to clean it all up.
[00:36:45.760]There are several other question types in Qualtrics
[00:36:48.140]including pick, group, and rank, drill down,
[00:36:52.460]signature, timing, tons of other options.
[00:36:55.150]I'm not gonna cover these in great detail
[00:36:57.530]but if you need more information,
[00:36:58.730]you can go to this link at the bottom here.
[00:37:00.854]I will say again that we have used the timing question
[00:37:04.130]and then just briefly I just wanted to kind of show
[00:37:08.270]what the drill down question is.
[00:37:09.590]So a drill down is basically when you have data
[00:37:12.540]that's collected in a structured format.
[00:37:15.750]For example, I've done some work for
[00:37:18.060]the Nebraska Department of Education
[00:37:19.500]when we're trying to collect data
[00:37:21.931]from particular schools or teachers.
[00:37:23.540]Drill down basically the respondent will select
[00:37:27.760]their district and then only the schools
[00:37:29.820]that appear in that district will pop up
[00:37:31.490]and then only the teachers in that school will pop up
[00:37:34.310]when they select their school
[00:37:35.750]so it's just a much better way to kind of
[00:37:38.020]collect that sort of data.
[00:37:40.320]You probably don't wanna use text entry for that
[00:37:42.680]because it's a lot of cleaning of the data
[00:37:44.480]and so I really like these drill down.
[00:37:46.490]They have been very very helpful for me
[00:37:48.400]in the surveys that I've conducted.
[00:37:51.380]Alright, we covered question format
[00:37:54.260]so now let's get into measurement errors
[00:37:56.450]due to survey length, the order of questions,
[00:37:58.970]as well as the response options.
[00:38:03.420]The length of the question as well as
[00:38:04.970]the length of the survey can have
[00:38:06.680]profound effect on measurement error.
[00:38:09.050]There are several functions in Qualtrics
[00:38:10.563]that can help with survey length.
[00:38:13.270]For example, you can implement skip and display logic
[00:38:15.900]which I explained earlier.
[00:38:18.100]Only displaying questions that are relevant
[00:38:19.780]to the respondent is in general a good practice
[00:38:22.420]and this is again a way that Qualtrics is
[00:38:24.833]probably preferred over paper and pencil measures
[00:38:28.000]just because it's a lot harder to follow
[00:38:30.390]some sort of logic or to get participants
[00:38:32.800]to skip a bunch of questions when it's
[00:38:34.640]a paper and pencil version.
[00:38:35.920]On Qualtrics, it's easily set up.
[00:38:37.820]You don't have to worry about them getting questions
[00:38:40.440]that don't apply to them if you set up
[00:38:41.910]your logic correctly.
[00:38:43.740]Another option is including a progress bar.
[00:38:46.240]While this obviously doesn't help
[00:38:47.480]with the length of the survey,
[00:38:49.260]the choice to include a progress bar is very important
[00:38:51.750]and again this is found in the look and feel
[00:38:53.931]of that general tab.
[00:38:55.120]There's a lot of research that shows
[00:38:56.430]that including a progress bar is preferred
[00:38:58.080]for short surveys but it actually may be counterproductive
[00:39:01.220]for longer surveys so if you have a survey
[00:39:03.900]that's like 100 questions long
[00:39:05.520]and on that first page maybe there's 15 questions
[00:39:08.190]and it took the respondent a while to go through,
[00:39:10.670]once they hit that page submit
[00:39:12.060]and then it's like, oh, you're only 5%
[00:39:14.300]done with the survey, they're gonna be like,
[00:39:16.070]this is terrible and they'll just end the survey
[00:39:18.420]so that is something that you kind of got to keep in mind.
[00:39:21.090]I usually don't use them for any survey
[00:39:22.840]longer than about 20 questions.
[00:39:28.210]The order of questions appear on a survey
[00:39:30.060]can also effect its measurement error
[00:39:31.920]so for example we can have things like context effects
[00:39:34.270]and this is the process in which prior questions
[00:39:36.330]affect responses to later questions
[00:39:38.320]and then we have the question of where to place
[00:39:39.910]demographic information and research on this
[00:39:42.610]is very conflicted so some researchers just prefer
[00:39:45.070]to put, to ask demographic questions
[00:39:47.270]at the beginning of the survey just because
[00:39:49.330]oftentimes when we're trying to analyze data,
[00:39:51.350]this is some of the most important things
[00:39:53.230]in order to analyze the data while on the other hand
[00:39:55.990]researchers like to put it at the end of the survey
[00:39:58.840]because maybe answering those demographic questions
[00:40:01.330]effects their survey responses
[00:40:03.471]when they're actually taking the survey.
[00:40:04.830]That's kind of a researcher's choice
[00:40:08.040]but you can also implement question randomization
[00:40:10.500]and so again this is set up in the block options.
[00:40:14.150]You just go down to question randomization
[00:40:15.960]and again you can control if you want to randomize
[00:40:17.840]the order of all questions or if you wanna set up
[00:40:20.050]an advanced randomization so if you wanna again
[00:40:22.240]keep all your demographics at the beginning
[00:40:23.900]or all at the end you can do so there.
[00:40:28.280]Next we have the response options,
[00:40:30.100]measurement errors due to the response options
[00:40:32.110]so there are several built in options in Qualtrics
[00:40:34.671]to assist with the response options
[00:40:36.830]and so I'll unpack each one of these here.
[00:40:39.720]First we have that none of the above or not applicable
[00:40:42.710]so none of the above or not applicable questions
[00:40:46.010]will automatically trigger an option
[00:40:47.660]excluding that answer choice from data exports and reports.
[00:40:51.350]You can change this setting easily by hitting that
[00:40:53.270]blue arrow next to the response option
[00:40:55.150]and then just going up and toggling
[00:40:56.740]the exclude from analysis.
[00:40:59.240]You can also add question validation
[00:41:01.313]so for example you can say are you sure
[00:41:04.150]this question is not applicable to you?
[00:41:06.240]Or you can allow for text entry so maybe if they
[00:41:09.020]select that none of the above you can ask them
[00:41:10.740]to explain which just provides a little bit of
[00:41:13.793]context on why they might have selected
[00:41:15.494]that not applicable response option.
[00:41:19.650]Next we have mark all that apply
[00:41:21.100]versus yes no questions.
[00:41:23.170]Respondents are instructed to select as many
[00:41:25.280]of the response options as are perceived to apply to them,
[00:41:28.251]that's what a mark all that apply question is.
[00:41:30.290]Research has shown that mark all that apply questions
[00:41:32.650]are actually less than optimal.
[00:41:34.710]The question type is sensitive to privacy
[00:41:36.570]and recency effects and or satisficing
[00:41:38.273]and burden avoidance.
[00:41:40.740]For example, maybe the respondent just
[00:41:42.360]selects the first few that apply to them
[00:41:43.970]and then they don't wanna read through the rest
[00:41:46.000]of the responses so they just move on.
[00:41:48.410]It's often preferred to actually use a yes or no format
[00:41:50.614]for each of the response option
[00:41:53.140]as it actually requires participants
[00:41:54.570]to mark a response for each of the response options.
[00:41:57.270]Hopefully they actually read each of them
[00:41:59.550]and then they have to decide yes or no for each of them.
[00:42:01.440]Again they can still do things like straight lining
[00:42:03.670]but it just is probably preferred to try to use
[00:42:06.760]that yes or no response scale.
[00:42:10.030]Again the yes no format respondents are asked
[00:42:12.670]to evaluate each forced choice response option
[00:42:16.807]individually and provide an answer yes or no
[00:42:17.930]before moving on to the next.
[00:42:19.770]Research has shown that there's a higher average
[00:42:22.290]number of response options selected per respondent
[00:42:24.800]in forced choice format, so again,
[00:42:26.540]it's often preferred to do so.
[00:42:28.720]The screenshots below just show please select
[00:42:31.730]the following comedy TV shows that you enjoy
[00:42:34.420]and it's a select all that apply
[00:42:36.250]and then the screenshot to the right
[00:42:37.610]I converted this to a yes or no response
[00:42:39.500]so it asks then to provide an answer
[00:42:43.410]for each of the response options
[00:42:45.050]so this is again is one case where matrices
[00:42:47.260]are actually probably preferred to not using a matrice.
[00:42:53.691]Next we have labeling and scale options.
[00:42:56.560]Research has shown that respondents are often
[00:42:58.580]unclear about the meaning of response options
[00:43:00.730]and that they often rely on visual cues
[00:43:02.900]in deciding what the options mean.
[00:43:04.480]For example, the numbers attached to the response scale,
[00:43:08.100]maybe sometimes it's negative five to five
[00:43:09.990]as opposed to zero to 10 do affect
[00:43:12.740]the respondents' answers by shaping
[00:43:14.180]their understanding of the scale.
[00:43:16.710]Maybe they see that negative five
[00:43:18.210]and they're scared off by that.
[00:43:19.810]I don't feel negatively, so sometimes those kinds of things
[00:43:22.980]do shape their understanding of the scale.
[00:43:25.320]Additionally, the visual representation
[00:43:27.740]of the response options may effect their relative popularity
[00:43:30.353]for example if it's horizontally or vertically aligned
[00:43:34.100]as well as the order of the Likert scale
[00:43:36.190]and again all of this is customizable in Qualtrics
[00:43:39.150]so you can add both scale and labels
[00:43:41.640]and you can edit the positioning
[00:43:44.066]of the response options.
[00:43:49.492]We're gonna get into the look and feel of the survey.
[00:43:59.206]The look and feel of a survey can affect
[00:44:00.570]its measurement error.
[00:44:02.020]For example things like the choice of the background
[00:44:04.040]and foreground color do affect legibility
[00:44:06.470]and readability.
[00:44:07.670]It's often preferred to use plain colored backgrounds
[00:44:09.900]or extremely subtle background patterns.
[00:44:12.770]The type face and font size that you use
[00:44:14.700]are also really important and it's another design issue
[00:44:17.814]that may affect the readability of the survey.
[00:44:20.560]Selective emphasis, so you can use bold, underlining,
[00:44:23.170]capitalization, color, all sorts of different things
[00:44:26.110]to help with survey design and make it really easy
[00:44:28.330]for your respondents to navigate throughout the survey
[00:44:31.050]and then additionally page layout and alignment.
[00:44:33.030]Headers often serve as branding or orienting function
[00:44:36.920]including directions or reminders
[00:44:38.950]and then links for additional information.
[00:44:41.410]I'll kind of unpack each one of these
[00:44:43.090]and show how Qualtrics, show how the built in function
[00:44:46.850]in Qualtrics can kind of help with this.
[00:44:52.170]Again this is the look and feel tab.
[00:44:55.120]Here we have some different options once you hit that.
[00:44:58.070]Basically along the side, there's layout, general style,
[00:45:01.010]motion, and background, logo and background.
[00:45:04.020]This was actually just overhauled about a month ago,
[00:45:06.620]their entire look and feel was so I'm still kind of
[00:45:08.970]getting used to it.
[00:45:10.513]There are some kind of new features that I'm
[00:45:11.910]a little unsure about but I'll kind of just provide
[00:45:14.770]a little bit of explanation of some of these.
[00:45:17.370]First off we do have one preset UNL theme
[00:45:20.260]so if you want some branding that looks like UNL,
[00:45:22.930]red and black colors, you have the option to do so.
[00:45:26.960]You also have the option to have just a blank design
[00:45:29.170]and you can customize that design to your liking.
[00:45:32.440]You can change the layout so now Qualtrics
[00:45:34.740]has three different options for the layout
[00:45:36.400]including flat, modern, and classic.
[00:45:39.070]I do usually use that flat one.
[00:45:41.380]It just seems to get the best survey responses
[00:45:43.860]and then finally along the top here
[00:45:45.610]you actually can see a preview of what your survey's
[00:45:47.950]gonna look like on both the desktop
[00:45:50.010]as well as the tablet and that happens by just selecting
[00:45:52.910]if you wanna see the tablet you can just select
[00:45:55.371]that one there and it is generally a good idea
[00:45:56.960]to kind of check how your survey looks
[00:45:59.100]on both of the platforms just so you ensure that
[00:46:02.020]it's performing exactly how you want it to.
[00:46:06.720]Next we have the general tab and this is where
[00:46:08.540]we can do things like add next and previous button text,
[00:46:12.454]you could also add that progress bar,
[00:46:13.530]you can control the number of questions per page
[00:46:15.800]and then you can have things like headers and footers.
[00:46:21.230]Next we have the look and feel style tab
[00:46:23.370]and this is where we can change the font style
[00:46:25.950]and color for the entire survey.
[00:46:28.170]We also have the ability to add things like custom CSS
[00:46:31.030]for fancier features and personalization.
[00:46:33.230]However, you do have to have programming knowledge
[00:46:35.890]of CSS yourself which I do not have
[00:46:39.050]but I have had somebody, Seth, help me out with this
[00:46:42.140]and so he created really cool CYFS branded survey for us
[00:46:46.530]and I'll show that on the next page.
[00:46:48.671]You can also do things like add page transitions
[00:46:52.089]and auto focus the question.
[00:46:54.147]Auto focus the question is pretty nice because
[00:46:56.550]it basically highlights the question that they're on
[00:46:58.932]and it just makes it a little easier for them
[00:47:00.294]and then don't get distracted by the rest of the survey.
[00:47:03.290]You could also do things like add logos
[00:47:04.480]and change the background color.
[00:47:06.600]These are just some examples of some of the
[00:47:08.070]surveys that I've created.
[00:47:09.580]The one to the left is the one that Seth helped with
[00:47:11.460]so this is the Nebraska State CYFS.
[00:47:14.010]At the bottom, we have our contact information
[00:47:16.273]and then the one to the right is a survey
[00:47:20.225]I put together for the tap it project.
[00:47:21.560]They wanted their survey to really look like their branding
[00:47:24.870]and so I kind of customized it to include their colors
[00:47:28.060]and the background to kind of fit in with that
[00:47:29.970]so the look and feel is really helpful
[00:47:33.010]and you can kind of customize your surveys to your liking
[00:47:36.390]and oftentimes to the funders liking.
[00:47:39.310]Next we do have the ability to add static content
[00:47:42.150]and so this is for descriptive text and graphics
[00:47:45.270]which can be used to help provide selective emphasis
[00:47:48.160]or assist with things like page layout
[00:47:49.960]or navigation of the survey.
[00:47:52.540]You can edit the text and color
[00:47:54.500]as well as the format and add graphics
[00:47:56.430]using the rich content editor and so again,
[00:47:59.740]the descriptive text and the graphics,
[00:48:01.970]these are actually question types that happen
[00:48:04.190]so once you click change question type,
[00:48:07.580]it's along the top here and then you can use
[00:48:10.130]that rich content editor to really personalize
[00:48:12.220]what that text actually looks like.
[00:48:14.490]Once you're clicked on maybe descriptive text,
[00:48:17.910]a little box will appear on the rich content editor
[00:48:20.720]and then you can go in and you can control
[00:48:22.570]exactly what your text look like
[00:48:23.970]so if you wanna center it, if you wanna make it huge,
[00:48:26.660]you wanna change the color,
[00:48:27.610]you have the ability to do all of that.
[00:48:33.240]The final helpful thing that Qualtrics provides
[00:48:35.600]is the survey review or IQ score.
[00:48:38.800]Again this is something that's brand new in Qualtrics.
[00:48:40.980]I just started using it recently.
[00:48:43.140]I think they implemented it around the same time
[00:48:45.070]as the new look and feel.
[00:48:47.030]It is found in that tools tab or also just below
[00:48:50.070]the survey preview button, that IQ score
[00:48:52.420]and if you just hit that, this fantastic little box
[00:48:56.070]will pop up on survey review.
[00:48:59.340]The survey review can help with things like grammar,
[00:49:01.870]the wording of the questions, navigation of the survey.
[00:49:05.550]It checks to make sure that the logic you implemented
[00:49:08.540]in your survey works correctly
[00:49:10.100]and it also looks at things like the use of matrix tables
[00:49:13.260]as well as if your surveys are optimized for mobile.
[00:49:16.470]For example, this survey I'm not even sure
[00:49:18.440]which survey this expert review is for but they did find
[00:49:21.473]two ways to improve this survey.
[00:49:24.420]They did score as a great survey
[00:49:27.770]so that's pretty cool I guess but then you also,
[00:49:31.430]it kind of provides some recommendations
[00:49:32.980]on how they can improve it and so their recommendations
[00:49:35.740]here are that five of the questions
[00:49:37.430]contain grammatical conjunctions
[00:49:39.320]so I must have used like an and or an or in this survey
[00:49:41.740]and so it suggests to try to cut down on the use of those.
[00:49:45.150]It'll also the other recommendation it's giving
[00:49:49.275]is for the number of text boxes.
[00:49:51.515]There's a lot of research showing that
[00:49:53.760]the more text entry boxes you add in your survey,
[00:49:58.050]the greater non response you have
[00:50:00.160]and so their recommendation is to try to keep
[00:50:01.960]the number of text boxes under three.
[00:50:04.230]Obviously in some of your surveys that's impossible to do
[00:50:06.890]but it's just helpful that Qualtrics is kind of just
[00:50:09.370]helping with this kind of thing and again,
[00:50:12.833]this is a newer feature and so I'm really excited
[00:50:15.835]to keep working and just see what kind of things
[00:50:17.050]it actually points out and then along the side
[00:50:20.070]it does kind of separate the survey errors
[00:50:23.260]into different things so severe, moderate,
[00:50:25.240]minor suggestion and passed and then you can filter it
[00:50:27.930]by issue type so it does separate it into survey errors,
[00:50:30.670]errors in methodology as well as compliance assist.
[00:50:34.001]This function is fantastic.
[00:50:36.540]Finally Qualtrics is a powerful software
[00:50:38.810]for collecting and analyzing data
[00:50:40.760]and its use can help minimize some forms of error in surveys
[00:50:44.530]including non response errors
[00:50:46.010]as well as errors of measurement.
[00:50:47.990]For example, errors due to the respondents
[00:50:50.513]and errors due to the survey instrument.
[00:50:52.370]However as with all research it is important
[00:50:54.650]to understand the goals of the research
[00:50:57.595]and the evaluation project
[00:50:58.913]and it requires us to make the correct design decision
[00:51:00.912]which in itself requires simultaneously
[00:51:03.570]considering quality and cost factors
[00:51:05.750]as well as the choosing the combination of design features
[00:51:08.130]and parameters that help minimize
[00:51:09.610]the total survey error within all the specified constraints.
[00:51:12.830]Again, it's trade offs as opposed to what mode you're using
[00:51:16.370]and the different questions you have on your survey.
[00:51:20.850]I did include some pretty helpful Qualtrics
[00:51:23.480]resources here.
[00:51:25.275]They're just a link to different websites
[00:51:27.390]that provide a lot of help
[00:51:29.915]in creating things in Qualtrics.
[00:51:33.290]In my experience again the support team's been
[00:51:34.960]very helpful so if you have any problems
[00:51:36.840]you can reach out to them and then again
[00:51:39.000]I do have a more general walkthrough of Qualtrics
[00:51:41.090]presentation that I put together and again
[00:51:43.440]it's a lot of the same material
[00:51:44.720]but it's more geared just towards the novice user
[00:51:47.840]and how to use Qualtrics so I can send that to you.
[00:51:50.710]Just send me an email and I can do that.
[00:51:54.340]Finally these are my references.
[00:51:56.200]Before I get into any questions,
[00:51:58.070]I did just include a bunch of other helpful
[00:52:00.700]Qualtrics information that I put together.
[00:52:02.340]Some of that is in the other presentation
[00:52:05.440]but it's just some helpful things,
[00:52:08.891]some tips that I've learned
[00:52:09.724]just being kind of an experienced Qualtrics user.
[00:52:13.153]Yeah.
[00:52:14.240]Alright, any questions?
[00:52:17.290]Awesome, well thank you guys.
[00:52:18.370]Thanks for coming.
[00:52:19.835](applauding)

The screen size you are trying to search captions on is too small!

You can always jump over to MediaHub and check it out there.

Comments

0 Comments

Jared Stevens: “Collecting Your Own Data”

Description

Searchable Transcript

Comments icon comment

Related Channels

Comments