How to create a survey

Intro: What is a survey and why do we use them?

Surveys are an invaluable tool for gathering insights, but designing one that yields useful data can be tricky. Surveys help understand user behavior and experiences, but crafting effective questions and avoiding bias takes careful planning.

This article will walk you through the essential steps to design and distribute a survey that generates valuable insights. While creating a survey may seem easy, thoughtful preparation is key to getting meaningful responses. With the right approach, your survey will provide actionable data to drive informed decisions.

Step 0: Pre-survey launch considerations & logistics

When planning for your survey, it’s important to think about what’s required before the launch. Surveys can have significant overhead, especially if they don't achieve the desired response rate. The average response rate typically ranges from 10% to 60%. There are a few considerations when it comes to research operations you’ll want to consider beforehand to ensure you have the time, resources, and budget to launch a survey.

Here are some important considerations to keep in mind before launching a survey:

Legal & Privacy

Ensure ethical handling of personal data. Familiarize yourself with General Data Protection Regulation (GDPR) for important matters related to obtaining consent and encrypting data for security. Have a system in place for ethical and safe handling of personal data and sensitive information.

Survey Tools

Select the right survey tool that offers the necessary features to complete your research, such as skip logic, branching, and customization options. More advanced survey tools can be pricey, so it's also important to consider costs. Finding a balance between a tool that fits your budget and your survey needs might require some investigation.

Survey Distribution

Decide if the survey will be one-time or recurring. Choose an optimal distribution time, avoiding exam periods or holidays, and aligning with the academic calendar. Allow a 2-3 week response window to maintain relevance. Use university-approved communication channels, such as email, social media, or newsletters, for distribution, making sure to get relevant permissions beforehand.

Sampling Strategy

Ensure your sample accurately reflects the population you are serving, considering factors like study year, major, and tenure. Strive for diversity in demographics, such as gender, ethnicity, and academic standing. Also, consider language needs for individuals whose first language may not be English.

Participant Management

Make a plan for participant communication for tasks such as sending recruitment emails, reminders, and thank-you notes. It's important to track participants' progress and interactions, using tools like spreadsheets and checklists to ensure no one gets left behind.

Incentives

Managing incentives for a study involves clearly communicating the rewards participants will receive and ensuring they are delivered promptly. It's important to align incentives with your study’s budget and follow any rules or guidelines regarding eligibility and distribution.

Step 1: Do I need a survey?

Not every situation requires a survey. Before committing, it’s essential to determine if a survey is the right tool for your needs. This decision depends on the context, the goals, and the timing. Are you ready to invest the time and effort necessary for a survey?

Here’s a task flow to help you decide:

Reasons you might not need a survey:

• A different research method (like interviews) would be more effective for understanding the ‘why’.
• You haven’t explored secondary research—check with the UXCoE for previous studies.
• You need a conversation, not a survey (i.e., you’re missing context or understanding the deeper ‘why’).
• You need a prototype to test ideas or behaviors instead of relying on what people say they would do.
• It’s not the right time for a survey (i.e., during periods of high stress or distraction).
• You’re asking for predictions, guesses, or speculative behavior.

Step 2: Determine what you want to learn

If everything from the last step points to clear skies, congratulations! You’re ready to move forward with creating your survey!

Like any research project, your survey should start with clear, well-defined goals to guide your process. Once you have your primary goal in mind, you can then use it as a north star to guide key questions to ask in your survey. Start by reflecting on what triggered the need for research— was there an event or decision that led to this point? Where do things currently stand? Where do you want them to be?

For further guidance, check out this article about defining research problems.

Before you begin designing your survey:

1. List everything you want to learn, don’t know, and have questions about. It can be anything. Don’t worry about whether it’s possible or not to get the answers yet.
2. Determine what can be learned effectively in a survey and what should be addressed later with some other research method. Refer to the previous step for more ideas about what is and is not appropriate for a survey and consult with the UXCoE for more personalized guidance.
3. Identify measurable elements that can demonstrate impact and change. If your work is to ultimately lead to improvement, you’ll need a way to articulate in such a way that can be used to make comparisons and draw conclusions.

Step 3: Developing questions

Now that you have defined some goals, you can start brainstorming questions. When you’re first starting out, don’t worry about getting the questions worded exactly right. Jot down any and all questions you can think of and we’ll talk about editing, wordsmithing, and paring them down later.

There are lots of types of questions you can ask, for now we will break them down into three types to keep it simple.

Demographic

Questions about social, economic, and background information

Who do you want to talk to? Collecting demographic and background information, such as age, gender, ethnicity, income, education, marital status, or occupation, can be valuable for segmenting your respondents later in analysis. Perhaps a certain challenge only affects students studying a particular major or people who have newer building facilities might be more satisfied with their environment than others.

Demographic questions can include anything that relates to how your participant might define themselves and their identity. It could include questions about what their access to resources, daily routines, and the communities or environments they are part of.

However, keep in mind this information can be sensitive for some of your respondents. As a general rule, only ask this information if it is relevant and needed. It is also good practice to offer an explanation as to why you are asking for this information. In your analysis be sure to be mindful of your own biases that examining data in demographics can expose. For more help on this, please consult the UXCoE.

Be mindful of evolving language and terminology as to be respectful of your respondents. Provide answer options like ‘Self-describe’ or ‘Prefer not to answer’ and reiterate that these questions are completely optional.

For ideas on how to phrase demographic questions, please review this Question Bank resource.

Quantitative

Questions that measure behavior, activity, and sentiment

Quantitative questions focus on information that can be counted and measured. These are things your audience should be able to easily recount about their behavior, habits, and impressions. They provide insight into how users interact with your product or service and what value they place on it. In addition they give you some context into the respondent’s history as it relates to a particular subject.

Quantitative data typically produces objective insights, making it easier to spot trends, patterns, and relationships within the data.

When collecting quantitative data, think about how you will present scaling to make participants feel comfortable with sharing their responses. For example, asking how many hours you spend on your phone using social media could be embarrassing or shameful for some respondents, leading them to misreport. Doing some research into typical habits could help you provide scaling that reflect common behaviors, even if some responses seem extreme. This way, respondents are more likely to provide honest answers without fear of judgement.

When asking for quantitative data, consider how you present scales to ensure participants feel comfortable giving their responses. For example, asking how many hours a day they spend on social media could feel embarrassing or shameful to some, potentially leading to inaccurate reporting. Researching common habits can help you create scales that reflect typical behavior, even if some responses may seem extreme. This way, respondents can answer more honestly without feeling judged.

Compare how the choice in scaling in these examples might change one’s opinion about their social media usage.

Version 1: How many hours a day on average do you spend on your phone using social media?

1. None
2. Less than 2 hours
3. 2 - 3 hours
4. 4 - 6 hours
5. 7 - 8 hours
6. Over 8 hours
7. Not sure/ Don’t know

Version 2: How many hours a day on average do you spend on your phone using social media?

1. None
2. Less than 2 hours
3. 2 - 6 hours
4. 7 - 12 hours
5. 13 - 18 hours
6. Over 18 hours
7. Not sure/ Don’t know

Qualitative

Questions that share a stories, thoughts, and ideas or describe feelings, opinions, and perceptions.

Qualitative questions focus more on people’s stories, feelings, opinions, motivations, and perceptions. These questions might also delve into looking at the contexts of use for products and services.

The open-ended question is one of the most powerful qualitative question forms. Open-ended questions allow respondents to express themselves in their own words, offering deeper insights into participants' thoughts, feelings, and experiences.

However, your survey will perform better in some cases if you are able to provide answers for your respondent to choose from. It takes time to develop these answers that can cover an exhaustive list of what your user may do, but it is essential to your analysis. Offering an ‘Other’ field with the opportunity to add free-text can sometimes be more effective. Reserve free text analysis for information that warrants a question that requires details and explanations of nuances.

Version 1: What has been the most rewarding part of your experience at the university so far? (Please answer in a few sentences.)

Version 2: What has been the most rewarding part of your experience at the university so far? (Select all that apply.)

• Learning about something I’m passionate about
• Pursuing creative projects or research
• Building strong relationships with professors and classmates
• Growing as a person and discovering more about myself
• Establishing professional connections and mentorships for my career
• Being in an inspiring and challenging academic environment
• Other (please specify) _______________________

Open-ended question data is often analyzed thematically or through content analysis to identify common themes, patterns, and narratives. There are methods that can be used to translate open-ended question data into measurable quantifiable data. This could be useful if one of the goals of your research is to demonstrate impact. Please reach out to UXCoE if you require this service.

Best practices

Once you've created your questions, it's important to refine them so they're clear and easy for your respondents to understand and answer. While the following list is not exhaustive, it highlights the most important factors and common adjustments that can help make your survey professional and polished. To ensure you collect reliable data, your questions must be free of ambiguity and easily interpreted in the same way by all respondents.

Ask one question at a time

Sometimes two topics might be related or intertwined, but it is important to give each the respect of its own individual question. Combining them can lead to unclear responses, as your respondent might have different experiences with each aspect.

For example, instead of asking:

Q: How would you rate the food and service?

Ask two questions:

Q: How would you rate the taste of the meal?

Q: How would you rate the service?

Perhaps the service was great, but the food was not. Make it easy for your respondent to answer by asking two separate questions.

Provide a frame of reference

Ensure all respondents are considering the same time period and context when answering your questions.

For example, if you ask:

Q: How often do you come into the office?

Some respondents might think about their overall frequency, while others might focus on just this month or week. External factors, like taking time off or increased visits for training, could skew responses.

To avoid confusion, provide a clear reference:

Q: On average, how often have you come into the office over the past three months? (Please select the option that best matches your attendance, even if it’s not exact.)

• Everyday
• 2-3 times a week
• Once a week
• 2-3 times a month
• Once a month
• Never

Make sure you’re getting answers

You have a limited amount of questions you can ask in your survey. If your data leads to more questions than answers, reconsider your wording or add a follow-up question when necessary.

For example, if you ask:

Q: What is your favorite or most used feature of the app?

• Getting notifications about events
• Updating my preferences
• Customizing products
• Checking the status of my inquiries
• Chatting instantly with support
• I don’t use the app

If a respondent selects the last option but you don’t ask follow-up questions about non-users, you miss the opportunity to understand why they’re not using the app. A better approach would be to ask earlier whether they use the app and include a screenshot or icon to help jog their memory.

If a respondent selects the last option but you don’t ask follow-up questions about non-users, you miss the opportunity to understand why they’re not using the app. A better approach would be to ask earlier whether they use the app and include a screenshot or icon to help jog their memory.

Q: Do you use our app?

(Then, use survey logic to follow up only with non-users.)

Why don’t you use the app? (Select all that apply.)

• I didn’t know it existed
• I prefer using the website
• I don’t use the service enough to warrant downloading the app
• It’s difficult to perform tasks using the app
• It doesn’t work well (e.g., slow, crashes)
• It drains my battery
• I have privacy concerns
• Not sure / Don’t know
• Other (please specify) _____________________________

Offering these options allows you to identify whether issues are technical or related to how the app’s value is perceived.

Avoid asking for outright validation

Sometimes you are looking for a simple “yay” or “nay.” This is traditionally what a poll is for, not a survey. Polling is a separate process from surveying and requires careful timing and context. If you're unsure whether your question is leaning toward validation, consult the UXCoE.

Asking for validation in itself can present bias. Respondents may agree with an idea just because you’ve presented it, even if they haven’t considered it fully. This can distort your data by encouraging answers that aren't fully thought through.

For example, asking:

Q: Would you like the library to be open 24 hours a day?

Is not ideal. A survey should explore respondents' values, how a change might impact them, and how important that change is.

Better questions might include:

Q: How often do you use the library during off-peak hours (4am - 8am, 10pm - 2am)?

Q: Have you ever gone elsewhere to study because the library was closed?

Alternatively, you could explore the impact on well-being, like:

Q: How do you usually decide when to take a break from work or studying?

• When I feel physically tired
• When the place I’m studying at closes When my study buddy leaves When I reach my set time limit Other (please specify) ___________

This approach avoids leading questions and allows respondents to share their experiences and values more accurately.

Step 4: Choosing conversational style and format style

The design and formatting of your survey can influence how easily respondents complete it. Elements like pagination, color, spacing, and typography can make your survey feel easier or more difficult to complete. There are lots of survey tools that will give you plenty of freedom and flexibility to design attractive, user-friendly surveys. Check out the Resources section for some suggestions.

Survey fatigue occurs when respondents lose interest or time. To avoid this, keep your survey concise and focused, encouraging thoughtful responses. Since respondents spend about 10 minutes on a survey, asking fewer questions can lead to more thoughtful answers. Focus on essential data while keeping the survey short.

Here are some design tips to make your survey more engaging. Using the concept of a survey as a conversation, we'll explain why these elements are important for creating a positive respondent experience.

What’s your tone?

If the opportunity presents itself, try to make the survey’s language as natural and conversational as possible. It can humanize the survey and make it feel like a personal conversation, rather than a clinical form they’re filling out.

Start with a friendly, upbeat welcome message. Introduce the survey as if it were a person, and express gratitude for the respondent's participation:

Hello [NAME], thank you for volunteering to take our survey. This survey is to help us learn how we can make the experience of students at the library even more aligned with how students work. We will be using your feedback to make decisions about [X,Y,Z]. Please do your best to answer questions. It should only take about 10 minutes to complete. As a thank you for your time, we’d also like to award you [Incentive].

Encourage respondents as they move through the survey. Celebrate progress and motivate them to continue, especially when transitioning between sections:

Thank you for sharing your study habits with us! Now, let’s talk about what you need in your study environment in the next few questions.

This friendly, engaging tone keeps respondents connected and increases the likelihood they’ll complete the survey.

Provide clear directions

Even if a question seems straightforward, it’s better to offer clear instructions than to leave them out. Make sure your directions are easy to understand so respondents can complete the survey independently, without needing your assistance. Remember, you won’t be there to guide them in person—think of it as sending a message in a bottle.

Let the flow of the survey match that of a conversation

Think about question ordering and timing. The questions should progress in a natural way that supports their energy and investment.

Begin with easy questions. Introduce sensitive questions and questions that require more thought like open-ended questions later.

Q: What was your GPA last semester?

This is a bold and jarring way to open a survey! Consider starting with lighter questions to help the participant get comfortable like:

Q: How do you usually prepare for exams?

If you are asking a series of similar questions, try grouping them together.

Q: How often do you visit the library each week?

Q: What time of day do you typically visit the library?

Q: How long do you usually stay at the library?

Building upon answers of earlier questions and staying in the same mindset can reduce cognitive load and make it easier for respondents to recall information and give you better quality data.

Offer variety in question types

Variety in question types not only helps with data collection, but also enhances the survey's visual appeal, keeping respondents engaged and aware of their progress. You have many options to choose from in your survey question toolkit, but you should always choose the question format that will deliver the most meaningful information. Consult with the UXCoE to determine the best times to introduce different types of questions.

Here’s a list of the most common survey question types. Click on each option to view an example:

• Multiple choice
• Single select
• Open-ended text
• Ranking order
• Rating questions
• Likert scale
• Matrix
• Graphic prompts
• A/B testing
• Uploaded media

Step 5: Audit your survey questions

Creating a strong survey rarely happens on the first try. Since a survey is typically seen by a large audience, it’s crucial to gather a team to evaluate its effectiveness. Relying on your own perspective alone can lead to bias, so collaborating with others increases your chances of creating a more inclusive, well-rounded survey.

Workshop

One approach is to print out each survey question on large sheets of paper or use a visual board, like Miro, to work through the questions and flow with your team. Group similar questions, evaluate the flow, and identify redundancies where questions can be removed. This method also helps with question prioritization and ordering.

Survey Checklist

• How long does it take to complete the survey?
• Does the question flow logically with the overall survey?
• Are there any ambiguous or confusing questions?
• Does the survey seem too long or tedious?
• Is the language simple and easy to understand?
• Are there any technical or formatting issues across devices and browsers?
• Are questions respectful, especially if sensitive topics are involved?
• Add in the survey goals you outlined earlier here too.

Pilot Testing

Test the survey with a small, diverse group (10-20 people) and conduct follow-up interviews with 3-5 participants. Ask them to share their thought process as they answer the questions and note any deviations from your intended outcomes. This will help you identify areas that need improvement.

Test accessibility

Ensure the survey is accessible to all, including individuals with disabilities. Follow Web Content Accessibility Guidelines (WCAG), such as adding alt texts for images, describing hyperlinks, providing closed captions for videos, and ensuring good colour contrast between text and background.

Ultimately, you won’t get valuable insights if you ask the wrong questions. Understanding your goals ensures that your survey delivers the data you need to make informed decisions.

How UXCoE can help with your survey

The UXCoE is available to assist with various aspects of your survey.

Here are a few services you are welcome to reach out to the UXCoE for help with your survey:

• Scheduling a meeting to refine your questions and improve response rates
• Reviewing timelines to determine optimal survey timing
• Conducting problem definition exercises
• Discussing impact measurement factors
• Reducing 'I don’t know' and 'Neutral' responses
• Analyzing survey data and crafting compelling data stories
• Coding and analyzing qualitative data
• Survey piloting
• Exploring alternative sampling methods
• Developing participant panels
• Conducting advanced surveys (multivariate testing, conjoint analysis, open-text analysis)