UX toolkit: survey design

Introduction

What is a Survey?

A survey is a research method consisting of a structured set of questions, or questionnaire, asked to a group of people to gather feedback about them and their opinions, behaviors, and experiences. Surveys help teams understand users at scale and identify patterns, needs, and opportunities to improve products and experiences.

You have probably heard the terms questionnaire and survey used interchangeably, but they are not the same. A questionnaire is a set of questions used to collect data. In effect, this means that a form like you might use to order a cake or renew your license is also a questionnaire. What you do with the data is what makes a questionnaire a survey. A survey is a tool that consists of a questionnaire designed for data analysis.

When should you use a survey?

1. When you want to validate qualitative data with the masses

Surveys are a great complement to other qualitative research methods, in which you may have a small sample. A survey will tell you how many people agree or disagree with your current findings and can add more perspective to what you learned.

2. When the data you want to collect is very direct

Surveys are unilateral tools, meaning that there usually is no room for you to interact with the survey taker and clarify anything. Use a survey if you are confident that you can ask the questions that will lead you to the data you want without the benefit of continuous communication.

3. Temperature checks to see how your product or service is performing

General ‘How am I doing?’ surveys can tell you if your users are satisfied, what they like and don’t like, and if they are having any issues with the product. A survey can help you identify big versus small issues and help you prioritize what to work on first. These types of surveys are especially useful if you don’t know where to start with strategy planning.

When many people think about conducting research and connecting with customers, their first instinct is often to launch a survey. But not every situation calls for a survey. Besides making sure a survey is the appropriate method for the information you want to collect, know that they are also a big investment of your time and resources, so choose this method only when it’s necessary and you can fully commit to and when you can fully commit to executing them properly.

For more help, see: Decision Tree: Do I need a survey?

back to top

Before you get started

Set up, planning, & securing resources

Like all user research, there is some planning and logistics involved before you start conducting your own primary research. Digital surveys

Resources to secure

• Account with a survey tool platform
• Mailing list or contacts in spreadsheet form (with permission to contact)
• Compensation or incentives for participants

UX docs to create

• Research Plan
• Questionnaire (survey questions and instructions)
• Participant Recruitment & Research Ops Plan (including MailMerge templates to handle participant correspondence)

Ask the team

If you have a longitudinal research program, it might be worth it to invest in developing a panel as opposed to one-off recruiting. The UX team can help you set up a panel for more regular feedback about your product.

In general, most research budgets will not allow for compensation for every respondent, especially if you are distributing to a large number of people. Talk to the UX team about alternative ways to incentivize surveys like raffles, discounts, donations, vouchers, and other non-monetary items.

Book a consultation meeting with the UX team. 

Decide if the survey will be one-time or recurring. Choose an optimal distribution time, avoiding exam periods or holidays, and aligning with the academic calendar. Allow a 2-3 week response window to maintain relevance. Use university-approved communication channels, such as email, social media, or newsletters, for distribution, making sure to get relevant permissions beforehand.

back to top

How to administrate a survey

Survey setup

1. Secure a survey tool.

Choose a survey platform and set aside time to practice using it. You can also schedule a demo or training session with the UX Team.

2. Draft your survey questions and instructions.

See: Developing questions for surveys Use existing research, anecdotal insights, metrics, service tickets, feedback forms, or stakeholder input to inform your questions.

3. Program the survey into your survey tool.

Add your questions and instructions into the survey tool. Test the survey with someone who has not seen it before, such as an internal team member. Check for clarity, comprehension, and flow.

4. Refine.

Make updates based on the pilot feedback before launching.

Survey launch

1. Activate the survey.

Most survey tools do not make your survey live until you have confirmed that you are done editing and ready to publish it.

Ask the UX Team…

If you do find that you want to change something after making your survey live, most survey tools will allow this, however you should talk to the UX team before making the decision to edit a live survey after receiving submissions to determine how it might affect your data and if the change is worth any negative consequences: book a consultation meeting with the UX team.

2. Distribute the survey

After publishing your survey, depending on your survey tool there are a few ways that you can distribute.

1. Copy a shareable distribution link that your survey tool provides you in an email that you mass send to invited. Use Mail Merge to send out mass emails to your contacts.
2. Your survey tool may also have a feature that allows you to import or manually add your contacts and mass send your survey as well as other distribution monitoring tools (open rate, bounce-backs, etc.)

3. Monitor incoming responses.

Preview incoming submissions regularly to ensure participants understand what they are supposed to do and aren’t encountering any major issues.

4. Continue Research Ops support.

Manage participant reminders, confirmations, and all other communication and logistics throughout the study according to your Research Ops Plan.

back to top

Developing questions for a survey

Good survey questions start with clearly defined research goals. If you ask the wrong questions, you will not obtain the meaningful insights needed to support informed decision-making. A survey’s success also depends on achieving a strong response rate, that means reaching enough people and getting them to complete it. Since most respondents spend about ten minutes* on a survey, it is important to make the experience as simple and easy as possible, while still encouraging thoughtful, high-quality answers.

for further information, see our guide: Improving survey experience.

The easier your survey is to complete, the more likely respondents are to provide honest, meaningful feedback that you can confidently use to guide decisions.

How long should a survey be? See Research-backed best practices.

Ask the UX Team…

Writing survey questions is both an art and a science. When you’re first starting out, don’t worry about making every question perfect right away. Start by jotting down all the questions that come to mind, then work with the UX team to edit, wordsmith, and narrow the list down to the most effective questions: book a consultation meeting with the UX team.

How to write good survey questions

The two tables below outline key principles for survey questions. While the lists are not exhaustive, they cover some of the most common survey design mistakes and offer practical guidance to avoid making them, helping you to create surveys that are more likely to generate meaningful insights.

Ensuring respondents have an opportunity to provide accurate responses

Sometimes respondents are unable to provide the information you need because the question’s design does not allow them to fully answer. The issue lies with the survey itself, not the respondents. As a result, even well-intentioned respondents may provide incomplete, inconsistent, or less reliable data. Thoughtful question design can help reduce these barriers and improves data quality.

Six rules for eliciting accurate responses in  user surveys

no.	Rule	Why this rule exists	Examples
1	Ask one question at a time.	Multi-part questions can produce inaccurate data, as each part may have a conflicting response.	“How would you rate the service?” How would you rate the food and the service?”
2	Use mutually exclusive answer options.	Overlapping options create confusion and compromise data.	“What is your age?” “a) 0–17 b) 18–24, c) 25–34” “a) 0–18 b) 18–25, b) 25–34”
3	Provide balanced and labelled answer scales.	Unbalanced and unlabelled scales introduce ambiguity, which results in respondents interpreting answer options in different ways.	Rate 1 - 5”, 1= Very dissatisfied, 5 = Very satisfied” Excellent → Poor” “Rate 1 - 5”
4	Do not force answer choices that may not fit.	Outlier data points are not accounted for when respondents do not have their experience represented in available options.	“How do you commute to work?” Include “None,” “Other,” or “Not applicable”. “a) Bike, b) Car c) Public transport
5	Choose the question format that best fits the data needs.	The wrong question type can limit insights or create difficult-to-interpret responses.	“On average, how often have you come into the office over the past three months?” Daily, Weekly, Once a month, Never "Do you come into the office regularly?” Yes/No/Not sure
6	Only ask relevant questions to relevant people.	Requiring respondents to answer questions that do not apply to them encourages unreliable responses.	"Have you contacted student support in the past 6 months?” If yes → “What issue were you trying to resolve?” “Do you find student support helpful?”

Ensuring questions are clear and unbiased

Sometimes respondents interpret a question differently than you intended, which can introduce inconsistent, inaccurate, or unusable data into your results. Because these misunderstandings are not always obvious, compromised data can easily make its way into your analysis and jeopardize results. Be mindful of unclear and biased wording that can affect the overall quality and reliability of your data.

Six rules for ensuring clear and unbiased questions

no.	Rule	Why this rule exists	Examples
1	Provide a frame of reference.	Respondents may assume a different context without clear boundaries.	“Over the past week, how healthy or unhealthy would you rate your eating habits while at home?” "How healthy are your eating habits?”
2	Use plain language.	Avoid technical terminology, internal language, abbreviations, and acronyms that may confuse respondents.	“Did you contact customer support for help?” "Did you create a JIRA ticket for a service escalation with the IT team?”
3	Do not make assumptions about background, experience, or knowledge.	Respondents may have different levels of familiarity or experience, and may unintentionally provide inaccurate or misleading responses because they are not knowledgeable to answer.	“How confident do you feel identifying when your car may need repairs?” “How do you go about diagnosing issues with your car?”
4	Ask questions that uncover insights, not lead to more unknowns.	Some questions need more context in order to garner meaningful insights that you can act on.	“How do you commute to work?” Which feature in the app do you use the least? Why? Which feature in the app do you use the least?
5	Don’t seek to do polling or quick validation	Asking for validation in itself can present bias. Respondents may agree with an idea just because it was presented to them. Avoid one-off questions that encourage respondents to hastily make decisions or react to new ideas without fully considering them.	“How would longer library hours affect your productivity?” “Would you like the library to be open 24 hours a day?”
6	Use visuals to provide clarity.	Respondents may not recognize products or may refer to them differently than your organization does. Visuals like screenshots, icons, or photos can help ensure respondents understand exactly what the question is referring to.	“Have you ever received mobile notifications about your account from the MyOxford app?” “Have you ever received mobile notifications from Oxford?”

There are lots of types of questions you can ask, for now we will break them down into three types based on the type of data that they collect.

Demographic and background

Questions about social, economic, and background information

Who do you want to talk to? Collecting demographic and background information, such as age, gender, ethnicity, income, education, marital status, or occupation, can be valuable for segmenting your respondents later in analysis. Perhaps a certain challenge only affects students studying a particular major or people who have newer building facilities might be more satisfied with their environment than others.

Demographic questions can include anything that relates to how your participant might define themselves and their identity. It could include questions about their access to resources, daily routines, and the communities or environments they are part of.

Best Practice

Keep in mind this information can be sensitive for some of your respondents. As a general rule, only ask this information if it is relevant and needed. It is also good practice to offer an explanation as to why you are asking for this information. In your analysis be sure to be mindful of your own biases that examining data in demographics can expose. For more help on this, please consult the UXCoE.

Also be mindful of evolving language and terminology as to be respectful of your respondents. Provide answer options like ‘Self-describe’ or ‘Prefer not to answer’ and reiterate that these questions are completely optional. Compare how the choice in presentation of these examples might affect one’s reaction and response.

Version 1: What is your household income? ___

Version 2: What is your approximate annual household income?

• Less than $25,000
• $25,000–$49,999
• $50,000–$74,999
• $75,000–$99,999
• $100,000 or more
• Prefer not to answer

Quantitative

Questions that measure behaviour, activity, and sentiment

Quantitative data typically produces objective insights, making it easier to spot trends, patterns, and relationships within the data. These questions focus on information that can be counted and measured. These are things your audience should be able to easily recount about their behaviour, habits, and impressions. They provide insight into how users interact with your product or service and what value they place on it. In addition, they give you some context into the respondent’s history as it relates to a particular subject.

Best Practice

When asking for quantitative data, consider how you present scales to ensure participants feel comfortable giving their responses. For example, asking how many hours a day they spend on social media could feel embarrassing or shameful to some, potentially leading them to misreport.

Doing some research into habits could help you provide scaling that reflects common behaviours, and in turn make respondents more likely to provide honest answers without fear of judgment.

Compare how the choice in scaling in these examples might affect how someone might answer a question about their social media usage.

Version 1: How many hours a day on average do you spend on your phone using social media?

• None
• Less than 2 hours
• 2 - 3 hours
• 4 - 6 hours
• 7 - 8 hours
• Over 8 hours
• Not sure / Don’t know

Version 2: How many hours a day on average do you spend on your phone using social media?

• None
• Less than 2 hours
• 2 - 6 hours
• • 12 hours
• 13 - 18 hours
• Over 18 hours
• Not sure / Don’t know

Qualitative

Questions that share stories, thoughts, and ideas or describe feelings, opinions, and perceptions.

Qualitative questions focus more on people’s stories, feelings, opinions, motivations, and perceptions. These questions might also delve into looking at the contexts of use for products and services.

Best Practice

The open-ended question is one of the most powerful qualitative question forms. Open-ended questions allow respondents to express themselves in their own words, offering deeper insights into participants' thoughts, feelings, and experiences.

However, your survey will perform better in some cases if you are able to provide answers for your respondent to choose from. It takes time to develop these answers that can cover an exhaustive list of what your user may do, but it is essential to your analysis. Offering an ‘Other’ field with the opportunity to add free-text can sometimes be more effective. Reserve free text analysis for information that warrants a question that requires details and explanations of nuances.

Compare how the choice in question format in these examples might affect the quality of data you collect about university experience.

Version 1: What has been the most rewarding part of your experience at the university so far? (Please answer in a few sentences.)

Version 2: What has been the most rewarding part of your experience at the university so far? (Select all that apply.)

• Learning about something I’m passionate about
• Pursuing creative projects or research
• Building strong relationships with professors and classmates
• Growing as a person and discovering more about myself
• Establishing professional connections and mentorships for my career
• Being in an inspiring and challenging academic environment
• Other (please specify) ___

Ask the UX Team…

Open-ended question data is often analysed thematically or through content analysis to identify common themes, patterns, and narratives. There are methods that can be used to translate open-ended question data into measurable quantifiable data. This could be useful if one of the goals of your research is to demonstrate impact. Please reach out to the UX team if you require this service: book a consultation meeting with the UX team.

back to top

Next steps: sharing & reporting

Piloting your survey

Test the survey with a small, diverse group of people and conduct follow-up interviews with at least two participants. Ask them to share their thought process as they answer the questions and note any deviations from your intended outcomes. This will help you identify areas that need improvement.

Survey Audit Checklist

• How long does it take to complete the survey?
• Does the question flow logically with the overall survey?
• Are there any ambiguous or confusing questions?
• Does the survey seem too long or tedious?
• Is the language simple and easy to understand?
• Are there any technical or formatting issues across devices and browsers?
• Are questions respectful, especially if sensitive topics are involved?
• Add in the survey goals you outlined earlier here too

How UXCoE can help with your survey

The UXCoE is available to assist with various aspects of your survey.

Here are a few services you are welcome to reach out to the UXCoE for help with your survey:

• Scheduling a meeting to refine your questions and improve response rates
• Reviewing timelines to determine optimal survey timing
• Conducting problem definition exercises
• Discussing impact measurement factors
• Reducing 'I don’t know' and 'Neutral' responses
• Analyzing survey data and crafting compelling data stories
• Coding and analyzing qualitative data
• Survey piloting
• Exploring alternative sampling methods
• Developing participant panels
• Conducting advanced surveys (multivariate testing, conjoint analysis, open-text analysis)

back to top

References and Resources

Additional reading

• Storytelling with Data: A Data Visualization Guide for Business Professionals (2015) by Cole Nussbaumer Knaflic
• Design, Evaluation, and Analysis of Questionnaires for Survey Research (2007) by Willem E. Saris, Irmtraud N. Gallhofer
• Should I Run A Survey? (2024) by Maddie Brown, Nielsen Norman Group
• Writing Survey Questions Pew Research Center

Survey Tools

• JISC Online survey tool (university approved)
• Survey sample size calculator
• Sample survey question bank
• Web Accessibility Standards

Glossary

There many common terms used in UX that may need explanation, to familiarise yourself with some of the terminology, please take a look at our glossary of UX terms.