In-situ logging is the most straightforward method to collect data from diaries. A concept testing interview focuses on the concept and covers the high-level areas of: You can conduct these types of interviews remotely, however it might be difficult if you have a tangible artifact or want someone to build on your concept using craft material. The two new members add 40+ years of combined of technology and startup experience to the company. We spent a couple of weeks on site collecting data from visitors to the zoos manatee exhibit. But in reality, people can become frustrated or make guesses that arent helpful when they are simply presented with the question what do you think of when I say financial command center? Instead, showing them a simple sketch of some charts representing the concept of personal financial data being monitored can generate a robust conversation where they provide deep insight into the direction and critical functionality of a financial command center.. Canvas Chrome, Google Jamboard, Miro, and Mural, are four examples of many online collaboration spaces that exist to serve the needs of remote co-creation. If you cant afford that, check the following tips. Weekly tips on front-end & UX.Trusted by 176.000 folks. A moderator leads a group of 36 participants through a discussion about a set of topics, giving verbal and written feedback through discussion and exercises. You should work with a researcher who has experience creating reliable and valid survey questions that do not lead to biased findings. We felt this made sense and aligned with the characteristics of the stakeholders wed spoken to: many of them had invested budgets or sold the existing tools, and they would need to start finding solutions on how to communicate the transition to the new consolidated tool to their teams or clients as soon as possible. The tangible representation of the concept allowed visitors to interact with it in a way similar to a touch screen. A concept should convey the key essence of a product. Although I changed my thinking and now believe artifacts are key to concept testing, I do think its important to keep it conceptual the point is to present ideas and let users expand on them, not grab users tightly by the hand and walk them to the conclusion that your idea is great and usable. In fact, UX research is the starting point for any projectit comes first in the UX design process because, without a proper research, product design can only be based on a designers own experiences and assumptions, which isnt objective. I suggest thinking of the following list as a good starting point, not as a complete guide for user testing. Our client provided us with a list of required stakeholders to participate in the interviews and testing. Between these two extremes lie the two most popular methods we use: usability studies andfield studies. More about You can answer each of the questions listed in the interview section of this article, and additionally, A/B testing will allow you to determine: A/B testing requires greater resources in terms of creating an artifact to test. The article was last updated and revised on July 17, 2022.). There are several types of concept testing: With comparison testing, survey respondents review two or more concepts. You can mix in key stakeholders who might also be users or work closely with users (for example, supervisors at an HR service provider call center might be good to include as part of the participants to test the concept of your new idea for a virtual assistant tool call center reps would use while servicing calls, along with using actual call center reps who might benefit from using the tool daily). Thats why getting their feedback during development is crucial to an effective launch. Interviews can be a good supplement to surveys and questionnaires; conducting an interview beforehand helps researchers refine questions for the survey while conducting an interview afterwards allows them to gather more detailed analysis. Focus groups: Groups of 312 participants are led through a discussion about a set of topics, giving verbal and written feedback through discussion and exercises. Co-design allows users to continue building out an idea you have. However, this structure also makes the survey lengthy, which can affect the completion rate and lead to nonresponse bias. However, these results come without context, as you wont have any data on why a respondent chose a particular concept over another. Having such numbers helps prioritize resources, for example to focus on issues with the biggest impact. This method helps create or refine the information architecture of a system (how to label your menus, how to group your content, etc.) This includes: You can invite these team members to join planning sessions or leave it as simple as inviting them to view a testing session so they see how users respond to the idea. To better understand when to use which method, it is helpful to view them along a3-dimensional frameworkwith the following axes: The following chart illustrates where 20 popular methods appear along these dimensions: This distinction can be summed up bycontrasting "what people say" versus "what people do"(very often the two are quite different). You can use concept testing to evaluate customers responses to product features, price points, advertisements and other aspects of the product youre planning to release. You can create or refine a user-focused backlog of features for the product with concept testing findings. Tree testing: A quantitative method of testing an information architecture to determine how easy it is to find items in the hierarchy. You can proactively account for some of these, while others might need additional research to complement your findings at a later point in the design process. Nextdoor partnered with Lucid (a Cint Group company) to measure the effectiveness of their customers advertising on the platform. We often use interviews combined with artifacts in UX research. It can be done one-on-one or with larger numbers of participants, and either in person or online. The insights in quantitative methods are typically derived from mathematical analysis, since the instrument of data collection (e.g., survey tool or analytics tool) captures such large amounts of data that are automatically coded numerically. Once you know what youre trying to learn and why you can start thinking about how to learn it. Concept testing, like any research method, is not without its potential downsides or shortcomings. Ascriptedstudy of product usage is done in order to focus the insights on specific product areas, such as a newly redesigned flow. Find new directions and opportunities, Research goal: This method is very similar to a field study and was developed to study complex systems and in-depth processes. Kara Pernice: You must have javascript and cookies enabled in order to display videos. Nextdoor and Lucid partner to deliver meaningful brand lift insights, Cint is the political research partner youve been looking for, Cint elect Tina Daniels and Liselotte Engstam to its board of directors. On the other end of this dimension, methods that focus mostly on behavior seek to understand "what people do" with the product or service in question.For exampleA/B testingpresents changes to a site's design to random samples of site visitors but attempts to hold all else constant, in order to see the effect of different site-design choices on behavior, whileeyetrackingseeks to understand how users visually interact with a design or visual stimulus. Researchers can determine whether the chosen concept from the comparison test aligns with the inferences gathered from the sequential monadic test. Asynchronous unmoderated A/B concept testing can allow you to include a larger number of participants without requiring you to dedicate time to conducting each session. Once a week. They ask for comments, feedback, suggestions, and other kinds of responses (e.g. With a commitment to quality content for the design community. Rather, studies that are qualitative in nature generate data about behaviors or attitudes based on observing or hearing themdirectly, whereas inquantitative studies, the data about the behavior or attitudes in question are gatheredindirectly, through a measurement or an instrument such as a survey or ananalytics tool. In the end, the success of your work will be determined by how much of an impact it has on improving the user experience of the website or product in question. Concept testing is giving users a rough approximation of a product or service in order to understand if they would want or need such product or service. User research is a process of understanding user needs, wants, and behaviors using different observation and feedback collection methods. Trusted by 190.000 friendly folks. You can also test ideas that look beyond what an MVP will bring to life. More typically, participants are recruited from an email message or reached through some other channel such as social media. Concept testing is the process of finding out how customers feel about your product idea before you launch it. As the former Vice President of Design, Research and Enterprise Services at Capital One 2015-2018, he headed up user research and design teams focused on identity, security and API Development. In usability testing, you recruit some test participants and give them a set of scenarios that lead to usage of specific aspects of a product (or prototype). I had the pleasure of serving on a design team creating a digital and physical experience conceptualized as an opportunity to enhance visitor engagement with a manatee exhibit at a major metropolitan US zoo and aquarium. If you are at a decision-making point, say you want to determine if mobile or desktop is the right way to focus your initial effort, you can introduce multiple artifacts to compare multiple versions of your idea. You shouldnt consider it anti-UX to also explore topics related to market acceptance. During this period, study participants are asked to keep a diary and log specific information about activities. This structure allows for a streamlined, cost-effective survey process. We attempted to remedy this on our final design by including a message that allowed the user to enter their own email into the message as well. Designers often use the rapid prototyping technique to create a concept. Im not saying this is a bad thing, but it led to us interacting with a different set of research participants than we would have preferred if we were focusing on meeting the needs of users. You havent identified the correct audience for your concept and you need to expand or redefine the user base. You can engage in concept testing as part of an interview with a participant in which they provide feedback on the concept. Unfortunately, many design teams only use one or two methods that they are most familiar with.The key question is what to use when. The sample size for a sequential monadic test is comparatively small, and this method can evaluate multiple concepts in one round. It requires the site or application to be instrumented properly in advance. Concept testing is a blend of market and UX research, two disciplines that share many methods in common. Unmoderated testing: An automated method that can be used in both quantitative and qualitative studies and that uses a specialized research tool to capture participant behaviors and attitudes, usually by giving participants goals or scenarios to accomplish with a site, app, or prototype. And other printed books. Concept testing is also highly flexible, proving useful to businesses of virtually any type or size.You can ask customers about any aspect of your idea to gain valuable insights into every detail of your product. The purpose of the artifact is to ground the research participants thinking in the reality the future concept would exist and allow them to think about how the concept would apply to this reality. A/B testing(akamultivariate testing, live testing, or bucket testing): A method of scientifically testing different designs on a site by randomly assigning groups of users to interact with each of the different designs and measuring the effect of these assignments on user behavior. This might be static such as having the user view the concept then respond to questions, or it can be similar to how UserZoom or Usabilitytesting.com present questions in the context of a participant walking through a site or prototype. For example, a benchmarking study is usually very tightly scripted, so that it can produce reliableusability metrics by ensuring consistency across participants. This research method is helpful when a company needs to gather a lot of insight in a short period of time. Limitedmethods use a limited form of a product to study a specific or abstracted aspect of the experience. However, you should not focus solely on internal stakeholders as a concept testing population. A diary study can be used to see how users interact with a product over an extended period of time (ranging from a few days to even a month or longer). We found out these stakeholders were more high-level than what would be considered an everyday user of the product. Before choosing a certain approach for user research its important to answer the following fundamental question: What do I want to know about my users and why do I want to know it? Youll identify potential roadblocks to implementing your idea that you may have missed if you ask the right questions during testing. 20062022. Without some type of user input guiding our process, we are left being directed by the same high-level stakeholder assumptions we often rally against as being the antithesis to UX: those holding the purse strings have the loudest voice in the room. This example of concept testing was not as focused on meeting the users needs as my previous example. As survey tools evolve online, you can even include audio or video descriptions of the concept for participants to consume and then provide their feedback/complete the survey. For example, you can start with small qualitative surveys to gather rich feedback. Diary studies can be used as a follow-up to a contextual inquiry: they provide organic behavioral insights and help develop a rich understanding of a participants context and environment. Useful tips on front-end & UX. The field of user experience has a wide range of research techniques available and quite often its not obvious which one you should choose for a given project. Surveys and questionnaires allow a researcher to get a larger volume of responses, which can open up the opportunity for more detailed analysis. After all, what good is a usable product if it serves no purpose or doesnt have any meaningful sized market willing to adopt it? Copyright 1998-2022 Nielsen Norman Group, All Rights Reserved. Improve usability of design, Research goal: You also make yourself open to gaining new insight and evolving your concept in directions you were not thinking of prior to engaging a broader group including users to provide feedback on the idea. Your full design team should be a part of creating and executing concept testing. While it's not realistic to use the full set of methods on a given project, nearly all projects would benefit from multiple research methods and from combining insights. You can test concepts in a number of ways, Ill cover a few in this article, but each way includes getting feedback from potential end-users on the validity and execution of the idea. Creating a product without good user research is like building a house without a solid foundation. Conducting user research allows you to dive deep beneath the surface of what you think your users want, to what they actually need. You can take the argument from there; Im aware these things dont always end the ideal way. Concept testing, as Im describing in this article, is the process of getting feedback from stakeholders and users on a solution that has yet to come to life through detailed design. It can also be done informally (where a researcher goes to the nearest coffee shop, finds participants, asks them to play with a product for 10 minutes, and then gives them a small treat as a thank you). Gathering information through direct dialog is a well-known user research technique that can give a researcher rich information about users. Summary:Modern day UX research methods answer a wide range of questions. We found less value in having these high-level stakeholders providing feedback on the concept and the experience because they were far removed from understanding some of the issues with the current tools, as well as the exact processes that were core to the purpose of the concept. However, with limited resources including time, money, and available labor, decisions need to be made: These are only a few questions of many that you can immediately come up with based on the concept of enrolling in direct deposit online a relatively simple idea. They can take place face-to-face, over the phone, or via video chat. Concept testing: A researcher shares an approximation of a product or service that captures the key essence (the value proposition) of a new concept or product in order to determine if it meets the needs of the target audience. His book Design for the Mind, is available from Manning Publications. A protomonadic test is an effective way to validate the results of a sequential monadic test. This method helpscreate or refine the information architectureof a site by exposing usersmental models. Each group is presented with a single concept for in-depth analysis. A large international financial advisory wanted to create a new product that would combine multiple existing applications and bring new features to advisors using their services. With this method, respondents examine multiple concepts and then select the one they prefer. The value in concept testing extends beyond UX and you should include members beyond the UX team in the process of carrying out testing. As Founder and Principal at XD Strategy, a UX strategy consultancy, he advised several companies on design strategy, research and insights, organizational design, career development and measuring UX. For example, lets say a large bank wants to make it easier for customers to enroll in direct deposit, you would be fairly safe to assume that allowing customers to complete the enrollment process online from start to finish would be an easy win to solve this problem. To maximize your chances of success, its extremely important to conduct insight research before making any product decisions. User groups (also called focus groups) are structured interviews that quickly (and, usually, inexpensively) reveal the desires and attitudes of a target audience. This testing method provides clear results on which concepts respondents prefer. Everyone who works on a design team should participate in a contextual inquiry from time to time. UX was not just a process, but a philosophy embraced by the team. More after jump! UX design teams are passionate about our approach to solving problems and providing users with experiences that lead to their desired outcomes. Please reply to this email and I can share more with you when I get home. (This was prior to ubiquitous smartphones.). These studies can be both qualitative and quantitative. Victor frequently writes If you havent reached a critical mass of which testing is a valuable use of the resources required, you should hold off until the concept becomes more thought out or you have additional concepts to include in testing to make it worthwhile. Participatory design: Participants are given design elements or creative materials in order to construct their ideal experience in a concrete way that expresses what matters to them most and why. July 17, 2022. You should involve as many members of the product team as possible in concept testing. You can use quantitative data from A-B testing to make decisions between multiple directions for a product and quantitative data from surveys to examine the market acceptance and prioritize features with statistical reliability if you conduct a large enough study. In a monadic test, researchers can ask specific follow-up questions for more context on what makes a certain concept better than others. The main goal of this user experience testing method is to identify usability problems, collect qualitative data, and determine participants overall satisfaction with the product. The client was pleased with the list because they wanted to raise awareness and buy-in of the concept at the top levels in their organization. For example, you might add an aspect of augmented reality to your concept testing and get some ideas around the expectations and challenges for implementation while knowing this isnt the main focus or immediate outcome of the concept testing. Here are the additional details of this concept testing: I view this as one of the most successful concept testing efforts Ive been a part of. Id recommend using an online whiteboard or collaboration tool to accomplish remote co-create. But with sequential monadic testing, each group is shown every concept. Contextual inquiry is a variety of field study where a researcher observes people in their natural environment and studies them as they go about their everyday tasks. This method helps researchers obtain information about the context of use: users are first asked a set of standard questions such as What is the most frequent task you typically do and then they are observed and questioned while they work in their own environments. Tip: Dont wait for a fully-formed product. We reflected our concern that we needed end-users who were closer to the issues the concept was meant to resolve in our report out to the client. We did the testing at the right time and with the right intention, but with the wrong audience. Surveys can provide a powerful argument that a market exists for your product when done at a scale allowing for statistical analysis and generalization of the results across your potential user population. In contrast, the kind of data collected in quantitative methods is predetermined it could include task time, success, whether the user has clicked on a given UI element or whether they selected a certain answer to a multiple-choice question. This allows key stakeholders to gain an understanding of the concept and potentially become advocates of the idea coming to fruition. You might hold workshops or working sessions with key stakeholders and users as the idea is grown from a seed in order to generate the artifacts we will discuss below, but I suggest waiting to build out the requirements of the product until youve done testing. This following table describes where many methods map to these stages in time: Research goal: We can avoid costly errors in our design if we determine direction upfront through user research in the form of concept testing. We learned from this how much of a potential negative impact on visitor flow the kiosks would have, and where we might best place the kiosks based on observed visitor flow through the exhibit. Youll need at least two versions of an idea to move forward with A/B testing. A sequential monadic test splits the target audience into groups like a monadic test. Concept testing at the beginning phase of product development allows users to share in the initial shaping of an idea to solve a problem. Concept testing allows you to validate an idea before you invest resources in building something that misses the mark. Many of the methods in the chart can move along one or more dimensions, and some do so even in the same study, usually to satisfy multiple goals. Youll have qualitative feedback that might be good for telling the story of the product and exploring how people you spoke with see the concept evolving. Intercept surveys and data mining or other analytic techniques are quantitative examples of this. Qualitative surveys ask open-ended questions. The sample size for a monadic test must be large to accommodate multiple testing groups. Another tactic for getting key stakeholder buy-in is to have them be participants in the testing sessions. To help you know when to use which user research method, each of 20 methods is mapped across 3 dimensions and over time within a typical product-development process. According to the Nielsen Norman Group, if you want to select just one type of user research for your project, it should be qualitative usability testing. Christian Rohrer is the VP of Design at TD Bank, Canadas number 2 bank and a top 10 bank in the US. This will help you to discover which questions you need to ask and the best way to ask them for a later quantitative survey. Studies where theproduct is not usedare conducted to examine issues that are broader than usage and usability, such as a study of the brand or discovering the aesthetic attributes that participants associate with a specific design style. Card sorting is asking users to organize items into groups (major features or topics related to the product), and then getting them to assign categories to each of those groups. We sometimes run into projects where the upfront discovery with users is rushed or overlooked as unnecessary. Due to thenature of their differences,qualitativemethods are much better suited for answering questions aboutwhyorhow to fixa problem, whereasquantitativemethods do a much better job answeringhow manyandhow muchtypes of questions. Remote moderated testing:Usability studies are conducted remotely,with the use of tools such as video conferencing, screen-sharing software, and remote-control capabilities. Upon reflection, I realize this testing was not ideal, but useful as an example of what to watch out for when setting up concept testing. Respondents may be asked to select the best concept or rank the options. Heres a short description of the user research methods shown in the above chart: Usability testing (aka usability-lab studies): Participants are brought into a lab, one-on-one with a researcher, and given a set ofscenarios that lead to tasksand usage of specific interest within a product or service. Now you have a few different techniques to choose from. Use surveys to engage your audience and increase revenue. He holds a Bachelors in Computer Science from UC Santa Cruz and a Ph.D in Cognitive Science and Education from Stanford University. Just the things you can actually use. Key internal stakeholders from outside the core product team. Any experience starts as an idea a concept and we are in line with the philosophy of putting users first when we take our ideas out to the end-user for feedback at the conceptual stage. Do you use product A on a daily basis? with answers being Yes or No). Good user research is key to designing a great user experience. 10 Usability Heuristics for User Interface Design, Empathy Mapping: The First Step in Design Thinking, When to Use Which User-Experience Research Methods, Between-Subjects vs. Within-Subjects Study Design, The Funnel Technique in Qualitative User Research, usability studies should rely on behavior, Focus groups tend to be less useful for usability, study participants in their own environment, create or refine the information architecture, User Research Methods: From Strategy to Requirements to Design, Operationalizing CX: Organizational Strategies for Delivering Superior Omnichannel Experiences, Marketing Email UX - User Research Methodology, How Inclusive Design Expands Business Value, Conversion Rate Optimization and Applied UX Research, Digital Diary Studies for Longitudinal Field Research. (An earlier version of this article was originally published October 12, 2014. Christian is a veteran of web and mobile user experience design and research, having participated directly in the development of some of the worlds most popular web destinations, including Yahoo!, eBay, and REALTOR.com (Move, Inc.). I initially thought it was a better idea to test concepts completely in the abstract as a researcher, I wanted to avoid leading people. It can help researchers find the answer to the following important questions: Despite the obvious advantages of this method, user groups are often criticized by many researchers. Desirability studies: Participants are offered different visual-design alternatives and are expected to associate each alternative with a set ofattributes selected from a closed list. More details about the methods and the dimensions of use in the full-day training courseUser Research Methods: From Strategy to Requirements to Design. Concept testing allowed us to shape the final design well before the materials were fabricated, screens were designed, or code was finalized. These classifications are meant to help you make the best choice at the right time. Card sorting: A quantitative or qualitative method that asks users to organize items into groups and assign categories to each group. This provides greater external validity but less control over what topics you learn about.Many ethnographic field studies attempt to do this, though there are always some observation biases. Ive changed my opinion on this as Ive grown as a researcher. Concept testing can help you identify a viable idea for the present and also identify ideas you might implement in the future when technology or society is ready. Measure product performance against itself or its competition, Field studies, diary studies, interviews, surveys, participatory design, concept testing, Card sorting, tree testing, usability testing, remote testing (moderated and unmoderated), Usability benchmarking, unmoderated UX testing, A/B testing, clickstream / analytics, surveys. Many of these platforms for gathering user feedback also allow you to have users give audio responses or provide feedback on the overall experience upon the conclusion of their session.