Data Collection Instruments
In the first and second installments of this series, we explored how generative AI tools, including ChatGPT and Claude, can be employed early in a research project to come up to speed, refine research objectives, develop hypotheses, determine a research methodology and develop an airtight screener.
In this article, we’ll dive into ways GenAI can help optimize both qual and quant data collection instruments like discussion guides and questionnaires.
As always, check your corporate policies for how you’re allowed to use GenAI, and which platforms are approved.
Co-creating Discussion Guides with GenAI
While Generative AI tools lack the nuanced contextual understanding of human researchers, they can provide valuable improvements to discussion guides. GenAI can ensure questions directly test the established hypotheses and answer the research objectives, eliminate redundant or unnecessary questions, and recommend a flow that promotes a natural conversation.
GenAI tools can also create customized discussion guides tailored to different audience segments based on provided prompts and can develop effective probes and follow-ups to elicit more detailed responses from participants.
Are there any questions missing in order to understand the influence of different AI features in a CRM message on its recall? Assume we will present the respondent with messages as stimuli emphasizing the different features.
Could you describe in your own words how the time-saving automation feature of GPT-powered functionality in the CRM system works? What benefits do you associate with this feature?
Similarly, let’s talk about the customer sentiment analysis feature. How would you describe this feature, and what advantages do you see in using it within a CRM system?
GenAI can also recommend specialized techniques such as laddering or projective, and can check for bias in the way questions are worded. It can even create neutral message stimuli that don’t introduce bias.
Create a series of laddered questions to arrive at the implicit receptivity to CRMs with ChatGPT features.
-
-
- Surface Question: “What are your thoughts on CRM systems that incorporate ChatGPT features?”
- Functional Benefits: “What specific functional advantages do you see in using CRM systems with ChatGPT features?”
- Impact on Efficiency: “How do you think these systems might improve the efficiency of business operations?”
- Customer Insights: “Can you explain how having ChatGPT in CRM systems might provide a deeper understanding of your customers?”
- Competitive Advantage: “Do you think businesses using these systems could have a competitive advantage? Why or why not?”
- Concerns or Reservations: “Are there any potential drawbacks or concerns you have about implementing CRM systems with ChatGPT?”
- Long-term Goals: “From a strategic perspective, how do you see these systems fitting into your long-term business goals?”
- Ultimate Benefits: “What would be the ultimate goal or benefits you aim to achieve by adopting CRM systems with ChatGPT features?”
-
Although the auto-generated questions themselves may be repetitive and uninterestingly phrased for a participant, it does scaffold a skeleton system for analytical probing that can improve a researchers’ thinking by being more comprehensive.
If the guide length needs to be adjusted, GenAI can assist by ranking questions as high, medium, or low priority, based on how crucial they are to answering the stated research objectives.
GenAI can also help simplify language for clarity and readability, generate a logical flow and structure, and ensure the questions are open-ended and unbiased. The AI can even suggest places for transitions to smooth out the flow of the conversation.
Architecting Questionnaires with GenAI
GenAI tools are great at enhancing questionnaires. They can propose additional measurement questions tied directly to the defined research objectives and ensure that the survey will address them. Given the hypotheses, GenAI can check that the necessary questions and logics are included to test them soundly, especially regarding measurement baselining:
Identify the research hypotheses that may have weak testing by the questionnaire and recommend ways to improve the testing.
-
- Hypothesis on Message Differentiation:
- Weak Testing: If the questionnaire doesn’t adequately assess whether the messaging effectively highlights the unique selling points of GPT integration compared to competitors, the hypothesis about differentiation perception may not be robustly tested.
- Improvement: Include questions that directly ask respondents about their perception of how well the messaging distinguishes your CRM product’s GPT integration from competitors. Use Likert scales for more nuanced responses.
- Hypothesis on Barriers Identification:
- Weak Testing: Identifying potential misunderstandings, misconceptions, or reservations might be challenging if the questions are not specific enough or if respondents are not encouraged to express detailed concerns.
- Improvement: Frame questions in a way that encourages respondents to share specific concerns or reservations. Include open-ended questions to capture nuanced feedback.
- Hypothesis on Message Effectiveness:
- Weak Testing: If the survey questions do not effectively gauge how well the messaging communicates the advantages of GPT-powered functionality, the hypothesis on message effectiveness may not be adequately tested.
- Improvement: Include questions that assess respondents’ perceived effectiveness of the messaging, possibly using Likert scales, and ask for specific examples of what worked or didn’t work in the messaging.
- Hypothesis on Message Differentiation:
GenAI can flag question design that makes interpretation of results ambiguous (e.g., the question refers to multiple concepts, but respondents can only select one option that may not apply to all). It can then rewrite those questions so interpretation of results is more straightforward. The GenAI tool can suggest different quantitative measurement types or logics to capture information from a question, such as converting an open-ended question into a quantitative one, recommending scale choices to effectively quantify responses, and ensuring the scales used throughout the survey are consistent. It can also help identify questions where the options may be incomplete and recommend additional responses. It can even use a discussion guide or transcripts to help fill out those options with more context.
Convert the open-ended question 5 into a closed-ended question by preserving the collection of content information and adding scales.
On a scale of 1 to 7 (where 1 = Not important at all, and 7 = Absolutely critical), please rate the importance of the following potential benefits of GPT-powered features in a CRM system for your business needs.
-
-
- Time-Saving Automation
- Enhanced Customer Insights
- Personalized Recommendations
- Streamlined Communication
- Improved Customer Support
-
You can also ask the GenAI tool to check the flow to ensure that the sequential logic and branching is sound and streamlined. It can optimize survey length for better completion rates and prioritize questions based on the research objectives.
Finally, GenAI can be used to minimize survey fraud by ‘red herring’ responses to help prevent bad actors from completing the survey. It can also recommend specific quality check techniques, such as inter-question consistency:
Enumerate any consistency checks that can be performed on this survey by cross-checking responses between two or more questions.
The GenAI tool returned ten different thematic inter-question consistency checks of the response content with specific actionable cross-validation comparisons between question responses. Instituting these QC guidelines will greatly improve the accuracy of your sample, by identifying and removing questionable respondents as you’re fielding.
On The Road to Results
To effectively use GenAI to refine a data collection instrument, it’s important to keep the objectives and hypotheses in focus. For quality assurance and critical thought partnership, the AI tool should be given clear criteria from which to evaluate the instrument – the specific research standards you seek to improve can serve as the axes for the GenAI tool to analyze your instrument. Having a set of such standard checks can help streamline this process and improve the overall objectivity of the research.
With optimized instruments driving research, GenAI can then assist with performing quality control and gaining a preliminary understanding of the results. This will be the topic of the next installment of this series.
Reach out to us today at [email protected] to learn more about how GenAI can help optimize both qual and quant data collection instruments.
Read more in our AI for Market Research series
Part 1: Objectives, Onboarding, Methodology
Part 2: Methodology Selection + Participant Screening
Part 3: Data Collection Instruments
Part 4: Quality Control + Preliminary Findings
Part 5: Final Deliverables
Part 6: The Future + Synth Data (Coming Soon)