At inVibe, we’ve been working with AI since our founding in 2013, and we were among the first market research agencies to adopt 

large language models (LLMs) back in 2021

, and we released our LLM tools to clients in 2023. AI isn’t a gimmick or a buzzword for us, it’s central to 

: To develop technology that provides cost-effective, high-speed healthcare market research while maintaining the highest quality.

As AI takes a center stage in healthcare with the release of popular LLMs like ChatGPT and Claude, we’ve continued to integrate the latest AI models, features, and functionality into more of the work we do at inVibe. Working with AI every day, we quickly started to see the cracks and have prioritized finding ways to ensure the accuracy and integrity of AI-generated content.

To ensure our clients have reliable AI insights, we designed and implemented an evaluation system to validate our AI tools. Our language experts and prompt engineers leverage this system to iteratively test the quality and reliability of the outputs. By testing different prompts and LLMs in this system, we’re able to ensure that our AI tools meet our clients’ needs and our high standards.

Practically speaking, we do this by having two trained language experts compare two outputs presented next to one another on one screen. Once we review the two outputs, we score each across 6 categories that are based on best practices for evaluating the quality of language data and conversational LLMs.

: To what extent does the output address everything it was asked to?

: How closely does the output align with our understanding of the voice data? (Rating: 1-4)

: How clear is the relationship between the findings and the verbatim evidence in the output? (Rating: 1-4)

: To what extent is the output presently coherently and in the prescribed way? (Rating: 1-4)

: Does the output include consistent formatting that enhances readability? (Rating: 1-2)

: Are findings in the output supported by citations in the prescribed format? (Rating: 1-2)

Let’s take a look at one of these comparisons in action. In the below image, we’re seeing a quality test of two different LLMs that were prompted to report the key findings from the same set of voice data. After reviewing both outputs, the language expert scrolls down to where they assign scores to each output and select a ‘winner.’

Once we have a significant number of ratings for what we’re assessing, we compare the total scores, sub-scores, and selected winners to determine an overall winner (i.e., the model that performed better). This overall winner becomes our new baseline for this task in the next round of testing and becomes the default way that our AI tools perform this operation in our dashboard—until the next upgrade at least.

Through many iterations of this testing, we’ve improved the quality of our models by 46.73% over the last 12 months (and we expect to continue this pace in the next 12 months)! This means that our clients can rest assured that our AI tools will remain best-in-class: offering comprehensive, accurate, transparent, and comprehensible outputs tuned to market research best practices for qualitative voice data.

Are you interested in learning more about how our AI tools can make your qualitative research simpler, more systematic, and more scalable? 

Pioneer in AI-powered healthcare research since 2013: See how inVibe validates enterprise LLMs with a rigorous testing framework that's improved insights quality by 46.73%. Discover our proven approach to reliable, scalable market research.

Responsible AI: Validating LLM Outputs

Good insights require good research, and good research requires high-quality data. At inVibe, this data comes from pre-designed voice response questions, which we use to elicit the depth of response of traditional qual research without the scheduling and moderation needs of an interview. The resulting voices can provide powerful insights into HCPs and patient behaviors.

However, our methodology demands particular care in how we ask questions – in an unmoderated survey, we need to make sure that we are prompting respondents in a way that elicits useful content while reducing the need to follow up. To accomplish this, we must strike a delicate balance between an open-ended design philosophy and goal-oriented lines of questioning.

Without the back-and-forth of an interview, it can be difficult to imagine respondents speaking both broadly and deeply enough to provide meaningful qualitative insights. Indeed, from the participant’s perspective, our voice response surveys may seem more like quick quant surveys, at least in terms of how long it takes. But getting expansive responses in this format is easier than you may think, mainly for one simple reason: 

When we reach out to stakeholders, whether they are doctors or patients, they are – in one way or another – an expert in some condition. Whether a patient is discussing their experience with an illness, or an HCP is describing their typical patient, our prompts place the respondent in the role of a teacher, an expert. By giving them that role, we empower respondents to speak with authority and confidence on whatever subject we’re trying to dig into.

This positioning sets us up for success and helps us avoid the dreaded “yes/no” quant-esque responses that come from reticent respondents or under-designed write-in surveys. When respondents feel engaged and confident, they will give us more than that.

So, what does this look like? Frequent readers (or partners) may have seen some examples of our voice-response questions in the past, but here is a good example of a prompt that gets someone talking:

“How would you describe what living with multiple sclerosis is like to someone who is unfamiliar with it? What frustrations or challenges do you experience?

The purpose of a question like this is clear: Elicit a patient description of the MS experience. Something that could be asked briefly and answered even more briefly – indeed, a perfectly valid line of inquiry would be to ask for a few adjectives. But by putting a patient directly into the role of an explainer, we set an implicit expectation. We want a story, not just a description. And by tapping into how MS makes patients feel, we can get just that:

Question 1, Response 34

We set the stage for this respondent, and she took it. By harnessing people’s desire to be the expert, we can get the rich voice data that our insights are made of.

Now that we see the benefit of getting a respondent talking, why not take it further, go broader? If we just need more voice to get the insights, why not just design for maximum response length?

First and perhaps most obviously, we need to respect our experts’ time. 

, and trying to push through it in an unmoderated environment can lead to lopsided or outright bad data.

But second, and equally important, is that designing around the fundamental research questions makes responses more coherent and insights more accessible. One standard part of our survey design process is making sure each prompt we write corresponds to a research question or goal our client has. If our client wants to know a patient’s thoughts on new treatments, we do not use half of the survey breaking down symptom impact. One prompt that gets us context on disease experience can be extremely helpful in interpreting a patient’s later eagerness or hesitation to try something new, but we build the prompt to that specific purpose.

There is a third ‘why-not’ that is more situational but often quite powerful: setting boundaries gives respondents the chance to meaningfully break them. Compare the following examples:

A.  What is your overall experience with Treatment X?

B.  How effective has Treatment X been at managing your condition?

If a patient focuses on side effects when answering A., we can infer that side effects are the most top-of-mind aspect of Treatment X. Efficacy goes un-discussed, which tells us something about the patient’s priorities.

If a patient focuses on side effects when answering B., however, that tells us a little more. Not only are side effects top-of-mind, but we’re seeing the respondent choose to focus on that aspect of treatment X at the expense of efficacy. This signals both the patient’s priorities (which we would have gotten from A) and that whatever efficacy X has is not enough to justify those side effects. We get more insight by providing a little more guidance, even if our respondent disregards it.

This strategy is an example of how we take the linguistic expertise we use when analyzing responses and put it to work on the design side. Awareness of language helps us not only find insights in the data, but elicit the kind of data that will give us plenty to fins.

There is always more to be said about survey design, but the best way to understand how we put these principles into action is through an example. If you want to see how we would handle your research questions, 

Learn how inVibe crafts effective voice response questions for healthcare research. Discover techniques for balancing open-ended prompts with goal-oriented design to elicit meaningful patient and HCP insights.

How we ask: How inVibe writes voice response questions

Crohn's & Colitis Awareness Week takes place annually from December 1-7 to raise awareness about inflammatory bowel diseases (IBD). Since 2011, this week has aimed to educate the public about two conditions that affect millions of Americans, Crohn's disease and ulcerative colitis.

In this post, we’ll be focusing on ulcerative colitis (UC). UC colitis is a chronic inflammatory bowel disease that causes inflammation and ulcers in the large intestine, or colon. Symptoms typically include persistent diarrhea, abdominal pain, rectal bleeding, fever, and weight loss, though the severity can vary among individuals.

Symptoms of UC and flare ups often seem unpredictable, and those living with the condition often exert some semblance of control via strict adherence to diet and medication routines.

Let’s listen to a 26- to 30-year-old woman with UC describe what it’s like for her to live with this condition and the “mental load” of navigating a flare-remission condition.

Q1, R132

Understanding the patient voice is at the heart of 

, which we approach holistically by analyzing what is said, how it’s said, and how it sounds.

Focusing on the content and language of her response, this patient describes the “unknowingness” of living with UC using the refrain “I don’t know” to introduce how her symptoms emerge unpredictably (if cyclically). This lack of control and knowledge she feels results in additional stress that wear her down mentally like symptoms wear her down physically.

 characterized by moderately negative emotionality (‘valence’) and decreased excitement (‘activation’), which indicates frustration at and acceptance of how UC limits adds stress to her life and forces her to make accommodations.

Overall, her response underscores the challenges of living with an unpredictable condition and the unmet need for therapeutic options that help those living with UC to exert control over their lives without overly burdensome diet and lifestyle changes. 

Our Listening Platform makes it simple for patients to speak with us about their experiences in an open and authentic way and easy for you to engage with their perspectives directly.

If you’re interested in hearing from more patients like this one so that you can gain insight into their experiences, beliefs, behaviors, and desires, 

Explore the daily challenges of living with ulcerative colitis through a patient's perspective on unpredictability, stress management, and lifestyle adaptations.

In Support of Crohn's & Colitis Awareness Week (Dec. 1-7)

World Lung Cancer Day, observed annually on August 1st, serves as a global platform to raise awareness about lung cancer, its risk factors, and the importance of early detection through screening and treatment.

Lung cancer is the leading cause of cancer-related deaths worldwide for both men and women. In 2020 alone, there were 2.21 million new cases of lung cancer and 1.8 million deaths due to the disease.

While each person’s experience of lung cancer is unique, it’s common for people to experience physical challenges associated with symptoms and treatment side effects; to feel overwhelmed, anxious, or uncertain about the future; and to endure significant changes to their lifestyle due to the needs of treatment and monitoring.

Let’s listen to a 65+-year-old woman with non-small cell lung cancer (NSCLC) describe the emotional toll of living with this condition and some of the challenges she faces.

Lung Cancer (Q1, R19)

Focusing on what she is saying and how, this patient tells us how she feels “anguish” with the cyclical anxiety of living with lung cancer. She anchors her emotional wellbeing to the schedule of her ongoing cancer management and the uncertain results of each scan, repeating how she feels like she can only live “3 months at a time.”

, we hear a measurable decrease in fluency and control (‘dominance’) when she describes the uncertainty around the results of her regular scans, which commonly occurs when someone is discussing something emotionally fraught.

As a whole, this patient’s focus on the lack of predictability and of control indicates a need for healthcare professionals to provide more comprehensive support systems that address the psychological aspects of living with lung cancer and highlights the importance of developing treatments and management strategies that provide more stability and predictability for patients.

In Recognition of World Lung Cancer Day (Aug. 1)

Thanks for reading!

Responsible AI: Validating LLM Outputs

How we ask: How inVibe writes voice response questions

In Support of Crohn's & Colitis Awareness Week (Dec. 1-7)

/voice

/product

/resources

/use-cases

/company

@social