You may be surprised to learn that there is often very little silence in conversation. When we speak, the typical gap between one person finishing what they are saying and the next person starting to speak is tiny, measuring just one fifth of a second on average. In fact, many times these gaps aren’t present at all, and speakers talk over each other and overlap their speech.

Since we are used to such short silences between conversational turns, anything longer than that is noticeable. The more time someone takes to reply, the more it seems like the silence is purposeful, conveying something by saying nothing at all.

“Do you want to go out for coffee with me?”

 linguists have found that an acceptance or a “

” often occurs quicker, with barely a pause at all between the question and the reply. By contrast, someone who is going to decline or say “

” tends to delay their rejection. Even a pause of half a second can be enough to indicate that they are probably not going to agree to go out to coffee with you.

Depending on context, silence can convey a wide variety of meanings, including avoidance of a subject, lack of knowledge, agreement, disagreement, indignation, disappointment, confusion, hesitation, and so on. By paying attention to these pauses, we at inVibe are able to more accurately interpret nuances in qualitative data.

Let’s dig into a relevant example. When a participant in qualitative research (including our own) is asked a question, there can often be a gap before they begin their answer. This could indicate that they are at a loss for words or speechless. It could also mean they are about to share something emotionally heavy, conveying deep feeling. Or it could simply be a sign that they are hesitating, trying to think of an answer to a particularly tricky question.

In our conference intelligence projects specifically, we find that the presence of long pauses and silence can indicate a lack of certainty on a topic. When they feel more uncertain, HCPs tend to pause more throughout their responses.

Take, for example, this HCP describing his typical treatment approach for non-small cell lung cancer patients.

HCP Audiogram 1

You can hear from the speed and fluency of his speech that this HCP is very certain of his treatment approach. The sentences almost appear to run into each other, with very little breath between them.

However, more pauses and hesitation appear in this same HCP’s responses later in the study when he reacts to new data released at the conference.

HCP Audiogram 2

We can hear the pause length increasing almost from the very beginning of this HCP’s response. After saying that the abstract was “very interesting,” he pauses for half of a second before he shares a more hesitant reaction, saying he would only use this regimen with “some patients” and that he is “still on the fence.” This pause clues us into his uncertainty, prefacing his negative evaluation that follows.

We see this hesitation come up again later in the response as well. When he begins to say that he 

be willing to try the combination, this is followed by a long pause before he clarifies that it’s “not data supported.” This pause – among other linguistic markers – indicates that these results will likely not affect this HCP’s current treatment choices.

In this context, listening to every response for signs of hesitation (including pauses) allows us to recommend concrete messaging strategies to encourage broad uptake of a client’s novel treatment across patient types.

By expanding our analyses beyond what is said to how it sounds, our trained linguists at inVibe are better able to interpret HCPs’ and patients’ beliefs, attitudes, and intended meanings. That’s why we listen to every response to every question.

 to learn more about how our listening team can help you to accomplish your business objectives!

Why We Listen, Part 2: Silences

, I discussed what emotions are and how we can identify emotions by recognizing subtle differences in vocal qualities. I’ll be focusing here on what Speech Emotion Recognition (SER) is, how computers can reliably measure emotions, and how we use SER to power our linguistic insights.

Speech emotion recognition (SER) is an application of AI that uses machine learning models to measure and classify emotions as conveyed in the 

 spoken. Our SER model takes into account vocal qualities that humans can easily distinguish (like pitch, tone, speed, loudness) 

 others that anyone would struggle to articulate but may be able to intuit with some training (like signal energy, voice quality, voice onset, vowel formant frequencies). These vocal qualities then get laddered up to three acoustic measures that have been shown to be more robust, more adaptable, and more accurate compared to measuring emotions directly: activation, valence, and dominance.

 in the voice. Increased activation indicates greater enthusiasm for a topic

in the voice. Increased valence indicates more pleasant feelings toward a topic

communicated by the voice. Increased dominance indicates a more confident way of speaking about a topic

Taken together, these three measures allow us to measure underlying emotions encoded in the voice and compare differences in emotion and in their intensity.

In order to accurately capture the emotionality cross-culturally, our model has been trained on large and diverse datasets that capture a wide range of emotional expressions from tens of thousands of speakers across dozens of languages and cultures. This rich dataset allows our model to perform in a way that is ‘speaker-independent,’ focusing on universal aspects of emotional expression instead of those specific to any individual speaker or speech community. More specifically, our model measures about 40 acoustic features, maps their contours statistically, and outputs normalized scores to ensure direct comparability across speakers.

To ensure that each speaker’s emotionality is reliable, we also measure each respondent’s score relative to a baseline that we collect from them at the beginning of the voice survey. Relativizing scores like this ensures that any emotionality we measure in the voice corresponds exclusively to the topic that the respondent is discussing. That is, if someone who is already 

 about a topic in the survey that they’re reacting to, the model alone would output 

. This is because the vocal qualities of 

 remain in their voice, coloring their emotional measures. Relativizing the scores effectively resets their emotional baseline so that the model recognizes their emotionality as 

Let’s turn now to an example where we uncover the underlying emotions in physician responses. In this example, we asked physicians to share their reactions to clinical trial data released at a professional oncology conference. I know it doesn’t sound like the kind of thing that one would get overly emotional about, but let’s take a listen and then a look.

First, let’s listen to two typical responses from oncologists reacting to two different abstracts. The first HCP (pink) is reacting to Abstract 1, which covers the results of a phase 1 study of a novel molecule. The second HCP (green) is reacting to Abstract 2, which covers phase 2 data from a novel combination involving two familiar molecules.

HCP Reactions to Novel Data

Both responses cover many of the same topics: “interest” in the results, concerns about sample size and generalizability, how this data compares to standard of care, and potential adoption into practice. However, I’m sure you heard a subtle difference between the two HCPs. With acoustic analysis, some of our intuitions about these differences in the HCPs’ voices can be plotted and shown plainly.

These two charts (called circumplexes) illustrate the different levels of enthusiasm (activation; y-axis) and positivity (valence; x-axis) of each respondent’s voice as they react to Abstract 1 (

). Comparing these two charts, we see that responses to Abstract 1 cluster in the lower left quadrants and responses to Abstract 2 in the upper right quadrants. These differences in clustering indicate that HCPs are 

 about Abstract 2 than they are about Abstract 1. In this case, enthusiasm for a treatment indicates a greater willingness to adopt it once it is approved by the FDA.

Measuring emotion, understanding behavior

Acoustic analysis is a potent tool in the belts of our language experts that helps our clients to understand the thoughts, beliefs, attitudes that explain and motivate behavior. If you’re interested in learning more about how inVibe can help you to collect and analyze the voices of your key stakeholders (patients, caregivers, HCPs, payors, and more!), 

Learn about Speech Emotion Recognition (SER), how computers measure emotions, and how inVibe uses SER technology to generate powerful linguistic insights.

Measuring and Identifying Emotion in the Voice

At inVibe, we use machine learning technology to recognize and measure emotionality in our respondents’ voices when they’re speaking. To humans who have spent a lifetime decoding emotion, this task seems very simple. However, getting a computer to decode emotions like a human can is a bit more complicated than it seems. To understand exactly what we’re asking computers to do, let’s explore the nature of emotions, what it means to “recognize” one, and how humans do it.

When we think about emotions, the first ones that come to mind tend to be “basic” emotions like 

. Social scientists refer to these emotions as “basic” because—compared to other emotions—they’re simple. In fact, they’re so one-dimensional and consistent that children tend to recognize them as young as 3 to 4 years old—about 4 years before they reliably recognize more complex emotions like 

After a lifetime of interacting with others, we all have intuitions about how people express their emotions. In real life, those intuitions are generated from a variety of sources: facial expressions, body language, tone of voice, and situational context, among others. What if we couldn’t rely on all of these behaviors and contextual elements? What if, like a computer, we had to recognize emotions from just the tone of voice? No context, no body language, no facial expressions, just voice. How would we do it?

Let’s take this example out of the abstract and give it a try for ourselves. Below are 5 examples of an actor saying “He told me he was moving away” with each performed with a different basic emotion (

). Listen to the following examples and try to intuit which audio corresponds to each emotion. Once you’re ready to see the answer, click the static below the “EMOTION” header to reveal the corresponding emotion.

How’d you do? Maybe this was harder than you expected, maybe your intuitions matched all 5 correctly. Tasks like this are hard for most people—certainly harder than recognizing the emotions of someone you’re speaking with directly—and most people are only able to guess emotions at just above random chance in tasks like this (about 1 in 5 or 20%). We’re asking much more from computers, which are expected to be at least 80% accurate, identifying 4 out of 5 emotions correctly.

Let’s explore how humans decode emotionality in speech. When the only thing that most of us are aware of is that person “sounds 

,” it’s hard to tease apart how we came to that intuition. Think about your own thought process as you tried to decode the above examples. What were you listening for? What clued you into the emotionality in the audio examples? Well, if you’re like most adults, you probably noticed differences in vocal qualities like 

 (smoothness), or any other of the 30-odd other human-perceptible vocal qualities. If your intuitions are anything like mine, your descriptions for how these 5 emotions sound may look something like this.

Taken together, subtle differences across these 4 vocal qualities are enough to uniquely identify these 5 basic emotions. But the differences within each feature are just that: “subtle.” Over the course of childhood, we learn to associate certain vocal qualities with particular emotions through repeated exposure in social interactions. And, through this exposure, we learn to adapt to tremendous amounts of variation by speaker and by circumstance. Take pitch for example.

Is my ‘quieter’ the same as yours? ‘Louder?’ ‘Neutral?’

Is my ‘quieter’ right now the same as it was yesterday or it will be tomorrow?

How much louder is a ‘loud’ emotion like 

Are these pitch contrasts consistent for speakers of American English and British English? What about speakers of French? Chinese?

) be? Does it pick a quality from one emotion, take the average, something else altogether?

Whether we can articulate answers to these questions or not, we all have robust intuitions on the answers to these questions in real life conversations. The problem becomes then “How do we move beyond intuition? How can we prove our intuitions are accurate?” At inVibe, we solve these problems by using Speech Emotion Recognition (SER) technology to measure emotionality in the voice. In 

, I’ll discuss what SER is, how it works, and how we use it to power our linguistic insights.

If you’re interested in learning more about inVibe, our listening-centered approach to research, or how we can support your business objectives, 

Discover how inVibe uses machine learning to decode emotions in voice data, exploring the complexities of getting computers to recognize feelings as humans do.

Hearing and Intuiting Emotion in the Voice

National Breast Cancer Awareness Month (NBCAM) is observed annually in October to help spread awareness, increase early detection rates, and provide public education about breast cancer.

Breast cancer is the most common type of cancer in women in the United States aside from skin cancers. The NBCAM initiative helps to encourage individuals to get screened, donate to relevant foundations, and take part in events that support research and patient care.

Living with breast cancer involves navigating a range of challenges, from managing physical symptoms like pain as well as treatment side effects like fatigue, nausea, and hair loss to coping with the emotional and mental impacts of the living with the condition.

Let’s listen to a 41- to 45-year-old woman with breast cancer describe what it’s like for her to live with this condition and some of the many challenges that she faces.

Question 1, Response 15

Understanding the patient voice is at the heart of 

, which we approach holistically by analyzing what is said, how it’s said, and how it sounds.

Looking first at what she says and how, this patient describes going through treatment as physically, financially, and emotionally difficult. She emphasizes the burdensome nature of living with breast cancer by repeating the linguistic intensifier ‘very’ three times: “

 sick.” She stresses how the physical impacts of living with cancer have changed even how she sees herself, describing how cancer has “all of a sudden” made her “not be the same person.”

, her description of the challenges and frustrations of living with breast cancer and going through treatment is expressed with strongly negative emotionality (‘valence’), which emphasizes the multifaceted difficulty of undergoing breast cancer treatment.

Taken together, her response suggests a need for advocacy groups to move beyond awareness to understanding and addressing the problems associated with living with and treating breast cancer.

Our Listening Platform makes it simple for patients to speak with us about their experiences in an open and authentic way and easy for you to engage with their perspectives directly.

If you’re interested in hearing from more patients like this one so that you can gain insight into their experiences, beliefs, behaviors, and desires, 

In Recognition of National Breast Cancer Awareness Month (October 2024)

The Patient Perspective

Decoding the Patient Voice

Simple, Systematic, Scalable

Thanks for reading!

Why We Listen, Part 2: Silences

Measuring and Identifying Emotion in the Voice

Hearing and Intuiting Emotion in the Voice

/voice

/product

/resources

/use-cases

/company

@social