NLU, Language ID and Diarization gRPC Integration

Transcription tells you what was said. The questions that follow are harder: what it meant, what language it was in, and who said it. Natural language understanding (NLU), audio language identification, and speaker diarization each answer one of them, and each runs as its own interaction on a session, so you request only the analysis a workflow needs. This article shows how to call each one on its own and how to combine them with transcription, including the ordering rules that are easy to get wrong.

Before you start. These interactions use the same session lifecycle as every speech interaction, described in ASR gRPC Integration, and the transcription steps are described in Transcription gRPC Integration. NLU settings are in the ASR & Transcription Settings Reference. For what each capability does and where it fits, see Speech Products: NLU, Neuron & Diarization.

Which interaction to use

You need to	Interaction	Input	Mode
Summarize, detect topic or outcome, analyze sentiment, translate, or detect language from text	`interaction_create_nlu`	Text, in the `input_text` field	No audio is needed
Identify the language from audio	`interaction_create_language_id`	Audio in the session	Batch
Find out who spoke when	`interaction_create_diarization`	Mono audio in the session	Batch

The platform also has interactions for sentiment on its own (interaction_create_sentiment, which accepts several phrases for batch analysis) and for Neuron intent and slot extraction (interaction_create_neuron). They follow the same pattern.

NLU

An NLU interaction processes text you supply. The text can come from any source, including a transcript from the platform, so NLU does not require audio at all. The text goes into the create request, so it must exist before you create the interaction.

Field	Purpose
`language`	The language to use for NLU processing, such as `en-US`.
`input_text`	The text string to process.
`nlu_settings`	Optional settings that choose which NLU steps run. Each step is enabled separately.
`general_interaction_settings`	Optional settings that apply to every interaction type.

NLU steps and where the results appear

Enable the steps you need in nlu_settings. Each result appears in nlu_interaction_result.

To get this	Set this in `nlu_settings`	Result field
A summary as bullet points	`summarization_bullet_points` (1 to 15)	`summary_bullet_points`
A summary as a number of words	`summarization_words` (10 to 1000)	`summary_words`
A translation	`translate_to_language`, and optionally `translate_from_language`	`translation_result`
Alternate translations	`enable_alternate_translations`	`translation_alternate_results`
The language of the text, and the dialect where possible	`enable_language_detect`	`language_detect_result`
The topic of the text	`enable_topic_detect`	`topic_detect_result`
Whether an outcome was achieved	`detect_outcome_type` (`PAYMENT` or `SUCCESSFUL`)	`outcome_result`
The sentiment	`enable_sentiment_analysis`	`sentiment_result`

Summarization by bullet points and by word count are mutually exclusive. Specify only one.
Translation can be less accurate when the source language is not specified. Set translate_from_language, or enable enable_language_detect so the source language is detected.
Outcome detection can detect whether a payment was successfully made (PAYMENT) or whether the call was successfully resolved (SUCCESSFUL). For other outcomes, contact Technical Support. See Logging a Support Ticket.
Sentiment analysis supports English, Spanish, and French.
request_timeout_ms sets how long NLU waits for results. The default is 5000 ms.

NLU on text you already have

Step	Your client sends	What happens
1	`session_create`	Creates the session and returns the `session_id`.
2	`interaction_create_nlu`	Creates the NLU interaction with the `input_text` and the steps you want. Returns an `interaction_id`.
3	(wait for the result)	The result arrives on the session response stream. If you miss it, call `interaction_request_results`.
4	`interaction_close`, then `session_close`	Ends the interaction and the session.

NLU on a transcript

To analyze speech, transcribe it first and pass the transcript to NLU. Because the NLU request carries the text, create the NLU interaction after the transcription result is available.

Step	Your client sends	What happens
1	`session_create`	Creates the session.
2	`session_audio_format`, then `audio_push`	Sets the audio format and pushes the audio.
3	`interaction_create_transcription`	Creates a transcription interaction in batch mode, with any phrase lists or normalization settings. The result arrives on the stream.
4	`interaction_create_nlu`	Creates the NLU interaction, with the transcript as `input_text`.
5	(wait for the result)	The NLU result arrives on the stream.
6	`interaction_close` for each interaction, then `session_close`	Ends the interactions and the session.

For example, to summarize a transcript in three bullet points and score its sentiment:

{
  "interaction_request": {
    "interaction_create_nlu": {
      "language": "en-US",
      "input_text": "<the transcript text>",
      "nlu_settings": {
        "summarization_bullet_points": 3,
        "enable_sentiment_analysis": true
      }
    }
  }
}

Field names are shown as they appear in the API reference.

Audio language identification

Language identification determines the language from the audio itself, which is what you need when you do not know the language before the call is transcribed. Use the result to set the language of a transcription interaction on the same session, so the audio is pushed only once.

Field	Purpose
`target_languages`	An optional list of the languages to detect, for example `"en, es, fr"`. Possible values are `en`, `es`, `fr`, `de`, `ru`, `zh`, `nl`, `ja`, `pt`, and `it`. By default, all languages are used. Restricting the list to the languages you expect narrows the choices.
`audio_consume_settings`	Optional settings for how the audio is consumed. Set the mode to batch.
`request_timeout_ms`	How long language identification waits for results. The default is 10,000,000 ms (about 2.7 hours).
`general_interaction_settings`	Optional settings that apply to every interaction type.

Identify the language, then transcribe

Step	Your client sends	What happens
1	`session_create`	Creates the session.
2	`session_audio_format`, then `audio_push`	Sets the audio format and pushes the audio, once.
3	`interaction_create_language_id`	Creates the language identification interaction with `target_languages` if you want to restrict the choice.
4	(read the result)	The result lists the detected languages, each with a confidence from 0 to 1000.
5	`interaction_create_transcription`	Creates the transcription interaction on the same audio, using the language the identification returned. Use batch mode.
6	(wait for the result)	The transcription result arrives on the stream.
7	`interaction_close` for each interaction, then `session_close`	Ends the interactions and the session.

The transcription depends on the identification result, so the two steps run one after the other, not in parallel.

{
  "interaction_request": {
    "interaction_create_language_id": {
      "target_languages": "en, es, fr",
      "audio_consume_settings": {
        "audio_consume_mode": "AUDIO_CONSUME_MODE_BATCH"
      }
    }
  }
}

Speaker diarization

Diarization takes mono audio that contains several speakers and reports which speaker spoke when. It does not know what was said. Combine it with a transcription of the same audio to attribute each phrase to a speaker.

Field	Purpose
`language`	The language to use for diarization, such as `en-US`.
`max_num_speakers`	The maximum number of speakers to identify, from 1 to 100.
`audio_consume_settings`	Optional settings for how the audio is consumed. Set the mode to batch.
`request_timeout_ms`	How long diarization waits for results. The default is 10,000,000 ms (about 2.7 hours).
`general_interaction_settings`	Optional settings that apply to every interaction type.

Diarization with transcription

Diarization and transcription both work from the same audio, so they can run in parallel on one session.

Step	Your client sends	What happens
1	`session_create`	Creates the session.
2	`session_audio_format`, then `audio_push`	Sets the audio format and pushes the audio, once.
3	`interaction_create_diarization`	Creates the diarization interaction with `max_num_speakers`.
4	`interaction_create_transcription`	Creates the transcription interaction in batch mode.
5	(wait for both results)	Both results arrive on the stream.
6	(align them in your client)	Match each word’s timing to the speaker segments.
7	`interaction_close` for each interaction, then `session_close`	Ends the interactions and the session.

Aligning speakers with words

The diarization result is a list of segments. Each segment has a speaker label (an integer), a start_time_ms, and an end_time_ms. The transcription result gives every word a start_time_ms and a duration_ms. Your client does the alignment: attribute each word to the speaker whose segment contains that word’s time. The platform does not merge the two results for you.

{
  "interaction_request": {
    "interaction_create_diarization": {
      "language": "en-US",
      "max_num_speakers": 2,
      "audio_consume_settings": {
        "audio_consume_mode": "AUDIO_CONSUME_MODE_BATCH"
      }
    }
  }
}

Results

Each result arrives in a final_result on the session response stream. Check final_result_status and status before you read the result.

Interaction	Result field	Contents
`interaction_create_nlu`	`nlu_interaction_result`	The summary, translation, language, topic, outcome, and sentiment results you enabled.
`interaction_create_language_id`	`language_id_interaction_result`	A list of detected languages, each with a `language` and a `confidence` from 0 to 1000.
`interaction_create_diarization`	`diarization_interaction_result`	A list of segments, each with a speaker `label`, a `start_time_ms`, and an `end_time_ms`.

Troubleshooting

Symptom	Likely cause and fix
The NLU result is missing a feature	Each NLU step is enabled separately. Enable the step in `nlu_settings`.
No summary is returned	Set either `summarization_bullet_points` or `summarization_words`, not both.
A translation is poor	Specify `translate_from_language`, or enable `enable_language_detect` so the source language is detected.
Sentiment is missing for a language	Sentiment analysis supports English, Spanish, and French.
The NLU interaction has no text to process	The text goes in `input_text` when you create the interaction. Create it after the transcription result is available.
Language identification returns the wrong language	Restrict `target_languages` to the languages you expect, and check the confidence values.
Diarization finds too many or too few speakers	Set `max_num_speakers` to the number you expect. Diarization expects mono audio.
Speakers do not line up with words	Compare each word’s `start_time_ms` with the speaker segments. Both are measured in milliseconds from the start of the audio.
A request times out	Check `request_timeout_ms`. The NLU default is 5000 ms; language identification and diarization default to 10,000,000 ms.

Was this article helpful?