Transcription gRPC Integration

Transcription turns speech into text without a grammar, so what a caller can say is open-ended. That makes the integration decisions different from grammar-based ASR: which mode fits your use case (recorded files, live dictation, live subtitles, or transcription combined with grammar matching), how to bias the engine toward your vocabulary, what form the output should take, and how long the audio can run. This article explains the transcription modes, the settings on a transcription interaction, and what comes back, so you can choose a mode with confidence and avoid the common integration mistakes.

Before you start. A transcription interaction uses the same session lifecycle as any other speech interaction: create a session, set the audio format, push audio, create the interaction, collect results, and close. That lifecycle is described in ASR gRPC Integration, and this article assumes it. Settings are described in the ASR & Transcription Settings Reference. For the supported languages, see Languages Available Overview.

Choose a mode

You create every transcription interaction with interaction_create_transcription. The mode comes from the audio mode you choose and a few fields on the request.

Mode	Use it for	How results arrive	How it stops
Batch	Recorded audio, such as call recordings	One final result when the audio has been processed	When the audio is processed
Streaming	A live caller who speaks one utterance	One final result when the caller pauses or stops, plus optional partial results	At end of speech (with VAD), or when you call `interaction_finalize_processing`
Continuous	Live subtitling, agent assist, and IVRs that need steady real-time text	A final result each time a pause is detected, so you receive several	When you call `interaction_finalize_processing`
Enhanced	Transcription plus slot filling and entity recognition, by adding grammars to batch or streaming mode	The transcription, with the grammar matches added to the result	As for the underlying batch or streaming mode

Two modes are easy to confuse. Streaming with partial results produces interim text every few words and can revise it as more audio arrives, which suits dictation and live displays that expect the text to settle as the caller finishes a thought. Continuous mode produces a result only when a pause is detected, and that result is final once produced, which suits subtitling, where each line is committed to the screen and should not change.

Keep transcription audio under 120 minutes per session. The platform supports a maximum of 120 minutes of transcribed audio per session, because of gRPC message size limits.

The transcription interaction

These are the fields of interaction_create_transcription. Most are optional.

Field	Purpose
`language`	The transcription language, either with a dialect such as `en-US` or dialect-independent such as `en`.
`continuous_utterance_transcription`	When true, transcription keeps recognizing, even if the caller pauses, until you stop it, and can return several final results. When false (the default), the recognizer detects a single utterance, returns one final result when the caller pauses or stops, and stops recognizing.
`phrases`	Optional phrase lists that make particular words and phrases more likely to be recognized. See Phrase lists.
`phrase_list_settings`	Optional boost settings for phrases.
`recognition_settings`	Optional recognition settings, including partial results and the number of alternatives.
`vad_settings`	Optional voice activity detection settings.
`audio_consume_settings`	Optional settings for how audio is consumed: batch or streaming mode, and where processing starts.
`normalization_settings`	Optional text normalization settings. See Text normalization.
`embedded_grammars`	Optional grammars for enhanced transcription. When a grammar matches during transcription, the semantic results of the grammar are returned too.
`embedded_grammar_settings`	Optional grammar settings for the embedded grammars.
`language_model_name`	Optional name of a language model (decoder). If you do not set it, the engine uses the default decoder for the language.
`acoustic_model_name`	Optional name of an acoustic model (encoder). If you do not set it, the engine uses the default encoder for the language.
`enable_postprocessing`	Optional custom postprocessing to enhance the decoder. If you do not set it, the engine uses its default postprocessing.
`neuron_model`	Optional name of a neuron model to apply to the final transcript. The neuron step runs only when this is set. See Speech Products: NLU, Neuron & Diarization.
`general_interaction_settings`	Optional settings that apply to every interaction type.

Batch mode

Create the session, set the audio format, and push the audio.
Create the transcription interaction with audio_consume_mode set to AUDIO_CONSUME_MODE_BATCH, plus any phrase lists or normalization settings.
Processing starts when the interaction is created. When the audio has been processed, the final result arrives on the stream.
If your client misses it, call interaction_request_results. Then close the interaction and the session.

In batch mode, the only valid stream start location is STREAM_START_LOCATION_STREAM_BEGIN.

Streaming mode

Create the session and set the audio format.
Create the transcription interaction with audio_consume_mode set to AUDIO_CONSUME_MODE_STREAMING. To receive interim text, set recognition_settings.enable_partial_results to true.
Push audio as the call produces it.
Start processing with interaction_begin_processing, or let it start automatically, depending on the stream start location.
With voice activity detection on, the platform detects the end of speech, stops processing, and returns the final result. You can call interaction_finalize_processing at any time to force it to stop and return the result.
Close the interaction and, when the call ends, the session.

WAV audio: when you use a WAV format, push the audio before you create the interaction.

Continuous mode

Set continuous_utterance_transcription to true on a streaming interaction. The recognizer keeps listening through pauses, so one interaction can cover a whole conversation.

Create the session and set the audio format.
Create the interaction with streaming mode and continuous_utterance_transcription set to true.
Push audio continuously and start processing as in streaming mode.
Each time a pause is detected, the platform returns a final result with status FINAL_RESULT_STATUS_TRANSCRIPTION_CONTINUOUS_MATCH, meaning an intermediate final result. Handle each one as it arrives.
When you want to stop, call interaction_finalize_processing. Then close the interaction and the session.

Enhanced transcription

Enhanced transcription combines open-vocabulary transcription with grammars, which lets you do slot filling and entity recognition on top of the transcript. It works in either batch or streaming mode. Supply the grammars in embedded_grammars (and optionally embedded_grammar_settings) when you create the interaction, in the same ways described for ASR grammars. When a grammar matches, its semantic interpretation is returned in grammar_results alongside the transcript.

Phrase lists

A phrase list is a set of words and phrases that the transcriber should be more likely to recognize, such as product names, commands, or domain terms. Phrase lists are optional, and you can add vocabulary the engine would not otherwise know. They can be supplied at three levels.

Level	How	When to use it
Interaction	List the phrases in the `phrases` field of `interaction_create_transcription`. Each entry holds a list of `phrases`, and can also reference a session or deployment list by label.	Most applications.
Session	Load a list once with `session_load_phrase_list` and a label, then reference it from any interaction in the session.	A list that many interactions in the same call reuse.
Deployment	Load a list with `global_load_phrase_list` on the global stream. It is then available to every session and interaction in the deployment.	A list that many sessions reuse.

Once processing has begun, the phrase lists cannot be changed for that interaction. Phrase lists are inherently language-independent, so the language on a load request only directs which language-specific resource processes the load. The label field on the load messages is named phase_list_label in the API reference. To control how strongly phrases influence recognition, see phrase_list_settings.probability_boost in the ASR & Transcription Settings Reference.

Text normalization

Text normalization post-processes the transcript so it reads as written text. There is no change to the call sequence: enable the steps you want in normalization_settings on the create request, and the output is returned with the transcription.

Setting	What it adds
`enable_inverse_text`	Inverse text normalization, which converts spoken form to written form, for example “twenty two” to “22”.
`enable_punctuation_capitalization`	Punctuation and capitalization.
`enable_redaction`	Redaction of sensitive information.
`enable_srt_generation`	A SubRip (SRT) subtitle file, returned in `srt_file`.
`enable_vtt_generation`	A WebVTT (VTT) subtitle file, returned in `vtt_file`.

The normalized text is returned in normalized_result. It contains the individual segments and four complete strings, so you can pick the form you need.

Field	Contents
`verbalized`	The output after inverse text normalization.
`verbalized_redacted`	The output after inverse text normalization and redaction.
`final`	The final output after inverse text normalization, punctuation, and capitalization.
`final_redacted`	The final output after inverse text normalization, punctuation, capitalization, and redaction.

For the concepts and the entity types that redaction covers, see Speech Products: ASR & Transcription.

Examples

A streaming transcription with a phrase list and written-form output:

{
  "interaction_request": {
    "interaction_create_transcription": {
      "language": "en-US",
      "phrases": [
        { "phrases": ["reschedule", "refund", "account balance"] }
      ],
      "normalization_settings": {
        "enable_inverse_text": true,
        "enable_punctuation_capitalization": true
      },
      "audio_consume_settings": {
        "audio_consume_mode": "AUDIO_CONSUME_MODE_STREAMING"
      }
    }
  }
}

A continuous transcription for live subtitles, with SRT output:

{
  "interaction_request": {
    "interaction_create_transcription": {
      "language": "en-US",
      "continuous_utterance_transcription": true,
      "vad_settings": {
        "use_vad": true
      },
      "normalization_settings": {
        "enable_srt_generation": true
      },
      "audio_consume_settings": {
        "audio_consume_mode": "AUDIO_CONSUME_MODE_STREAMING"
      }
    }
  }
}

Field names are shown as they appear in the API reference.

Results

Results arrive on the session response stream. A final_result carries the interaction ID, the result, and a final status. If you enabled partial results, partial_result messages arrive while the caller is still speaking.

Final status

Status	Meaning
`FINAL_RESULT_STATUS_TRANSCRIPTION_MATCH`	A transcription result was returned.
`FINAL_RESULT_STATUS_TRANSCRIPTION_CONTINUOUS_MATCH`	An intermediate final result was returned in continuous mode.
`FINAL_RESULT_STATUS_TRANSCRIPTION_GRAMMAR_MATCHES`	A transcription result was returned that contains one or more embedded grammar matches.
`FINAL_RESULT_STATUS_TRANSCRIPTION_PARTIAL_MATCH`	An enhanced transcription result was returned, but without semantic interpretation.
`FINAL_RESULT_STATUS_NO_INPUT`	No voice audio was detected. The result is empty.
`FINAL_RESULT_STATUS_ERROR`	An error occurred that stopped processing.
`FINAL_RESULT_STATUS_CANCELLED`	The interaction was cancelled or closed before results could be returned.

What a result contains

A transcription result (transcription_interaction_result) holds a list of the best matches, n_bests, and the language. The number of matches is set by recognition_settings.max_alternatives. Each match can contain:

Field	What it holds
`asr_result_meta_data`	The raw transcript: `transcript` as one string, the individual `words` with start time, duration, and confidence, and an overall `confidence` from 0 to 1000.
`normalized_result`	The normalized text, if normalization is enabled.
`grammar_results`	The grammar matches, if you used enhanced transcription.
`srt_file`	The SRT subtitle file, if SRT generation is enabled.
`vtt_file`	The VTT subtitle file, if VTT generation is enabled.
`blended_score`	The transcription quality score, if a fine-tuned model is enabled.
`neuron_result`	The neuron results, if a neuron model is set.

Troubleshooting

Symptom	Likely cause and fix
A very long recording does not process reliably	The supported maximum is 120 minutes of transcribed audio per session. Split longer audio across sessions.
No result arrives in continuous mode	Continuous mode returns a result when it detects a pause. Check that audio is flowing and review the VAD settings. Call `interaction_finalize_processing` when you want to stop.
Interim text changes, or final text never changes	Partial results can be revised as more audio arrives; continuous results are final once returned. Choose the mode that matches how your application displays text.
A phrase list has no effect	Phrase lists cannot be changed after processing has begun. Supply them when the interaction is created, or load them at the session or deployment level first.
No subtitle file in the result	Set `enable_srt_generation` or `enable_vtt_generation` in `normalization_settings`.
The transcript lacks punctuation or numerals	Text normalization is off by default. Enable the steps you need in `normalization_settings`.
The status is `TRANSCRIPTION_PARTIAL_MATCH`	An enhanced transcription returned a result with no semantic interpretation. Review the embedded grammars.
The status is `NO_INPUT`	No speech was detected in the audio that was processed. Check the audio channel and the stream start location.
Results were missed	Call `interaction_request_results` with the interaction ID.

Was this article helpful?