ASR gRPC Integration

Integrating speech recognition directly over gRPC gives you control that a telephony bridge hides: you decide when the platform starts counting audio, when to stop playing a prompt because the caller started speaking, how long to wait for an answer, and what to do when nothing matches. That control comes with a sequence you have to get right. This article explains the session and interaction model, the two ways to supply audio, how grammars are loaded, what the platform sends back, and the mistakes that cost integrators the most time. It covers grammar-based ASR, where the caller’s speech is matched against grammars and returned as a structured result.

Scope. Free-form transcription, NLU, language identification, and diarization use the same session lifecycle but different interactions, and are covered in their own articles. For the message and field definitions, see the gRPC API reference. If you connect through a telephony platform over MRCP instead, see the Media Server integration articles. Settings are described in the ASR & Transcription Settings Reference.

Sessions and interactions

Two objects organize everything you send to the platform.

• A session represents one conversation, typically one phone call. It holds the audio you stream in and the settings that apply to the work done on that audio. You create it once and close it when the call ends.
• An interaction is one unit of work against the session’s audio, such as one ASR recognition. A session can run many interactions, one after another or in parallel. Each interaction is a child of its session, so it inherits the session’s settings at the moment it is created.

You send requests and receive results over a bidirectional stream. Requests are wrapped in a session_request, audio_request, interaction_request, or dtmf_request message, and everything the platform returns arrives as a response on the same stream. Each request can carry an optional correlation_id so you can match responses to requests; the platform generates one if you do not.

The call sequence

Every ASR integration follows the same sequence. The differences between batch and streaming mode are in the audio and processing steps and are explained in the sections that follow.

Supplying audio

Audio is pushed into the session with audio_push as raw bytes in the format you declared with session_audio_format.

Push WAV audio before you create the interaction. When you use a WAV format, the audio must already be in the session when you create the interaction.

For multi-channel audio, choose the channel to recognize with audio_consume_settings.audio_channel. Channel 0 is used by default.

Choosing a mode: batch or streaming

You select the mode with audio_consume_settings.audio_consume_mode on the interaction. Streaming is the default.

Batch mode

1. Create the session, set the audio format, and push the whole audio file.
2. Create the ASR interaction with audio_consume_mode set to AUDIO_CONSUME_MODE_BATCH and your grammars.
3. The platform processes the audio as soon as the interaction is created, and returns the final result on the stream.
4. Close the interaction, and close the session when you are done.

In batch mode, the only valid stream start location is STREAM_START_LOCATION_STREAM_BEGIN.

Streaming mode

1. Create the session and set the audio format.
2. Create the ASR interaction with your grammars and, if you want it, voice activity detection settings.
3. Push audio continuously as the call produces it.
4. Start processing with interaction_begin_processing, or let it start automatically, as described below.
5. Watch the stream for voice activity events and results. With VAD enabled, the platform detects the end of speech, stops processing, and returns the final result.
6. Close the interaction and, when the call ends, the session.

Where processing starts

In streaming mode, stream_start_location decides which audio the interaction processes.

Choose BEGIN_PROCESSING_CALL when you want to control the start yourself, for example to ignore audio from a prompt that is still playing.

Ending processing: VAD or finalize

Voice activity detection (VAD) is controlled by vad_settings.use_vad.

• When use_vad is true, VAD detects when the caller starts and stops speaking, and processing stops at the end of speech. In batch mode, the engine looks for the start of speech within the audio and stops at end of speech, so not all loaded audio may be processed.
• When use_vad is false, all audio selected by the audio consume settings is processed. In streaming mode you must then call interaction_finalize_processing to finish.
• With VAD on, you can still call interaction_finalize_processing at any time to force processing to stop and return the result.

Voice activity events

While VAD runs, the platform sends vad_event messages that tell your application what is happening in the audio. Each carries the interaction ID and the offset in milliseconds from the start of the audio.

The timing settings that drive these events, barge_in_timeout_ms, end_of_speech_timeout_ms, and eos_delay_ms, are in the ASR & Transcription Settings Reference.

DTMF input

To include keypad input in the recognition, send each key press to the interaction with a dtmf_request (a DtmfPushRequest) that names the interaction_id and the dtmf_key. Valid keys are 0 to 9, A to F, *, and #.

Grammars

An ASR interaction needs at least one grammar. Grammars tell the recognizer what to listen for and how to turn what it hears into a structured result. To learn how to write them, see A SRGS Introduction and Intro to Semantic Interpretation.

Where grammars are loaded

Loading grammars at the deployment (global) level is not available, and the global_grammar_label field is deprecated. Once processing has begun, the grammar list cannot be changed for that interaction.

Ways to supply a grammar

Each entry in grammars is one of the following.

Example: load once, reuse in an interaction

First, load the grammar at the session level:

Then create a streaming ASR interaction that uses it, with VAD enabled:

Field names are shown as they appear in the API reference. The grammar label must be one you loaded with session_load_grammar, and a label can contain only letters, digits, hyphens, and underscores. The language selects the language for the grammars, either with a dialect such as en-US or dialect-independent such as en.

Results

Results arrive on the session response stream. The messages you handle in an ASR integration are:

Final status

Check the final status before you read the result. These are the statuses an ASR interaction returns.

What a result contains

An ASR result (asr_interaction_result) holds a list of the best matches, n_bests, plus the input_mode (for example speech or DTMF) and the language. Each match contains:

The number of matches returned is set by recognition_settings.max_alternatives, which is 1 by default. Confidence values are integers from 0 to 1000.

If your client misses a result on the response stream, call interaction_request_results with the interaction ID to retrieve it.

Settings you can supply

The ASR create request accepts these settings groups: grammar_settings, recognition_settings, vad_settings, audio_consume_settings, and general_interaction_settings. You can also set defaults at the session or deployment level, and an interaction takes a snapshot of its settings when it is created. See Speech Settings: Hierarchy, Inheritance and Value Types for how that works, and the ASR & Transcription Settings Reference for every setting.

Troubleshooting

Related Articles

• ASR & Transcription Functional Overview & API Integration
• Speech Products: ASR & Transcription
• A SRGS Introduction
• Intro to Semantic Interpretation
• Builtin Grammars
• HTTP Caching
• Meta Data Within Grammars
• Portal Help Guide