Asterisk PHP generic speech interface

The Asterisk AGI/PHP Speech Interface is a PHP library that exposes automatic speech recognition (ASR) and text-to-speech (TTS) functionality to applications running on Asterisk, the open-source PBX. It is designed for developers who want to build speech applications in PHP — invoked over the Asterisk Gateway Interface (AGI) — rather than expressing complex speech logic directly in dialplan.

Download the library: speech_interface.zip

Background

Asterisk has several ways to interact with speech servers, but the preferred mechanism is a community-built, open-source set of modules called UniMRCP. These modules provide Asterisk dialplan applications that expose speech functionality over the Media Resource Control Protocol (MRCP), including:

• MRCPRecog — perform speech recognition.
• MRCPSynth — perform speech synthesis.
• SynthAndRecog — perform TTS and ASR at the same time.

As a contributor to the UniMRCP project, Capacity Private Cloud fully supports working with these modules; see Installing and Configuring UniMRCP Modules for setup instructions.

While these dialplan applications are powerful, building entire speech applications in dialplan becomes problematic as complexity grows. Recognition works cleanly in dialplan as long as the questions asked of callers are simple. Asking whether a caller said yes or no might look like this:

This is a simple matter of calling MRCPRecog, checking the RECOG_INSTANCE variable for the first answer (denoted 0/0), and acting on it. However, speech results can be complex objects represented as XML strings. Consider an application that returns a list of names matched against a database, with multiple results ("Did you want to talk to Joe Smith or Joe Jones?") and extra data such as each person's extension. Manipulating such a return in dialplan is clunky — you must iterate through the results and often hand the XML off to an external process to parse, which dialplan does not do well.

This interface solves that problem. It is meant to be called from scripts invoked over the Asterisk Gateway Interface (AGI), and it provides methods for common speech functions — including handling complex returns — while abstracting the more idiosyncratic dialplan mechanics into patterns familiar to any PHP developer.

Goals and Caveats

The interface was created first to provide example applications demonstrating complex speech concepts on Asterisk. In building them, it became clear that abstracting common speech functionality into a few methods kept the sample applications clean and readable — and gave any PHP-comfortable Asterisk developer an easy-to-use starting point. The interface aims to:

• Abstract the trickier parts of recognition, such as navigating an NLSML return, so developers can build applications without first learning a stack of standards.
• Provide intuitive PHP methods to change speech settings, specify grammars, and so on, without learning the quirks of the dialplan applications.
• Be easy to read, understand, and extend.

Be aware of the following caveats:

• The interface has only been tested with Capacity Private Cloud. Because vendors implement the NLSML specification differently, the parse_nlsml method is unlikely to behave as expected with other speech vendors.
• It has been tested against only a few sample applications. As with any new software, bugs are likely; bug reports and patches are welcome.
• It does not implement every useful piece of speech functionality. A language like VoiceXML is a more feature-complete, abstract interface for speech applications. This interface is designed to cover the 80–90% of needs that most developers have, prioritizing ease of use.
• It does not use the PHPAGI library, so it may duplicate some of that library's functionality.

Using the Speech Interface

The interface is called from PHP applications invoked with Asterisk's AGI command. Familiarity with basic Asterisk dialplan, AGI, and PHP development is assumed.

Include the speech_interface.php file in your PHP/AGI scripts to gain access to the classes and methods described below. The general pattern for performing ASR/TTS is:

1. Create a new object from the speech class.
2. Use the class's get and set methods to set any parameters.
3. Call one of the three primary methods to perform ASR, TTS, or both.
4. Check the result:
   • If you performed TTS, the result is a string indicating success or failure.
   • If you performed ASR, the result is a speech_return object.
5. Repeat steps 2–4 as needed.

The speech Class

The speech class exposes all of the methods needed to perform ASR and TTS. Creating a new speech object is the starting point for every speech application.

mrcp_synth(text)

Performs synthesis on text. It takes one required parameter:

• text — a string containing the text to synthesize. This may be plaintext, an SSML document, or the URI to an SSML document. Use URIs whenever possible.

Returns a string indicating one of three results:

• OK — synthesis performed as expected.
• ERROR — an error occurred during synthesis.
• INTERRUPTED — the caller hung up while synthesis was in progress.

mrcp_recog(grammar)

Performs speech recognition against the supplied grammar. It takes one required parameter:

• grammar — a string containing the recognition grammar to use. The string may contain the grammar itself, or the URI to a grammar document. Use URIs whenever possible. Multiple grammars can be sent by delimiting them with a comma.

Returns a speech_return object.

mrcp_synth_and_recog(text, grammar)

Performs synthesis while doing ASR. It takes two required parameters:

• text — a string containing the text to synthesize (plaintext, an SSML document, or a URI to one). Use URIs whenever possible.
• grammar— a string containing the recognition grammar (the grammar itself, or a URI to a grammar document). Use URIs whenever possible.
  • Multiple grammars can be sent by delimiting them with a comma. When sending multiple grammars, a backslash and double quote are required before and after.
  • Example: '\"http://example.com/grammar1.grxml,http://example.com/grammar2.grxml\"'

Returns a speech_return object.

check_match(speech_return)

Checks a speech_return object for a match. Returns a Boolean indicating whether there was at least one match. This method also calls no_input() or no_match() when the result is a no-input or no-match. It is a convenience helper — not required, but it simplifies a common task.

check_ambiguity(speech_return)

Checks a speech_return object for ambiguity. Returns a Boolean indicating whether the result is ambiguous. Ambiguous results are not always possible, but when they occur they must be handled carefully. This is an advanced method that most applications will not need.

ask_yes_or_no(question)

Asks the user a yes-or-no question (synthesizing the text supplied in question) and returns a Boolean: true for yes, false for no.

no_input()

Plays a synthesized message — by default, "I'm sorry, but I did not hear what you said."

no_match()

Plays a synthesized message — by default, "I'm sorry, but I did not understand what you said."

Setting Parameters

A number of parameters can be set on a per-interaction basis (defaults are controlled in configuration files). The speech class exposes a get and a set method for each.

General Settings

• mrcp_profile — the name of the profile to use in mrcp.conf. Affects mrcp_synth(), mrcp_recog(), and mrcp_synth_and_recog().

ASR Settings

These settings affect the mrcp_recog() and mrcp_synth_and_recog() methods.

• confidence_threshold — the confidence threshold for a recognition, on a 0.0–1.0 scale. If the confidence falls below this value, the recognizer returns a NO-MATCH instead of a result. Default: 0.5
• barge_in — whether barge-in is disabled (0), enabled (1), or performed by Asterisk (2). Default: 1. We strongly recommend using only 0 or 1.
• sensitivity — barge-in sensitivity, on a 0.0–1.0 scale. Higher values make it easier for callers to barge in. Default: 0.5
• speed_vs_accuracy — whether the ASR favors speed or accuracy, on a 0.0–1.0 scale. Higher values favor speed over accuracy. Default: 0.5
• max_nbest — the maximum number of results the recognizer should return. Increasing this provides matches beyond the single best, useful where complex disambiguation is required. Default: 1
• recognition_timeout — once barge-in is detected, how long the caller has to speak, in milliseconds, before the recognizer returns a timeout event. Default: 15000
• no_input_timeout — how long the caller has to start speaking, in milliseconds, before the recognizer returns a no-input-timeout event.
• speech_complete_timeout — how long the caller can pause before the recognizer treats the audio as a complete utterance, in milliseconds. Default: 800
• dtmf_interdigit_timeout — when entering DTMF, how long the recognizer waits between key presses before processing the result, in milliseconds. Default: 5000
• dtmf_terminate_timeout — the total time the user has to enter DTMF, in milliseconds. Default: 10000

TTS Settings

These settings apply to mrcp_synth() and mrcp_synth_and_recog().

• synth_language — the language for synthesis (can be overridden by SSML documents), in standard language-COUNTRY format, e.g. en-US or es-MX. Default: en-US
• voice_name — the voice name for synthesis (can be overridden by SSML documents). Default: null; the platform auto-selects a voice.
• voice_gender — the gender for synthesis, "male" or "female" (can be overridden by SSML documents). Default: null; the platform auto-selects the gender.
• prosody_volume — the synthesis volume. Valid settings: silent, x-soft, soft, medium, loud, x-loud, default. Default: medium
• prosody_rate — how quickly the synthesizer speaks. Valid settings: x-slow, slow, medium, fast, x-fast, default. Default: medium

The speech_return Object

A speech_return represents the return from an ASR interaction. It provides no methods, but exposes several properties you can examine to get the status, result, and answer(s) of an ASR request.

recognition_status

A string indicating whether the recognition worked (OK), failed (ERROR), or the caller hung up during the interaction (INTERRUPTED).

recognition_result

A string indicating whether there was input from the caller (000), a no-match (001), a no-input (002), or a timeout (003). This value is what speech->check_match() checks.

mode

A string indicating whether the user responded by speaking (voice) or by keypress (dtmf).

nlsml_string

A string containing the complete NLSML-formatted XML returned by the recognizer over MRCP. In general you should not parse this directly — the answers array contains a pre-parsed version. It is provided in case a developer needs it for another purpose. (For details on NLSML, see NLSML.)

answers

An array of speech_answer objects containing the actual results of the interaction. Assuming there was a valid recognition, each element is a speech_answer with the following properties:

• confidence — an integer confidence score on the 0–100 scale.
• grammar — a string identifying the grammar that was matched.
• input — a string containing the text that was recognized.
• interpretation — an array of one or more semantic interpretations for the input. Each semantic interpretation is a separate element.

Understanding Ambiguity

Developers new to speech recognition may find the distinction between the answers array and the interpretation array confusing at first, but it is a necessary one that makes ambiguous results easier to understand.

Each element in answers corresponds to one n-best result from the recognizer. Since the default n-best value is 1, in most cases answers has a single element. Each element in interpretation corresponds to a possible semantic meaning for that answer; in most cases there is only one.

For example, if a caller says "Yeah" using the built-in Boolean grammar with the default n-best of 1, there is one answer, and the built-in grammar allows only one interpretation, "true":

This is easy to check in PHP:

For the vast majority of interactions, it is safe — and good design — to assume a single answer with a single interpretation.

Some results, however, are ambiguous. Imagine a directory with two people named "John." If the caller just says "John," there is one answer with two interpretations:

Here the application should check the length of the interpretation array (e.g. with sizeof()) and behave accordingly.

You can also get multiple n-best answers where phonetic similarity is expected:

Here there are two separate answers, each with one interpretation. And you can of course get returns with multiple answers and multiple interpretations:

Dealing with ambiguity is a complex topic, addressed further in the example PHP applications. The key points:

• Most applications never need to worry about it — they only check answers[0]->interpretation[0].
• If your application or grammar returns ambiguous results, make sure that is intended. Many new developers unintentionally write grammars that allow multiple parses. Use the Grammar Tester in the Deployment Portal to verify your grammars are not accidentally doing so.
• If you do intend to allow ambiguous results, have a strategy for handling them — commonly iterating through the answers/interpretation arrays with a foreach, or passing the object to an external process to disambiguate.

Multi-Slot Returns

Beyond ambiguity, an interpretation can be a "multi-slot" return. In the examples above the result was always a string such as "John Smith" or "New York" — single-slot returns. A multi-slot return carries more than one piece of information. If we expand the directory application to return a name and an extension, we might get:

Here an interpretation element is itself an object with properties holding the relevant information. The speech class handles this automatically as part of its NLSML parsing. These complex returns are controlled by the grammar (see SISR Basics); most grammars return simple strings, but if you write a grammar that returns a multi-slot result, your code must account for it — you cannot always assume interpretation[n] is a string.

Example Applications

The following sample applications and the library itself are available for download. They demonstrate the concepts described above, including handling complex and ambiguous returns.

Related Articles

• Building Asterisk Speech Applications with AGI
• Installing and Configuring UniMRCP Modules
• MRCPRecog() · MRCPSynth() · SynthAndRecog()
• NLSML
• SISR Basics · Intro to Semantic Interpretation