mmir.MediaManager
mmirf/env/media
for available plugins.
This "class" is a singleton - so that only one instance is in use.Members
-
A logger for the MediaManager and its plugins/modules.
This logger MAY be used by media-plugins and / or tools and helpers related to the MediaManager.
This logger SHOULD NOT be used by "code" that non-related to the MediaManager
- Default Value:
- mmir.Logger (logger instance for mmir.MediaManager)
-
ctxObject
-
Execution context for plugins: dictionary for non-default execution contexts (as specified via plugin configuration "ctx")
- Default Value:
- Object (empty context, i.e. plugins are loaded into the "root context", and no plugins loaded into the execution context)
-
pluginsArray.<PluginLoadConfig>
-
List of loaded media plugins
{mod:
, type: "asr" | "tts" | "audio" | "custom", config?: any, disabled?: boolean | NonFunctionalInfo} -
waitReadyImpl
-
Wait indicator, e.g. for speech input:
provides 2 functions:
preparing()
: if called, the implementation indicates that the "user should wait"
ready()
: if called, the implementation stops indicating that the "user should wait" (i.e. that the system is ready for user input now)
If not set (or functions are not available) will do nothing
- Default Value:
- Object (no implementation set)
- See:
Example
//define custom wait/ready implementation: var impl = { preparing: function(str){ console.log('Media module '+str+' is preparing...'); }, ready: function(str){ console.log('Media module '+str+' is ready now!'); } }; //configure MediaManager to use custom implementation: mmir.MediaManager.waitReadyImpl = impl; //-> now plugins that call mmir.MediaManager._preparing() and mmir.MediaManager._ready() // will invoke the custom implementation's functions.
Methods
-
protected_addListenerObserver(eventName, observerCallback)
-
Add an observer for registration / removal of event-handler. The observer gets notified,when handlers are registered / removed for the event. The observer-callback function will be called with the following arguments
(eventName, ACTION_TYPE, eventHandler)
where- eventName: String the name of the event that should be observed
- ACTION_TYPE: the type of action: "added" if the handler was registered for the event, "removed" if the the handler was removed
- eventHandler: the handler function that was registered or removed
Name Type Description eventName
String observerCallback
function -
protected_emitEvent(eventName, args)
-
Helper for firing / triggering an event. This should only be used by media plugins (that handle the eventName).
Name Type Description eventName
String args
any optional repeatable OPTIONAL the arguments (event data) with which the event-handlers will be called. - See:
Example
// will invoke listeners for "someevent": mmir.media._emitEvent('someevent'); // will invoke listeners for "otherevent" with the 2 event data arguments: mmir.media._emitEvent('otherevent', withTwo, dataParameters);
-
protected_fireEvent(eventName, argsArray)
-
Helper for firing / triggering an event. This should only be used by media plugins (that handle the eventName).
Name Type Description eventName
String argsArray
Array the list of arguments with which the event-handlers will be called. - Deprecated
- use
_emitEvent
instead!
- use
- See:
-
protected_notifyObservers(eventName, actionType, eventHandler)
-
Helper for notifying listener-observers about changes (adding/removing listeners). This should only be used by media plugins (that handle the eventName).
Name Type Description eventName
String actionType
String the change-type that occurred for the event/event-handler: one of ["added" | "removed"]
.eventHandler
function the event-handler function that has changed. -
protected_preparing(moduleName)
-
This function is called by media plugin implementations (i.e. modules) to indicate that they are preparing something and that the user should wait.
The actual implementation for
_preparing(String)
is given bywaitReadyImpl
.preparing (if not set, then calling_preparing(String)
will have no effect.Name Type Description moduleName
String the module name from which the function was invoked - See:
-
protected_ready(moduleName)
-
This function is called by media plugin implementations (i.e. modules) to indicate that they are now ready and that the user can start interacting.
The actual implementation for
_ready(String)
is given by thewaitReadyImpl
implementation (if not set, then calling_ready(String)
will have no effect.Name Type Description moduleName
String the module name from which the function was invoked - See:
-
protected_removeListenerObserver()
-
Remove an observer that gets notified on registration / removal of event-handler.
- See:
-
addListener(eventName, eventHandler)
-
Adds the handler-function for the event. This function calls
_notifyObservers
for the eventName withactionType "added"
. Event names (and firing events) are specific to the loaded media plugins. TODO list events that the default media-plugins support * "miclevelchanged": fired by AudioInput plugins that support querying the microphone (audio input) levels A plugin can tigger / fire events using the helper_emitEvent
of the MediaManager. Media plugins may observe registration / removal of listeners via_addListenerObserver
and_removeListenerObserver
. Or get and iterate over listeners viagetListeners
.Name Type Description eventName
String eventHandler
function -
cancelRecognition()
-
Cancel currently active speech recognition. Has no effect, if no recognition is active.
-
cancelSpeech()
-
Cancel current synthesis.
-
createEmptyAudio(){mmir.env.media.IAudio}
-
Get an empty audio object. This can be used as dummy or placeholder for a "real" audio object. The audio object exports the following functions:
play() stop() release() enable() disable() setVolume(number) getDuration() isPaused() isEnabled()
Note:enable()
anddisable()
will set the internal enabled-state, which can be queried viaisEnabled()
.play()
andstop()
will set the internal playing-state, which can be queried viaisPaused()
(note however, that this empty audio does not actually play anything.setVolume()
sets the internal volume-value.getDuration()
will always return0
.Returns:
Type Description mmir.env.media.IAudio the audio -
destroyRecognition(successCallback, failureCallback)
-
Destroy the speech recognition instance and free up system resources. NOTE: may not be supported by all recognition implementations (e.g. if the impl. does not block system resources etc). NOTE: If it is not supported,
successCallback(false)
is triggered. IMPORTANT: pluins that support destroyRecognition() should also support initializeRecognition().Name Type Description successCallback
function optional callback in case of success: successCallback(didDestroy: boolean)
in case, the plugin does not support destroyRecognition(),successCallback(false)
will be invokedfailureCallback
function optional callback that will be invoked in case of an error: failureCallback(error)
-
destroySpeech(successCallback, failureCallback)
-
Destroy the speech synthesizer instance and free up system resources. NOTE: may not be supported by all synthesizer implementations (e.g. if the impl. does not block system resources etc). NOTE: If it is not supported,
successCallback(false)
is triggered. IMPORTANT: pluins that support destroySpeech() should also support initializeSpeech().Name Type Description successCallback
function optional callback in case of success: successCallback(didDestroy: boolean)
in case, the plugin does not support destroySpeech(),successCallback(false)
will be invokedfailureCallback
function optional callback that will be invoked in case of an error: failureCallback(error)
- See:
-
getAudio()
-
Get an audio object for the audio file specified by URL URL or by WAV data. NOTE that getWAVAsAudio may not be supported by all modules! Convenience function for
getURLAsAudio
andgetWAVAsAudio
: if first argument is a String, thengetURLAsAudio
will be invoked, otherwisegetWAVAsAudio
(if the module supports this function). -
getFunc(ctx, funcName)
-
Returns function
funcName
from "sub-module"ctx
.If there is no
funcName
in "sub-module"ctx
, thenfuncName
from the "main-module" (i.e. from the MediaManager instance itself) will be returned.NOTE that the returned functions will always execute within the context of the MediaManager instance (i.e.
this
will refer to the MediaManager instance).Name Type Description ctx
String the execution context, i.e. "sub-module", in which to execute funcName.
Iffalsy
, the "root-module" will used as execution context.funcName
String the function name Throws:
-
if
funcName
does not exist in the requested Execution context.
Or ifctx
is notfalsy
but there is no valid execution contextctx
in MediaManager. - Type
- ReferenceError
Example
//same as mmir.MediaManager.ctx.android.textToSpeech("...", function...): mmir.MediaManager.getFunc("android", "textToSpeech")("some text to read out loud", function onFinished(){ console.log("finished reading."); } ); //same as mmir.MediaManager.textToSpeech("...", function...): //... IF the defaultExecId is falsy // (i.e. un-changed or set to falsy value via setDefaultExec()) mmir.MediaManager.getFunc(null, "textToSpeech")("some text to read out loud", function onFinished(){ console.log("finished reading."); } );
-
-
getListeners(){Array.<function()>}
-
Get list of registered listeners / handlers for an event.
Returns:
Type Description Array.<function()> of event-handlers. Empty, if there are no event handlers for eventName -
getRecognitionLanguages()
-
get list of supported languages for ASR (may not be supported by all plugins).
-
getSpeechLanguages()
-
get list of supported languages for TTS (may not be supported by all plugins).
-
getURLAsAudio(url, onPlayedCallback, failureCallback, onLoadedCallback){mmir.env.media.IAudio}
-
Get an audio object for the audio file specified by URL. The audio object exports the following functions:
play() stop() release() enable() disable() setVolume(number) getDuration() isPaused() isEnabled()
NOTE: the audio object should only be used, after theonLoadedCallback
was triggered.Name Type Description url
String onPlayedCallback
function optional OPTIONAL failureCallback
function optional OPTIONAL onLoadedCallback
function optional OPTIONAL Returns:
Type Description mmir.env.media.IAudio the audio -
getVoices(options, successCallback, failureCallback)
-
get list of supported voices for TTS (may not be supported by all plugins).
Name Type Description options
String | VoiceOptions optional OPTIONAL if String, the language code (optionally with country code) for which the voices should be listed. if VoiceOptions: options.language: {String} OPTIONAL the language code options.details: {Boolean} OPTIONAL if TRUE the returned list contains VoiceDetail objects with {name: STRING, language: STRING, gender: "female" | "male" | "unknown"} successCallback
function the success callback: successCallback(Array ) failureCallback
function the error callback: failureCallback(err) -
hasListeners(){Boolean}
-
Check if at least one listener / handler is registered for the event.
Returns:
Type Description Boolean true
if at least 1 handler is registered for eventName; otherwisefalse
. -
asyncinit(successCallback, failureCallback, listenerList){Object}
-
Object containing the instance of the class {{#crossLink "audioInput"}}{{/crossLink}} If listenerList is provided, each listener will be registered after the instance is initialized, but before media-plugins (i.e. environment specfific implementations) are loaded. Each entry in the listenerList must have fields name (String) and listener (Function), where
name: is the name of the event
listener: is the listener implementation (the signature/arguments of the listener function depends on the specific event for which the listener will be registered)Name Type Description successCallback
function optional OPTIONAL callback that gets triggered after the MediaManager instance has been initialized. failureCallback
function optional OPTIONAL a failure callback that gets triggered if an error occurs during initialization. listenerList
Array.<Object> optional OPTIONAL a list of listeners that should be registered, where each entry is an Object with properties: { name: String the event name, listener: Function the handler function }
Returns:
Type Description Object a Deferred object that gets resolved, after the mmir.MediaManager
has been initialized. -
initializeRecognition(successCallback, failureCallback)
-
Re-initialize the speech recognition instance: should be called after invoking
destroyRecognition()
(and its success-callback returned
true
) before continuing to use the recognition instance. NOTE: may not be supported by all recognition implementations. NOTE: If it is not supported,successCallback(false)
is triggered. IMPORTANT: pluins that support initializeRecognition() should also support destroyRecognition().Name Type Description successCallback
function optional callback in case of success: successCallback(didDestroy: boolean)
in case, the plugin does not support initializeRecognition(),successCallback(false)
will be invokedfailureCallback
function optional callback that will be invoked in case of an error: failureCallback(error)
- See:
-
destroyRecognitio
-
initializeSpeech(successCallback, failureCallback)
-
Re-initialize the speech synthesizer instance: should be called after invoking
destroySpeech()
(and its success-callback returned
true
) before continuing to use the synthesizer instance. NOTE: may not be supported by all synthesizer implementations. NOTE: If it is not supported,successCallback(false)
is triggered. IMPORTANT: pluins that support initializeSpeech() should also support destroySpeech().Name Type Description successCallback
function optional callback in case of success: successCallback(didDestroy: boolean)
in case, the plugin does not support initializeSpeech(),successCallback(false)
will be invokedfailureCallback
function optional callback that will be invoked in case of an error: failureCallback(error)
- See:
-
protectedloadPlugin()
-
loads a file. If the file implements a function initialize(f) where the function f is called with a set of functions e, then those functions in e are added to the visibility of audioInput, and will from now on be applicable by calling mmir.MediaManager.
(). Example
NOTE should only be used by plugin implementations for loading (dependent/sub-) plugins.
-
off(eventName, eventHandler)
-
Add an event listener.
Name Type Description eventName
String the name of the event eventHandler
function the event handler / callback function - See:
-
on(eventName, eventHandler)
-
Add an event listener.
Name Type Description eventName
String the name of the event eventHandler
function the event handler / callback function - See:
-
perform(ctx, funcName, args)
-
Executes function
funcName
in "sub-module"ctx
with argumentsargs
.If there is no
funcName
in "sub-module"ctx
, thenfuncName
from the "main-module" (i.e. from the MediaManager instance itself) will be used.Name Type Description ctx
String the execution context, i.e. "sub-module", in which to execute funcName.
Iffalsy
, the "root-module" will used as execution context.funcName
String the function name args
Array the arguments for function "packaged" in an array Throws:
-
if
funcName
does not exist in the requested Execution context.
Or ifctx
is notfalsy
but there is no valid execution contextctx
in MediaManager. - Type
- ReferenceError
Example
//same as mmir.MediaManager.ctx.android.textToSpeech("...", function...): mmir.MediaManager.perform("android", "textToSpeech", ["some text to read out loud", function onFinished(){ console.log("finished reading."); } ]); //same as mmir.MediaManager.textToSpeech("...", function...) //... IF the defaultExecId is falsy // (i.e. un-changed or set to falsy value via setDefaultExec()) mmir.MediaManager.perform(null, "textToSpeech", ["some text to read out loud", function onFinished(){ console.log("finished reading."); } ]);
-
-
play()
-
Play audio file from the specified URL or WAV data. Convenience function for
playWAV
andplayURL
: if first argument is a String, thenplayURL
will be invoked, otherwiseplayWAV
. -
playURL()
-
Play audio file from the specified URL.
-
playWAV()
-
Play PCM audio data.
-
asyncrecognize(options, statusCallback, failureCallback)
-
Start speech recognition with end-of-speech detection: the recognizer automatically tries to detect when speech has finished and triggers the status-callback accordingly with results.
Name Type Description options
PlainObject optional OPTIONAL options for Automatic Speech Recognition: { success: OPTIONAL Function, the status-callback (see arg statusCallback) , error: OPTIONAL Function, the error callback (see arg failureCallback) , language: OPTIONAL String, the language for recognition (if omitted, the current language setting is used) , intermediate: OTPIONAL Boolean, set true for receiving intermediate results (NOTE not all ASR engines may support intermediate results) , results: OTPIONAL Number, set how many recognition alternatives should be returned at most (NOTE not all ASR engines may support this option) , mode: OTPIONAL "search" | "dictation", set how many recognition alternatives should be returned at most (NOTE not all ASR engines may support this option) , eosPause: OTPIONAL "short" | "long", length of pause after speech for end-of-speech detection (NOTE not all ASR engines may support this option) , disableImprovedFeedback: OTPIONAL Boolean, disable improved feedback when using intermediate results (NOTE not all ASR engines may support this option) }
statusCallback
function optional OPTIONAL callback function that is triggered when, recognition starts, text results become available, and recognition ends. The callback signature is: callback( text: String | "", confidence: Number | Void, status: "FINAL"|"INTERIM"|"INTERMEDIATE"|"RECORDING_BEGIN"|"RECORDING_DONE", alternatives: Array<{result: String, score: Number}> | Void, unstable: String | Void, custom: any | Void )
Usually, for status"FINAL" | "INTERIM" | "INTERMEDIATE"
text results are returned, where"INTERIM": an interim result, that might still change "INTERMEDIATE": a stable, intermediate result "FINAL": a (stable) final result, before the recognition stops
If present, theunstable
argument provides a preview for the currently processed / recognized text. Thecustom
argument is dependent on the ASR engine / plugin: specific implementations may return some custom results.
NOTE that when usingintermediate
mode, status-calls with"INTERMEDIATE"
may contain "final intermediate" results, too.
NOTE: if used in combination withoptions.success
, this argument will supersede the optionsfailureCallback
function optional OPTIONAL callback function that is triggered when an error occurred. The callback signature is: callback(error)
NOTE: if used in combination withoptions.error
, this argument will supersede the options -
removeListener(eventName, eventHandler){Boolean}
-
Removes the handler-function for the event. Calls
_notifyObservers
for the eventName withactionType "removed"
, if the handler was actually removed.Name Type Description eventName
String eventHandler
function Returns:
Type Description Boolean true
if the handler function was actually removed, andfalse
otherwise. -
setDefaultCtx(ctxId)
-
Set the default execution context. If not explicitly set, or set to a
falsy
value, then the "root" execution context is the default context.Name Type Description ctxId
String the new default excution context for loaded media modules (if falsy
the default context will be the "root context")Throws:
-
if
ctxId
is no valid context - Type
- ReferenceError
Example
//if context "nuance" exists: mmir.MediaManager.setDefaultCtx("nuance") // -> now the following calls are equal to mmir.MediaManager.ctx.nuance.textToSpeech("some text") mmir.MediaManager.perform(null, "textToSpeech", ["some text"]); mmir.MediaManager.getFunc(null, "textToSpeech")("some text"); //reset to root context: mmir.MediaManager.setDefaultCtx(false); // -> now the following call is equal to mmir.MediaManager.textToSpeech("some text") again mmir.MediaManager.perform("textToSpeech", ["some text"]);
-
-
setTextToSpeechVolume(newValue)
-
Set the volume for the speech synthesis (text-to-speech).
Name Type Description newValue
Number TODO specify format / range -
asyncstartRecord(options, statusCallback, failureCallback)
-
Start continuous speech recognition: The recognizer continues until
stopRecord
is called.Name Type Description options
PlainObject optional OPTIONAL options for Automatic Speech Recognition: { success: OPTIONAL Function, the status-callback (see arg statusCallback) , error: OPTIONAL Function, the error callback (see arg failureCallback) , language: OPTIONAL String, the language for recognition (if omitted, the current language setting is used) , intermediate: OTPIONAL Boolean, set true for receiving intermediate results (NOTE not all ASR engines may support intermediate results) , results: OTPIONAL Number, set how many recognition alternatives should be returned at most (NOTE not all ASR engines may support this option) , mode: OTPIONAL "search" | "dictation", set how many recognition alternatives should be returned at most (NOTE not all ASR engines may support this option) , eosPause: OTPIONAL "short" | "long", length of pause after speech for end-of-speech detection (NOTE not all ASR engines may support this option) , disableImprovedFeedback: OTPIONAL Boolean, disable improved feedback when using intermediate results (NOTE not all ASR engines may support this option) }
statusCallback
function optional OPTIONAL callback function that is triggered when, recognition starts, text results become available, and recognition ends. The callback signature is: callback( text: String | "", confidence: Number | Void, status: "FINAL"|"INTERIM"|"INTERMEDIATE"|"RECORDING_BEGIN"|"RECORDING_DONE", alternatives: Array<{result: String, score: Number}> | Void, unstable: String | Void, custom: any | Void )
Usually, for status"FINAL" | "INTERIM" | "INTERMEDIATE"
text results are returned, where"INTERIM": an interim result, that might still change "INTERMEDIATE": a stable, intermediate result "FINAL": a (stable) final result, before the recognition stops
If present, theunstable
argument provides a preview for the currently processed / recognized text. Thecustom
argument is dependent on the ASR engine / plugin: specific implementations may return some custom results.
NOTE that when usingintermediate
mode, status-calls with"INTERMEDIATE"
may contain "final intermediate" results, too.
NOTE: if used in combination withoptions.success
, this argument will supersede the optionsfailureCallback
function optional OPTIONAL callback function that is triggered when an error occurred. The callback signature is: callback(error)
NOTE: if used in combination withoptions.error
, this argument will supersede the options- See:
-
asyncstopRecord(options, statusCallback, failureCallback)
-
Stops continuous speech recognition: After
startRecord
was called, invoking this function will stop the recognition process and return the result by invoking thesuccesCallback
. Note, that thestatusCallback
may not return an actual text result (i.e. the last text result may have been return in thestatusCallback
of thestartRecord()
call)Name Type Description options
PlainObject optional OPTIONAL options for stopping the Automatic Speech Recognition: { success: OPTIONAL Function, the status-callback (see arg statusCallback) , error: OPTIONAL Function, the error callback (see arg failureCallback) }
statusCallback
function optional OPTIONAL callback function that is triggered when, recognition starts, text results become available, and recognition ends. The callback signature is: callback( text: String | "", confidence: Number | Void, status: "FINAL"|"INTERIM"|"INTERMEDIATE"|"RECORDING_BEGIN"|"RECORDING_DONE", alternatives: Array<{result: String, score: Number}> | Void, unstable: String | Void, custom: any | Void )
Usually, for status"FINAL" | "INTERIM" | "INTERMEDIATE"
text results are returned, where"INTERIM": an interim result, that might still change "INTERMEDIATE": a stable, intermediate result "FINAL": a (stable) final result, before the recognition stops
If present, theunstable
argument provides a preview for the currently processed / recognized text. Thecustom
argument is dependent on the ASR engine / plugin: specific implementations may return some custom results.
NOTE that when usingintermediate
mode (as option instartRecord()
), status-calls with"INTERMEDIATE"
may contain "final intermediate" results, too.failureCallback
function optional OPTIONAL callback function that is triggered when an error occurred. The callback signature is: callback(error)
- See:
-
textToSpeech()
-
- Deprecated
- use
tts
instead
- use
-
tts(options, onPlayedCallback, failureCallback, onReadyCallback)
-
Synthesizes ("read out loud") text.
Name Type Description options
String | Array.<String> | PlainObject optional OPTIONAL if String
orArray
ofString
s synthesizes the text of the String(s).
For an Array: each entry is interpreted as "sentence"; after each sentence, a short pause is inserted before synthesizing the the next sentence
for aPlainObject
, the following properties should be used:{ text: String | String[], text that should be read aloud , pauseDuration: OPTIONAL Number, the length of the pauses between sentences (i.e. for String Arrays) in milliseconds , language: OPTIONAL String, the language for synthesis (if omitted, the current language setting is used) , voice: OPTIONAL String, the voice (language specific) for synthesis; NOTE that the specific available voices depend on the TTS engine , success: OPTIONAL Function, the on-playing-completed callback (see arg onPlayedCallback) , error: OPTIONAL Function, the error callback (see arg failureCallback) , ready: OPTIONAL Function, the audio-ready callback (see arg onReadyCallback) }
onPlayedCallback
function optional OPTIONAL callback that is invoked when the audio of the speech synthesis finished playing: onPlayedCallback()
NOTE: if used in combination withoptions.success
, this argument will supersede the optionsfailureCallback
function optional OPTIONAL callback that is invoked in case an error occurred: failureCallback(error: String | Error)
NOTE: if used in combination withoptions.error
, this argument will supersede the optionsonReadyCallback
function optional OPTIONAL callback that is invoked when audio becomes ready / is starting to play. If, after the first invocation, audio is paused due to preparing the next audio, then the callback will be invoked with false
, and then withtrue
(as first argument), when the audio becomes ready again, i.e. the callback signature is:onReadyCallback(isReady: Boolean, audio: IAudio)
NOTE: if used in combination withoptions.ready
, this argument will supersede the options