Low-latency audio streams for Mob apps, with the DSP in the app's own C.
The plugin owns the platform's audio streams: AAudio on Android, a RemoteIO
(or VoiceProcessingIO) unit and the AVAudioSession on iOS. An app's
static NIF registers a processor through the C interface in
priv/include/mob_audio_io.h (mai_register/3), and the plugin calls it
from the audio thread: render for output, capture for input, both (in
that order, in one callback) for duplex. Audio never passes through the
BEAM; Elixir opens and closes streams, reads status/0, lists devices,
decodes files (in chunks, cancellably) and receives the stream's messages.
Platforms
| Android (API 28+) | iOS (15+) | Host (mix test, dev) | |
|---|---|---|---|
:output | AAudio, low latency, shared or exclusive | RemoteIO | {:error, :unsupported} |
:input | AAudio with an input preset | RemoteIO input (VoiceProcessingIO for preset: :voice) | {:error, :unsupported} |
:duplex | AAudio output, the input read in its callback | one RemoteIO unit, input pulled in the render | {:error, :unsupported} |
devices/1 | AudioManager.getDevices (JNI) | AVAudioSession inputs, the current route's outputs | {:error, :unsupported} |
decode_stream/3, decode/2 | AMediaExtractor + AMediaCodec | ExtAudioFile | {:error, :unsupported} |
device: (output) | an AAudio device id | {:error, :unsupported} (iOS routes output itself) | |
| Presentation time | AAudio timestamps on 12+; an estimate from the buffer on 8-11 | AudioTimeStamp.mHostTime + the session's latency |
MobAudioIo.Fake runs all of it on the host for tests.
Processors
Two are built in: "mai_silence" (plays nothing and drops its input,
what MobAudioIo.SelfTest opens) and "mai_sine" (440 Hz at -18 dBFS,
for a listening check). Everything else comes from the app's C, by name:
processors/0 lists what's registered. A processor exists once the app's
NIF module has loaded, which Mob does lazily: pass module: :my_dsp to
open/3 and it is loaded first.
Messages
The owner process receives, as {:mob_audio_io, kind, stream, payload}:
:started,%{rate:, channels:, input_channels:, burst:, buffer:, sharing:, direction:, reopened:}: the stream runs (again, after a reopen, withreopened: true).:stopped,%{reason: :closed | :device_lost | :interrupted | :route_changed | :owner_exit, posts_dropped: n}. Posts the processor queued before a stop reach the owner first;posts_droppedcounts those that could not (the owner had exited).:device_lost,%{}: the device went away (headphones, USB, a Bluetooth headset). The plugin reopens on the new route;:startedfollows.:interrupted,%{}and:resumed,%{}: iOS session interruptions (a call, Siri, another app). After:resumedthe stream reopens and:startedfollows. An interruption that never sends its end (the app was suspended) gets one reopen try when the app is back in the foreground (:resumed,:startedif it works; an:errorand the stream stays interrupted if a call still holds the session).:route_changed,%{reason: :new_route | :new_rate | :media_reset}: a new route. The stream is rebuilt for a new rate or a media-services reset.:xrun,%{xruns:, buffer:}: underruns (output) and overruns (input) since open; on Android the buffer grows by a burst (up to its capacity) unless a fixed size was asked for.:error,%{reason:, detail:, while:}: a reopen failed; it is tried again after 0.5 s, 1 s, 2 s ... up to every 8 s.:post,binary: a message the processor queued withmai_post.
A screen keeps these in assigns; Mob.Test.send_message/2 stands in for
them in tests. Every one but :post is also logged with a
[mob_audio_io] prefix (MobAudioIo.Log), so they reach Sloppy Joe's
logs.
File decodes (decode_stream/3) send their own messages, as
{:mob_audio_io, kind, decode, payload} (a decode's id is never a
stream's), logged the same way but for the chunks: see "Decoding files".
Decoding files
decode_stream/3 decodes a file on a thread of its own and hands the
PCM out in chunks as it decodes, so an app can analyse a long file
without holding all of it, and stop it at once. The owner gets, in
order:
:decode_started,%{rate:, channels:, frames:}: the format, before any chunk.rateis the file's (no resampling),channels1 withmono: true,framesthe length the file states atrate(nilif it states none; an estimate for compressed formats).:decoded,binary:chunk_framesframes (the last chunk fewer) of interleaved 32-bit floats in the device's byte order. The owner callsdecode_ack/2once it is done with a chunk; at mostwindowchunks go out unacknowledged, and the decode waits for an ack before the next, so memory stays at aboutwindow + 1chunks whatever the file's length. None withprocessor:.- Then exactly one of
:decode_done,%{frames:, rate:, channels:};:decode_cancelled,%{frames:};:decode_error,%{reason:, detail:, frames:}.reason::not_found,:unsupported(a format the platform can't decode),:no_audio,:too_long(pastmax_ms:),:no_memoryor:decode_error;detailis the platform's words ornil.
decode_cancel/1 stops a decode within one codec buffer (an
ExtAudioFileRead of 4096 frames, an AMediaCodec buffer), or at once
while it waits for an ack, and releases the codec; :decode_cancelled
follows, and once decode_cancel/1 has answered :ok nothing else
arrives for that decode. The decode is cancelled when its owner exits.
With processor: name, the chunks go to that registered processor's
capture on the decode's thread instead, so C analysis never passes
through the BEAM: stream_event(ctx, decode, MAI_EVENT_STARTED, t)
first (t->rate, t->channels), then capture(ctx, chunk, frames, t)
per chunk (t->frame is the chunk's first frame in the file), then
MAI_EVENT_STOPPED after the last, whether the decode finished, failed
or was cancelled. Every call of one decode runs on that decode's
thread, in that order; a processor used by several decodes (or by
streams) at once gets their calls concurrently. The decode's thread is
not real-time (a processor may allocate or lock there), but it waits
for capture. The owner still gets :decode_started and the end.
Errors
open/3 answers {:error, reason}:
:unsupported: this platform (the host) or this combination (an outputdevice:on iOS).:permission: an:inputor:duplexstream without the microphone permission. Ask withMob.Permissions.request(socket, :microphone)first; iOS answers:permissionuntil the user chose.:unknown_processor: nothing registered under that name.:busy: 8 streams are open.{:module, reason}: the:moduledidn't load.{:invalid_option, key}: a bad option (checked before any native call).{:unavailable, detail}: the platform refused the stream;detailis its own words ("open: AAUDIO_ERROR_... (-889)","AudioUnitInitialize: -10875").{:session, detail}: iOS couldn't configure or activate the audio session (another app holds it, a call is active).{:invalid_device, detail}: no such:device/:input_device.
Sessions (iOS)
One AVAudioSession serves every stream, so it is configured from all
the open ones:
:play(output default): the playback category; the silent switch doesn't mute it and other apps' audio stops.:mix: plays beside other apps' audio (mixWithOthers, FMutate's).:play_and_record(input and duplex default): PlayAndRecord with the speaker (not the earpiece) as the default output and Bluetooth A2DP allowed (Choir Coach's).
Any open input makes the session PlayAndRecord; it mixes only if every
open stream asked for :mix. The :buffer is the preferred IO buffer
(5 ms for :low_latency); iOS rounds it to what the hardware does
(status().session.io_buffer_ms). The input :preset picks the mode:
:unprocessed is Measurement, :voice is VoiceChat with the
VoiceProcessingIO unit (echo cancellation).
Background audio
A stream keeps running with the screen locked only if the app may run in the background:
- iOS: the app's
Info.plistdeclaresUIBackgroundModes [audio](this plugin doesn't add it: Apple rejects the mode for apps without an audio feature). Recording keeps going in the background once started; iOS doesn't let the microphone start from the background. - Android: output keeps running; the microphone needs a foreground
service of type
microphonewhile in the background, whichmob_background(0.3.1+) provides.
Fake
config :mob_audio_io, backend: MobAudioIo.Fake makes this module run on
the host: see MobAudioIo.Fake.
Summary
Types
A file decode's id, as decode_stream/3 returned it.
An Android AAudio device id or an iOS port UID, as devices/1 lists them.
Functions
Stop and close a stream. Posts still queued reach the owner, then
:stopped. {:error, :not_open} for an unknown or closed id.
Decode a whole audio file to PCM: {:ok, %{rate:, channels:, frames:, data:}}, data being frames * channels interleaved 32-bit floats in
the device's byte order (little-endian on every supported phone).
The owner is done with n of a decode's chunks (default 1): as many
more may come. {:error, :not_running} once the decode has ended.
Stop a decode within one codec buffer and release its codec. :ok:
:decode_cancelled follows and nothing else arrives for it.
{:error, :not_running}: it had already ended, and its last message
(:decode_done, :decode_error or :decode_cancelled) was sent.
Decode an audio file for owner in chunks, on a thread of its own:
{:ok, decode}, then the messages "Decoding files" describes
(:decode_started, :decoded chunks the owner acknowledges with
decode_ack/2, and one end). Everything that goes wrong once it has
started, a missing file too, arrives as :decode_error.
The devices of a direction, in the platform's order:
[%{id: device_id, name: String.t(), kind: atom}].
Open a stream for owner and start it.
Names of the registered processors.
Zero a stream's worst-case counters (cb_max_us, gap_max_us, cb_frames_min/max, in_peak_db), e.g. after its start.
Everything the plugin knows, as a plain map (for RPC and agents)
One stream's map from status/0.
Types
@type decode() :: pos_integer()
A file decode's id, as decode_stream/3 returned it.
@type decode_option() :: {:mono, boolean()} | {:chunk_frames, pos_integer()} | {:window, pos_integer()} | {:max_ms, pos_integer() | nil} | {:processor, String.t() | nil} | {:module, module()}
An Android AAudio device id or an iOS port UID, as devices/1 lists them.
@type direction() :: :output | :input | :duplex
@type option() :: {:processor, String.t()} | {:module, module()} | {:rate, pos_integer() | nil} | {:buffer, :low_latency | {:frames, pos_integer()}} | {:channels, 1 | 2} | {:input_channels, 1 | 2} | {:sharing, :shared | :exclusive} | {:session, :play | :mix | :play_and_record} | {:preset, preset()} | {:device, device_id() | nil} | {:input_device, device_id() | nil}
@type preset() :: :default | :generic | :performance | :unprocessed | :voice
@type stream() :: pos_integer()
A stream id, as open/3 returned it.
Functions
@spec close(stream()) :: :ok | {:error, :not_open | :unsupported}
Stop and close a stream. Posts still queued reach the owner, then
:stopped. {:error, :not_open} for an unknown or closed id.
Decode a whole audio file to PCM: {:ok, %{rate:, channels:, frames:, data:}}, data being frames * channels interleaved 32-bit floats in
the device's byte order (little-endian on every supported phone).
The calling process runs a decode_stream/3 and collects its chunks
(selectively receiving that decode's messages), so it waits for the
whole file and holds all of it: decode_stream/3 is the way to analyse
a long file in bounded memory, or to stop early.
Options:
:max_bytes: the most PCM to produce (default 64 MiB, about 2.9 minutes of 48 kHz stereo; at most 1 GiB). A longer file answers{:error, :too_large}rather than a partial result: at once when the length the file states is too long, else when the PCM passes it.:mono: fold every frame to one channel (defaultfalse).
Errors: :not_found, :unsupported (the host, or a format the platform
can't decode), :no_audio, :too_large, :no_memory, :decode_error,
each possibly as {reason, detail} with the platform's words,
:busy, :cancelled (another process cancelled it) and
{:invalid_option, key}.
@spec decode_ack(decode(), pos_integer()) :: :ok | {:error, :not_running | :unsupported}
The owner is done with n of a decode's chunks (default 1): as many
more may come. {:error, :not_running} once the decode has ended.
@spec decode_cancel(decode()) :: :ok | {:error, :not_running | :unsupported}
Stop a decode within one codec buffer and release its codec. :ok:
:decode_cancelled follows and nothing else arrives for it.
{:error, :not_running}: it had already ended, and its last message
(:decode_done, :decode_error or :decode_cancelled) was sent.
@spec decode_stream(Path.t(), pid(), [decode_option()]) :: {:ok, decode()} | {:error, term()}
Decode an audio file for owner in chunks, on a thread of its own:
{:ok, decode}, then the messages "Decoding files" describes
(:decode_started, :decoded chunks the owner acknowledges with
decode_ack/2, and one end). Everything that goes wrong once it has
started, a missing file too, arrives as :decode_error.
WAV, MP3, AAC/M4A and whatever else the platform's decoders take, at the file's own rate.
Options:
:mono: fold every frame to one channel (defaultfalse).:chunk_frames: frames per chunk, 256 to 65 536 (default 4096).:window: chunks the owner may hold unacknowledged, 1 to 64 (default 4).:max_ms: refuse a file longer than this many milliseconds (:too_long): at once when its stated length is longer, else as soon as the decoded frames pass it. Defaultnil, no limit.:processor: a registered processor's name: itscapturegets the chunks on the decode's thread instead of the owner.:module: the app's NIF module that registers the processor; loaded (Code.ensure_loaded/1) first.
Errors: :busy (4 decodes run), :unknown_processor, :no_memory,
:unsupported (the host), {:module, reason} and {:invalid_option, key}.
The devices of a direction, in the platform's order:
[%{id: device_id, name: String.t(), kind: atom}].
kind: :builtin_mic, :builtin_speaker, :builtin_earpiece,
:wired_headset, :wired_headphones, :bluetooth, :bluetooth_a2dp,
:usb, :hdmi, :airplay, :car or :other. Android lists every
input or output AudioManager knows (ids for :device); iOS lists the
session's available inputs (port UIDs for :device) and the current
route's outputs (informational: iOS picks the output itself).
Open a stream for owner and start it.
Options:
:processor(required): a registered processor's name.:module: the app's NIF module that registers the processor; loaded (Code.ensure_loaded/1) before the stream opens.:rate: frames per second;nil(default) for the device's own, which avoids resampling.:buffer::low_latency(default: two bursts on Android, a 5 ms IO buffer on iOS) or{:frames, n}.:channels: output channels, 1 or 2 (default 2).:input_channels: input channels, 1 (default) or 2.:sharing: Android:shared(default) or:exclusive(falls back to shared when refused).:session: iOS:play,:mixor:play_and_record(see "Sessions"); defaults to:playfor output and:play_and_recordwith input.:playwith input is an option error.:preset(input and duplex)::default(Android's own, VOICE_RECOGNITION),:generic,:performance(Android VOICE_PERFORMANCE, low-latency singing),:unprocessed(no AGC or noise suppression where the device allows; iOS Measurement mode) or:voice(echo cancellation for calls: Android VOICE_COMMUNICATION, iOS VoiceChat + VoiceProcessingIO).:device: a device fromdevices/1(the input's for:input, the output's otherwise);nil(default) for the system's choice.:input_device(duplex): the input's device.
The stream closes when owner exits.
@spec processors() :: [String.t()] | {:error, :unsupported}
Names of the registered processors.
@spec reset_stats(stream()) :: :ok | {:error, :not_open | :unsupported}
Zero a stream's worst-case counters (cb_max_us, gap_max_us, cb_frames_min/max, in_peak_db), e.g. after its start.
@spec status() :: map() | {:error, :unsupported}
Everything the plugin knows, as a plain map (for RPC and agents):
%{platform: :android | :ios | :fake, abi: 1, processors: ["mai_silence", ...],
streams: [stream_status], session: nil | %{...}}Each stream:
id, processor, direction, running, interrupted, session, sharing.rate,channels(output),input_channels.burst: the platform's own block (AAudio's burst, iOS's IO buffer);cb_frames: the frames the last callback actually had, withcb_frames_min/cb_frames_maxsincereset_stats/1(Android's legacy path and the iOS simulator call with other sizes than the burst).bufferandcapacity(frames).latency_ms: output. iOS: the session's output latency plus one IO buffer. Android 12+: the frames written but not yet presented (AAudio timestamps). Android 8-11: only the frames written but not yet read by the device, becauseAAudioStream_getTimestampcan abort the process there (seedocs/VERIFICATION.md).input_latency_ms: iOS's input latency plus one IO buffer;nilelsewhere.present: wheremai_time.present_nscomes from::timestampor:estimate.xruns(underruns and overruns since open),input_underflows(duplex callbacks whose input came short and was zero-filled).callbacks,frames,cb_max_us(the worst time in one callback: the processor and the plugin's bookkeeping),cb_avg_us,gap_max_us(the worst time between two callbacks' starts).in_peak_db(sincereset_stats/1) andin_rms_db(the last callback's block), measured by the plugin on every input;nilfor output.preset,preset_granted(Android'sgetInputPreset),device_id,input_device_id.reopens,posts_dropped,last_error.
session (iOS): active, category, mode, rate, io_buffer_ms, preferred_io_buffer_ms, output_latency_ms, input_latency_ms, input_available, outputs, inputs.
One stream's map from status/0.