MobAudioIo (mob_audio_io v0.1.0)

Copy Markdown View Source

Low-latency audio streams for Mob apps, with the DSP in the app's own C.

The plugin owns the platform's audio streams: AAudio on Android, a RemoteIO (or VoiceProcessingIO) unit and the AVAudioSession on iOS. An app's static NIF registers a processor through the C interface in priv/include/mob_audio_io.h (mai_register/3), and the plugin calls it from the audio thread: render for output, capture for input, both (in that order, in one callback) for duplex. Audio never passes through the BEAM; Elixir opens and closes streams, reads status/0, lists devices, decodes files (in chunks, cancellably) and receives the stream's messages.

Platforms

Android (API 28+)iOS (15+)Host (mix test, dev)
:outputAAudio, low latency, shared or exclusiveRemoteIO{:error, :unsupported}
:inputAAudio with an input presetRemoteIO input (VoiceProcessingIO for preset: :voice){:error, :unsupported}
:duplexAAudio output, the input read in its callbackone RemoteIO unit, input pulled in the render{:error, :unsupported}
devices/1AudioManager.getDevices (JNI)AVAudioSession inputs, the current route's outputs{:error, :unsupported}
decode_stream/3, decode/2AMediaExtractor + AMediaCodecExtAudioFile{:error, :unsupported}
device: (output)an AAudio device id{:error, :unsupported} (iOS routes output itself)
Presentation timeAAudio timestamps on 12+; an estimate from the buffer on 8-11AudioTimeStamp.mHostTime + the session's latency

MobAudioIo.Fake runs all of it on the host for tests.

Processors

Two are built in: "mai_silence" (plays nothing and drops its input, what MobAudioIo.SelfTest opens) and "mai_sine" (440 Hz at -18 dBFS, for a listening check). Everything else comes from the app's C, by name: processors/0 lists what's registered. A processor exists once the app's NIF module has loaded, which Mob does lazily: pass module: :my_dsp to open/3 and it is loaded first.

Messages

The owner process receives, as {:mob_audio_io, kind, stream, payload}:

  • :started, %{rate:, channels:, input_channels:, burst:, buffer:, sharing:, direction:, reopened:}: the stream runs (again, after a reopen, with reopened: true).
  • :stopped, %{reason: :closed | :device_lost | :interrupted | :route_changed | :owner_exit, posts_dropped: n}. Posts the processor queued before a stop reach the owner first; posts_dropped counts those that could not (the owner had exited).

  • :device_lost, %{}: the device went away (headphones, USB, a Bluetooth headset). The plugin reopens on the new route; :started follows.
  • :interrupted, %{} and :resumed, %{}: iOS session interruptions (a call, Siri, another app). After :resumed the stream reopens and :started follows. An interruption that never sends its end (the app was suspended) gets one reopen try when the app is back in the foreground (:resumed, :started if it works; an :error and the stream stays interrupted if a call still holds the session).
  • :route_changed, %{reason: :new_route | :new_rate | :media_reset}: a new route. The stream is rebuilt for a new rate or a media-services reset.

  • :xrun, %{xruns:, buffer:}: underruns (output) and overruns (input) since open; on Android the buffer grows by a burst (up to its capacity) unless a fixed size was asked for.
  • :error, %{reason:, detail:, while:}: a reopen failed; it is tried again after 0.5 s, 1 s, 2 s ... up to every 8 s.
  • :post, binary: a message the processor queued with mai_post.

A screen keeps these in assigns; Mob.Test.send_message/2 stands in for them in tests. Every one but :post is also logged with a [mob_audio_io] prefix (MobAudioIo.Log), so they reach Sloppy Joe's logs.

File decodes (decode_stream/3) send their own messages, as {:mob_audio_io, kind, decode, payload} (a decode's id is never a stream's), logged the same way but for the chunks: see "Decoding files".

Decoding files

decode_stream/3 decodes a file on a thread of its own and hands the PCM out in chunks as it decodes, so an app can analyse a long file without holding all of it, and stop it at once. The owner gets, in order:

  • :decode_started, %{rate:, channels:, frames:}: the format, before any chunk. rate is the file's (no resampling), channels 1 with mono: true, frames the length the file states at rate (nil if it states none; an estimate for compressed formats).
  • :decoded, binary: chunk_frames frames (the last chunk fewer) of interleaved 32-bit floats in the device's byte order. The owner calls decode_ack/2 once it is done with a chunk; at most window chunks go out unacknowledged, and the decode waits for an ack before the next, so memory stays at about window + 1 chunks whatever the file's length. None with processor:.
  • Then exactly one of :decode_done, %{frames:, rate:, channels:}; :decode_cancelled, %{frames:}; :decode_error, %{reason:, detail:, frames:}. reason: :not_found, :unsupported (a format the platform can't decode), :no_audio, :too_long (past max_ms:), :no_memory or :decode_error; detail is the platform's words or nil.

decode_cancel/1 stops a decode within one codec buffer (an ExtAudioFileRead of 4096 frames, an AMediaCodec buffer), or at once while it waits for an ack, and releases the codec; :decode_cancelled follows, and once decode_cancel/1 has answered :ok nothing else arrives for that decode. The decode is cancelled when its owner exits.

With processor: name, the chunks go to that registered processor's capture on the decode's thread instead, so C analysis never passes through the BEAM: stream_event(ctx, decode, MAI_EVENT_STARTED, t) first (t->rate, t->channels), then capture(ctx, chunk, frames, t) per chunk (t->frame is the chunk's first frame in the file), then MAI_EVENT_STOPPED after the last, whether the decode finished, failed or was cancelled. Every call of one decode runs on that decode's thread, in that order; a processor used by several decodes (or by streams) at once gets their calls concurrently. The decode's thread is not real-time (a processor may allocate or lock there), but it waits for capture. The owner still gets :decode_started and the end.

Errors

open/3 answers {:error, reason}:

  • :unsupported: this platform (the host) or this combination (an output device: on iOS).
  • :permission: an :input or :duplex stream without the microphone permission. Ask with Mob.Permissions.request(socket, :microphone) first; iOS answers :permission until the user chose.
  • :unknown_processor: nothing registered under that name.
  • :busy: 8 streams are open.
  • {:module, reason}: the :module didn't load.
  • {:invalid_option, key}: a bad option (checked before any native call).
  • {:unavailable, detail}: the platform refused the stream; detail is its own words ("open: AAUDIO_ERROR_... (-889)", "AudioUnitInitialize: -10875").
  • {:session, detail}: iOS couldn't configure or activate the audio session (another app holds it, a call is active).
  • {:invalid_device, detail}: no such :device / :input_device.

Sessions (iOS)

One AVAudioSession serves every stream, so it is configured from all the open ones:

  • :play (output default): the playback category; the silent switch doesn't mute it and other apps' audio stops.
  • :mix: plays beside other apps' audio (mixWithOthers, FMutate's).
  • :play_and_record (input and duplex default): PlayAndRecord with the speaker (not the earpiece) as the default output and Bluetooth A2DP allowed (Choir Coach's).

Any open input makes the session PlayAndRecord; it mixes only if every open stream asked for :mix. The :buffer is the preferred IO buffer (5 ms for :low_latency); iOS rounds it to what the hardware does (status().session.io_buffer_ms). The input :preset picks the mode: :unprocessed is Measurement, :voice is VoiceChat with the VoiceProcessingIO unit (echo cancellation).

Background audio

A stream keeps running with the screen locked only if the app may run in the background:

  • iOS: the app's Info.plist declares UIBackgroundModes [audio] (this plugin doesn't add it: Apple rejects the mode for apps without an audio feature). Recording keeps going in the background once started; iOS doesn't let the microphone start from the background.
  • Android: output keeps running; the microphone needs a foreground service of type microphone while in the background, which mob_background (0.3.1+) provides.

Fake

config :mob_audio_io, backend: MobAudioIo.Fake makes this module run on the host: see MobAudioIo.Fake.

Summary

Types

A file decode's id, as decode_stream/3 returned it.

An Android AAudio device id or an iOS port UID, as devices/1 lists them.

A stream id, as open/3 returned it.

Functions

Stop and close a stream. Posts still queued reach the owner, then :stopped. {:error, :not_open} for an unknown or closed id.

Decode a whole audio file to PCM: {:ok, %{rate:, channels:, frames:, data:}}, data being frames * channels interleaved 32-bit floats in the device's byte order (little-endian on every supported phone).

The owner is done with n of a decode's chunks (default 1): as many more may come. {:error, :not_running} once the decode has ended.

Stop a decode within one codec buffer and release its codec. :ok: :decode_cancelled follows and nothing else arrives for it. {:error, :not_running}: it had already ended, and its last message (:decode_done, :decode_error or :decode_cancelled) was sent.

Decode an audio file for owner in chunks, on a thread of its own: {:ok, decode}, then the messages "Decoding files" describes (:decode_started, :decoded chunks the owner acknowledges with decode_ack/2, and one end). Everything that goes wrong once it has started, a missing file too, arrives as :decode_error.

The devices of a direction, in the platform's order: [%{id: device_id, name: String.t(), kind: atom}].

Open a stream for owner and start it.

Names of the registered processors.

Zero a stream's worst-case counters (cb_max_us, gap_max_us, cb_frames_min/max, in_peak_db), e.g. after its start.

Everything the plugin knows, as a plain map (for RPC and agents)

One stream's map from status/0.

Types

decode()

@type decode() :: pos_integer()

A file decode's id, as decode_stream/3 returned it.

decode_option()

@type decode_option() ::
  {:mono, boolean()}
  | {:chunk_frames, pos_integer()}
  | {:window, pos_integer()}
  | {:max_ms, pos_integer() | nil}
  | {:processor, String.t() | nil}
  | {:module, module()}

device_id()

@type device_id() :: integer() | String.t()

An Android AAudio device id or an iOS port UID, as devices/1 lists them.

direction()

@type direction() :: :output | :input | :duplex

option()

@type option() ::
  {:processor, String.t()}
  | {:module, module()}
  | {:rate, pos_integer() | nil}
  | {:buffer, :low_latency | {:frames, pos_integer()}}
  | {:channels, 1 | 2}
  | {:input_channels, 1 | 2}
  | {:sharing, :shared | :exclusive}
  | {:session, :play | :mix | :play_and_record}
  | {:preset, preset()}
  | {:device, device_id() | nil}
  | {:input_device, device_id() | nil}

preset()

@type preset() :: :default | :generic | :performance | :unprocessed | :voice

stream()

@type stream() :: pos_integer()

A stream id, as open/3 returned it.

Functions

close(stream)

@spec close(stream()) :: :ok | {:error, :not_open | :unsupported}

Stop and close a stream. Posts still queued reach the owner, then :stopped. {:error, :not_open} for an unknown or closed id.

decode(path, opts \\ [])

@spec decode(Path.t(), keyword()) :: {:ok, map()} | {:error, term()}

Decode a whole audio file to PCM: {:ok, %{rate:, channels:, frames:, data:}}, data being frames * channels interleaved 32-bit floats in the device's byte order (little-endian on every supported phone).

The calling process runs a decode_stream/3 and collects its chunks (selectively receiving that decode's messages), so it waits for the whole file and holds all of it: decode_stream/3 is the way to analyse a long file in bounded memory, or to stop early.

Options:

  • :max_bytes: the most PCM to produce (default 64 MiB, about 2.9 minutes of 48 kHz stereo; at most 1 GiB). A longer file answers {:error, :too_large} rather than a partial result: at once when the length the file states is too long, else when the PCM passes it.
  • :mono: fold every frame to one channel (default false).

Errors: :not_found, :unsupported (the host, or a format the platform can't decode), :no_audio, :too_large, :no_memory, :decode_error, each possibly as {reason, detail} with the platform's words, :busy, :cancelled (another process cancelled it) and {:invalid_option, key}.

decode_ack(decode, n \\ 1)

@spec decode_ack(decode(), pos_integer()) ::
  :ok | {:error, :not_running | :unsupported}

The owner is done with n of a decode's chunks (default 1): as many more may come. {:error, :not_running} once the decode has ended.

decode_cancel(decode)

@spec decode_cancel(decode()) :: :ok | {:error, :not_running | :unsupported}

Stop a decode within one codec buffer and release its codec. :ok: :decode_cancelled follows and nothing else arrives for it. {:error, :not_running}: it had already ended, and its last message (:decode_done, :decode_error or :decode_cancelled) was sent.

decode_stream(path, owner, opts \\ [])

@spec decode_stream(Path.t(), pid(), [decode_option()]) ::
  {:ok, decode()} | {:error, term()}

Decode an audio file for owner in chunks, on a thread of its own: {:ok, decode}, then the messages "Decoding files" describes (:decode_started, :decoded chunks the owner acknowledges with decode_ack/2, and one end). Everything that goes wrong once it has started, a missing file too, arrives as :decode_error.

WAV, MP3, AAC/M4A and whatever else the platform's decoders take, at the file's own rate.

Options:

  • :mono: fold every frame to one channel (default false).
  • :chunk_frames: frames per chunk, 256 to 65 536 (default 4096).
  • :window: chunks the owner may hold unacknowledged, 1 to 64 (default 4).
  • :max_ms: refuse a file longer than this many milliseconds (:too_long): at once when its stated length is longer, else as soon as the decoded frames pass it. Default nil, no limit.
  • :processor: a registered processor's name: its capture gets the chunks on the decode's thread instead of the owner.
  • :module: the app's NIF module that registers the processor; loaded (Code.ensure_loaded/1) first.

Errors: :busy (4 decodes run), :unknown_processor, :no_memory, :unsupported (the host), {:module, reason} and {:invalid_option, key}.

devices(direction)

@spec devices(:input | :output) :: [map()] | {:error, :unsupported | term()}

The devices of a direction, in the platform's order: [%{id: device_id, name: String.t(), kind: atom}].

kind: :builtin_mic, :builtin_speaker, :builtin_earpiece, :wired_headset, :wired_headphones, :bluetooth, :bluetooth_a2dp, :usb, :hdmi, :airplay, :car or :other. Android lists every input or output AudioManager knows (ids for :device); iOS lists the session's available inputs (port UIDs for :device) and the current route's outputs (informational: iOS picks the output itself).

open(owner, direction, opts)

@spec open(pid(), direction(), [option()]) :: {:ok, stream()} | {:error, term()}

Open a stream for owner and start it.

Options:

  • :processor (required): a registered processor's name.
  • :module: the app's NIF module that registers the processor; loaded (Code.ensure_loaded/1) before the stream opens.
  • :rate: frames per second; nil (default) for the device's own, which avoids resampling.
  • :buffer: :low_latency (default: two bursts on Android, a 5 ms IO buffer on iOS) or {:frames, n}.
  • :channels: output channels, 1 or 2 (default 2).
  • :input_channels: input channels, 1 (default) or 2.
  • :sharing: Android :shared (default) or :exclusive (falls back to shared when refused).
  • :session: iOS :play, :mix or :play_and_record (see "Sessions"); defaults to :play for output and :play_and_record with input. :play with input is an option error.
  • :preset (input and duplex): :default (Android's own, VOICE_RECOGNITION), :generic, :performance (Android VOICE_PERFORMANCE, low-latency singing), :unprocessed (no AGC or noise suppression where the device allows; iOS Measurement mode) or :voice (echo cancellation for calls: Android VOICE_COMMUNICATION, iOS VoiceChat + VoiceProcessingIO).
  • :device: a device from devices/1 (the input's for :input, the output's otherwise); nil (default) for the system's choice.
  • :input_device (duplex): the input's device.

The stream closes when owner exits.

processors()

@spec processors() :: [String.t()] | {:error, :unsupported}

Names of the registered processors.

reset_stats(stream)

@spec reset_stats(stream()) :: :ok | {:error, :not_open | :unsupported}

Zero a stream's worst-case counters (cb_max_us, gap_max_us, cb_frames_min/max, in_peak_db), e.g. after its start.

status()

@spec status() :: map() | {:error, :unsupported}

Everything the plugin knows, as a plain map (for RPC and agents):

%{platform: :android | :ios | :fake, abi: 1, processors: ["mai_silence", ...],
  streams: [stream_status], session: nil | %{...}}

Each stream:

  • id, processor, direction, running, interrupted, session, sharing.
  • rate, channels (output), input_channels.
  • burst: the platform's own block (AAudio's burst, iOS's IO buffer); cb_frames: the frames the last callback actually had, with cb_frames_min / cb_frames_max since reset_stats/1 (Android's legacy path and the iOS simulator call with other sizes than the burst).
  • buffer and capacity (frames).
  • latency_ms: output. iOS: the session's output latency plus one IO buffer. Android 12+: the frames written but not yet presented (AAudio timestamps). Android 8-11: only the frames written but not yet read by the device, because AAudioStream_getTimestamp can abort the process there (see docs/VERIFICATION.md). input_latency_ms: iOS's input latency plus one IO buffer; nil elsewhere.
  • present: where mai_time.present_ns comes from: :timestamp or :estimate.
  • xruns (underruns and overruns since open), input_underflows (duplex callbacks whose input came short and was zero-filled).
  • callbacks, frames, cb_max_us (the worst time in one callback: the processor and the plugin's bookkeeping), cb_avg_us, gap_max_us (the worst time between two callbacks' starts).
  • in_peak_db (since reset_stats/1) and in_rms_db (the last callback's block), measured by the plugin on every input; nil for output.
  • preset, preset_granted (Android's getInputPreset), device_id, input_device_id.
  • reopens, posts_dropped, last_error.

session (iOS): active, category, mode, rate, io_buffer_ms, preferred_io_buffer_ms, output_latency_ms, input_latency_ms, input_available, outputs, inputs.

status(stream)

@spec status(stream()) :: map() | {:error, :not_open | :unsupported}

One stream's map from status/0.