SDL3: Mixer changes, what to do?

[Starbuck5]

As many of you may know, SDL3_mixer features substantial changes over SDL2_mixer. See https://wiki.libsdl.org/SDL3_mixer/README-migration. It has yet to be formally released, and also relies on an unreleased SDL version.

I've spent quite a bit of time this weekend going through the API and writing a stub file that exposes all SDL3_mixer functionality the way I would do it with a fresh port. I also had to do the same for SDL3's audio module, because SDL3_mixer relies on input/output with audio types like formats, specs, and devices, to various degrees.

mixer2.pyi

from typing import Type, TypeVar, TypedDict
from collections.abc import Callable

from pygame.typing import FileLike
import audio

from typing_extensions import Buffer


def init() -> None: ...
def quit() -> None: ...
def get_sdl_mixer_version(linked: bool = True) -> tuple[int, int, int]: ...
def ms_to_frames(sample_rate: int, ms: int) -> int: ...
def frames_to_ms(sample_rate: int, frames: int) -> int: ...
def get_decoders() -> list[str]: ...

T = TypeVar("T")
track_stopped_callback = Callable[[T, Track], None]
track_mix_callback = Callable[[T, Track, audio.AudioSpec, Buffer], None]
group_mix_callback = Callable[[T, Group, audio.AudioSpec, Buffer], None]
post_mix_callback = Callable[[T, Mixer, audio.AudioSpec, Buffer], None]

class Mixer:
    def __init__(self, device: audio._AudioDevice = audio.DEFAULT_PLAYBACK_DEVICE, spec: audio.AudioSpec | None = None) -> None: ...
    @property
    def gain(self) -> float: ...
    @gain.setter
    def gain(self, value: float): ...
    def play_tag(self, tag: str, **kwargs) -> None: ...
    def stop_tag(self, tag: str, fade_out_ms: int) -> None: ...
    def pause_tag(self, tag: str) -> None: ...
    def resume_tag(self, tag: str) -> None: ...
    def set_tag_gain(self, tag: str, gain: float) -> None: ...
    def play_audio(self, audio: Audio)-> None: ...
    def stop_all_tracks(self, fade_out_ms: int) -> None: ...
    def pause_all_tracks(self) -> None: ...
    def resume_all_tracks(self) -> None: ...
    @property
    def format(self) -> audio.AudioSpec: ...
    def set_post_mix_callback(self, callback: post_mix_callback | None, userdata: T) -> None: ...


class MemoryMixer(Mixer):
    def __init__(self, spec: audio.AudioSpec) -> None: ...
    def generate(self, buffer: Buffer, buflen: int) -> None: ...


class Audio:
    def __init__(self, file: FileLike, predecode: bool = False, preferred_mixer: Mixer | None = None) -> None: ...
    @classmethod
    def from_raw(cls, buf: Buffer) -> Audio: ...
    @classmethod
    def from_sine_wave(hz: int, amplitude: float) -> Audio: ...
    @property
    def duration_frames(self) -> int: ...
    @property
    def duration_ms(self) -> int: ...
    @property
    def format(self) -> audio.AudioSpec: ...  
    def ms_to_frames(self, ms: int) -> int: ...
    def frames_to_ms(self, ms: int) -> int: ...
    def get_metadata() -> _MusicMetadataDict: ...


class Group:
    def __init__(self, mixer: Mixer) -> None: ...
    @property
    def mixer(self) -> Mixer: ...
    def set_post_mix_callback(self, callback: group_mix_callback | None, userdata: Type[T]) -> None: ...


class Track:
    def __init__(self, mixer: Mixer) -> None: ...
    def set_audio(self, audio: Audio | None) -> None: ...
    def get_audio(self) -> Audio | None: ...
    def set_audiostream(self, audiostream: audio.AudioStream | None) -> None: ...
    def get_audiostream(self) -> audio.AudioStream | None: ...
    def set_filestream(self, file: FileLike) -> None: ...
    def play(self, loops: int = 0, max_frame: int = -1, max_ms: int = -1, start_frame: int = 0, start_ms: int = 0, loop_start_frame: int = 0, loop_start_ms: int = 0, fadein_frames: int = 0, fadein_ms: int = 0, append_silence_frames: int = 0, append_silence_ms: int = 0) -> None: ...
    @property
    def mixer(self) -> Mixer: ...
    def add_tag(self, tag: str) -> None: ...
    def remove_tag(self, tag: str) -> None: ...
    def set_group(self, group: Group | None) -> None: ...
    def set_playback_position(self, frames: int) -> None: ...
    def get_playback_position(self) -> int: ...
    def get_remaining_frames(self) -> int | None: ...
    def ms_to_frames(self, ms: int) -> int: ...
    def frames_to_ms(self, ms: int) -> int: ...
    def stop(self, fade_out_frames: int) -> None: ...
    def pause(self) -> None: ...
    def resume(self) -> None: ...
    @property
    def playing(self) -> bool: ...
    @property
    def paused(self) -> bool: ...
    @property
    def looping(self) -> bool: ...
    @property
    def gain(self) -> float: ...
    @gain.setter
    def gain(self, value: float): ...
    @property
    def frequency_ratio(self) -> float: ...
    @gain.setter
    def frequency_ratio(self, value: float): ...
    def set_output_channel_map(self, channel_map: list[int] | None) -> None: ...
    def set_stereo(self, left_gain: float, right_gain: float) -> None: ...
    def set_3d_position(self, position: tuple[float, float, float]) -> None: ...
    def get_3d_position(self) -> tuple[float, float, float]: ...
    def set_stopped_callback(self, callback: track_stopped_callback | None, userdata: T) -> None: ...
    def set_raw_callback(self, callback: track_mix_callback | None, userdata: T) -> None: ...


class AudioDecoder:
    def __init__(self, file: FileLike) -> None: ...
    @property
    def format(self) -> audio.AudioSpec: ...
    def decode(buffer: Buffer, spec: audio.AudioSpec) -> int: ...


class _MusicMetadataDict(TypedDict):
    title: str
    album: str
    artist: str
    copyright: str

audio.pyi

from dataclasses import dataclass
from collections.abc import Callable
from typing import TypeVar

from typing_extensions import Buffer

from pygame.typing import FileLike

def get_current_audio_driver() -> str: ...
def get_audio_drivers() -> list[str]: ...
def get_playback_devices() -> list[PhysicalAudioDevice]: ...
def get_recording_devices() -> list[PhysicalAudioDevice]: ...
def mix_audio(dst: Buffer, src: Buffer, format: AudioFormat, volume: float) -> None: ...
def load_wav(file: FileLike) -> tuple[AudioSpec, bytes]: ...
def convert_samples(src_spec: AudioSpec, src_data: Buffer, dst_spec: AudioSpec) -> bytes: ...

DEFAULT_RECORDING_DEVICE: PhysicalAudioDevice
DEFAULT_PLAYBACK_DEVICE: PhysicalAudioDevice

T = TypeVar("T")
stream_callback = Callable[[T, AudioStream, int, int], None]
post_mix_callback = Callable[[T, AudioStream, Buffer], None]
iteration_callback = Callable[[T, _AudioDevice, bool], None]

class AudioFormat:
    def __init__(self, value: int) -> None: ...
    # So that PyLong_AsLong will get it as an integer
    def __index__(self) -> int: ...
    @property
    def name(self) -> str: ...
    @property
    def bitsize(self) -> int: ...
    @property
    def bytesize(self) -> int: ...
    @property
    def is_big_endian(self) -> bool: ...
    @property
    def is_float(self) -> bool: ...
    @property
    def is_int(self) -> bool: ...
    @property
    def is_little_endian(self) -> bool: ...
    @property
    def is_signed(self) -> bool: ...
    @property
    def is_unsigned(self) -> bool: ...
    @property
    def silence_value(self) -> int: ...


formats: list[AudioFormat]

U8: AudioFormat
S8: AudioFormat
S16LE: AudioFormat
S16BE: AudioFormat
S32LE: AudioFormat
S32BE: AudioFormat
F32LE: AudioFormat
F32BE: AudioFormat
S16: AudioFormat
S32: AudioFormat
F32: AudioFormat


@dataclass
class AudioSpec:
    format: AudioFormat
    channels: int
    freq: int
    @property
    def framesize(self) -> int: ...


class _AudioDevice:
    def open(self, spec: AudioSpec | None) -> LogicalAudioDevice: ...
    def open_stream(self, spec: AudioSpec | None, callback: stream_callback, userdata: T) -> AudioStream: ...
    def bind(self, *args: AudioStream) -> None: ...
    def unbind(self, *args: AudioStream) -> None: ...
    @property
    def is_playback(self) -> bool: ...
    @property
    def name(self) -> str: ...
    @property
    def paused(self) -> bool: ...
    def get_channel_map(self) -> list[int] | None: ...

class PhysicalAudioDevice(_AudioDevice):
    pass

class LogicalAudioDevice(_AudioDevice):
    def pause(self) -> None: ...
    def resume(self) -> None: ...
    @property
    def gain(self) -> float: ...
    @gain.setter
    def gain(self, value: float): ...
    def set_iteration_callbacks(self, start: iteration_callback | None, end: iteration_callback | None, userdata: T) -> None: ...
    def set_post_mix_callback(self, callback: post_mix_callback | None, userdata: T) -> None: ...


class AudioStream:
    def __init__(self, src_spec: AudioSpec, dst_spec: AudioSpec) -> None: ...
    def clear(self) -> None: ...
    def flush(self) -> None: ...
    def get_available_bytes(self) -> int: ...
    def get_queued_bytes(self) -> int: ...
    def get_data(self) -> bytes: ...
    def put_data(self, data: Buffer) -> None: ...
    # pause_device, resume_device, device_paused could all just be called on device?
    # .device.pause(), .device.resume
    def pause_device(self) -> None: ...
    def resume_device(self) -> None: ...
    @property
    def device_paused(self) -> bool: ...
    @property
    def device(self) -> _AudioDevice: ...
    @property
    def src_format(self) -> AudioSpec: ...
    @src_format.setter
    def src_format(self, value: AudioSpec) -> None: ...
    @property
    def dst_format(self) -> AudioSpec: ...
    @dst_format.setter
    def dst_format(self, value: AudioSpec) -> None: ...
    @property
    def gain(self) -> float: ...
    @gain.setter
    def gain(self, value: float): ...
    @property
    def frequency_ratio(self) -> float: ...
    @gain.setter
    def frequency_ratio(self, value: float): ...
    def set_input_channel_map(self, channel_map: list[int] | None) -> None: ...
    def get_input_channel_map(self) -> list[int] | None: ...
    def set_output_channel_map(self, channel_map: list[int] | None) -> None: ...
    def get_output_channel_map(self) -> list[int] | None: ...
    def lock(self) -> None: ...
    def unlock(self) -> None: ...
    def set_get_callback(self, callback: stream_callback | None, userdata: T) -> None: ...
    def set_put_callback(self, callback: stream_callback | None, userdata: T) -> None: ...

I also have these stubs on a branch: 0c12dba

Simple code examples I believe would work:

# Broadcast audio to all speakers
mixers: list[pygame.Mixer] = []

for device in pygame.audio.get_playback_devices():
    mixers.append(pygame.Mixer(device))

output_audio = pygame.Audio("test.mp3")

for mixer in mixers:
    mixer.play_audio(output_audio)

# Loop mic input into speaker output
microphone_stream = pygame.audio.DEFAULT_RECORDING_DEVICE.open_stream()

speakers = pygame.mixer.Mixer()
speakers_track = pygame.mixer.Track(speakers)
speakers_track.set_audiostream(microphone_stream)
speakers_track.play()

[Starbuck5]

To summarize some of the changes, the explicit "music" doesn't exist now, such that any Track object could be streaming from a file. Channels don't exist now, but Tracks do... The doc I linked does a way better job of summarizing.

Initially, my plan was to do a fresh port of SDL3_mixer and then put in a compatibility layer in Python. This would allow us to move forward with an interface closer to what SDL natively has, with a best effort compatibility layer for folks to not be too inconvenienced. The docs would have these two parts of the mixer module separated into different pages, even if at runtime it's all in the same namespace. So if you can imagine, the top bar would have "mixer", which would link to mixer2.html, and then mixer2.html would contain a link to mixer.html as the "legacy API." (This would also save old URLs)

But now that I've written it all out maybe it wouldn't be too bad to just natively support something pretty similar to what we've got now. Like when you initialize the module it could just create a mixer for the default speakers, and an array of preallocated Tracks. And then if you squint at them the Audio and Track objects are similar to the existing Sound and Channel objects. We could make it so there's a play() method on Audio that finds a preallocated track on the default mixer. And then we could smooth out the terminology differences like gain vs volume, unpause vs resume, etc. And we'd still be able to expand the functionality, just without abandoning our existing paradigm quite as much. I think mixer.music would still be a Python compat layer though.

[Starbuck5]

Hoping to get some other opinions on this, I know it's a lot of information to sort through!

I think right now I'm back to leaning with fresh-port-full-compat-layer.

[OlaKenji]

I don’t fully understand all the details, but the current music/mixer setup feels quite limited. Any changes that could improve or expand it would be a welcome improvement.