Skip to content

Backends

A backend runs a model on one platform: MLX on macOS, diffusers and PyTorch on CUDA for Windows and Linux. Everything above the backends is shared. This page holds the design rules of each backend family and the reasons for them; the code's docstrings hold the details.

Protocols and selection

Image backends implement ImageBackend (core/image_backend.py), video backends VideoBackend (core/video_backend.py), the prompt-enhancer LLM PromptEnhancer (core/prompt_enhancer.py), plain upscaling SeedVR2Upscaler (core/seedvr2.py) and the cut-out chain MatteChain (core/matte.py). The macOS implementations live in backends/mac/, the Windows and Linux ones in backends/cuda/, and the two folders never import from each other. A task with helpers of its own gets a folder (cuda/image/, cuda/video/); what two tasks share stays one level up.

backends/__init__.py is the only place that picks a platform; the platform rule says what that allows elsewhere. It exposes factories (get_backend() for images, get_video_backend(family), get_audio_backend(family), create_matte_chain()), the download-plan dispatch, and capability functions such as supports_seedvr2(), supports_video_extend(), video_file_upscale_method() and accelerator_free_bytes(). Other code calls those instead of choosing a backend itself.

The factories differ by task, and a new family fits each differently:

  • Image: one ImageBackend per platform (MfluxBackend on macOS, DiffusersBackend on CUDA), which dispatches on the model's family inside itself. A new image family goes into those two classes, and get_backend() takes no family.
  • Video: a registry keyed by platform and then family (get_video_backend(family)), with one backend class per pair. A new video family adds a class and a registry entry for each platform.
  • Audio: a registry keyed by family only; the model's metadata says which family runs on which platform (see Audio).

Importing the package, or running a code path that does not load a model, must not need torch, diffusers, mflux or a GPU. Validation that a CUDA device exists happens lazily when a backend loads.

Two methods exist for memory control: ImageBackend.release_model(model) (and the same on SeedVR2Upscaler) frees a loaded model's weights in place and is idempotent; accelerator_free_bytes() returns the free accelerator memory or None when unknown. Background removal uses them (see Background removal).

Download plans

Which Hugging Face files a model needs is not part of any backend protocol, because a plan must exist without loading a backend. Each platform has a light planner module (backends/mac/download_plan.py, backends/cuda/download_plan.py), and backends/__init__.py dispatches to it, so the platform is still chosen only there. See Model downloads.

SeedVR2

The plain upscale of existing images is SeedVR2Upscaler, implemented by backends/mac/seedvr2.py (mflux) and backends/cuda/seedvr2.py (the vendored PyTorch core, see Vendored packages). The files each tier reads are the seedvr2-7b and seedvr2-3b entries of the model metadata, and seedvr2_loader.py keeps one hidden weights folder per tier and precision. On macOS it loads through a stored 8-bit copy, as image models do; on CUDA the engine downloads the FP8 transformer and the VAE into the folder, because the vendored core reads both from one folder. The CUDA 7B tier uses the FP8 file whose last block is FP16, since the plain FP8 file draws streaks in flat areas.

The CUDA engine also upscales video frames (SeedVR2VideoUpscaler). The macOS engine upscales images only; a Mac upscales a video file with LTX instead (LtxVideoBackend.upscale_video). video_file_upscale_method() names the platform's method (seedvr2, ltx or none), so callers ask instead of checking the platform. tests/contracts/test_mflux_seedvr2_contract.py records the mflux surface the macOS engine relies on; re-run it on every mflux bump.

Audio

All audio models share one protocol, AudioBackend (core/audio_backend.py): load(preset, device, cancelled=None), generate(request, on_progress, cancelled) and close(). load polls cancelled between tensors and raises AudioCancelled after freeing what it held, so Stop ends a long load early. There is one module per family in backends/audio/ (yue2, qwen3_tts, qwen3_tts_mlx, stable_audio_3). Preset keys are cuda on Windows and Linux and mps on macOS (audio_platform_key() in backends/__init__.py returns it, and it is the torch device the families receive). Three families run on PyTorch on that device; qwen3_tts_mlx runs on MLX on macOS and uses the key only to pick its settings.

  • Data decides the backend. A model's platforms.<key> entry in the metadata may name its own family, so data, not code, picks which backend runs a model on each platform. AudioPreset.on_platform resolves a preset for this machine, and the CLI and the Web UI resolve before anything reads its family.
  • A preset joins two sources, by model id (utils/audio_preset.py): the facts from the model's entry in model_metadata.yaml (kinds, family, limits, voices, measured memory, pinned downloads) and the knobs from audio_model_presets.<id>.settings in the config. What a model can do follows from its kinds.
  • Backends keep no process-wide bookkeeping. The vendored kits change process-wide torch flags, and cached accelerator memory outlives a model, so audio_runner._backend_session snapshots the flags before load() and, after close(), restores them and releases the memory. Platform differences are defaults overridable in the preset's settings (YuE2's weight_swap: on for CUDA, off for MPS). Backends import torch lazily.
  • Steering settings are plain values on the request (tempo, key, delivery, sound_type). utils/audio_settings.py holds the vocabulary and parsers that audio_cli.py and web/audio_job.py share, so both refuse the same values with the same words, and the CLI refuses a flag the (model, kind) does not take before anything downloads. Per-model rules live in the backend of the model they belong to.

The families:

  • YuE2 runs the vendored kit stage by stage. utils/music_mode.music_mode decides song or instrumental from the lyrics. Weights stream in tensor by tensor, and on CUDA a WeightSwap (yue2_swap.py) keeps them in pinned CPU memory and moves only the running phase's group to the GPU, which is how it fits a 10 GB card with output identical to the stock kit's. On MPS, yue2_mps.py speeds up the token phases around the kit's calls, also with identical output. Tempo and key go through a written score header (yue2_score.py, pure); a step that fails falls back to the model's own plan with a note in AudioClip.notes.
  • Qwen3-TTS has two engines for one take logic (speech_takes.py): qwen3_tts.py on CUDA through the vendored kits, qwen3_tts_mlx.py on macOS through mlx-audio. The model has three checkpoints declared as download targets: CustomVoice (the built-in voices), Base (follows a recording or a saved voice, role voice) and VoiceDesign (makes a voice from a description, role design). An engine loads one checkpoint, once, so one batch needs one checkpoint, and audio_runner passes the preset with the matching file list. read_speech is the engine-neutral part: it splits long text into parts, reports progress per part, checks Stop between parts and joins the parts. Delivery reaches the model as one instruction sentence per part.
  • Stable Audio 3 builds the vendored model in fp16 and fills it tensor by tensor from the fp32 checkpoint. The library clamps its output to -1..1, so the backend keeps the waveform from before the clamp and audio_runner levels a loud take instead of saving a clipped one. sa3_caption.py wraps the prompt in the caption shell the model was trained on.
  • Voice design is an optional capability, not part of AudioBackend: the VoiceDesigner protocol. A caller checks isinstance(backend, VoiceDesigner), not the family. Its checkpoint is not part of audio_plan, so it never downloads with the model.

Background removal

The cut-out chain is MatteChain, implemented once by backends/matte/chain.TorchMatteChain on a torch device. create_matte_chain() picks the device (cuda or mps) and the cleanup after each cut-out; there is no per-platform implementation. BiRefNet loads from packages/birefnet, its DeformableConv2d modules are swapped for a grid_sample equivalent (deform_conv.py) because torchvision's op has no MPS kernel and a second code path would be worse, and ViTMatte follows. The pure steps are in processing/matte.py. The pinned repositories are the matte entry of the model metadata. See Background removal for the memory guard.

Image editing

ImageBackend.edit_region(model, image, reference, paint, prompt, seed, steps, guidance, ...) redraws the paint area of a 16 px aligned crop at strength 1.0 and returns an image of any size, or None when skipped. Only FLUX.2 Klein does it (supports_mask_edit, checked where the job is made, never by family).

Both platforms do two things. The model gets reference as a reference image, so it sees the picture around the paint. And after every step the latents outside the paint are replaced by the source's clean latents noised to that step's level with the run's own starting noise, so at the last step they are the clean ones. The mask is the painted fraction of each 16 px latent cell (soft_cell_mask). Both libraries shrink a reference over 1024 x 1024 px, and an Extend's reference, the drawn canvas, can pass that.

On Windows and Linux backends/cuda/image/edit.py prepares the noise, the clean latents and the mask; DiffusersBackend.edit_region passes the noise as latents= and runs the blend in the step-end callback before the preview reads the latents, so Stop and previews behave as for a generation. On macOS mflux's loop callbacks can't replace latents, and at strength 1.0 it never encodes the source, so backends/mac/image_edit.py encodes the crop itself, draws the same noise again and wraps the run's config.scheduler.step; MfluxBackend.edit_region then runs klein_references.generate_with_references. With fix_join both keep the last step's latents so match_strip can colour-match the strip to the source (match_join).

The Mac hooks depend on mflux's private loops, so tests/contracts/test_mflux_edit_contract.py pins them and mflux is bounded below 0.25; re-run it on every bump. tests/contracts/test_diffusers_image_edit_contract.py pins the diffusers names. The mflux contract test and tests/backends/mac/test_mac_image_edit_real.py need real MLX arrays, so they run on a Mac and are skipped elsewhere.

LoRA tensor names

Trainers name a LoRA's tensors in different ways, and each loader reads only some of them: diffusers' Krea 2, FLUX.2 and LTX-2 converters read no PEFT adapter name, its Krea 2 converter reads no lora_down/lora_up or alpha, its Ideogram 4 converter reads no Kohya lora_unet_ name, mflux's Ideogram 4 mapping reads neither an adapter name nor a bare key, and mflux and ltx-pipelines-mlx read only a path. utils/lora_keys.normalise_lora_keys(names, family, loader=...) is the one place that decides the names. It is pure and imports no torch. The detector (utils/lora_family) calls it with no family; every loader calls it for the model's family and its own loader (diffusers, mflux, mlx, moss or h3).

  • Policy. _POLICIES holds, per loader and family, only what that loader needs: drop the adapter name, write Kohya names dotted, rename lora_down/lora_up to lora_A/lora_B, give a bare key a namespace prefix, fold alpha into the weights. What a loader reads as it is stays as it is. A file with LyCORIS tensors keeps all its names, because the LoKr code reads its own.
  • Namespace. Which prefix a bare key gets depends on the file: native_roots and diffusers_roots are the first segments that tell the namespace. A file that mixes the two, or tells neither, is left alone unless the policy names a shared_namespace for it, as LTX does. The LTX root sets come from the modules of the diffusers and ltx_core_mlx models, and the test compares the trainer ones with diffusers' converter.
  • Kohya names. A flattened name is un-flattened by matching the whole name against LORA_MODULES, the family's real module names, never by splitting on _: attn_gate and to_q contain underscores themselves. A name no template spells, or two templates spell, is left alone, and the tests check that no two templates flatten to the same name.
  • Files. utils/lora_files.normalised_lora_state returns a file's tensors renamed, or None when the loader can read it as it is. readable_lora_files is a context manager that yields the path to hand the loader: the file itself, or a copy whose header carries the new names. A copy lives in a fresh folder under <data dir>/cache/loras/, is removed when the block ends, also on an exception, and is never reused. The process that made it holds an flock on it, and every call first deletes the copies it can lock, which are the ones a crashed process left.
  • When a copy goes. mflux reads a LoRA while the model is built, so MfluxBackend.load_model removes its copies as soon as the model is resident. ltx-pipelines-mlx reads its LoRAs when the transformer loads, and with low-RAM streaming every denoising step reads the file again, so the LTX handle owns its copies and a weakref.finalize removes them when the job drops the handle.

To teach it a new spelling, add the rule to lora_keys.py with a test in tests/common/test_lora_keys.py, and bump LORA_FAMILY_VERSION if what the detector sees changes.

MOSS-Image on Windows and Linux

backends/cuda/image/moss.py is the moss family's loader and generator; backend.py dispatches to it by family. The family has two built model folders, moss (Turbo) and moss-sft, built through built_models.ensure_built_model from the sources pinned in the model metadata. Turbo is the SFT checkpoint plus a 4-step LoRA kept in its folder. The same folders serve macOS.

  • Own loop. The vendored pipeline has no step callback, interrupt, start index or image input, so moss_sampler owns the denoising loop and mirrors the vendored one; a golden test pins both loop and decode against it.
  • Placement. The text encoder is NF4 at every level and sits on the CPU except while it encodes. At q4 the transformer stays on the GPU but steps aside for the text encoder and the VAE decode; at q8 and unquantized it streams block by block. The VAE is FP32 on the GPU. utils/model_memory._cuda_moss_estimate mirrors this.
  • Keep list. The vendored transformer reads weight.dtype on several Linears to choose its activation dtype, so NF4 or FP8 storage there corrupts the activations. KEEP_IN_COMPUTE_DTYPE in moss.py is the one list, expanded for bitsandbytes and for FP8 layerwise casting.
  • Adapters. moss_lora installs LoRA and LoKr files as unmerged wrappers around the target Linears, so they behave the same on BF16, FP8 and NF4 layers. It skips LoHa, .diff and DoRA with a warning. The Turbo LoRA goes first, in float32.
  • Stored copies. moss@q4, moss@q8 and the SFT equivalents are written from files with no LoRA. Turbo and SFT hold the same transformer and encoder, so the second alias hard-links the first's copy when it is current.

MOSS-Image on macOS

backends/mac/moss.py runs the moss family from mflux's parts instead of mflux's own MOSSImage, which has no start image. tests/contracts/test_mflux_moss_contract.py pins the mflux names it uses.

  • Two stages, never together. load reads the tokenizer and text encoder. prepare_prompts encodes the job's texts, frees the encoder and then loads the transformer and VAE. utils/model_memory._mlx_moss_estimate takes the larger stage.
  • Own loop. generate drives the transformer, scheduler and VAE and reports each step to mflux's callbacks, so skip, progress and previews work as for the other families. Above a pixel limit the VAE runs without the transformer resident.
  • LoRAs load unmerged (bake_lora=False) through moss_lora.moss_lora_mapping. Because mflux reads LoRAs only from files, a file mflux cannot fully apply goes to it as a filtered copy in a temporary folder, removed once the transformer has loaded.
  • Stored copies are written one stage at a time and leave nothing loaded, so runs_after_save is False and the job runs from the copy.

LTX on Windows and Linux

Video resolution is metadata-driven: ltx-4 and ltx-8 both declare the pinned diffusers repo, the quantize of each entry gives its level (there is no unquantized path), and the family's companions add the single-file transformer and the latent upsampler. utils.ltx_facts.ltx_load_options turns the declarations into the backend's load arguments for both the CLI and the Web UI. Only distilled LTX checkpoints are supported (LTX-2.3 and, as ltx25-4, LTX-2.5).

Versions. The family block describes LTX-2.3. A version is video_families.<family>.versions.<id> in the metadata: any keys of the family block, overlaid on it for the models whose entry names the version (dicts merge, every other value replaces). The merge order is CLI flags, then the version, then the family. ltx25 declares its own distilled LoRA, converter_base, image_crf and sampler, and a companions list with the latent upsampler only, so the repo's own transformer folder is used. Step defaults and other settings stay family-level in config.yaml. ltx_version returns the id of the version a model names (for a converted folder, the one whose model_type is the folder's manifest kind).

LTX-2.5 uses a Gemma 4 text encoder, selected by the text_encoder class in the repo's model index. Completeness checks ignore the LTX-2 components the app never loads (LTX2_UNLOADED_COMPONENTS), and the backend passes them as None.

cuda/video/backend.load_model keeps LTX within the system memory of a 32 GB machine and the GPU memory of a 10 GB one. The rules a change must keep:

  • The text encoder and the transformer never share system memory unless both fit. When they do not, they take turns: the transformer is released while the text encoder encodes each new prompt. LoRAs are applied when the transformer is attached.
  • The transformer is the single-file checkpoint declared as the family's transformer companion, or the repo's own folder when the version declares none. cuda/video/weights.py prepares it and the text encoder once per level under <models>/<alias>@q<bits>/, streaming tensor by tensor through safetensors_stream so a 46 GB file never sits in memory.
  • In low-memory mode the transformer streams to the GPU block by block (block_streaming): FP8 with diffusers' group offloading, NF4 with stream_blocks, because group offloading leaves bitsandbytes' quantization state on the CPU. The VAEs and vocoder visit the GPU only to encode or decode. Video never uses enable_model_cpu_offload().
  • Nothing may hold a full-length activation per token beyond the hidden states: token-wise layers run in chunks (token_chunking), image-to-video's per-token timesteps are applied per run of equal timesteps (timestep_runs), and the final pass returns latents that streaming_decode decodes a few frames at a time. Each keeps diffusers' output unchanged, and its tests compare against diffusers.

Sampler. A version's sampler selects how the distilled sigmas are walked. euler_ancestral (LTX-2.5) swaps in diffusers' ancestral scheduler for each denoising pass through cuda/video/sampler.py. A pass that carries conditions, which is every Extend pass, always uses the stock deterministic scheduler, because the ancestral step would re-noise the pinned tail; macOS follows the same rule. A version's image_crf re-compresses an image-to-video conditioning image before the app's fit to the clip's size, as upstream does.

Distilled sigmas. The backend passes the distilled sigma tables from diffusers.pipelines.ltx2.utils on every call, with num_inference_steps equal to their length and every guidance term neutral (_DISTILLED_GUIDANCE), because diffusers' own defaults for the audio, spatio-temporal and modality terms over-cook a distilled model.

Two-stage upscale (--upscale 2): stage 1 runs at half size with output_type="latent"; an LTX2LatentUpsamplePipeline upsamples the latents 2x (built around the generation pipeline's own VAE and never moved with .to() or given CPU offload, since either would re-hook that shared VAE); stage 2 calls the same LoRA-bearing pipeline at full size with the upsampled latents, the stage-1 audio latents and the stage-2 sigmas, with a static-shift scheduler for that call only, restored afterwards even on failure. Both stages report under the video_upscale_stage_1 and video_upscale_stage_2 phases, as macOS does.

LTX on macOS

Packs. The pack repositories hold files the app never loads, so each pinned pack is downloaded as a literal file list. The macOS downloads of ltx-4, ltx-8 and ltx25-4 each pin one pack (<repo>@<revision>) and list the files the vendored loader reads from it, and ltx_load_options turns them into HubFiles keyed by the pinned reference. utils.hf_snapshot.pack_for_reference is the one lookup of a pack by reference, shared by the planner, the resolver and the converter: a repo id typed by hand, a Web UI model and a custom alias all get the pinned file list, while a reference pinned to another revision names no pack and is downloaded whole. mac/video._resolve_model_dir loads the same list, and raises FileNotFoundError naming the revision and the files when a revision lacks one. Unless the pack bundles a text encoder, the pinned Gemma 3 reference is resolved the same way and passed to the pipeline as gemma_model_id.

downloads/plans.video_model_downloaded is the one offline answer to whether a model is on disk. utils.paths.canonical_model_name maps a Hugging Face reference to the alias on the current platform whose repo it is, so ziv-video runs a typed repo id as its alias, with the alias's pin and recorded name.

After changing a pin, list the files of the new revision that the loader reads in the model's downloads entry and run tests/contracts/test_ltx_mlx_contract.py. It checks every file the vendored loader picks from a folder of exactly the listed names, and that mlx_text_encoder names the pipeline's default Gemma 3 repo.

Aliases and text encoders. On macOS the alias is the first downloads target of its metadata entry, so an alias, its download plan and its load name one revision; version membership does not choose the pack. The ltx25-4 pack bundles its Gemma 4 text encoder at the folder root, and its own file list says so, so the planner skips the Gemma 3 repo for it. mac/video.load_model ignores sampler, image_crf and repo_ignore on purpose: the pack decides the sampler (is_ltx25_pack) and the MLX conditioner sets the image quality. It resolves a pinned owner/repo@revision to a snapshot folder itself, because the vendored pipeline downloads by repo id only.

Stopped generations. Under low_memory the vendored pipeline frees its parts only when a generation runs through. _generation_scope releases them after a stopped or failed generation and clears the MLX cache at the start of the next one, once the exception's traceback no longer holds the transformer.

LTX last frame

image_to_video(..., end_image=path) pins a second image on the clip's last frame. Callers pass load_model(end_image=True) for an image-to-video batch with a last frame, because the two backends need it differently. On macOS the last image becomes a keyframe condition whose tokens are appended at pixel frame num_frames - 1, and the output slice drops the appended tokens. On Windows and Linux LTX2ConditionPipeline replaces LTX2ImageToVideoPipeline and _keyframe_on_last_frame moves diffusers' keyframe coordinates to the real last frame for the call. Every pass keeps the stock scheduler, because the ancestral step would re-noise the appended tokens. MiniMax H3 takes end_image on its own clip methods.

Extending a clip

Extending is an optional capability, not part of VideoBackend: the VideoExtender protocol (extend_video(...)), which both LTX backends and both H3 backends implement. Callers gate on supports_video_extend(); the Web UI contract drops the extend workflow, and the CLI refuses --extend, when it is false.

On Windows and Linux, load_model(mode="extend") loads LTX2ConditionPipeline, and extend_video pins the tail frames in a one-stage window at the source's size and frame rate. diffusers gives every audio token one timestep, so cuda/video/audio_carry.py pins the carried sound itself, as the macOS denoise mask does: the tail's sound becomes a log-mel spectrogram encoded by the audio VAE, a transformer pre-hook puts those tokens clean at the start of the audio hidden states with timestep 0 on every forward pass, and the clean latent is put back before the audio is decoded. A user's audio track is held the same way for the whole clip, so the model makes only the picture. The track is read and encoded before the transformer is attached, so in low-memory mode the audio VAE is not on the GPU beside the streamed transformer.

On macOS an audio track is held as a frozen audio state, the way the vendored A2VidDistilledPipeline does it; without Upscale, and in Extend, the window's sound is encoded before the transformer loads and passed as a frozen initial latent, so both denoising loops condition the model on sigma 0 for a frozen stream. Extend with a track writes no segment.wav.

tests/contracts/test_diffusers_ltx2_contract.py and tests/contracts/test_ltx_mlx_contract.py pin the names and layouts this relies on. Re-run them on every diffusers or LTX bump.

Video families

LTX and MiniMax H3 share one seam, so code asks a family what it can do instead of branching on its name.

  • Registry. backends/__init__.py holds the (platform, family) map of backend factories. utils/video_model_detect maps an alias, a repository prefix or a local folder's contents to a family (ltx or h3); an H3 folder is recognised before the LTX path rule, since its path may contain "ltx".
  • Description. video_families.<family> in the model metadata, with the step and frame settings of video_model_presets.<family> in the config, is read by utils.config.video_family into a frozen core.video_family.VideoFamily: the label, the frame_rule, the capabilities (i2v, end_frame, extend, upscale, upscale_review, lora, audio, low_memory), step defaults and options, default_frames, extend_frame_step, carry_frames, and the optional extend_fps and upscale steps.
  • Platform overlay. platform_capabilities replaces some capabilities per sys.platform (H3 loses upscale on darwin), and platform_carry_frames overrides carry_frames. backends.video_capabilities(config, family) returns the result limited further by what this platform's backend supports, and backends.video_family_on_platform applies the carry. utils/video_support.unsupported_video_request turns a missing capability into the one refusal message the CLI and the Web UI share.
  • Frame rule. utils.alignment.align_frames(frames, rule) applies it (LTX 8k+1, H3 17k+5). The rule is part of the Workspace payload, so the frontend's duration maths uses it instead of a copy.
  • Steps. core.video_family.settle_video_steps moves a request to the family's step options or lowers it to the maximum, and returns a message when it changed the request.
  • Extend window. utils.video_extend.window_frames adds the carry to the new frames and moves the sum to the family's frame rule, and every consumer of the window (memory estimate, rewrite context, request) takes the family so they all see the same count. A source whose rate differs from extend_fps becomes a frame_rate problem that the Extend source route, the job submit and the CLI all refuse.
  • Hidden controls. web.workspace_contract.hidden_video_controls(capabilities) lists the controls a model lacks. The Workspace payload carries them beside the frame rule and step options.
  • Per-family estimates and downloads. utils.model_memory dispatches on family to LTX's estimators or utils/h3_memory, and downloads/plans.py dispatches the plan, so a new family adds a branch to each rather than to every caller.

A family's facts sit at video_families.<family> in model_metadata.yaml, with per-version overlays under versions and the downloads every model of the family shares under companions. Its settings (default_steps, default_frames, upscale.*) sit at video_model_presets.<family> in config.yaml. The macOS H3 preset carries the same frame_rule and schedules as CUDA, so both platforms run the same step counts, shifts and clip lengths.

MiniMax H3 on CUDA

backends/cuda/video/h3/backend.py loads one model for each stage just before it runs and frees it right after (text encoder, keyframe and tail encoders, denoise, video decode, audio decode), so the stages never overlap in memory. It never imports a private name from the LTX backend; helpers both need live in cuda/video/runtime.py. The diffusers pieces it uses are pinned by contract tests against one diffusers version (the tests assert the version, so an upgrade fails them first). After one, re-check tests/backends/cuda/test_cuda_h3_*.py and update cuda/video/h3/decode.py, which copies the VAE's chunked decode.

  • Conditioning rows. The packed sequence is [text | keyframe rows | audio | video]. A keyframe adds one latent frame of rows in front of the video block at timestep 0.999; an Extend tail holds the first latent frames of the video block plus its audio latents. Held rows are mixed with noise once and never written again: each step applies the schedulers to the generated rows only, selected by a boolean mask. A user's audio track holds every audio row, so the model makes only the picture.
  • Fold keys. One forward serves every row, each at its own timestep, and a block's AdaLN output depends only on the distinct timesteps of that forward. The sorted tuple of those timesteps is the fold key. schedule.ROW_SETS and the 4- and 8-step schedules decide which keys occur, and adaln.py builds a table per block and key; the stored copy holds the union, so every step count and mode loads a subset of one file. A clip whose audio rows are all held uses row sets whose tables are assembled from rows of stored keys (adaln.read_tables), since a row depends only on its own timestep. The key list is part of the copy's manifest: a changed schedule, row set or turbo LoRA scale rebuilds the copy.
  • User LoRAs. lora.split_user_lora routes a LoRA by target: AdaLN modules become a delta added to the fold tables, token-refiner modules are merged into the bf16 refiner, everything else goes to PEFT on the NF4 layers. A target on the timestep embedding is refused, because the tables are a function of it.
  • Long clips. The feed-forward layers run in chunks (ffn_chunk_tokens in the preset), which bounds their temporaries and lets 1280x736 and 345-frame clips fit a 10 GB GPU. The decode moves each chunk to system memory as 8-bit frames at once, so its GPU use does not grow with the clip.
  • Upscale. Stage 1 runs at half size and its waveform is kept; cuda/video/h3/latent_upsampler.py enlarges the latents 2x; the refine (sampler.refine) re-noises them to step 6 of the 8-step schedule and runs the last 2 forwards with the user LoRAs, in overlapping windows of latent frames planned from the clip length alone, so a seed gives the same clip on every card. The upsampler is a port of LBH-123-AI's LatentResizer3D with its attribute names unchanged, so the pinned weights load strictly. The sampler's latents are already normalised and the upsampler normalises them a second time: the reference node feeds the network that way and the upscale was validated that way, so the double normalisation is deliberate, and a test pins it.
  • One transformer load when it fits. Without a review the transformer is loaded once (_shared_transformer, with the union of the stage-1 and refine fold keys) and stays loaded through the waveform decode, the latent upscale and the keyframe encode. With a review the half-size decode needs the memory, so the stage-1 transformer is released and the refine loads its own. h3_memory.h3_upscale_keeps_transformer decides from the card's memory, and h3_clip_fit and h3_expected_seconds take the same decision, so the estimate has no second load when the backend will not make one.

MiniMax H3 on macOS

backends/mac/h3_video.py wraps the vendored port (see Vendored packages). It resolves the files, converts the text encoder once into the 4-bit h3@mlx-te4 copy, combines user LoRAs with the turbo LoRA into one rank-concatenated adapter, and extends a clip as image to video from its last frame. Upscale and review are refused, and the Workspace hides them through the platform overlay. It is tested with a fake minimax_h3_mlx module, syntax-tree contract tests and the staged path on tiny real MLX arrays; running a model needs a Mac.

  • Text to video runs the fork's low-memory pipeline and has no step hook, so Next and Quit take effect when the clip finishes.
  • Image modes (image to video, first and last frame, Extend) run through backends/mac/h3_staged.generate_conditioned, because the fork's image path keeps the text encoder, vision tower, both VAEs and the transformer resident at once. Each phase loads one component, uses it, drops it and returns host arrays, and release_memory() runs after every phase, so on unified memory the peak is about the largest component and the cost is re-reading weights from the SSD. h3_staged reimplements only the denoise loop; the other phases call the fork's functions. The step callback runs after every denoise step, so image modes report per-step progress and can be cancelled at a step boundary.
  • Audio track. A track holds every audio row of the clip, so the model makes the picture only: the audio VAE encodes the window, the rows replace the audio noise draw and run at timestep 1.0, the audio is never stepped, and the audio decode is skipped. The clip is written with silence and a workflow stage replaces it with the track.

tests/contracts/test_h3_mlx_contract.py pins the vendored names this uses.