Audio samples · Flickr8k test split · zero-shot multispeaker

Monotonicity-Guided Semantic Alignment for Zero-shot Multispeaker Image-to-Speech Synthesis

A Monotonic Query Aligner turns an image into a unit-rate sequence of conditioning states; a blockwise masked-diffusion generator writes speech units from them, end-to-end from random initialization. No text anywhere. The voice comes from a short reference recording used only for speaker conditioning, never as a prompt — including speakers the model has never heard.

Same sentence

Speaker conditioning from the utterance’s own recording (Flickr8k).

Cross sentence

Speaker conditioning from another sentence by the same speaker (Flickr8k).

Unseen speaker

Speaker conditioning from a speaker absent from all training data (LibriTTS-R).

Scores on the full test split ×100 · 4,997 utterances · wav2vec2-large-960h-lv60-self

Speaker conditioningB1B4METEORROUGE-LCIDErDistillMOSUTMOSSIM
Same sentence62.419.120.746.345.03.663.940.940
Cross sentence62.619.020.646.144.93.663.950.935
Unseen speaker (LibriTTS-R)63.818.421.247.246.23.984.440.935
Flickr8kAudio human recordings3.653.500.928

Best value per column in green. MOS scores are no-reference predictions on the rendered waveform. SIM is the WavLM-SV cosine similarity between the generated speech and the recording that supplied its speaker conditioning; the human line is two recordings of the same speaker (different speakers score {SIM_DIFF:.2f}). Inference uses no unit prompt; the reference recording only supplies the speaker conditioning. Unseen-speaker conditioning comes from LibriTTS-R test-clean recordings (38 speakers); the Flickr8k split is by image, so its own test speakers all occur in training.

Fifteen images, three voices each

Each card: the image with the human recording of that sentence (ground truth) on the left; then one generated clip per speaker-conditioning source — what the model says about the image in that voice. Compare each generated clip’s MOS with the ground truth’s. Transcripts under each clip are produced by the same wav2vec2-large-960h-lv60-self recognizer used for scoring. Per-clip BLEU-4 and CIDEr are single-utterance scores shown ×100 (CIDEr ×10).

Same sentencethe utterance’s own recording
Generated95 units · 3.8 s
DistillMOS 2.86UTMOS 3.64

ASR transcript · wav2vec2-large-960h-lv60-self“Two white dogs running in a grassy field with”

BLEU-4 51.3CIDEr 31.2

Cross sentenceanother sentence by the same speaker
Generated74 units · 3.0 s
DistillMOS 2.87UTMOS 3.44

ASR transcript · wav2vec2-large-960h-lv60-self“Two white dogs running in a grassy”

BLEU-4 68.0CIDEr 36.5

Unseen speakera speaker absent from all training data
Generated77 units · 3.1 s
DistillMOS 2.40UTMOS 4.48

ASR transcript · wav2vec2-large-960h-lv60-self“Two white dogs running in the grass”

BLEU-4 100.0CIDEr 44.5

Same sentencethe utterance’s own recording
Generated58 units · 2.3 s
DistillMOS 3.35UTMOS 4.34

ASR transcript · wav2vec2-large-960h-lv60-self“Two dogs play in the snow”

BLEU-4 67.3CIDEr 23.0

Cross sentenceanother sentence by the same speaker
Generated59 units · 2.4 s
DistillMOS 3.59UTMOS 4.35

ASR transcript · wav2vec2-large-960h-lv60-self“Two dogs playing in the snow”

BLEU-4 67.3CIDEr 27.9

Unseen speakera speaker absent from all training data
Generated63 units · 2.5 s
DistillMOS 3.19UTMOS 4.47

ASR transcript · wav2vec2-large-960h-lv60-self“Two dogs playing in the snow”

BLEU-4 67.3CIDEr 27.9

Same sentencethe utterance’s own recording
Generated66 units · 2.6 s
DistillMOS 3.11UTMOS 4.42

ASR transcript · wav2vec2-large-960h-lv60-self“Two white dogs playing in the water”

BLEU-4 79.5CIDEr 28.7

Cross sentenceanother sentence by the same speaker
Generated63 units · 2.5 s
DistillMOS 3.90UTMOS 4.18

ASR transcript · wav2vec2-large-960h-lv60-self“Two white dogs playing in the surf”

BLEU-4 59.5CIDEr 21.5

Unseen speakera speaker absent from all training data
Generated111 units · 4.4 s
DistillMOS 3.90UTMOS 4.48

ASR transcript · wav2vec2-large-960h-lv60-self“Two white dogs playing in the water in beach”

BLEU-4 57.7CIDEr 22.5

Same sentencethe utterance’s own recording
Generated46 units · 1.8 s
DistillMOS 3.04UTMOS 2.55

ASR transcript · wav2vec2-large-960h-lv60-self“A black dog walks on a beach near”

BLEU-4 64.3CIDEr 32.0

Cross sentenceanother sentence by the same speaker
Generated46 units · 1.8 s
DistillMOS 2.46UTMOS 3.77

ASR transcript · wav2vec2-large-960h-lv60-self“A black dog walking on the beach near”

BLEU-4 94.6CIDEr 22.8

Unseen speakera speaker absent from all training data
Generated79 units · 3.2 s
DistillMOS 3.95UTMOS 4.48

ASR transcript · wav2vec2-large-960h-lv60-self“A black dog walking on the beach”

BLEU-4 93.1CIDEr 21.1

Same sentencethe utterance’s own recording
Generated63 units · 2.5 s
DistillMOS 3.96UTMOS 3.66

ASR transcript · wav2vec2-large-960h-lv60-self“Two small white dogs running in the grass”

BLEU-4 54.1CIDEr 20.3

Cross sentenceanother sentence by the same speaker
Generated68 units · 2.7 s
DistillMOS 4.07UTMOS 4.10

ASR transcript · wav2vec2-large-960h-lv60-self“Two small white dogs running in the grass”

BLEU-4 54.1CIDEr 20.3

Unseen speakera speaker absent from all training data
Generated111 units · 4.4 s
DistillMOS 3.84UTMOS 4.47

ASR transcript · wav2vec2-large-960h-lv60-self“Two small white dogs running in the grass”

BLEU-4 54.1CIDEr 20.3

Same sentencethe utterance’s own recording
Generated59 units · 2.4 s
DistillMOS 3.88UTMOS 4.21

ASR transcript · wav2vec2-large-960h-lv60-self“Two children playing in the water in front”

BLEU-4 94.6CIDEr 22.6

Cross sentenceanother sentence by the same speaker
Generated62 units · 2.5 s
DistillMOS 3.80UTMOS 4.25

ASR transcript · wav2vec2-large-960h-lv60-self“Two children playing in the water in”

BLEU-4 93.1CIDEr 21.6

Unseen speakera speaker absent from all training data
Generated104 units · 4.2 s
DistillMOS 3.78UTMOS 4.52

ASR transcript · wav2vec2-large-960h-lv60-self“Two children playing in the water in the room”

BLEU-4 60.5CIDEr 16.2

Same sentencethe utterance’s own recording
Generated47 units · 1.9 s
DistillMOS 3.73UTMOS 4.39

ASR transcript · wav2vec2-large-960h-lv60-self“Two women walking down the street”

BLEU-4 100.0CIDEr 25.3

Cross sentenceanother sentence by the same speaker
Generated63 units · 2.5 s
DistillMOS 3.67UTMOS 4.19

ASR transcript · wav2vec2-large-960h-lv60-self“Two women walking on the sidewalk in the street”

BLEU-4 46.7CIDEr 15.6

Unseen speakera speaker absent from all training data
Generated63 units · 2.5 s
DistillMOS 3.80UTMOS 4.53

ASR transcript · wav2vec2-large-960h-lv60-self“Two women walking down the sidewalk”

BLEU-4 84.1CIDEr 20.9

Same sentencethe utterance’s own recording
Generated63 units · 2.5 s
DistillMOS 3.74UTMOS 4.45

ASR transcript · wav2vec2-large-960h-lv60-self“A surfer riding a wave in the ocean in the wave”

BLEU-4 32.1CIDEr 14.6

Cross sentenceanother sentence by the same speaker
Generated63 units · 2.5 s
DistillMOS 3.89UTMOS 4.38

ASR transcript · wav2vec2-large-960h-lv60-self“A surfer riding a wave in the ocean waved”

BLEU-4 40.8CIDEr 17.3

Unseen speakera speaker absent from all training data
Generated111 units · 4.4 s
DistillMOS 3.54UTMOS 4.54

ASR transcript · wav2vec2-large-960h-lv60-self“A surfer riding a wave on a surf board”

BLEU-4 41.5CIDEr 17.6

Same sentencethe utterance’s own recording
Generated78 units · 3.1 s
DistillMOS 3.79UTMOS 4.13

ASR transcript · wav2vec2-large-960h-lv60-self“A man a woman sitting on a branch with braycolor”

BLEU-4 46.7CIDEr 13.6

Cross sentenceanother sentence by the same speaker
Generated94 units · 3.8 s
DistillMOS 3.36UTMOS 4.02

ASR transcript · wav2vec2-large-960h-lv60-self“A man and woman sitting on the beach near a bird area”

BLEU-4 43.4CIDEr 14.0

Unseen speakera speaker absent from all training data
Generated110 units · 4.4 s
DistillMOS 3.49UTMOS 4.54

ASR transcript · wav2vec2-large-960h-lv60-self“A man and a woman sitting on a bench”

BLEU-4 86.3CIDEr 33.8

Same sentencethe utterance’s own recording
Generated63 units · 2.5 s
DistillMOS 3.20UTMOS 4.11

ASR transcript · wav2vec2-large-960h-lv60-self“A man riding a horse in front of a mountain”

BLEU-4 73.1CIDEr 19.3

Cross sentenceanother sentence by the same speaker
Generated79 units · 3.2 s
DistillMOS 3.42UTMOS 4.28

ASR transcript · wav2vec2-large-960h-lv60-self“A man riding a horse on a snowy mountain side”

BLEU-4 39.3CIDEr 14.7

Unseen speakera speaker absent from all training data
Generated79 units · 3.2 s
DistillMOS 3.86UTMOS 4.43

ASR transcript · wav2vec2-large-960h-lv60-self“A man riding a horse on a city slope”

BLEU-4 43.2CIDEr 13.0

Same sentencethe utterance’s own recording
Generated78 units · 3.1 s
DistillMOS 3.25UTMOS 4.43

ASR transcript · wav2vec2-large-960h-lv60-self“A man standing on an elephant watches a water”

BLEU-4 57.0CIDEr 18.3

Cross sentenceanother sentence by the same speaker
Generated78 units · 3.1 s
DistillMOS 3.76UTMOS 4.40

ASR transcript · wav2vec2-large-960h-lv60-self“A man stands on top of an elephant in the water in the”

BLEU-4 82.4CIDEr 30.7

Unseen speakera speaker absent from all training data
Generated95 units · 3.8 s
DistillMOS 3.56UTMOS 4.38

ASR transcript · wav2vec2-large-960h-lv60-self“A man standing on an elephant paring a bake”

BLEU-4 55.2CIDEr 12.7

Same sentencethe utterance’s own recording
Generated75 units · 3.0 s
DistillMOS 4.34UTMOS 4.39

ASR transcript · wav2vec2-large-960h-lv60-self“A man sits on a bench in a red shirt sits”

BLEU-4 59.0CIDEr 14.8

Cross sentenceanother sentence by the same speaker
Generated95 units · 3.8 s
DistillMOS 4.34UTMOS 4.22

ASR transcript · wav2vec2-large-960h-lv60-self“A man sits on a bench and talking on a sidewalk at a”

BLEU-4 41.9CIDEr 12.5

Unseen speakera speaker absent from all training data
Generated109 units · 4.4 s
DistillMOS 3.86UTMOS 4.45

ASR transcript · wav2vec2-large-960h-lv60-self“A man sitting on a bench in the city on a celphone”

BLEU-4 70.7CIDEr 14.5

Same sentencethe utterance’s own recording
Generated63 units · 2.5 s
DistillMOS 3.05UTMOS 4.28

ASR transcript · wav2vec2-large-960h-lv60-self“Two children playing on a swing”

BLEU-4 51.2CIDEr 14.9

Cross sentenceanother sentence by the same speaker
Generated63 units · 2.5 s
DistillMOS 3.41UTMOS 3.82

ASR transcript · wav2vec2-large-960h-lv60-self“Two children are playing on a swing”

BLEU-4 100.0CIDEr 19.8

Unseen speakera speaker absent from all training data
Generated79 units · 3.2 s
DistillMOS 3.48UTMOS 4.19

ASR transcript · wav2vec2-large-960h-lv60-self“A couple of children playing on a swing”

BLEU-4 39.8CIDEr 12.1

Same sentencethe utterance’s own recording
Generated79 units · 3.2 s
DistillMOS 4.02UTMOS 4.31

ASR transcript · wav2vec2-large-960h-lv60-self“A group of people sitting in front of a building outside”

BLEU-4 61.8CIDEr 14.1

Cross sentenceanother sentence by the same speaker
Generated94 units · 3.8 s
DistillMOS 4.05UTMOS 4.24

ASR transcript · wav2vec2-large-960h-lv60-self“A group of people sitting outside in front of a building”

BLEU-4 50.8CIDEr 13.2

Unseen speakera speaker absent from all training data
Generated95 units · 3.8 s
DistillMOS 4.13UTMOS 4.37

ASR transcript · wav2vec2-large-960h-lv60-self“A group of people sitting outside a front of a building”

BLEU-4 38.0CIDEr 12.0

Same sentencethe utterance’s own recording
Generated95 units · 3.8 s
DistillMOS 3.34UTMOS 4.25

ASR transcript · wav2vec2-large-960h-lv60-self“A person riding a dirt fike on a dirt hill”

BLEU-4 54.1CIDEr 14.1

Cross sentenceanother sentence by the same speaker
Generated62 units · 2.5 s
DistillMOS 2.88UTMOS 4.28

ASR transcript · wav2vec2-large-960h-lv60-self“A person riding a motorcycle”

BLEU-4 54.8CIDEr 11.2

Unseen speakera speaker absent from all training data
Generated118 units · 4.7 s
DistillMOS 3.97UTMOS 4.46

ASR transcript · wav2vec2-large-960h-lv60-self“A person riding a motor bike down a dirt road”

BLEU-4 39.3CIDEr 13.8