@celesta/aivis-speech

@celesta/aivis-speech reads an AivisSpeech AudioQuery together with the WAV synthesized from it: where each mora, phoneme, and pause falls in the WAV, and a LipSyncTrack built from that timing. Dialogue components and LipSyncTrack are in @celesta/character; VOICEVOX voices use @celesta/voicevox.

AivisSpeech serves the VOICEVOX API, but its model chooses every phoneme length itself, so the query’s lengths are dummy zeros. The query gives the moras in order, including ん, っ, and punctuation; the WAV gives where they fall. The voice’s start and end are exact; the times inside it are estimated from the WAV’s loudness, so pauses land on the silences the model made.

The desktop app and CLI bundle this package separately. Building the runtime builds it too, and File Set Up TypeScript includes its declarations and import mapping. It is private and is not published to npm. The browser playground does not support it.

Drive a dialogue line from its query and WAV

Use the AudioQuery that produced the WAV: its speedScale, prePhonemeLength, and postPhonemeLength locate the voice in the WAV. Read the WAV in prepare() and pass its bytes. The package does not synthesize audio or call an AivisSpeech server. Save the AudioQuery as aivis-speech-query.json beside aivis-speech.tsx, and use your own voice and mouth images in the example below.

import * as React from 'react';
import { Assets, Composition, Rect } from '@celesta/react';
import { Character, CharacterView, Dialogue } from '@celesta/character';
import type { AssetReference } from '@celesta/react';
import type { CharacterViewReference, LipSyncTrack } from '@celesta/character';

import { readFile } from 'node:fs/promises';
import { join } from 'node:path';
import { lipSyncFromAivisSpeech } from '@celesta/aivis-speech';
import query from './aivis-speech-query.json';

// The WAV is needed too: AivisSpeech queries hold no phoneme lengths.
let track: LipSyncTrack;
export async function prepare() {
  track = lipSyncFromAivisSpeech(query, await readFile(join(import.meta.dirname, 'voices/hello.wav')));
}
const mira = React.createRef<AssetReference>();
const miraView = React.createRef<CharacterViewReference>();

export default function Root() {
  return (
    <Composition width={1920} height={1080} fps={30} durationInFrames={Math.ceil(track.durationInSeconds * 30)}>
      <Rect width={1920} height={1080} fill="#20243a" />
      <Assets>
        <Character
          ref={mira}
          name="Mira"
          portrait={{
            defaultExpression: 'calm',
            expressions: {
              calm: './mira/calm.png',
              smile: './mira/smile.png',
            },
            lipSync: {
              a: './mira/mouth-a.png', i: './mira/mouth-i.png',
              u: './mira/mouth-u.png', e: './mira/mouth-e.png',
              o: './mira/mouth-o.png', closed: './mira/mouth-closed.png',
            },
          }}
          subtitle={{
            x: 960, y: 940, anchorX: 0.5, anchorY: 0.5, maxWidth: 1600,
            style: {
              fontSize: 56,
              fill: { type: 'solid', color: '#ffffff' },
              stroke: { paint: { type: 'solid', color: '#3a2d52' }, width: 6 },
              align: 'center',
            },
          }}
        />
      </Assets>

      <CharacterView ref={miraView} character={mira} x={1400} y={200} />

      <Dialogue character={miraView} audio="./voices/hello.wav" lipSync={track}>
        こんにちは、はじめまして。
      </Dialogue>
    </Composition>
  );
}

@celesta/aivis-speech

API	Use it for
`lipSyncFromAivisSpeech(query, wav)`	Convert an AudioQuery and its WAV (bytes or the result of decodeWav) into a LipSyncTrack, with time zero at the start of the WAV, including leading silence.
`lipSyncFromAivisSpeech(timing)`	The same track from a timing already computed, so the mouth and the timing cannot disagree.
`timingFromAivisSpeech(query, wav)`	Seconds from the start of the WAV for every mora, phoneme, pause, and accent phrase, shaped like the query (accentPhrases[i].moras[j] is accent_phrases[i].moras[j]). Use it to light moras as they are spoken or change a face at a pause.

AivisSpeech keeps punctuation as moras whose vowel is pau and whose text is the normalized mark (, . ! ? …). They stay in moras and are also listed in pauses, sentence-final marks included; a mark the model did not pause at is short or empty.

Both engines’ timings are a SpeechTiming from @celesta/character: exact: true from timingFromVoicevox, exact: false from timingFromAivisSpeech. Code that reads timings, such as lit moras or a face change at a pause, takes either, and lipSyncFromSpeechTiming builds the mouth from any of them.

Against VOICEVOX voices with known phoneme times (138 lines, six voices, three speeds), mora starts were within 14 ms at the median and 71 ms at the 90th percentile, and the mouth matched 83% of the time, against 51% for loadLipSync. Pauses anchor the estimate; in the middle of a long clause voiced without a break, errors grow. For VOICEVOX voices use timingFromVoicevox, which is exact.

Exports also include AivisSpeechTiming, AivisSpeechAudioQuery, AivisSpeechAccentPhrase, and AivisSpeechMora. SpeechTiming, its parts, speechVowelShape, and LipSyncTrack are imported from @celesta/character.

Drive a dialogue line from its query and WAV

Try it without installing.