Skip to content本文へ移動
Celesta
Download
Download

Character dialogue

Portraits, subtitles, voices, and lip sync.

Dialogue scenes are built from three pieces. Each works in React and in JSON projects:

  • A character defines a portrait (one image per expression, or a layered PSD), optional lip-sync mouths, and how that character’s subtitles look.
  • A character view places the portrait on screen.
  • A dialogue line shows a subtitle, plays a voice recording, and can change the view’s expression or mouth while it is on screen.

Your first conversation in React

Save the file below with two portrait images and two voice recordings beside it:

import * as React from 'react';
import { Assets, Composition, Rect, Sequence } from '@celesta/react';
import { Character, CharacterView, Dialogue } from '@celesta/character';
import type { AssetReference } from '@celesta/react';
import type { CharacterViewReference } from '@celesta/character';

const mira = React.createRef<AssetReference>();
const miraView = React.createRef<CharacterViewReference>();

export default function Root() {
  return (
    <Composition width={1920} height={1080} fps={30} durationInFrames={180}>
      <Rect width={1920} height={1080} fill="#20243a" />
      <Assets>
        <Character
          ref={mira}
          name="Mira"
          portrait={{
            defaultExpression: 'calm',
            expressions: {
              calm: './mira/calm.png',
              smile: './mira/smile.png',
            },
          }}
          subtitle={{
            x: 960, y: 940, anchorX: 0.5, anchorY: 0.5, maxWidth: 1600,
            style: {
              fontSize: 56,
              fill: { type: 'solid', color: '#ffffff' },
              stroke: { paint: { type: 'solid', color: '#3a2d52' }, width: 6 },
              align: 'center',
            },
          }}
        />
      </Assets>

      <CharacterView ref={miraView} character={mira} x={1400} y={200} />

      <Sequence durationInFrames={90}>
        <Dialogue character={miraView} audio="./voices/line-01.wav">
          Welcome back. Shall we begin?
        </Dialogue>
      </Sequence>
      <Sequence from={90} durationInFrames={90}>
        <Dialogue character={miraView} expression="smile"
          audio="./voices/line-02.wav">
          Every scene starts with a single line.
        </Dialogue>
      </Sequence>
    </Composition>
  );
}
  1. Declare the character inside <Assets>. The ref lets views refer to it. subtitle takes the same position and style props as Text.
  2. Place a view with <CharacterView>. It shows the defaultExpression unless a line says otherwise, and takes x, y, scale, and anchors like any layer.
  3. Time each line by wrapping <Dialogue> in a Sequence. The subtitle, voice, and expression apply only while it is active, then the view returns to its default.

Adjust a line with volume, startFrom, playbackRate, or muted, just as on Audio. Setting x, y, or opacity on a Dialogue moves or fades its subtitle, which is handy inside a Transition.

Emphasize part of a line with <Span> inside the Dialogue. A subtitle.render receives the line as written in content: draw it with <Text style={style}>{content}</Text> to keep the emphasis; text is the same line as a plain string.

Automatic lip sync

loadLipSync() listens to a voice recording and spreads the vowels of its transcript across the spoken parts, producing a mouth shape (a, i, u, e, o, or closed) for every moment. Call it from prepare() and pass the result to a Dialogue or CharacterView as lipSync.

import * as React from 'react';
import { Assets, Composition } from '@celesta/react';
import { Character, CharacterView, Dialogue, loadLipSync, loadPsdPreset } from '@celesta/character';
import type { AssetReference } from '@celesta/react';
import type { CharacterViewReference, LipSyncTrack } from '@celesta/character';

const FPS = 30;
const VOICE = './voices/hello.wav';
const LINE = 'こんにちは、はじめまして！';

const hana = React.createRef<AssetReference>();
const hanaView = React.createRef<CharacterViewReference>();

// Filled once by prepare(), before the first frame renders.
let pose: string[] = [];
let voice: LipSyncTrack | null = null;

export async function prepare() {
  pose = await loadPsdPreset({ src: './hana/hana.pfv', favorite: 'smile' });
  voice = await loadLipSync({ src: VOICE, text: LINE });
}

export default function Root() {
  const seconds = (voice?.durationInSeconds ?? 2) + 0.5;
  return (
    <Composition width={1920} height={1080} fps={FPS}
      durationInFrames={Math.ceil(seconds * FPS)}>
      <Assets>
        <Character
          ref={hana}
          name="Hana"
          portrait={{
            type: 'psd',
            src: './hana/hana.psd',
            layers: pose,
            lipSync: {
              a: 'face/mouth/a', i: 'face/mouth/i', u: 'face/mouth/u',
              e: 'face/mouth/e', o: 'face/mouth/o',
              closed: 'face/mouth/closed',
            },
          }}
          subtitle={{
            x: 960, y: 960, anchorX: 0.5, anchorY: 0.5,
            style: {
              fontSize: 60, align: 'center',
              fill: { type: 'solid', color: '#ffffff' },
              stroke: { paint: { type: 'solid', color: '#c2577a' }, width: 6 },
            },
          }}
        />
      </Assets>

      <CharacterView ref={hanaView} character={hana} x={1300} y={120} />
      {voice && (
        <Dialogue character={hanaView} audio={VOICE} lipSync={voice}>
          {LINE}
        </Dialogue>
      )}
    </Composition>
  );
}
  • Voices must be uncompressed WAV files. Write the transcript in kana or romaji so its vowels can be read; kanji are skipped.
  • PSD portraits name their mouth layers by path, such as face/mouth/a. Layered “tachie” files usually save every folder hidden, so pick visible layers with layers: an array of layer paths, a PSDTool layer-state string, or a .pfv favorite loaded with loadPsdPreset().
  • Image portraits can lip-sync too. Give portrait.lipSync one transparent mouth image per shape, the same size as the portrait, so each lines up when drawn over it.
  • To drive a mouth yourself, pass mouth="a" (or another shape) to a CharacterView or Dialogue. useLipSync(track) returns the current shape if you want to react to it elsewhere.

Sequence a conversation automatically

Call planDialogue(lines, { fps, gap, sceneLeadIn }) in prepare(). It measures each audio file unless durationInFrames is supplied, then places the lines back to back. gap and leadIn use seconds; durations use frames. Every line needs audio or a positive durationInFrames. Set Composition durationInFrames from the plan and pass speaker names to DialogueSeries through views. The plan exposes line(), startOf(), range(), and scene() for other direction.

import * as React from 'react';
import { Assets, Composition, Rect } from '@celesta/react';
import { Character, CharacterView, DialogueSeries, planDialogue } from '@celesta/character';
import type { AssetReference } from '@celesta/react';
import type { CharacterViewReference, DialoguePlan } from '@celesta/character';

const mira = React.createRef<AssetReference>();
const miraView = React.createRef<CharacterViewReference>();

let plan: DialoguePlan;

export async function prepare() {
  plan = await planDialogue([
    { id: 'hello', speaker: 'mira', text: 'Welcome back. Shall we begin?',
      audio: './voices/line-01.wav' },
    { id: 'start', speaker: 'mira', text: 'Every scene starts with a single line.',
      audio: './voices/line-02.wav', expression: 'smile' },
  ], { fps: 30, gap: 0.25 });
}

export default function Root() {
  return (
    <Composition width={1920} height={1080} fps={30} durationInFrames={plan.durationInFrames}>
      <Rect width={1920} height={1080} fill="#20243a" />
      <Assets>
        <Character
          ref={mira}
          name="Mira"
          portrait={{
            defaultExpression: 'calm',
            expressions: {
              calm: './mira/calm.png',
              smile: './mira/smile.png',
            },
          }}
          subtitle={{
            x: 960, y: 940, anchorX: 0.5, anchorY: 0.5, maxWidth: 1600,
            style: {
              fontSize: 56,
              fill: { type: 'solid', color: '#ffffff' },
              stroke: { paint: { type: 'solid', color: '#3a2d52' }, width: 6 },
              align: 'center',
            },
          }}
        />
      </Assets>

      <CharacterView ref={miraView} character={mira} x={1400} y={200} />

      <DialogueSeries plan={plan} views={{ mira: miraView }} holdThroughGap />
    </Composition>
  );
}

holdThroughGap keeps the previous line’s text and expression through its gap. holdSubtitle lets a custom subtitle.render keep its background between lines; its held flag identifies the gap. subtitle.render receives text, character (including displayName), metrics, style, maxWidth, frame, and durationInFrames so you can draw a measured subtitle panel. See the repository’s with-dialogue-series.tsx for a complete example.

For voices generated with VOICEVOX or AivisSpeech, use the synthesis AudioQuery with @celesta/voicevox or @celesta/aivis-speech instead of spreading the transcript’s vowels over the WAV.

See it with real artwork

packages/cli/examples/with-dialogue-series.tsx demonstrates an automatically timed conversation; packages/cli/examples/with-lip-sync.tsx shows a PSD portrait with lip sync. On GitHub, choose Code Download ZIP, extract the complete archive including assets, and open an entry with File Open…. No source build is needed.

Try it without installing.

The playground runs Celesta in your browser, export included.

Open the playground