lipsync-localize

I'd like you to add a skill to Wideframe called "lipsync-localize".

Save the SKILL.md below exactly as written, then confirm I can run it by saying /lipsync-localize.

---
name: lipsync-localize
description: Translate and dub video into other languages with HeyGen lip-synced localization. Use when the user asks to "translate this video", "dub into Spanish", "localize for", "lip sync translate", "make a German version", "multi-language version", or wants a video re-voiced in another language with matched mouth movement. Handles suitability pre-flight, progressive trust (test segment before full run), multi-language batching, QC transcript comparison, and sequence delivery.
argument-hint: [path-to-video] [target-language]
---

# Lip-Sync Localize

Translate video into other languages with HeyGen lip-synced dubbing. You are the **orchestrator** — you coordinate subagents, review their work, manage user trust, and never run HeyGen API calls directly.

**Core philosophy: don't waste time and tokens.** Flag problems before expensive API calls. Build confidence with small test runs before committing to full localizations. Never batch multiple languages until the first one proves out. Always check with the user — they have taste and insight that models do not.

## Architecture

```
You (Orchestrator)
  ├── Pre-flight Subagent    → media analysis, suitability assessment
  ├── HeyGen Subagent        → API calls, job monitoring, output handling
  └── QC Subagent            → mechanical verification, transcript comparison
```

You delegate work, review results, and own the user conversation. Subagents do the heavy lifting but never talk to the user directly.

## Phase 0: Workspace & Source Setup

1. If no workspace exists, create one with `workspace-init` — name it after the content being localized (e.g., "Product Demo Localization"). Write a `BRIEF.md` capturing the source video, target languages, and user goals.
2. Symlink (never copy) source video into `source/video/`.
3. Create the workspace directory structure:
   ```
   source/
     video/          # Symlinked source
     localized/      # HeyGen outputs, organized by language
       test-segments/ # Progressive trust test clips
   sequences/        # Source + localized sequences
   analysis/         # Content analysis + transcripts
   ```
4. Link the workspace to the user immediately after creation.

## Phase 1: Pre-flight & Suitability

**Delegate to a `general-purpose` subagent** for analysis. The subagent should:

1. Run `media-info-batch` on the source to get resolution, duration, codec, audio channels.
2. Run `media-analyze-content-batch` and `media-transcript-get-batch` in parallel on the source.
3. After results arrive, read all analysis and transcript outputs.
4. Produce a structured pre-flight report covering:
   - **Source profile**: resolution, duration, frame rate, audio quality
   - **Speaker analysis**: number of speakers, overlapping speech segments, speech rate (WPM from transcript)
   - **Visual analysis**: face visibility, occlusion moments, side-profile shots, on-screen text/graphics
   - **Audio analysis**: background music/noise levels, audio clarity
   - **Content summary**: what the video is about (1-2 sentences)

**Review the subagent's report yourself, then assess suitability.** Read [HeyGen Limitations](references/heygen-limitations.md) and score each risk factor. Present the user with:

- A brief content summary
- The source profile (resolution, duration, speakers)
- **Suitability assessment** — flagged concerns with severity and what to expect:
  - HIGH risk: recommend against proceeding or warn about likely poor results
  - MEDIUM risk: proceed with awareness, results may need manual touch-up
  - LOW/CLEAR: good candidate for localization
- The target language(s) confirmed
- Mode recommendation: `lip-sync` (default) vs `audio-only` (if face visibility is poor)
- Whether a test segment is recommended (videos > ~2 min) or if full run is appropriate (short videos)

**Do not proceed until the user acknowledges the assessment.** If suitability is HIGH risk, explicitly recommend the user consider whether it's worth the run.

## Phase 2: Progressive Trust

**Goal:** prove quality on a small segment before committing to a full run.

**When to use progressive trust:**
- Source video > ~2 minutes: always
- Source video ≤ 2 minutes: skip to Phase 3 (full run) unless suitability flags are MEDIUM or higher

**Read [Progressive Trust Protocol](references/progressive-trust.md) for segment selection rules.**

### Test Segment Workflow

1. **Select a representative segment** (~30-60s) from the source that contains the most challenging content identified in Phase 1 (face occlusion, fast speech, multiple speakers, expressive delivery). If no challenges were flagged, pick a segment from the middle third of the video with continuous speech.

2. **Extract the test segment** using `ffmpeg` — copy codec, no re-encode:
   ```
   ./ffmpeg -ss <start> -to <end> -i <source> -c copy <workspace>/source/localized/test-segments/<filename>_test.mp4
   ```

3. **Delegate to a `general-purpose` subagent** for the HeyGen call:
   - Run `caltools media-heygen-localize-video --source <test-segment> --target-lang "<language>" --workspace-path <workspace> --output <workspace>/source/localized/test-segments/`
   - The subagent reports back the output path and any errors.

4. **After the HeyGen job completes**, delegate QC to a `general-purpose` subagent:
   - Transcribe the localized test segment with `media-transcript-get-batch` using the target language (`--language <code>`)
   - Transcribe the corresponding source segment (already done in Phase 1 — extract the matching time range from existing transcript)
   - Compare: are the same ideas/concepts conveyed? Flag semantic drift, missing sentences, or garbled output.
   - Check mechanical properties: duration match (within 1s), audio presence throughout, no silence gaps > 0.5s, video plays without corruption.
   - Produce a QC report.

5. **Review the QC report yourself.** Then create a short sequence with the test segment (source on V1, localized on V2 or as a separate sequence) so the user can compare. Present to the user:
   - The localized test segment on a sequence (link it)
   - QC findings (mechanical: pass/fail; semantic: summary of transcript comparison)
   - Your assessment of quality
   - Ask: "Does this quality level work for the full video? Should I proceed with the complete localization?"

**Do not proceed to Phase 3 until the user confirms.** If the user is unhappy, discuss options: try `audio-only` mode, try a different segment, or abandon.

## Phase 3: Full Localization

Once trust is established (test segment approved or short video with low risk):

1. **Delegate to a `general-purpose` subagent** for the HeyGen call:
   - Run `caltools media-heygen-localize-video --source <full-source> --target-lang "<language>" --workspace-path <workspace> --output <workspace>/source/localized/<language>/`
   - Report back output path, duration, and any errors.

2. **Multi-language handling:**
   - If the user requested multiple languages, run the **first language only** initially.
   - After the first language passes QC and user review (Phase 4), decide whether to batch the rest:
     - If user expressed high confidence after reviewing the first language ("looks great, do the rest"), run remaining languages in parallel (respecting the 2-job concurrency limit).
     - If user was lukewarm or had notes, run the next language individually and check in again.
     - Never run all languages simultaneously without established confidence.

3. **Output organization:**
   ```
   source/localized/
     spanish/
       video_spanish.mp4
     german/
       video_german.mp4
     test-segments/
       video_test.mp4
       video_test_spanish.mp4
   ```

## Phase 4: QC & Delivery

For each localized video:

1. **Delegate QC to a `general-purpose` subagent:**
   - Transcribe the localized video in the target language (`media-transcript-get-batch --language <code>`)
   - Compare semantic content against source transcript — same ideas, same order, no omissions, no hallucinated additions
   - Mechanical checks: total duration within 2s of source, continuous audio (no dead gaps), video integrity (no frozen frames, no artifacts at segment boundaries)
   - Produce a structured QC report

2. **Review the QC report yourself.** The subagent catches mechanical issues; you catch anything it missed and form an overall assessment.

3. **Build delivery sequences:**
   - Create one sequence per language, named descriptively (e.g., `Product Demo - Spanish`, `Product Demo - German`)
   - Each sequence contains the full localized video as a single clip on V1/A1
   - Match sequence settings to source resolution/frame rate
   - Optionally create a comparison sequence with source and localized side-by-side if the user wants to review

4. **Present to the user:**
   - Link each localized sequence
   - QC summary per language (mechanical pass/fail + semantic comparison highlights)
   - Your overall assessment
   - **Always ask the user to review.** Never declare the work "done" — the user has final say on quality, tone, naturalness, and brand voice in the translated version. Phrases like "everything looks good on my end, but please review — you'll catch things I can't" are appropriate.

5. **If the user has notes or wants changes:**
   - For re-runs: use `--force-refresh` to bypass cache
   - For mode changes: switch between `lip-sync` and `audio-only`
   - For different segments: extract and re-run
   - Update `BRIEF.md` with any new constraints or preferences

## Subagent Delegation Guide

All subagent calls use `Task` with `subagent_type: "general-purpose"`. Structure each delegation clearly:

**Pre-flight subagent prompt pattern:**
> Analyze the source video at `<path>` for HeyGen lip-sync localization suitability. Run `media-info-batch`, `media-analyze-content-batch`, and `media-transcript-get-batch`. After all results arrive, read the analysis and transcript outputs and produce a pre-flight report covering: [list fields from Phase 1].

**HeyGen execution subagent prompt pattern:**
> Run HeyGen lip-sync localization on `<path>`. Execute: `caltools media-heygen-localize-video --source <path> --target-lang "<lang>" --output <output-dir>`. Report back the output file path, job status, and any errors.

**QC subagent prompt pattern:**
> QC the localized video at `<path>` against the source at `<source-path>`. Transcribe the localized video with `media-transcript-get-batch --language <code>`. Read both transcripts. Compare semantic content — same ideas conveyed, same order, no omissions. Check mechanical properties: duration within 2s of source, continuous audio, video integrity. Produce a structured QC report with pass/fail per category.

## Failure Modes

| Symptom | Cause | Recovery |
|---|---|---|
| HeyGen job times out | Source too long, server load | Retry with `--timeout 1200`; if still failing, split source into segments |
| Lip sync looks wrong | Side profile, occlusion, fast speech | Flag in suitability; try `audio-only` mode; pick cleaner segment for test |
| Translation semantically wrong | HeyGen transcription error upstream | Compare source transcript against what HeyGen received; if source has errors, the translation will too |
| Audio gaps in localized video | Speaker pauses interpreted differently | Mechanical QC catches this; report to user |
| Test segment looks good but full video doesn't | Challenge concentrated outside test segment | Select a second test segment from the problem area; re-evaluate |
| Multiple languages, first is bad | Source material fundamentally unsuitable | Stop the batch; discuss with user before proceeding |
| User unhappy with voice quality | Voice cloning limitations | Suggest `audio-only` if not already; note this is a HeyGen platform limitation |

## What This Skill Does NOT Do

- **Translate on-screen text or graphics** — HeyGen translates speech only. On-screen text remains in the original language. Flag this in pre-flight if text is prominent.
- **Replace the HeyGen voice model** — voice quality is determined by HeyGen's voice cloning. If the user dislikes the cloned voice, that's a platform limitation.
- **Edit the localized video** — this skill delivers localized footage on sequences. Further editing (trimming, adding captions, B-roll overlay) uses the `edit-sequence` skill.
- **Guarantee translation accuracy** — QC compares semantic content but is not a professional human translation review. Always note this to the user.