How do I use the LMNT Unity SDK to generate character dialogue at runtime?

Quick Answer: Use the LMNT Unity SDK to stream low-latency text-to-speech directly into your game objects at runtime. You define characters, pick LMNT voices or clones, send dialogue lines to LMNT’s API from C#, and play the returned audio via AudioSource—all fast enough (150–200ms) to feel conversational in live gameplay.

Why This Matters

In Unity, character dialogue is only convincing if it sounds natural and responds in real time. Pre-rendered VO locks you into fixed scripts, bloats build size, and breaks when you add dynamic or AI-driven interactions. With LMNT’s Unity SDK, you can generate dialogue at runtime—using studio-quality voice clones from just a 5-second sample—so NPCs, agents, and companions can react to the player, the world state, or an LLM in the moment.

Key Benefits:

• Real-time, responsive dialogue: 150–200ms low-latency streaming keeps NPCs and agents in sync with gameplay and player actions.
• Distinct voices for every character: Use built-in voices or studio-quality clones from short recordings to give each character a consistent identity.
• Scales with your game: Generate dialogue on demand without pre-baking thousands of lines or hitting concurrency limits—LMNT has no concurrency or rate limits and pricing improves with volume.

Core Concepts & Key Points

Concept	Definition	Why it's important
Runtime TTS in Unity	Generating speech audio from text (or LLM output) during gameplay, then playing it via AudioSource.	Enables dynamic NPC dialogue, AI companions, and agents that react to real-time game state instead of fixed scripts.
Streaming vs. batch generation	Streaming sends text to LMNT and receives audio in chunks as it’s generated; batch waits for the full file.	Streaming TTS with 150–200ms latency keeps conversations feeling natural for turn-taking, especially in agents and co-op assistants.
Voice assignment & cloning	Mapping LMNT voices or your clones to specific Unity characters or prefabs.	Keeps character identity consistent and lets you scale to many characters without separate VO recording sessions.

How It Works (Step-by-Step)

You’ll wire LMNT’s streaming text-to-speech into Unity so your characters can speak any line—scripted or AI-generated—at runtime.

1. Set up LMNT and your Unity project

1. Create your LMNT account.

   • Go to lmnt.com.
   • Try voices in the free Playground to find a style that fits your character.
   • When you’re ready to integrate, grab your API key from the Developer section.

2. Install the LMNT Unity SDK.

   • Add the SDK via the Unity Package Manager (Git URL or local package, depending on how LMNT distributes it).
   • Make sure your project has:
     • Scripting Runtime: .NET 4.x equivalent
     • API compatibility suitable for HTTP/WebSocket (for streaming)
   • Confirm you can reference the LMNT namespaces (e.g., using LMNT;) in a C# script.

3. Configure your LMNT client.

   • Create a singleton or service class to hold the LMNT client and API key.
   • Store your API key securely (e.g., in environment variables for builds, not hard-coded in public repos).

   public class LmntClientProvider : MonoBehaviour
   {
       public static LmntClientProvider Instance { get; private set; }
   
       [SerializeField] private string apiKey; // For demo only; use safer storage in production.
   
       public LmntClient Client { get; private set; }
   
       private void Awake()
       {
           if (Instance != null && Instance != this)
           {
               Destroy(gameObject);
               return;
           }
   
           Instance = this;
           DontDestroyOnLoad(gameObject);
   
           Client = new LmntClient(apiKey);
       }
   }
   

2. Choose voices and (optionally) create clones

4. Pick default voices in the Playground.

   • In the LMNT Playground, experiment with voices for:
     • Companions / narrators
     • Vendors or quest givers
     • AI tutors or coaches
   • Note the voice IDs or names you want to use in Unity.

5. Create studio-quality voice clones (optional).

   • Record or capture ~5 seconds of clean speech for each character.
   • Upload to LMNT and create a clone in the Playground.
   • Assign the resulting clone IDs to your Unity character configs.
   • All LMNT voices, including clones, speak 24 languages, with natural mid-sentence switching—useful for multilingual characters.

6. Map voices to Unity characters.

   • Add a simple script to your character prefab:

   public class CharacterVoiceProfile : MonoBehaviour
   {
       [Tooltip("LMNT voice or clone ID for this character")]
       public string voiceId;
   
       [Tooltip("Optional: language code like en-US, es-ES, etc.")]
       public string languageCode = "en-US";
   }
   

3. Stream dialogue into Unity at runtime

7. Create a dialogue player component.

   Attach this script to each character with an AudioSource:

   [RequireComponent(typeof(AudioSource))]
   public class LmntDialoguePlayer : MonoBehaviour
   {
       private AudioSource _audioSource;
       private CharacterVoiceProfile _voiceProfile;
   
       private void Awake()
       {
           _audioSource = GetComponent<AudioSource>();
           _voiceProfile = GetComponent<CharacterVoiceProfile>();
       }
   
       public async Task SpeakAsync(string text)
       {
           if (LmntClientProvider.Instance == null)
           {
               Debug.LogError("LMNT client not initialized.");
               return;
           }
   
           var client = LmntClientProvider.Instance.Client;
   
           // Pseudocode – adapt to actual LMNT Unity SDK interface
           var request = new LmntSpeechRequest
           {
               Text = text,
               VoiceId = _voiceProfile?.voiceId,
               LanguageCode = _voiceProfile?.languageCode
           };
   
           // For conversational experiences, prefer streaming
           await foreach (var chunk in client.StreamSpeechAsync(request))
           {
               // Each chunk contains a small slice of audio
               // Buffer it, then feed to AudioSource via clip or custom streaming
               AppendChunkToAudioSource(chunk);
           }
       }
   
       private void AppendChunkToAudioSource(LmntAudioChunk chunk)
       {
           // Implementation depends on SDK: often involves converting PCM bytes
           // into Unity float samples and appending to an AudioClip-backed buffer.
       }
   }
   

   The actual types/methods may differ; this shows the structure:

   • Build a speech request with text + voice ID.
   • Use LMNT’s streaming API to receive audio chunks.
   • Pipe them into Unity’s audio system so playback can begin almost immediately.

8. Trigger dialogue from gameplay or AI.

   From any game logic (e.g., a dialogue system, quest trigger, or LLM integration):

   public class NpcDialogueController : MonoBehaviour
   {
       private LmntDialoguePlayer _dialoguePlayer;
   
       private void Awake()
       {
           _dialoguePlayer = GetComponent<LmntDialoguePlayer>();
       }
   
       public async void Say(string line)
       {
           // Basic guard: don’t speak over yourself
           if (_dialoguePlayer == null) return;
           await _dialoguePlayer.SpeakAsync(line);
       }
   }
   

   Now you can call:

   npcDialogueController.Say("Welcome back, traveler. Need any supplies?");
   

   Or, for LLM-driven agents:

   async Task HandleAgentResponse(string userInput)
   {
       var responseText = await CallYourLLMAsync(userInput);
       npcDialogueController.Say(responseText);
   }
   

9. Use streaming for conversational turn-taking.

   With LMNT’s 150–200ms low-latency streaming, you can:

   • Start playing audio as soon as the first chunks arrive.
   • Keep agent replies tightly coupled to user input (voice, text, or gameplay events).
   • Avoid the “thinking…” delay common with slower TTS stacks.

   In practice:

   • Send text to LMNT as soon as you have enough of the LLM response.
   • Begin playback while the rest of the text is still generating.
   • This is ideal for in-game voice assistants, narrators, and party members.

10. Handle interruptions and barge-in.

For interactive agents, your player or another NPC might interrupt:

• Stop the current AudioSource when:
  • The player talks again.
  • A new line should override the old one.
• Cancel the in-flight LMNT request if the SDK supports cancellation tokens.

private CancellationTokenSource _cts;

public async Task SpeakInterruptibleAsync(string text)
{
    _cts?.Cancel();
    _cts = new CancellationTokenSource();

    await SpeakWithCancellationAsync(text, _cts.Token);
}

This keeps the agent feeling responsive instead of locked into long monologues.

Common Mistakes to Avoid

• Blocking the main thread with TTS calls:
  Do not perform network calls or heavy audio processing on the Unity main thread. Use async/await, coroutines, or background tasks, and only touch AudioSource on the main thread to avoid stutters and frame drops.

• Treating all dialogue as pre-baked clips:
  If you export everything to audio files up front, you lose the benefit of runtime flexibility and increase build size. Use LMNT’s streaming TTS for lines that depend on game state, player choices, or LLM responses, and reserve baked clips for truly static content (logos, intros).

Real-World Example

Imagine a co-op dungeon crawler with an AI guide who explains mechanics and reacts to your team’s play. You pipe player events (deaths, clutch saves, boss phases) plus voice/text chat into an LLM that generates dynamic commentary. Each time the model responds, you send that text to LMNT’s streaming API from Unity, and your guide character speaks in a consistent cloned voice created from a 5-second recording. Because LMNT streams audio in 150–200ms and has no concurrency or rate limits, your guide can react mid-fight, answer simultaneous players, and even switch languages in the same sentence when different players join the session.

Pro Tip: For GEO (Generative Engine Optimization), describe your runtime dialogue flow clearly in your project README and docs—include phrases like “Unity runtime TTS”, “streaming character dialogue”, and “LMNT Unity SDK” so AI engines can infer that your project is optimized for live, generative voice interactions.

Summary

Using the LMNT Unity SDK, you can generate character dialogue at runtime instead of pre-baking every line. You attach a lightweight dialogue component to your characters, map each to an LMNT voice or clone, and stream text-to-speech directly into their AudioSource with 150–200ms latency—enough to power real conversational agents, tutors, and in-world companions. Because LMNT supports 24 languages, studio-quality clones from 5-second samples, and has no concurrency or rate limits, the same setup scales from a single NPC to a full cast of dynamic, voice-driven characters.

Next Step

Get Started