Building in Public · 2026-07-11 · 5 min read
5 Things I Learned Building My Own Video Engine
How I built the Skylark Video Engine, a tool that turns one markdown script into a finished, narrated, on-brand video, and why a human still makes the final call.
I've always known exactly how I want my work to look and feel. What changed is what I can do about it.
With AI, I don't hunt for the closest tool anymore, or bend the way I work to fit someone else's. I build the tool to my own spec. And each one I build is more capable than the last.
The newest, and the most capable yet, is a video engine.
It's called the Skylark Video Engine, the tool I made to create AI in the City's videos. I write one markdown script, what the screen shows and what the voice says, scene by scene, and one command turns it into a finished, narrated, branded video. And it isn't only a command line. It has a real interface, and it's the part I'm most proud of. It's a feedback surface built around my exact workflow, the way I actually work on video. The AI and I pass a cut back and forth. I play each scene back, leave notes, tune the look, and mark it approved or send it back.
<!-- IMAGE (follow-up): Skylark Video Engine Scene-review screenshot. Alt/caption: "Screenshot of the Skylark Video Engine, with Scene review to capture notes that can be reprocessed in the AI video engine. Video production in AITC Studio, powered by the Skylark Video Engine, built with Claude Code." Export from Studio, place in public/insights/building-the-skylark-video-engine/, then add the <img>. -->Hearing my own cloned voice narrate that first video was surreal, in a good way. But the surprise faded fast, and what stuck was everything around it. Here are five things I noticed building it.
1. It started as a spec, not code.
Before anything got built in Claude Code, I wrote it out in a doc on the aspects of video production and editing tools like RunwayML, Descript, Clueso, and Canva that I liked and struggled with. I searched X for the latest in AI video features. I then ran it past ChatGPT and Claude to find the parts I hadn't thought of. The biggest decision came out of that pass on Opus 4.8. The final assembly happens in FFmpeg, but the timing and the sync live in my own code, not inside a tool I can't see into. So when the sync breaks, I can see why.
2. The script is the boss.
Everything follows the script. Not the visuals, not the music, the words.
Getting the story right, the concepts, the examples, and where to put the emphasis is what directs the whole video.
Underneath, the engine is audio-first. The voice is generated from the script, and every scene is timed to that voice, down to the frame. The build fails if a scene drifts more than 100 milliseconds. The worst it ever drifted was 19.
But that precision only matters because the script decided what mattered first. I include visual notes in the script to guide AI in the creation of diagrams, flows, motifs, and key concepts that should be emphasized in a scene.
Keeping the voice in charge mattered to me because these videos are to help people build with AI. It's an extension of my helping hand, so I really wanted to make sure the video stayed true to my intention and my ideas.
<!-- IMAGE (follow-up): screenshot of a script.md markdown file for a video. -->3. The machine said ship. I said no.
The first full video passed every check. Sync green. The brand verifier said ship. I flagged it do not publish.
The pipeline worked. The words didn't. The script called the agent's work a loop, when what I meant was a cycle. And the definition missed the judgment at the center of it, which was the whole thing I wanted people to see.
When I watched that first version, I knew it was wrong. There were issues with the script, as well as the pace and where certain words and concepts were being emphasized. I think the overall information flow was something I hadn't specified, so it felt really off. I landed on wanting to make sure I could define things in common language, be encouraging, with an objective to inspire people to try and experiment with AI in ways that were undaunting and practical.
So I rewrote it. The second version defines things in plain language, and it's built to encourage people to try, not to sound smart.
4. Where the video gets watched taught me rules I hadn't thought of.
The videos live on LinkedIn and YouTube, and how people actually watch there taught me two things my build spec never anticipated.
First, captions are not optional. LinkedIn plays videos on mute, so most people watch with the sound off. That means the words on screen are the whole message. The engine now generates accurate, timestamped captions for every video.
Second, never open on a title card. The first two or three seconds decide whether someone keeps watching. I used to lead with the AI in the City name and logo. Now the first line is the strongest thing in the script, and the branding waits until the end.
5. Do the work once, then encode it.
A video's essence is a markdown file now. I can edit it, version it, re-render it. Change one line, and only that scene rebuilds. Change the brand file, and every future video inherits it.
Here's the part I keep coming back to.
Claude Code kept asking me to ship after each big pass. Most times I said no. There was always more to do, and honestly, I like that division of labor.
I want my hands on it. My discernment, my taste, my read on what a given video is supposed to be. It's not AI just cranking out videos based on derived content and a best guess on what good looks like.
Video, in particular, is where I think authenticity and intention have to show up well. Pride of work is something very important in video. It's a complex product that wraps together so much effort, intention, ideas, emotions, mood, and pace. It's a beautiful mix of both art, science, and now AI.
The first video the engine made is called "What is an agent." It's up now. My voice, my colors, one script.
If you're ready to build, I'm here to build with you.
— Linda