Codex Is a Harness. Swap the Tools, Get a Video Editor.

2 min read

I edited this video 100% with Codex. Part 3.

Raw input I started with: source.mp4. Plain green-screen recording, one take.

Everything you see in was this video was edited by Codex (not in real-time).

Ball tracking, recoloring, the apple swap, cropping me out of the green screen, dropping in a studio background, the depth text floating behind me.

And also the part where I explain how it works and some animations and images that float it… also Codex.

The tools I gave it

Codex is a harness. A model in a loop with tools. By default those tools are for writing code, but the harness doesn’t care. Swap in video tools and you get a video editor.

The model is “general” enough to figure out and work with any tools.

Specifically, in here we have four of them.

  • SAM 3.1. Segmentation and tracking. That’s how the ball gets followed across the frame.

  • MatAnyone. Clean person matte. Gets you a cutout boundary around me, which is what makes the green screen swap and the depth text look good.

  • Remotion and FFmpeg. The actual video editor layer. Remotion gives Codex a programmatic timeline (basically React), and FFmpeg handles the heavy lifting.

  • Transcription with word-level timestamps. So it knows when exactly when I say a specific word to land what effect. Say “track the ball” and the ring appears the frame after “ball.”

So how do you do this stuff Adi?

  1. I shoot the video against a green screen with a rough storyboard in my head.
  2. I open my terminal and tell Codex what’s in my head: here are the tools, here’s what I want, go.
  3. We go back and forth. It render slices, I look at them, I yap some more and then they adjust.
  4. Eventually it’s done.

Used to take me a couple of hours. Now it’s closer to 45 minutes.

A lot of these experiments don’t work. I throw stuff at the harness, it produces something weird, I realize the idea was bad, I delete it and try something else.

The video you watched is one that survived.

Code

Everything is open source:

Same warning as last time: working dump, not a starter kit. Read it for ideas.

I’ll keep building video primitives. More soon.