A benchmark by Coral Bricks AI
Can a model animate?
A one-minute launch film asked for in plain words, a real Open Design brief with its own skill, and sixteen short motion briefs. Each model builds every one as an HTML + GSAP composition in an agent harness, three times. Then every frame is measured: easing curves, wind-ups, overshoot, squash, arcs, beat hits, token rates. No model judges anything.
Results
Preliminary. The score is the sixteen short tasks: one technique each (does the ball keep its volume, does a word land on the beat, does the camera stop without drifting), each grader proven on a reference and a naive solution, 48 runs a model. The long briefs, appeal (how enjoyable the motion is to watch) and Kowalski's taste and polish rules are shown beside it, unweighted. A measure is still not a viewer: a blind, side-by-side vote by a panel of people who work in motion will carry 70% of the overall, shown once every model has 30 panel votes: watch two unnamed versions of the same brief and pick the one you would ship.
The score is the share of motion checks passed on the sixteen short tasks, three attempts each; the whisker is a 95% bootstrap interval. Long is the two long briefs' checks, 5 : 3: a one-minute launch film asked for in plain words by a founder who knows nothing about animation, and Open Design's own one-shape UI brief done with its motion skill. The coral line is the Pareto frontier: the models no other model beats on both score and the measure across the bottom.
Cost per task prices every run's own tokens at the provider's list price (CoralBricks' standard plan for DeepSeek and GLM), so models compare whatever plan they ran on; the Claude and GPT runs used subscriptions. Steps are tool calls. Contract (under each score) is the shared checks: determinism, offline, brand palette and fonts, no glow, no jitter.
Every task, every model
Craft score per task (share of its motion checks passed), averaged over three attempts; darker coral means more checks missed. The free piece has no brief to check against, so its rows show its appeal, polish and taste, unshaded.
Experimental
Tasks contributed outside the core set. Every model runs them, three attempts each, and they stay out of the score until they settle.
Gallery
The models' own compositions, playing live in your browser. Pick a task and an attempt; click a video or press Space to play and pause, or scrub.
Method
The long briefs
The launch film is a founder's message with no animation words in it: a ball that "bounces like a real rubber ball", a tag that "keeps swinging", typing "at our model's real speed", words "with the music". Each scene is found by itself and held to the same physics checks as the short tasks. The second is the starter prompt of Open Design's "One-Shape UI Motion" example, verbatim, with Open Design's motion skill installed: one shape that never cuts, something on every beat at 120 BPM, a cursor behind every change, tiny overshoot, a seamless loop.
Free, polish, taste
A third piece has no brief at all: "Make a motion graphic. Anything you like." Every piece, briefed or not, is also read element by element for glitches a viewer notices (tooling words such as "Export ready" on screen, one-frame flashes, text on text, text cut off at rest, characters the font cannot draw, empty screen, unreadably small type) and for Emil Kowalski's written rules: nothing grows from scale 0, entrances ease out and never ease in, exits are no slower than entrances, UI changes stay quick, the real ellipsis, tracked-out capitals, tabular figures, sensible line lengths.
Appeal
Read from the rendered frames at 30 fps by optical flow, so canvas, SVG and HTML pieces are measured alike. Six parts, each from research on what people enjoy in moving images, averaged: alive (attention fades on an unchanging screen: something changes in most seconds), continuity (good continuation and processing fluency: change carried by things moving or reshaping, not by fades and cuts), smooth (smooth motion is liked and abrupt motion disliked: speed changes gradually), complexity (Berlyne's inverted U: two to four coherent motions at once, and noise counts for nothing), variety (novelty against habituation: slides, scales, rotations, several motions at once, reshapes, wipes), pacing (Cutting, DeLong & Nothelfer 2010: the rhythm of popular films fluctuates as 1/f, matching attention). Reported beside the score, not in it. It was set on test pages before any model was measured: the reference films score 0.73 and 0.70, a lone spinner 0.41, random squares 0.28, a crossfading slideshow 0.22.
Elo
The blind vote is turned into Elo ratings (1000 is the average model; 400 points is ten-to-one odds) with 95% intervals from resampling the votes, shown once every model has 30 panel votes. The overall is 70% the panel's vote and 30% the score, both put on one scale: the chance of beating an average other model (in the vote, being preferred; on the score, coming out higher when both models' runs are resampled). Anyone can vote; only the invited panel's votes count toward it. It stays preliminary until every model has 30 panel votes.
Principles
Eight of the twelve principles of animation that can be measured on a flat frame: slow in and slow out, anticipation, follow-through, overlapping action, squash and stretch, arcs, secondary action, staging.
Studio
Eight moves and rules from the CoralBricks motion studio: precise expo pans between stations, one element from dot to hairline to divider, tokens at their real 340 tok/s, a count that lands on time, a curtain between surfaces, words that arrive on the beat, the lockup, a line-mask rise.
Grading
Headless Chromium seeks each composition's timeline to every frame at 60 fps and measures the elements the brief names: box, opacity, rotation, visible text, colour. Some checks read pixels. The same frame must come back whatever order it is seeked in. A reference solution scores 1.000 on every task; a naive one loses exactly the checks its task is about.
Harness
Claude, Grok, DeepSeek and GLM run in Claude Code; GPT runs in Codex CLI, its own harness. Each run is sandboxed to its folder with no network and gets the same brief, contract, brand sheet, fonts and two tools: one prints what the grader measures, one saves frames. The thresholds are never shown.
DeepSeek on CoralBricks ignores Claude Code's adaptive-thinking request, so "DeepSeek V4.1 Flash" ran with reasoning switched on explicitly; "no reasoning" is what a Claude Code user gets by default. A small local proxy also moved snapshot images out of tool results for the CoralBricks runs (the gateway refused them there); GLM-5.3 is text-only there, so it worked from the numeric probe alone. Grok 4.7 ran through xAI's API with Claude Code's mid-conversation system messages folded into the user turn. Seven checks were corrected after the first model runs (listed in the README) and every run was re-graded the same way. Three principles (exaggeration, solid drawing, straight-ahead vs pose-to-pose) are not here: none has a measurement that is not taste. Appeal is measured as watchability (above), not as character design.