A Repeatable Evaluation Protocol for Text-to-Video APIs
Teams often compare video models with different prompts, clip lengths, aspect ratios, or reference inputs. The result may look persuasive while answering the wrong question. A small, versioned evaluation set is more useful than a collection of hand-picked samples.
This guide is a test-design template, not a benchmark or a claim that one model performs better than another.
1. Define the workload before choosing a model
Write down the tasks the application actually needs: for example, a product shot, a short social clip, a scene transition, or a reference-guided edit. For each case, define what counts as usable before generating anything. Keep acceptance criteria separate from stylistic preferences.
Use representative prompts rather than prompts designed to favor a particular model. Keep a versioned manifest with:
- a case ID and prompt version;
- the exact model ID and the request parameters accepted by that model;
- requested duration, resolution, and aspect ratio;
- hashes for any reference assets, with the originals stored privately;
- the date the model documentation and configuration were checked.
Record a seed only when the selected model explicitly supports one. Do not assume that a repeated request will reproduce identical output.
2. Separate hard checks from human scoring
First apply pass/fail checks that do not depend on taste: did the task complete, can the file be decoded, is the delivered clip close to the requested duration and format, and is the result accessible to the intended user?
Then score quality dimensions that matter to the use case. A simple 1–5 rubric can cover prompt adherence, subject continuity, motion coherence, visible artifacts, and composition. Skip dimensions that do not apply. Have reviewers score outputs without seeing the model name where practical, and write down a short reason for each score.
A score is only meaningful beside its case definition and review method. Do not collapse different workloads into one universal “best model” number.
3. Keep a traceable run ledger
For every run, store the case ID, request time, application request ID, returned task ID, chosen configuration, completion state, output checksum, review scores, and the matching billing record. Keep credentials out of the manifest and logs.
For asynchronous APIs, retain the returned task ID and query that task as it progresses. If a create request times out, first determine whether the original job exists; blindly creating another job can produce duplicate work and charges. The Infinite Diffusion API documentation describes its video submit, query, and download endpoints and advises clients to save and reuse the returned ID.
4. Compare like with like
Use the same evaluation cases and comparable settings for each candidate. If a model does not support a requested input or parameter, mark the case unsupported instead of silently changing the test. Report the number of runs, failed or unsupported cases, reviewer rubric, and the spread of scores alongside any summary statistic.
Choose based on the application’s acceptance criteria and current budget. A visually preferred clip may still fail delivery requirements; a technically valid file may still be unsuitable for the intended audience.
5. Re-run when the configuration changes
Model IDs, available parameters, service availability, and billing can change. Date each evaluation and re-check the live documentation and model catalog before relying on an old result. Compare actual invoice records when estimating operating cost; a published example is not a substitute for the current bill.
Infinite Diffusion (无限扩散) provides third-party API access and is not the official Seedance model provider. This article contains an evaluation method, not measured performance results. Check the current model catalog and pricing, API documentation, and actual billing for current IDs, capabilities, availability, and charges. No generation tests were run for this guide.