AuraCaptioner: Grounded Multi-Style Captions

Streamlit
application badge
Created by team AuraStorm on July 11, 2026
Video Captioning

Most captioning pipelines fail on reliability before they fail on quality, missing a style, timing out, or returning malformed JSON zeroes out an otherwise-good submission. The core pipeline follows a two-stage grounding-then-styling architecture. Stage 1 extracts a style-agnostic factual description of the video (subjects, actions, setting, mood, on-screen text, audio) from scene-aware keyframes rather than blind uniform sampling. A Chain-of-Verification pass then drafts verification questions against those facts and re-checks them independently, dropping unsupported claims, measurably reducing hallucination versus "just be grounded" prompting alone. Speech transcription and OCR are gated behind cheap voice-activity and text-likelihood checks, so silent or text-free clips skip those calls entirely. Stage 2 rewrites the verified facts into four distinct tones, formal, sarcastic, humorous_tech, humorous_non_tech each with explicit rubrics, negative constraints, and brevity limits to counter known LLM-judge biases (verbosity and style-invariance). A self-judge pass, deliberately run on a different model than the generator to avoid self-preference bias, scores each caption on accuracy and style match and regenerates weak ones within a runtime-bounded retry budget.

Category tags: