🤓 Latest Submissions
StyleCap
The problem. Most captioning systems output a single generic caption. StyleCap generates four captions per clip, each in a distinct style. That requires identifying the right frames, understanding the scene, and expressing it in different voices. We solve each step with specialized Gemma models. Gemma-native architecture. Gemma 4 — Orchestrator. Combines a verified scene description with a timeline to generate four styled captions. In our A/B evaluation, it matched the baseline on style quality (0.87). EmbeddingGemma — Frame selection. We oversample ~20 candidate frames, caption and embed them, then use maximal marginal relevance to keep the most informative, least-redundant frames instead of uniform sampling. PaliGemma 2 — Vision grounding. Produces a verified factual description of selected frames. It runs on an AMD Radeon PRO W7900 (48GB) via vLLM + ROCm, with automatic fallback to a hosted AMD-powered vision service when unavailable. ShieldGemma 2 — Safety. Planned as a Responsible AI layer to screen frames and generated captions. Engineering. Every stage degrades gracefully: smart selection → uniform sampling, PaliGemma → hosted grounder, Gemma orchestrator → one-shot captioning. The pipeline is fully containerized for the hackathon harness (reads /input/tasks.json, writes /output/results.json, processes four clips in parallel within the 10-minute limit) and verified inside Docker. Why Gemma? The intelligence pipeline is entirely Gemma-powered: EmbeddingGemma decides what to observe, PaliGemma verifies visual facts, and Gemma 4 generates the final styled captions—creating a coherent, end-to-end Gemma-on-AMD system.
13 Jul 2026
