Ghost Expert –– A! mentor

In factories, a new technician learns a procedure like lockout/tagout by shadowing a senior expert. That expert's time is scarce, and a skipped step can mean injury. Ghost Expert clones the expert. 1. Author: the expert films the procedure once while narrating. Whisper transcribes the narration, SigLIP finds scene changes, and Gemma 3 groups the segments into steps, each with an instruction, a visual check, a safety flag and a common mistake. The expert fixes anything in a simple editor and approves. No coding. 2. Coach: the trainee points a camera at the work area. A fast tier (SigLIP embeddings) tracks which step they seem to be on every frame. A slow tier (Gemma 3 vision on vLLM) verifies steps on 4 recent frames and returns strict JSON. A deterministic rule engine owns the state: a step counts as done only when Gemma confirms it, and before accusing anyone it re-checks skipped steps. Skip a safety-critical step and the screen turns red, a voice says what was missed, and the expert's own clip of that step loops until it is done. Minor slips get a hint. If the model is unsure, it asks instead of accusing. 3. Ask: the trainee holds a button and asks out loud ("where does the new filter go?"). Gemma answers in two sentences, grounded in the procedure and the expert's narration, and plays the relevant expert clip. 4. Record: at the end, a printable training record shows the verdict, trainee vs expert time per step, the mistakes caught and the questions asked, for the training or EHS manager. Coaching works in English and Italian, with the voice from Kokoro TTS. Everything runs on a single AMD Instinct GPU on ROCm: Gemma 3, SigLIP, Whisper and Kokoro together, with a live benchmark strip showing VRAM, frames/s and Gemma latency. Frames are never stored: a coach, not surveillance. Who benefits: trainees get a patient expert at their side, experts teach once instead of per hire, and plants catch mistakes before they cause harm.