Falcon 2 11B VLM

Falcon2-11B-VLM is an 11B parameters causal decoder-only model built by TII and trained on over 5,000B tokens of RefinedWeb enhanced with curated corpora. To bring vision capabilities, we integrate the pretrained CLIP ViT-L/14 vision encoder with our Falcon2-11B chat-finetuned model and train with image-text data. For enhancing the VLM's perception of fine-grained details w.r.t small objects in images, we employ a dynamic encoding mechanism at high-resolution for image inputs. The model is built on the same robust foundation as Falcon 2 11B, featuring 11 billion parameters. It matches or exceeds the performance of other leading models, such as Meta’s Llama 3 8B and Google’s Gemma 7B, particularly in tasks that require vision-language integration.

General
Relese date	May 13, 2024
Author	AI71
Dataset	https://huggingface.co/tiiuae/falcon-11B
Type	Vision Language Model

Falcon 2 Tutorials

ForgeSense: AI Maintenance Copilot on AMD MI300X

Factories already collect sensor streams, camera footage, work orders and thick PDF manuals, yet technicians still lose hours deciding what an alarm means and what to do. ForgeSense is a maintenance copilot for shift technicians that closes that gap. How it works: deterministic detectors watch the data first. An anomaly and remaining-useful-life model monitors vibration and temperature signals, and a vision model flags surface defects on camera frames. When one fires, an agent running an open LLM on AMD Instinct GPUs (ROCm + vLLM) gathers evidence with tools: the recent sensor window, similar past work orders, and a retrieval index of the equipment manuals. It then writes a plain-language alert: what is wrong, the evidence chart, ranked likely causes, step-by-step repair actions with manual page citations, and required parts and safety lockout notes. Responsible by design: the LLM never makes the detection and never stops a machine. Every alert carries a confidence level and cites its sources. Low-confidence or safety-critical cases are escalated to a human supervisor with a pre-filled work order, and the technician approves before anything is created. Impact: we measured time-to-diagnosis, detection recall and false alerts against a manual-search baseline on public industrial datasets (NASA C-MAPSS, MVTec AD), plus tokens/second and cost per alert on a single MI300X, where the vision-language model, embedding model and reranker run side by side in 192 GB of memory

Two-Arm Table Setter

Say "set the table" - or "put the red thing top left of the plate and the grey utensil on the right" - and two SO-101 arms in MuJoCo do it. HOW IT WORKS. An overhead colour-and-depth camera finds every object from its height and shape (0.5 mm average error over 20 random tables); the simulator's true positions are never used to act. Qwen3-VL-4B, running locally as Intel's INT4 OpenVINO build, reads the command together with the camera image, so it knows "the red thing" is the mug. A planner decides which arm does what: cutlery starting on the wrong side is passed hand to hand, and round items are set down where both arms can reach. After every item the robot looks again, and anything knocked out of place is put back first. LEARNING. Our scripted skills became the teacher, recording 1,219 successful demonstrations automatically. A LeRobot ACT policy trained on a laptop CPU learned the mug skill - 19/20 within 1.5 cm on tables it had never seen. It took seven versions; the write-up covers what failed and why. INTEL OPTIMISATION, on a Core Ultra 7 155H across all three chips. OpenVINO runs the policy 2.9x faster than PyTorch on the CPU (0.72 ms vs 2.07 ms) with no loss of task success. The same policy runs on the AI Boost NPU at 0.90 ms and places 10/10 mugs, freeing the CPU and iGPU. The Arc iGPU reads a command in 1.3 s, 2.8x faster than the CPU. An honest negative result too: OpenVINO 2026.3's NPU compiler cannot build Qwen3-VL's vision tower, so the language model falls back to the iGPU automatically. RESULTS. 30/30 random tables set correctly, 29/30 on harder ones, 9/10 on the Core Ultra. Every number is checked against simulator truth, reproducible from one command, and the raw logs ship in the repo. Not included, said plainly: the drawer and pouring, and most skills are scripted rather than learned.

ManipulaX

This project is an end-to-end Physical AI system for bimanual robotic manipulation, built for the Intel Physical AI Online Challenge. It uses two simulated SO-101 robot arms in MuJoCo to understand natural-language instructions, interpret camera observations, reason about the current scene, and execute multi-step manipulation tasks for setting up a dinner table. The system combines vision-language-action reasoning with coordinated dual-arm control, enabling actions such as opening a drawer, retrieving utensils, picking up and placing a plate or mug, and performing complementary actions between both arms. The environment supports randomized scene conditions to evaluate robustness across different object placements and physical or visual variations. The inference pipeline is designed for Intel Core Ultra hardware, using OpenVINO for model optimization and efficient edge inference while maintaining task performance. The project also includes reproducible simulation, evaluation, benchmarking, and demonstration workflows.

Fresno LUX: A Living City

Fresno LUX is a prototype for a living intelligence layer across an entire city. Fresno already contains enormous amounts of intelligence: residents know where problems exist, nonprofits know where help is needed, businesses know where opportunities exist, educators understand gaps in learning, researchers generate knowledge, and government maintains critical infrastructure and data. The problem is that much of this intelligence is fragmented across disconnected systems. Fresno LUX connects those pieces. The platform creates a shared city graph where people, organizations, resources, needs, projects, observations, experiments, and outcomes become connected nodes. Instead of simply collecting data, LUX continuously asks: What is happening? What do we know? What is missing? What can we test? What changed? A community problem can become an observable signal. That signal can become an experiment. The experiment produces evidence. Evidence changes the system's understanding. That updated understanding informs the next action. Observe → Connect → Understand → Act → Measure → Learn → Converge. The goal is to move Fresno from disconnected information toward collective intelligence—a system where knowledge can travel across boundaries and successful solutions can be discovered, tested, measured, and replicated. LUX is designed to make the invisible connections visible: the relationship between a neighborhood and its resources, between a student and educational opportunity, between a business and workforce capacity, between environmental conditions and community health, and between an idea and measurable real-world impact. This hackathon prototype demonstrates the beginning of a Living Fresno Graph: a city that can observe itself, learn from evidence, and coordinate toward better outcomes.