Most AR medical demos stop at "here's a brain floating on a marker." This one tracks a simulated probe against that brain in real time, computes tool-to-tissue distance, and raises safety warnings on a tablet HUD.
1. Abstract
A marker-based AR prototype built in Unity with Vuforia tracking, deployed as an Android tablet APK. A 3D brain model is anchored to a printed base marker; a pen with fiducial fins stands in for a surgical probe. The system computes live tip-to-anatomy distance, classifies it into SAFE / CAUTION / DANGER zones with hysteresis, and shows both on a high-contrast HUD. That tells the trainee where the tool is, relative to what matters: the operative judgement they have to build before the first patient.
Tracking is Vuforia image targets at 6-DOF, with two redundant markers on the pen so a temporarily occluded face never breaks the track. Collider.ClosestPoint() against capsule and sphere colliders approximating the brain computes distance fast enough for frame-rate feedback and accurately enough to separate the training scenario's safety bands.
The novelty is the loop, not the rendering. Every commercial surgical-AR demo on the market stops at "here is the anatomy floating on a marker." Few close the loop into spatial reasoning about the tool, and none of the consumer-grade ones do it on a $400 Android tablet with paper-printed image targets. This build closes that loop on commodity hardware.
6-DOF pose tracking on three image targets: BrainTarget for anatomy, FrontTarget and BackTarget for tool redundancy.
Real-time mm-scale distance via Collider.ClosestPoint() against capsule and sphere colliders on the brain. Frame-rate-stable on commodity Android.
Hysteresis on zone transitions prevents flicker between SAFE, CAUTION, and DANGER when the tip hovers near a threshold.
Multi-touch gesture controls: pinch-to-scale, drag-to-rotate, lock, and measurement-mode toggle.
Self-contained Android tablet APK. No headset, no operating-room infrastructure, no PC tether.
Figure 1. Tablet AR view. As the pen approaches the tracked brain model, the HUD distance updates in millimetres and the zone indicator steps through SAFE, CAUTION, and DANGER.
2. Introduction
AR for surgical visualization has a ceiling. Training needs interaction awareness: where the tool is, how close it is to the structure that matters, and which side of a threshold the trainee is on. Until a demo answers those in real time, it is a presentation prop.
Visualization without interaction is inert. A 3D model on a fiducial is visually striking, but it shows what is there, not what to do.
The trainee is building spatial judgement. Millimetre-scale awareness of the tip relative to the anatomy takes the longest to develop. That is what a training surface has to teach.
Commodity hardware unlocks reach. An Android tablet with paper-printed image targets reaches more trainees in more places than a HoloLens, a wet lab, or an OR rotation.
2.1 Why marker tracking, not headset SLAM
Vuforia image targets sit at one extreme of AR tracking: cheap, robust, deterministic, and obviously markered. Headset SLAM (HoloLens, Quest 3, ARKit) sits at the other: hands-free and operating-room compatible, but expensive and harder to debug. For a tablet training prototype, markers are the right trade.
Vuforia 6-DOF on a tablet is robust under handheld motion and frame-rate-stable on commodity Android. The trainee wears nothing.
Three image targets, redundantly placed. One on the anatomy, two on the tool, so losing one face of the tool does not break the track.
Deterministic registration. A printed marker is a known anchor in the camera frame. No SLAM drift, no calibration at session start, no anatomy sliding off the desk.
2.2 What is actually new
The marker tracking and rendering are off-the-shelf. The contribution is the per-frame spatial reasoning on top, and the HUD that makes it actionable for a trainee.
Per-frame proximity. The tip-to-anatomy distance is a live number, not a one-shot measurement.
Classified safety state with hysteresis. SAFE, CAUTION, and DANGER thresholds are tunable, and the HUD does not flicker at a boundary.
Trainee-facing HUD, not researcher-facing telemetry. A large numeric readout and a color-coded zone indicator. The legibility budget is "readable on a tablet under variable lighting," not "complete in a research paper."
Everything below makes those three commitments hold inside the tablet.
3. System
3.1 AR anchoring
BrainTarget. Printed base marker that anchors the 3D brain model in world space.
FrontTarget and BackTarget. Dual markers on opposite faces of the pen. Either face recovers the tool pose.
6-DOF pose per target from Vuforia keeps brain-to-tool spatial relationships consistent under typical camera or marker motion.
3.2 Tool tracking
ToolRigFollower picks the marker with the highest Vuforia tracking confidence each frame and smoothly follows its pose into the tool rig.
Marker loss triggers interpolation and graceful reacquisition, not snapping. If both markers vanish, the rig coasts on its last known velocity for a few frames.
Temporal smoothing on the raw pose suppresses high-frequency jitter at the tool tip when the camera is still.
3.3 Proximity logic
The tool tip is a dedicated transform at the probe's working point, so distance is measured from the point of interest, not the marker centroid.
Capsule and sphere colliders approximate the brain: fast distance queries, enough resolution for training scenarios that care about gross proximity, not surface curvature.
Collider.ClosestPoint() runs every frame, and the distance is classified against tunable SAFE, CAUTION, and DANGER thresholds.
Hysteresis: a band transition commits only when the distance has held the new band for a configurable hold-time.
3.4 HUD
Bottom-left. Reset, Lock, and Measure-mode toggles.
Bottom-right. A large distance readout in millimetres, plus a color-coded zone indicator (green for SAFE, amber for CAUTION, red for DANGER).
Transparent black panels and high-contrast typography keep the HUD readable under classroom, lab-bench, or study-desk lighting.
3.5 Gesture controls
Pinch-to-scale matches the anatomy model to the marker arrangement on the desk.
Drag-to-rotate around the marker axis inspects the anatomy from different angles without moving the camera.
Lock freezes the current placement during a measurement, so accidental input cannot disturb a careful approach.
Measure-mode switches the HUD between the live distance and a snapshot held at the moment of the toggle.
4. Evaluation
Validated qualitatively on Android tablet hardware, as a proof-of-concept for proximity-aware AR on commodity tablets, not a validated training instrument.
Vuforia tracking holds under the typical motion of a seated trainee, with the markers within roughly 30 cm of the camera in normal indoor lighting.
Distance updates at frame rate on the target Android tablet, with no observable lag between probe motion and HUD.
Zone transitions are stable across the typical SAFE-to-CAUTION and CAUTION-to-DANGER hover ranges, held steady by the hysteresis hold-time.
Multi-touch gestures (pinch, drag, lock, measure-mode) behave as expected across the four configurations exercised during development.
This is a single-tablet training prototype, not a clinical evaluation: not a surgical guidance system, not an OR-ready instrument, not on an FDA track. The contribution is the proximity-plus-safety-state loop on commodity hardware, not a benchmark number against any clinical AR system.
5. Conclusion and Implications
A tablet AR prototype that closes the proximity loop is a concrete step toward more accessible surgical training. Moving even a fraction of the spatial-judgement learning curve, usually built on cadavers, simulators, or expensive headsets, earlier onto a $400 Android tablet with paper markers means more reps before the first patient, more consistent feedback across the cohort, and a lower threshold for a teaching surgeon to drop in and supervise. Marker tracking, hysteresis, and a trainee-facing HUD make this a training surface, not a research demo.
5.1 Limitations and future work
Whole-brain colliders only. The system measures "distance to brain," not to a specific region. Per-region colliders from an MRI-segmented anatomy mesh would enable warnings like "3 mm from the motor cortex" or "8 mm from the optic chiasm."
Single tool geometry. Thresholds were tuned for one pen-and-marker tool. A lookup of instrument geometries to thresholds, set from the HUD at session start, would let one tablet support several training scenarios.
Marker tracking constrains context. It needs line-of-sight to the targets in normal indoor lighting; occluded markers and surgical-suite lighting are out of scope. A HoloLens or SLAM-tablet port would allow hands-free practice.
No quantitative trainee study. Training efficacy against cadaver labs or commercial simulators is unproven. A cohort study against a standard simulator, with per-session trajectory logging, is the biggest step from "engineering prototype" to "training claim."
6. Acknowledgments
Marker tracking primitives provided by Vuforia. Rendering, gesture handling, and the proximity loop are built on Unity. Built independently.