2024

Dermatological triage with cited medical evidence.

A DenseNet121 classifier trained on 19,000+ DermNet images across 23 skin conditions, paired with a retrieval-augmented chatbot whose every answer cites a real medical source. Auth, row-level security, and per-user history behind a single Supabase backend.

1. Abstract

A dermatological triage layer that pairs an image classifier with a retrieval-augmented chatbot, behind a Supabase-backed account with auth, row-level security, and persistent conversation history. Where dermatologist access is limited, how fast a patient reaches a clinician often separates a survival event from a routine appointment. An evidence-grounded opinion, on a phone, on whether a skin condition warrants urgent attention shortens that distance without pretending to replace the appointment.

The classifier is a DenseNet121 backbone (Huang et al., 2017), picked because dense connections give efficient gradient flow and feature reuse on a parameter budget suited to medical images. It trains on the DermNet dataset, roughly 19,000 dermatoscopic images across 23 categories, with class-weighted loss that penalises errors on rare malignant categories more heavily. The chatbot embeds each query with OpenAI text-embedding-3-small (1536-d), pulls the top-5 chunks from a pgvector HNSW index over dermatology textbooks and clinical guidelines, and hands generation to Gemini 2.0 Flash under a system prompt that requires every answer to cite specific retrieved chunks or refuse.

The novelty is the integration, not the parts. DenseNet121 is off-the-shelf, DermNet is public, RAG is standard. What is new is the product shape: a classifier with a "see a clinician" fallback below 60% confidence, a chatbot grounded in a cited source by construction so it cannot hallucinate clinical advice, and per-user history that makes both reusable across sessions, all behind a real auth surface. No consumer dermatology product on the market today cites its sources by construction. This build targets that gap.

19,000+ dermatoscopic images across 23 condition categories in training, spanning malignant, precancerous, and benign skin conditions.
Citation-by-construction. Every answer must carry a cited chunk. Uncited responses are regenerated; refusal is the fallback when context is insufficient.
Sub-50 ms vector retrieval over a pgvector HNSW index. First-token chat latency under 1 s via Vercel AI SDK streaming.
Calibrated "see a clinician" fallback below 60% top-1 confidence, so ambiguous predictions surface as low-confidence, not as a confidently-wrong label.
Supabase Auth and row-level security. Users see only their own classifications and conversations. Signed image URLs expire after 24 hours.

2. Introduction

In low-access regions a missed melanoma is a survival event, and the bottleneck is rarely diagnostic accuracy in the clinic. It is the time it takes a patient to reach a clinic at all.

The path-to-clinic is the bottleneck. In front of a dermatologist, accuracy is high. The friction is deciding whether the lesion warrants the appointment in the first place.
Black-box labels are not actionable. "This is melanoma" with no explanation does not help a non-clinician decide whether to seek care, or what to say at intake.
Hallucinated medical advice is a real failure mode. Off-the-shelf LLMs confidently produce plausible-but-wrong clinical statements. A triage tool anyone might trust cannot afford that.

2.1 Why a classifier and a RAG chatbot, not either alone

A classifier alone gives a label and a probability, leaving the patient to judge whether it warrants a visit. An ungrounded chatbot cannot reliably tell a benign mole from one that needs urgent attention. Together, the classifier sets a calibrated starting point, the chatbot turns it into language a patient can act on, and grounding anchors that language in cited medical sources.

DenseNet121. Dense connections carry early-layer features into late layers without retraining them, giving favourable accuracy-per-parameter on small medical datasets. 19,000 images is small by ImageNet standards but representative for dermatoscopic capture.
RAG over an authoritative corpus. Generation is constrained to retrieved chunks from dermatology textbooks and clinical guidelines, not generic training-data priors. A required citation slot forces the model to attribute or refuse.
Authenticated persistence. Predictions and chat history sit behind Supabase Auth with row-level security. A clinician can be granted temporary read access through a signed URL rather than a screenshot.

2.2 What is actually new

The classifier architecture, embedding model, vector store, and LLM are existing infrastructure. The contribution is how they compose into a triage product.

Citation-by-construction. Hallucinated medical advice cannot leave the system, because the response template will not accept one.
Calibrated "I don't know." Failure is redirection, not a confidently-wrong answer. The patient is steered to professional evaluation exactly where the system is least sure.
Per-user persistent context. A user returns to a previous classification and continues the conversation across devices, turning a one-shot classifier into a longitudinal triage layer.

Everything below makes those three commitments hold inside one deployed product.

3. System

3.1 Image classification

DenseNet121 backbone. Its feature reuse compensates for a small training set.
DermNet, 19,000+ dermatoscopic images, 23 categories spanning malignant (melanoma, basal-cell carcinoma, squamous-cell carcinoma), precancerous (actinic keratosis), and benign conditions.
Augmentation: rotation, horizontal and vertical flips, color jitter, Gaussian noise. Addresses class imbalance and capture variability across patient phones.
Class-weighted cross-entropy penalises errors on rare malignant categories more than on common benign ones. A missed melanoma costs more than a missed benign mole.
Top-3 with calibrated confidence. The model outputs the full probability distribution; the user sees the top-3 labels and their probabilities. Below 60% top-1 confidence, a "see a clinician" prompt replaces the label.
FP16 quantization at deployment keeps inference responsive on the Flask host.

3.2 Retrieval-augmented generation

Generation is constrained to retrieved sources, so the chatbot answers follow-up questions about a prediction without inventing clinical facts.

Ingestion. Dermatology textbooks and clinical guidelines chunked into 500 to 1000-token overlapping segments, embedded with OpenAI text-embedding-3-small (1536-d).
Storage. Supabase Postgres with the pgvector extension and an HNSW index for approximate-nearest-neighbour search. Per-chunk metadata carries source authority and recency for filtered retrieval.
Retrieval. The query is embedded into the same 1536-d space; the top-5 chunks are ranked by cosine similarity weighted with source authority, so a textbook chunk outranks a forum post at equal cosine.
Generation. Gemini 2.0 Flash receives the prediction context, the retrieved chunks, and a system prompt: answer only from those chunks, cite sources by chunk identifier, refuse when context is insufficient. Uncited responses are regenerated under stricter grounding instructions.

3.3 Stack

Frontend. Next.js 15 App Router with React server components, Tailwind, and shadcn/ui. Drag-and-drop upload, streaming chat, and markdown responses with inline citation badges.
Backend. Flask serves the PyTorch classifier on /predict (port 5001). Next.js API routes own RAG orchestration on /api/chat and the database I/O. The two independent services meet at the Next.js layer.
Auth and storage. Supabase Auth with OAuth (Google and GitHub) and HTTP-only session cookies. Row-level security on every table. Uploaded images live in Supabase Storage behind signed URLs that expire after 24 hours.
Streaming. The Vercel AI SDK pipes Gemini's output to the client token by token.

4. Evaluation

Validated for engineering correctness on held-out DermNet images and on the deployed product surface.

Sub-50 ms vector retrieval on the pgvector HNSW index across the ingested corpus.
First-token chat latency under 1 s via Vercel AI SDK streaming, even on responses that consume the full top-5 retrieved context.
FP16 quantization reduces classifier memory roughly 50% with negligible accuracy impact on the held-out DermNet split.
Calibrated uncertainty routes ambiguous cases to "see a clinician" instead of overconfident top-1 labels. The 60% threshold is configurable, picked to err toward routing borderline cases to clinical care.
Citation enforcement validated on a representative query set. Uncited responses are caught and regenerated; insufficient-context queries surface as refusals, not fabricated answers.

The accuracy numbers come from held-out DermNet images, single split, single seed. They are not a leaderboard claim against published dermatology AI benchmarks (different datasets, different label conventions), and not a clinical-grade diagnostic claim. This triage layer helps a patient decide whether to seek care, not what they have.

5. Conclusion and Implications

Pairing a calibrated classifier with citation-grounded follow-up is a concrete step toward earlier dermatologist intake for populations that cannot easily reach one. Recovering even a fraction of the path-to-clinic friction means faster intake on time-critical conditions like melanoma, fewer late-stage diagnoses, and lower triage burden on under-resourced dermatology services. Citation-by-construction, the calibrated fallback, and per-user history make this a deployable triage layer, not something that demos well and helps nobody.

5.1 Limitations and future work

Training-set bias toward lighter Fitzpatrick skin types. A documented equity issue in dermatology AI that augmentation cannot close. Multi-clinic data partnerships for representative coverage can.
Smartphone capture is not dermoscopy. Phones lack the magnification and polarised lighting of clinical dermoscopes. Next: dermoscopy attachment support and a dermoscopy-trained model variant.
Educational and pre-clinical, not regulated. Engineering correctness is validated; clinical efficacy against a standard dermatology intake protocol is unproven and is the headline next step.
Citation quality is bounded by the corpus. Out-of-corpus conditions surface as refusals: correct behaviour, but no help resolving the patient's question.
Next features. Longitudinal lesion tracking, a multilingual chatbot, and a signed-URL clinician handoff that makes the triage layer a referral-ready intake document.