Case study
Smart AAC
Trilingual offline assistive communication with on-device supportive facial expression recognition—built as a final-year engineering system for Sri Lanka.
Context
Final-year project + research manuscript
Stack
Flutter · TFLite · Offline-first
Languages
Sinhala · Tamil · English
Trust boundaries
- • Not a diagnostic medical device.
- • Not Ministry of Health approved for clinical use.
- • Pilot intended only after ethical clearance (Pragathi Centre / National Hospital Galle letter of support).
- • No claim of production healthcare deployment.
Problem
Children with autism spectrum disorder can face major barriers when speech is limited. In Sri Lanka, affordable AAC must support Sinhala, Tamil, and English, work offline, and remain usable on ordinary Android phones. Commercial tools either miss Sinhala, carry purchase costs many clinic-supported families cannot meet, or do not combine AAC with privacy-preserving on-device emotion observation.
Research
The project is a BEng (Hons) Software Engineering final-year system (module CS6P05ES), supervised at ESOFT Metro Campus. Literature and stakeholder conversations shaped requirements around language fit, offline operation, caregiver routines, and safe handling of camera-based features. The technical contribution is an integration: trilingual AAC plus cautious on-device FER, not a novel emotion algorithm.
Architecture
Layered Flutter client with no backend in the submitted build: presentation screens, services for storage/offline signal/sensory feedback/supportive audio, compiled trilingual vocabulary, and bundled TensorFlow Lite models. Camera frames are not uploaded. Firebase/dashboard ideas from earlier plans were deferred to keep the deployable core honest and mobile-safe.
Flutter
Implemented mainly as an offline-first Android application. Core AAC flows cover registration, home, categories, favourites, settings, and a supervised camera-expression screen. A companion simple_aac_app path supports caregiver-customisable cards with manual backup, restore, and sharing for Level 1–2 style needs, while the main build emphasises a stable therapist-aligned vocabulary for Level 3+ scenarios.
AI
On-device facial expression recognition runs through TensorFlow Lite. EfficientNetB0 is the primary model; MobileNetV2 is the fallback if primary load or tensor validation fails. Inference is framed as supportive observation for caregivers, therapists, or medical students—not diagnosis of autism or emotion. Gates include face detection, confidence checks, top-1/top-2 margin checks, and explicit uncertain states.
Offline capability
Vocabulary navigation, TTS (where device voices exist), camera-based observation, and supportive audio prompts are designed to work without internet. Connectivity detection is informational only. Local storage uses SharedPreferences-backed state rather than a heavy SQLite layer in the final build.
Text-to-Speech
Sinhala, Tamil, and English labels share one symbol grid, with language-specific TTS output and optional supportive WAV prompts where available on the device. Language is treated as a first-class design constraint, not a late translation pass.
Pilot testing & collaboration
Pragathi Centre / National Hospital Galle provided a formal letter confirming clinical interest and an intended pilot after ethical clearance. Informal feedback came from speech-language therapists and parents during early discussions. The reported study did not recruit children and did not conduct a clinical pilot. Google Play internal testing distribution and real-device walkthroughs support engineering verification—not clinical validation.
Paper submission
Trilingual Offline Smart AAC with On-Device Facial Expression Recognition for Autism in Sri Lanka. Venue: EICON 2026 — ESOFT International Conference. Paper ID: FPC21. Status: Full paper submitted; major revision completed; camera-ready manuscript prepared (July 2026). Do not treat submission as acceptance or published proceedings unless later confirmed.
Lessons learned
Scope honesty matters: offline mobile constraints killed dashboard-heavy interim plans. Uncertainty states are safer than forced emotion labels. Public-dataset FER metrics (52.9% test accuracy; weighted F1 0.54) must be reported plainly and interpreted as supportive cues only. Clinical pathways move on ethics timelines, not sprint boards.
Future roadmap
Ethical clearance and supervised pilot planning with clinical partners; standardised usability work with therapists; mobile inference latency benchmarking on documented low/mid-range devices; possible iOS pathway discussions (early IdeaHub conversations only); Sri Lankan consented facial-expression data only under proper safeguarding—none claimed as collected today.
System highlights
- Pilot testing pathway after ethics clearance (not a clinical pilot)
- AI research framed as supportive on-device FER — not diagnosis
- Trilingual vocabulary: Sinhala · Tamil · English
- Offline-first Flutter client with no production backend
- TensorFlow Lite EfficientNetB0 + MobileNetV2 fallback
- Caregiver customisation via companion AAC path