Vault

handout_idea6_event_driven_neuromorphic

3 min read

Idea 6: Sparse Event-Driven Vision/Audio with ESP32 (Neuromorphic Edge)

Overview

Pair an ESP32 with an event-based sensor (event camera or silicon cochlea) and develop a sparse neural network that processes raw event addresses/timestamps directly. Eliminate the “sense-then-compress” paradigm by computing directly on sparse event streams.

Research Gap

  • All TinyML vision/audio on ESP32 converts to dense frames/spectrograms, wasting bandwidth and compute
  • Event-based sensors output sparse asynchronous data, but existing processing requires FPGA or GPU
  • No prior work demonstrates event-driven inference on a commodity microcontroller like ESP32
  • The neuromorphic computing community focuses on custom silicon, not commodity MCUs

Key Innovation

First demonstration of sparse event-driven inference on commodity ESP32 hardware. Develop custom data structures and sparse kernels that exploit event sparsity for 100x lower effective compute compared to frame-based processing.

Hardware Requirements

ComponentSpecificationPurpose
Main MCUESP32-S3-DevKitCCompute platform
Event SensorProphesee Metavision (if budget allows) or TDK/InvenSense IMU with event modeEvent input
AlternativeLow-cost event camera (e.g., iniVation DVXplorer mini) or DIY event sensorBudget option
Alternative 2Custom microphone array with threshold-based event detectionUltra-low-cost
TimingLogic analyzerEvent timestamp validation

Technical Approach

Phase 1: Event Representation

  • Events: tuples of (x, y, t, p) for vision or (channel, t, p) for audio
  • p = polarity (+1 for ON event, -1 for OFF event)
  • Temporal resolution: microseconds
  • Spatial resolution: depends on sensor (e.g., 320x240 for event camera)

Phase 2: Sparse Data Structures on ESP32

  • Event Queue: Circular buffer in PSRAM storing last N events (e.g., 10,000 events = ~160 KB)
  • Sparse Tensor: Hash map or coordinate list (COO format) for active regions
  • Event Surface: 2D grid where each cell stores timestamp of last event (for vision)
  • Design for cache efficiency: group nearby events for batch processing

Phase 3: Sparse Neural Network

  • Spiking Neural Network (SNN):

    • Leaky Integrate-and-Fire (LIF) neurons
    • Event-driven: neurons update only when they receive spikes
    • Synaptic weights stored in flash
    • Implement in C with event-driven scheduling

  • Event-Driven CNN:

    • Process events in small temporal windows (e.g., 10 ms)
    • Sparse convolution: only compute where events occurred
    • Use hash map for sparse activation storage
    • Output: classification or regression

Phase 4: Comparison Baselines

  • Frame-based: Convert events to frames (accumulate over 33 ms), run standard CNN on ESP32
  • Dense processing: Process all pixels/timesteps regardless of events
  • Measure: accuracy, latency, energy, memory

Phase 5: Applications

  • Gesture Recognition: Hand gestures captured by event camera; classify in real-time
  • Anomaly Detection: Industrial vibration monitoring with event-based IMU
  • Audio Keyword Spotting: Event-based silicon cochlea for always-on listening

Expected Results

  • 50-100x lower effective compute (FLOPs) compared to frame-based equivalent
  • <10% accuracy loss compared to frame-based baseline
  • Latency: <50 ms end-to-end (event to classification)
  • Memory: <200 KB for event buffer + model
  • Energy: <5 mJ per inference

Target Venues

  • Frontiers in Neuroscience (Neuromorphic Engineering)
  • IEEE Transactions on Very Large Scale Integration (TVLSI)
  • Nature Electronics
  • IEEE International Conference on Neuromorphic Systems (ICONS)
  • Event-based vision: Prophesee, iniVation, DAVIS sensor papers
  • Spiking neural networks: SNN literature (e.g., BindsNET, snnTorch)
  • Sparse neural networks: Lottery ticket hypothesis, structured sparsity
  • Neuromorphic computing: Intel Loihi, IBM TrueNorth (contrast with commodity MCU)
  • Event-based audio: AER cochlea, silicon cochlea papers

Risk Mitigation

RiskMitigation
Event sensor too expensiveUse IMU with threshold-based event detection as proxy; or build DIY event sensor
Sparse kernels too slow on ESP32Batch events into small groups; use lookup tables for common operations
SNN training difficultUse ANN-to-SNN conversion (e.g., rate coding) instead of direct SNN training
Event data hard to getUse public event-based datasets (DVS-Gesture, N-Cars, N-Caltech101)

Timeline (Approximate)

  • Weeks 1-2: Event sensor setup, data collection, dataset preparation
  • Weeks 3-4: Sparse data structure design and implementation
  • Weeks 5-6: Sparse neural network kernel development
  • Weeks 7-8: End-to-end pipeline and frame-based baseline
  • Weeks 9-10: Evaluation and paper writing

Graph View