Dynamic Neural Accelerator for Reconfigurable & Energy-efficient Neural Network Inference

Abstract

Unique Challenges for AI Inference Hardware at the Edge • Peak TOPS or TOPS/Watt are not ideal measures of performance at the edge. Cannot prioritize performance over power efficiency (throughput/watt) • Many AI Hardware rely on batching to improve utilization. Unsuitable for streaming data (batch size 1) use-case at the edge • AI hardware architectures that fully cache network parameters using large on-chip SRAM cannot be scaled down easily to sizes applicable for edge workloads. • Need adaptability to new workloads and the ability to deploy multiple AI models • AI-specific accelerator needs to operate within heterogenous compute environments • Need for efficient compiler & scheduling to maximize compute utilization • Need for high software robustness and usability