Pub Date : 2025-10-22DOI: 10.1109/LCA.2025.3624272
Sookyung Choi;Myunghyun Rhee;Euiseok Kim;Kwangsik Shin;Youngpyo Joo;Hoshik Kim
Processing multi-million tokens for advanced Large Language Models (LLMs) poses a significant memory bottleneck for existing AI systems. This bottleneck stems from a fundamental resource imbalance, where enormous memory capacity and bandwidth are required, yet the computational load is minimal. We propose NELSSA (Processing Near Memory for Extremely Long Sequences with Sparse Attention), an architectural platform that synergistically combines the high-capacity Processing Near Memory (PNM) with the principles of dynamic sparse attention to address this issue. This approach enables capacity scaling without performance degradation, and our evaluation shows that NELSSA can process up to 20M-token sequences on a single node (Llama-2-70B), achieving an 11× to 40× speedup over a representative DIMM-based PNM system. The proposed architecture radically resolves existing inefficiencies, enabling previously impractical multi-million-token processing and thus laying the foundation for next-generation AI applications.
为高级大型语言模型（llm）处理数百万个令牌对现有人工智能系统构成了显著的内存瓶颈。这种瓶颈源于基本的资源不平衡，需要巨大的内存容量和带宽，但计算负载却很小。我们提出了NELSSA (Processing Near Memory for Extremely Long Sequences with Sparse Attention)，这是一个将高容量处理近内存（PNM）与动态稀疏注意原理协同结合的架构平台来解决这个问题。这种方法可以在不降低性能的情况下实现容量扩展，我们的评估表明，NELSSA可以在单个节点（Llama-2-70B）上处理多达20m个令牌序列，比典型的基于dimm的PNM系统实现11倍到40倍的加速。所提出的架构从根本上解决了现有的低效率问题，实现了以前不切实际的数百万令牌处理，从而为下一代人工智能应用奠定了基础。
{"title":"PNM Meets Sparse Attention: Enabling Multi-Million Tokens Inference at Scale","authors":"Sookyung Choi;Myunghyun Rhee;Euiseok Kim;Kwangsik Shin;Youngpyo Joo;Hoshik Kim","doi":"10.1109/LCA.2025.3624272","DOIUrl":"https://doi.org/10.1109/LCA.2025.3624272","url":null,"abstract":"Processing multi-million tokens for advanced Large Language Models (LLMs) poses a significant memory bottleneck for existing AI systems. This bottleneck stems from a fundamental resource imbalance, where enormous memory capacity and bandwidth are required, yet the computational load is minimal. We propose <monospace>NELSSA</monospace> (Processing <underline>N</u>ear Memory for <underline>E</u>xtremely <underline>L</u>ong <underline>S</u>equences with <underline>S</u>parse <underline>A</u>ttention), an architectural platform that synergistically combines the high-capacity Processing Near Memory (PNM) with the principles of dynamic sparse attention to address this issue. This approach enables capacity scaling without performance degradation, and our evaluation shows that <monospace>NELSSA</monospace> can process up to 20M-token sequences on a single node (Llama-2-70B), achieving an 11× to 40× speedup over a representative DIMM-based PNM system. The proposed architecture radically resolves existing inefficiencies, enabling previously impractical multi-million-token processing and thus laying the foundation for next-generation AI applications.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"353-356"},"PeriodicalIF":1.4,"publicationDate":"2025-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145560750","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
As distributed machine learning (ML) workloads scale to thousands of GPUs connected by ultra-high-speed interconnects, tail latency in collective communication has emerged as a primary bottleneck. Prior RDMA designs, like RoCE, IRN, and SRNIC, enforce strict reliability and in-order delivery, relying on retransmissions and packet sequencing to ensure correctness. While effective for general-purpose workloads, these mechanisms introduce complexity and latency that scale poorly, where even rare packet losses or delays can consistently degrade system performance. We introduce Celeris, a domain-specific RDMA transport that revisits traditional reliability guarantees based on ML’s tolerance for lost or partial data. Celeris removes retransmissions and in-order delivery from the RDMA NIC, enabling best-effort transport that exploits the robustness of ML workloads. It retains congestion control (e.g., DCQCN) and manages communication with software-level mechanisms such as adaptive timeouts and data prioritization, while shifting loss recovery to the ML pipeline (e.g., using the Hadamard Transform). Early results show that Celeris reduces 99th-percentile latency by up to 2.3×, cuts BRAM usage by 67%, and nearly doubles NIC resilience to faults—delivering a resilient, scalable transport tailored for ML at cluster scale.
{"title":"Reimagining RDMA Through the Lens of ML","authors":"Ertza Warraich;Ali Imran;Annus Zulfiqar;Shay Vargaftik;Sonia Fahmy;Muhammad Shahbaz","doi":"10.1109/LCA.2025.3624158","DOIUrl":"https://doi.org/10.1109/LCA.2025.3624158","url":null,"abstract":"As distributed machine learning (ML) workloads scale to thousands of GPUs connected by ultra-high-speed interconnects, tail latency in collective communication has emerged as a primary bottleneck. Prior RDMA designs, like RoCE, IRN, and SRNIC, enforce strict reliability and in-order delivery, relying on retransmissions and packet sequencing to ensure correctness. While effective for general-purpose workloads, these mechanisms introduce complexity and latency that scale poorly, where even rare packet losses or delays can consistently degrade system performance. We introduce Celeris, a domain-specific RDMA transport that revisits traditional reliability guarantees based on ML’s tolerance for lost or partial data. Celeris removes retransmissions and in-order delivery from the RDMA NIC, enabling best-effort transport that exploits the robustness of ML workloads. It retains congestion control (e.g., DCQCN) and manages communication with software-level mechanisms such as adaptive timeouts and data prioritization, while shifting loss recovery to the ML pipeline (e.g., using the Hadamard Transform). Early results show that Celeris reduces 99th-percentile latency by up to 2.3×, cuts BRAM usage by 67%, and nearly doubles NIC resilience to faults—delivering a resilient, scalable transport tailored for ML at cluster scale.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"393-396"},"PeriodicalIF":1.4,"publicationDate":"2025-10-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145778198","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Modern processors rely on the last-level cache to bridge the growing latency gap between the CPU core and main memory. However, the memory access patterns of contemporary applications exhibit increasing complexity, characterized by significant temporal locality, irregular reuse, and high conflict rates. We propose a partial tag-data decoupling architecture that leverages temporal locality without modifying the main cache structure or replacement policy. A lightweight auxiliary tag path is introduced, where data is allocated only upon reuse confirmation, thus minimizing resource waste caused by low-reuse blocks. The experimental results show that the proposed design achieves an average IPC improvement of 1.55% and a 5.33% reduction in MPKI without prefetching. With prefetching enabled, IPC improves by 1.96% and MPKI is further reduced by 10.91%, while overall storage overhead is decreased by approximately 2.59%.
{"title":"A Partial Tag–Data Decoupled Architecture for Last-Level Cache Optimization","authors":"Honghui Liu;Xian Lin;Xin Zheng;Qiancheng Liu;Huaien Gao;Shuting Cai;Xiaoming Xiong","doi":"10.1109/LCA.2025.3623137","DOIUrl":"https://doi.org/10.1109/LCA.2025.3623137","url":null,"abstract":"Modern processors rely on the last-level cache to bridge the growing latency gap between the CPU core and main memory. However, the memory access patterns of contemporary applications exhibit increasing complexity, characterized by significant temporal locality, irregular reuse, and high conflict rates. We propose a partial tag-data decoupling architecture that leverages temporal locality without modifying the main cache structure or replacement policy. A lightweight auxiliary tag path is introduced, where data is allocated only upon reuse confirmation, thus minimizing resource waste caused by low-reuse blocks. The experimental results show that the proposed design achieves an average IPC improvement of 1.55% and a 5.33% reduction in MPKI without prefetching. With prefetching enabled, IPC improves by 1.96% and MPKI is further reduced by 10.91%, while overall storage overhead is decreased by approximately 2.59%.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"333-336"},"PeriodicalIF":1.4,"publicationDate":"2025-10-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145455958","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-10-17DOI: 10.1109/LCA.2025.3622724
Yunhua Fang;Rui Xie;Asad Ul Haq;Linsen Ma;Kaoutar El Maghraoui;Naigang Wang;Meng Wang;Liu Liu;Tong Zhang
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference.
{"title":"Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System","authors":"Yunhua Fang;Rui Xie;Asad Ul Haq;Linsen Ma;Kaoutar El Maghraoui;Naigang Wang;Meng Wang;Liu Liu;Tong Zhang","doi":"10.1109/LCA.2025.3622724","DOIUrl":"https://doi.org/10.1109/LCA.2025.3622724","url":null,"abstract":"Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"337-340"},"PeriodicalIF":1.4,"publicationDate":"2025-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145455932","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-10-16DOI: 10.1109/LCA.2025.3622588
Jiahao Xiang;Lang Li
The emergence of quantum computing threatens classical cryptographic systems, necessitating efficient architectural designs for post-quantum algorithms. This paper presents a novel architectural approach for implementing the FIPS 205 Stateless Hash-based Digital Signature Algorithm (SLH-DSA) on GPUs through execution model optimizations that maximize hardware utilization. We introduce a two-tier architectural framework: first, an Adaptive Thread Allocation mechanism that dynamically configures thread-level parallelism based on empirical performance modeling, optimizing the mapping between cryptographic workloads and GPU execution resources. Second, our Function-Level Parallelism design decomposes cryptographic components into fine-grained computational units with optimized memory access patterns and execution flows that better utilize the SIMT architecture of modern GPUs. Performance evaluation on an NVIDIA RTX 4090 demonstrates that our architectural design achieves 62,239 signatures per second for the SHA2-128f parameter set, representing a 1.16× improvement over prior implementations. Architectural analysis reveals that this throughput enhancement stems primarily from optimized thread-memory interactions and reduced resource contention in the GPU’s execution units.
{"title":"Thread-Adaptive: High-Throughput Parallel Architectures of SLH-DSA on GPUs","authors":"Jiahao Xiang;Lang Li","doi":"10.1109/LCA.2025.3622588","DOIUrl":"https://doi.org/10.1109/LCA.2025.3622588","url":null,"abstract":"The emergence of quantum computing threatens classical cryptographic systems, necessitating efficient architectural designs for post-quantum algorithms. This paper presents a novel architectural approach for implementing the FIPS 205 Stateless Hash-based Digital Signature Algorithm (SLH-DSA) on GPUs through execution model optimizations that maximize hardware utilization. We introduce a two-tier architectural framework: first, an Adaptive Thread Allocation mechanism that dynamically configures thread-level parallelism based on empirical performance modeling, optimizing the mapping between cryptographic workloads and GPU execution resources. Second, our Function-Level Parallelism design decomposes cryptographic components into fine-grained computational units with optimized memory access patterns and execution flows that better utilize the SIMT architecture of modern GPUs. Performance evaluation on an NVIDIA RTX 4090 demonstrates that our architectural design achieves 62,239 signatures per second for the SHA2-128f parameter set, representing a 1.16× improvement over prior implementations. Architectural analysis reveals that this throughput enhancement stems primarily from optimized thread-memory interactions and reduced resource contention in the GPU’s execution units.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"329-332"},"PeriodicalIF":1.4,"publicationDate":"2025-10-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145455891","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Deploying large language models (LLMs) on edge devices has the potentials for low-latency inference and privacy protection. However, meeting the substantial bandwidth demands of latency-oriented edge devices is challenging due to the strict power constraints of edge devices. Resistive random-access memory (RRAM)-based processing-in-memory (PIM) is an ideal solution for this challenge, thanks to its low read power and high internal bandwidth. Moreover, applying quantization methods, which require different precisions for weights and activations, is a common practice in edge inference. But existing accelerators cannot fully leverage the benefits of quantization, as they lack multiply-accumulate (MAC) units optimized for mixed-precision operands. To achieve low-latency edge inference, we design an RRAM-based PIM die that integrates dedicated energy-efficient MAC units, providing both computation and storage capabilities. Coupled with a dynamic random-access memory (DRAM) die for storing the key-value (KV) cache, we propose Lyla, an accelerator for low-latency edge LLM inference. Experimental results show that Lyla achieves 3.8×, 2.4×, and 1.2× latency improvements over a GPU and two DRAM-based PIM accelerators, respectively.
{"title":"Low-Latency PIM Accelerator for Edge LLM Inference","authors":"Xinyu Wang;Xiaotian Sun;Wanqian Li;Feng Min;Xiaoyu Zhang;Xinjiang Zhang;Yinhe Han;Xiaoming Chen","doi":"10.1109/LCA.2025.3618104","DOIUrl":"https://doi.org/10.1109/LCA.2025.3618104","url":null,"abstract":"Deploying large language models (LLMs) on edge devices has the potentials for low-latency inference and privacy protection. However, meeting the substantial bandwidth demands of latency-oriented edge devices is challenging due to the strict power constraints of edge devices. Resistive random-access memory (RRAM)-based processing-in-memory (PIM) is an ideal solution for this challenge, thanks to its low read power and high internal bandwidth. Moreover, applying quantization methods, which require different precisions for weights and activations, is a common practice in edge inference. But existing accelerators cannot fully leverage the benefits of quantization, as they lack multiply-accumulate (MAC) units optimized for mixed-precision operands. To achieve low-latency edge inference, we design an RRAM-based PIM die that integrates dedicated energy-efficient MAC units, providing both computation and storage capabilities. Coupled with a dynamic random-access memory (DRAM) die for storing the key-value (KV) cache, we propose Lyla, an accelerator for low-latency edge LLM inference. Experimental results show that Lyla achieves 3.8×, 2.4×, and 1.2× latency improvements over a GPU and two DRAM-based PIM accelerators, respectively.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"321-324"},"PeriodicalIF":1.4,"publicationDate":"2025-10-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145352109","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-10-06DOI: 10.1109/LCA.2025.3617159
Diamantis Patsidis;Georgios Vavouliotis
Set Dueling (SD) is an effective and widely adopted arbitration mechanism but is limited by its single-counter decision logic, relying solely on aggregate hit and miss counts of Leader Sets and ignoring richer contextual information (e.g., control-flow, sequence of memory accesses). This observation motivates our proposal, Context-Aware Set Dueling (CASD), which extends the original context-oblivious SD by incorporating contextual information in the decision logic, providing a framework that can be used to design runtime arbitration mechanisms capable of selecting between competing policies. As a prototype of CASD framework, we design DuelCeptron, a microarchitectural prediction scheme that replaces the single-counter decision logic of SD with hashed perceptrons to make more informed and accurate arbitration decisions. To showcase the benefits of DuelCeptron, we apply it in a case study, showing that it significantly outperforms SD across a diverse set of 145 workloads. DuelCeptron is one instantiation prototype of CASD, but the broader objective of this work is to advance SD into a general-purpose, context-aware arbitration mechanism applicable across different microarchitectural domains.
{"title":"Context-Aware Set Dueling for Dynamic Policy Arbitration","authors":"Diamantis Patsidis;Georgios Vavouliotis","doi":"10.1109/LCA.2025.3617159","DOIUrl":"https://doi.org/10.1109/LCA.2025.3617159","url":null,"abstract":"Set Dueling (SD) is an effective and widely adopted arbitration mechanism but is limited by its single-counter decision logic, relying solely on aggregate hit and miss counts of Leader Sets and ignoring richer contextual information (e.g., control-flow, sequence of memory accesses). This observation motivates our proposal, <italic>Context-Aware Set Dueling (CASD)</i>, which extends the original context-oblivious SD by incorporating contextual information in the decision logic, providing a framework that can be used to design runtime arbitration mechanisms capable of selecting between competing policies. As a prototype of CASD framework, we design <italic>DuelCeptron</i>, a microarchitectural prediction scheme that replaces the single-counter decision logic of SD with hashed perceptrons to make more informed and accurate arbitration decisions. To showcase the benefits of DuelCeptron, we apply it in a case study, showing that it significantly outperforms SD across a diverse set of 145 workloads. DuelCeptron is one instantiation prototype of CASD, but the broader objective of this work is to advance SD into a general-purpose, context-aware arbitration mechanism applicable across different microarchitectural domains.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"301-304"},"PeriodicalIF":1.4,"publicationDate":"2025-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145352231","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-10-06DOI: 10.1109/LCA.2025.3618627
Sanya Srivastava;Fletch Rydell;Andrés Goens;Vijay Nagarajan;Daniel J. Sorin
Traditional schemes for avoiding deadlocks compose techniques for both protocol deadlocks (virtual networks) and network deadlocks (virtual channels). Recent work has shown how to use fewer virtual networks by analyzing protocol stalls instead of just considering the longest chain of causally dependent messages. We identify a shortcoming in this work, which can lead to deadlocks, and show that combining stall analysis with analyses of message dependencies and topology can avoid deadlocks while using fewer buffers than the conventional approach.
{"title":"Efficient Deadlock Avoidance by Considering Stalling, Message Dependencies, and Topology","authors":"Sanya Srivastava;Fletch Rydell;Andrés Goens;Vijay Nagarajan;Daniel J. Sorin","doi":"10.1109/LCA.2025.3618627","DOIUrl":"https://doi.org/10.1109/LCA.2025.3618627","url":null,"abstract":"Traditional schemes for avoiding deadlocks compose techniques for both protocol deadlocks (virtual networks) and network deadlocks (virtual channels). Recent work has shown how to use fewer virtual networks by analyzing protocol stalls instead of just considering the longest chain of causally dependent messages. We identify a shortcoming in this work, which can lead to deadlocks, and show that combining stall analysis with analyses of message dependencies and topology can avoid deadlocks while using fewer buffers than the conventional approach.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"305-308"},"PeriodicalIF":1.4,"publicationDate":"2025-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145352177","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-10-01DOI: 10.1109/LCA.2025.3616810
Rui Xie;Asad Ul Haq;Yunhua Fang;Linsen Ma;Sanchari Sen;Swagath Venkataramani;Liu Liu;Tong Zhang
High-Bandwidth Memory (HBM) delivers exceptional bandwidth and energy efficiency for AI workloads, but its high cost per bit, driven in part by stringent on-die reliability requirements, poses a growing barrier to scalable deployment. This work explores a system-level approach to cost reduction by eliminating on-die ECC and shifting all fault management to the memory controller. We introduce a domain-specific ECC framework combining large-codeword Reed–Solomon (RS) correction with lightweight fine-grained CRC detection, differential parity updates to mitigate write amplification, and tunable protection based on data importance. Our evaluation using LLM inference workloads shows that, even under raw HBM bit error rates up to $10^{-3}$, the system retains 78% of throughput while maintaining at least 97% PIQA accuracy and 94% MMLU accuracy relative to error-free HBM. By treating reliability as a tunable system parameter rather than a fixed hardware constraint, our design opens a new path toward low-cost, high-performance HBM deployment in AI infrastructure.
{"title":"Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure","authors":"Rui Xie;Asad Ul Haq;Yunhua Fang;Linsen Ma;Sanchari Sen;Swagath Venkataramani;Liu Liu;Tong Zhang","doi":"10.1109/LCA.2025.3616810","DOIUrl":"https://doi.org/10.1109/LCA.2025.3616810","url":null,"abstract":"High-Bandwidth Memory (HBM) delivers exceptional bandwidth and energy efficiency for AI workloads, but its high cost per bit, driven in part by stringent on-die reliability requirements, poses a growing barrier to scalable deployment. This work explores a system-level approach to cost reduction by eliminating on-die ECC and shifting all fault management to the memory controller. We introduce a domain-specific ECC framework combining large-codeword Reed–Solomon (RS) correction with lightweight fine-grained CRC detection, differential parity updates to mitigate write amplification, and tunable protection based on data importance. Our evaluation using LLM inference workloads shows that, even under raw HBM bit error rates up to <inline-formula><tex-math>$10^{-3}$</tex-math></inline-formula>, the system retains 78% of throughput while maintaining at least 97% PIQA accuracy and 94% MMLU accuracy relative to error-free HBM. By treating reliability as a tunable system parameter rather than a fixed hardware constraint, our design opens a new path toward low-cost, high-performance HBM deployment in AI infrastructure.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"313-316"},"PeriodicalIF":1.4,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145352136","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2025-09-22DOI: 10.1109/LCA.2025.3612852
Haoyu Wang;Noa Zilberman;Ahmad Atamli;Amro Awad
Confidential computing is increasingly becoming a cornerstone for securely utilizing remote services and building trustworthy cloud infrastructure. Confidential computing builds on hardware-anchored root-of-trust that can attest the identity and authenticity of the remote machine, the configuration, and the running software stack, in an unforgeable way. In addition to the hardware-rooted verifiable attestation mechanism, confidential computing depends on strict run-time isolation of confidential computing tasks’ data and code from each other and the other tasks, including privileged ones. Such isolation is achieved via on-chip access control and cryptographically once off-chip. Despite the wide support of confidential computing in most modern processors, e.g., AMD SEV-SNP and ARM CCA, there is minimal discussion of the effect of such support on the performance of conventional on-chip access control. Thus, in this paper we highlight the key changes in virtual memory support required for access control in confidential computing environments, and quantify their overheads. We propose an optimized design that enables improved performance by caching confidential computing access control metadata effectively. Two design options are proposed to balance hardware overhead and performance. We evaluate two configurations with different TLB entry coverage, which mirror Arm CCA GPC and AMD RMP, respectively. Our design improves performance by 12% over the baseline access control design and 6% over the state-of-the-art.
机密计算正日益成为安全利用远程服务和构建可信云基础设施的基石。机密计算建立在硬件锚定的信任根基础上，可以以一种不可伪造的方式验证远程机器、配置和正在运行的软件堆栈的身份和真实性。除了基于硬件的可验证认证机制外，机密计算还依赖于机密计算任务之间以及其他任务（包括特权任务）之间数据和代码的严格运行时隔离。这种隔离是通过芯片上的访问控制和芯片外的加密实现的。尽管在大多数现代处理器中广泛支持机密计算，例如AMD SEV-SNP和ARM CCA，但很少讨论这种支持对传统片上访问控制性能的影响。因此，在本文中，我们强调了机密计算环境中访问控制所需的虚拟内存支持的关键变化，并量化了它们的开销。我们提出了一种优化设计，通过有效地缓存机密计算访问控制元数据来提高性能。提出了两种设计方案来平衡硬件开销和性能。我们评估了两种具有不同TLB入口覆盖的配置，分别反映了Arm CCA GPC和AMD RMP。我们的设计比基线访问控制设计提高了12%的性能，比最先进的性能提高了6%。
{"title":"Revisiting Virtual Memory Support for Confidential Computing Environments","authors":"Haoyu Wang;Noa Zilberman;Ahmad Atamli;Amro Awad","doi":"10.1109/LCA.2025.3612852","DOIUrl":"https://doi.org/10.1109/LCA.2025.3612852","url":null,"abstract":"Confidential computing is increasingly becoming a cornerstone for securely utilizing remote services and building trustworthy cloud infrastructure. Confidential computing builds on hardware-anchored root-of-trust that can attest the identity and authenticity of the remote machine, the configuration, and the running software stack, in an unforgeable way. In addition to the hardware-rooted verifiable attestation mechanism, confidential computing depends on strict run-time isolation of confidential computing tasks’ data and code from each other and the other tasks, including privileged ones. Such isolation is achieved via on-chip access control and cryptographically once off-chip. Despite the wide support of confidential computing in most modern processors, e.g., AMD SEV-SNP and ARM CCA, there is minimal discussion of the effect of such support on the performance of conventional on-chip access control. Thus, in this paper we highlight the key changes in virtual memory support required for access control in confidential computing environments, and quantify their overheads. We propose an optimized design that enables improved performance by caching confidential computing access control metadata effectively. Two design options are proposed to balance hardware overhead and performance. We evaluate two configurations with different TLB entry coverage, which mirror Arm CCA GPC and AMD RMP, respectively. Our design improves performance by 12% over the baseline access control design and 6% over the state-of-the-art.","PeriodicalId":51248,"journal":{"name":"IEEE Computer Architecture Letters","volume":"24 2","pages":"317-320"},"PeriodicalIF":1.4,"publicationDate":"2025-09-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145352179","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：