I. Choi, Nimish Wadekar, Raj Joshi, Joshua Fried, Dan R. K. Ports, Irene Zhang, Jialin Li
Latency-critical μs-scale data center applications are susceptible to server load spikes. The issue is particularly challenging for services using long-lived TCP connections. This paper introduces Capybara, a highly efficient and versatile live TCP migration system. Capybara builds atop a deterministic, kernel-bypassed TCP stack running in a library OS to realize its μs-scale TCP migration mechanism. Using modern programmable switches, Capybara implements migration-aware dynamic packet forwarding and transient packet buffering, further reducing system interference during live TCP migration. Capybara can transparently migrate a running TCP connection in 4 μs on average. It improves the average migration host latency by about 12 times compared to a Linux kernel-based solution.
{"title":"Capybara: μSecond-Scale Live TCP Migration","authors":"I. Choi, Nimish Wadekar, Raj Joshi, Joshua Fried, Dan R. K. Ports, Irene Zhang, Jialin Li","doi":"10.1145/3609510.3609813","DOIUrl":"https://doi.org/10.1145/3609510.3609813","url":null,"abstract":"Latency-critical μs-scale data center applications are susceptible to server load spikes. The issue is particularly challenging for services using long-lived TCP connections. This paper introduces Capybara, a highly efficient and versatile live TCP migration system. Capybara builds atop a deterministic, kernel-bypassed TCP stack running in a library OS to realize its μs-scale TCP migration mechanism. Using modern programmable switches, Capybara implements migration-aware dynamic packet forwarding and transient packet buffering, further reducing system interference during live TCP migration. Capybara can transparently migrate a running TCP connection in 4 μs on average. It improves the average migration host latency by about 12 times compared to a Linux kernel-based solution.","PeriodicalId":149629,"journal":{"name":"Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131535361","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Tchana, Dorian Goepp, Stella Bitchebe, Renaud Lachaize
Process and Thread are first-order abstractions of the operating system (OS), whose implementation is wired into the OS core. Several research works have shown the inadequacy of these two main abstractions for modern isolation needs, leading to the introduction of additional abstractions with new isolation and communication features. Despite their usefulness, these new proposals are introduced in a somewhat ad-hoc manner, compromising their broad and consensual adoption. This position paper presents xOS, an OS design that does not introduce yet another first-class isolation abstraction but instead investigates how the OS can help application programmers, libraries, and OS developers integrate and easily use new abstractions. To our knowledge, xOS is the first work in this area. Similar to file system development built around a Virtual File System (VFS), xOS introduces the concept of Isolation Context (IC), which should be the unique first-class abstraction wired into the OS core. ICs can be realized in several pluggable Isolation Context Factories (ICFs) such as ProcessFactory (provides processes), Thread-Factory (provides threads), Docker Engine (provides Docker containers), KVM (provides KVM virtual machines), Wasp (provides virtines), etc. We discuss our plan to redesign a general-purpose OS from these foundations, the required APIs, and how to support new and legacy applications.
{"title":"xOS: The End Of The Process-Thread Duo Reign","authors":"A. Tchana, Dorian Goepp, Stella Bitchebe, Renaud Lachaize","doi":"10.1145/3609510.3609817","DOIUrl":"https://doi.org/10.1145/3609510.3609817","url":null,"abstract":"Process and Thread are first-order abstractions of the operating system (OS), whose implementation is wired into the OS core. Several research works have shown the inadequacy of these two main abstractions for modern isolation needs, leading to the introduction of additional abstractions with new isolation and communication features. Despite their usefulness, these new proposals are introduced in a somewhat ad-hoc manner, compromising their broad and consensual adoption. This position paper presents xOS, an OS design that does not introduce yet another first-class isolation abstraction but instead investigates how the OS can help application programmers, libraries, and OS developers integrate and easily use new abstractions. To our knowledge, xOS is the first work in this area. Similar to file system development built around a Virtual File System (VFS), xOS introduces the concept of Isolation Context (IC), which should be the unique first-class abstraction wired into the OS core. ICs can be realized in several pluggable Isolation Context Factories (ICFs) such as ProcessFactory (provides processes), Thread-Factory (provides threads), Docker Engine (provides Docker containers), KVM (provides KVM virtual machines), Wasp (provides virtines), etc. We discuss our plan to redesign a general-purpose OS from these foundations, the required APIs, and how to support new and legacy applications.","PeriodicalId":149629,"journal":{"name":"Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems","volume":"119 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129100458","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Despite recent attempts to reduce the number of parameters of large language models (LLMs), their parameter data is still too large to fit into a single GPU. With the emergence of throughput-oriented tasks, high-throughput generative inference frameworks for LLMs on a single commodity GPU leverage GPU, DRAM, and NVMe to run inference on large models with terabytes of data. Our analysis of the technique shows that the runtime is dominated by data transfers of the weights, leading to a low utilization of both the GPU and the CPU. In this paper, we increase the throughput and decrease the total latency of state-of-the-art frameworks by including the CPU as a compute device and overlapping computations on the CPU with GPU data transfers. Our work shows a promising improvement of around 40% in throughput and total latency, with potential room for further improvements.
{"title":"Improving Throughput-oriented Generative Inference with CPUs","authors":"Daon Park, Sungbin Jo, Bernhard Egger","doi":"10.1145/3609510.3609815","DOIUrl":"https://doi.org/10.1145/3609510.3609815","url":null,"abstract":"Despite recent attempts to reduce the number of parameters of large language models (LLMs), their parameter data is still too large to fit into a single GPU. With the emergence of throughput-oriented tasks, high-throughput generative inference frameworks for LLMs on a single commodity GPU leverage GPU, DRAM, and NVMe to run inference on large models with terabytes of data. Our analysis of the technique shows that the runtime is dominated by data transfers of the weights, leading to a low utilization of both the GPU and the CPU. In this paper, we increase the throughput and decrease the total latency of state-of-the-art frameworks by including the CPU as a compute device and overlapping computations on the CPU with GPU data transfers. Our work shows a promising improvement of around 40% in throughput and total latency, with potential room for further improvements.","PeriodicalId":149629,"journal":{"name":"Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114960572","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems","authors":"","doi":"10.1145/3609510","DOIUrl":"https://doi.org/10.1145/3609510","url":null,"abstract":"","PeriodicalId":149629,"journal":{"name":"Proceedings of the 14th ACM SIGOPS Asia-Pacific Workshop on Systems","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124105427","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：