BenchCouncil Transactions on Benchmarks, Standards and Evaluations最新文献

Rapid business expansion of various companies has placed growing demand on office desktops recent decades. However, improper evaluation of system performance and inexplicit awareness of practical use conditions often hamper the efforts to make a consummate selection among multiple alternatives. From the perspective of end users, to optimize the evaluation process of desktop performance in centralized procurement, we present CpsMark+, a coherent benchmark system that evaluates office desktop performance based on simulated user experience. Specifically, CpsMark+ includes scenario-oriented workloads portraying representative user behaviors modeled from the cooperative workflow in modern office routines, and flexibly adapted metrics properly reflecting end-user experience according to different task types. The contrast experiment between state-of-the-art benchmarks demonstrates high sensitivity of CpsMark+ to various hardware components, e.g., CPU, and high repeatability with a Coefficient of Variation less than 3%. In a practical case study, we also demonstrate the effectiveness of CpsMark+ in simulating user experience of tested computer systems under modern office-oriented scenarios for improving the quality of office desktop performance evaluation in centralized procurement.

In recent years, the convergence of High Performance Computing (HPC) and artificial intelligence (AI) makes the community desperately need a benchmark to guide the design of next-generation scalable HPC AI systems. The success of the HPL benchmarks and the affiliated TOP500 ranking indicates that scalability is the fundamental requirement to evaluate HPC systems. However, being scalable in terms of these emerging AI workloads like deep learning (DL) raises nontrivial challenges. This paper formally and systematically analyzes the factor that limits scalability in DL workloads and presents HPC AI500 v3.0, a scalable HPC AI benchmarking framework. The HPC AI500 V3.0 methodology is inspired by bagging, which utilizes the collective wisdom of an ensemble of base models and enables the benchmarks to be adaptively scalable to different scales of HPC systems. We implement HPC AI500 V3.0 in a highly customizable manner, maintaining the space of various optimization from both system and algorithm levels. By reusing the representative workloads in HPC AI500 V2.0, we evaluate HPC AI500 V3.0 on typical HPC systems, and the results show it has near-linear scalability. Furthermore, based on the customizable design, we present a case study to perform a trade-off between AI model quality and its training speed. The source code of HPC AI500 V3.0 is publicly available from the HPC AI500 project homepage  https://www.benchcouncil.org/aibench/hpcai500/.

Open Artificial Intelligence (AI) published an AI chatbot tool called ChatGPT at the end of November 2022. Generative Pre-trained Transformer (GPT) architecture is the foundation of ChatGPT. On the internet, ChatGPT has been rapidly growing. This chatbot enables users to discuss with the AI by inputting prompts, and it is based on OpenAI’s language model. Although ChatGPT is fantastic and produces exciting results for writing tales, poetry, songs, essays, and other things, it has certain restrictions. Users may ask the bot questions, and it will reply with pertinent, convincing subjects and replies. ChatGPT has now risen to the top of several academic agendas. Administrators create task teams and hold institution-wide meetings to react to the tools, with most of the advice being to adopt this technology. This paper briefs about the ChatGPT and its need. Further, various Progressive Work Flow Processes of the ChatGPT Tool are stated diagrammatically. Specific features and capabilities of the ChatGPT Support System are studied in this paper. Finally, we identified and discussed the significant roles of ChatGPT in the current scenario. The neural language models that form the foundation of character AI have been developed from the bottom up with talks in mind. This technology implies that the programme uses deep learning methods to analyse and produce text. The model “understands” the subtleties of human-produced natural language using vast amounts of data from the internet.

Sponsored and organized by the International Open Benchmark Council (BenchCouncil), the Bench conference encompasses a wide range of topics in benchmarking, measurement, evaluation methods, and tools. Bench’s multi-disciplinary emphasis provides an ideal environment for developers and researchers from the architecture, data management, algorithm, system, network, dataset, and application communities to discuss practical and theoretical work covering workload characterization, benchmarks and tools, evaluation, measurement and optimization, and dataset generation. The Bench conferences have been successfully held for four series from 2018 to 2021 and attracted plenty of paper submissions and participants. Bench 2022 will be held virtually on Nov. 7–9, 2022, and invites manuscripts describing original work in benchmarking, measuring, and optimizing. The conference website is https://www.benchcouncil.org/bench2022/index.html.

Financial service providers find blockchain technology useful to enhance authenticity, security, and risk management. Several institutions are adopting blockchain in trade and finance systems to build smart contracts between participants, improve efficiency and transparency, and open up newer revenue opportunities. Blockchain’s unique recording capabilities make the existing clearing and settlement process redundant. Banks and other financial entities are adopting blockchain-enabled IDs to identify people. Better results come from organisations’ capacity to foresee emerging trends in financial blockchain applications and develop blockchain functionality. The transfer of asset ownership and addressing the maintenance of a precise financial ledger. Measurement, communication, and analysis of financial information are three significant areas to be focussed on by accounting professionals. Blockchain clarifies asset ownership and the existence of obligations for accountants, and it has the potential to improve productivity. This paper identifies and studies relevant articles related to blockchain for finance. This paper focuses on Blockchain technology and its importance for financial services. Further takes up various tools, strategies, and featured services in Blockchain-based financial services. Finally, the paper identifies and evaluates the significant applications of Blockchain technology in financial services. Credit reports significantly impact the financial lives of customers. Recent data breaches demonstrate the superior security of blockchain-based credit reporting over conventional server-based reporting. Blockchain-based systems enable the faster, more cost-effective, and more customised issuance of digital securities. With its adoption, the market for investors can be expanded, costs for issuers can be reduced, and counterparty risk can be reduced due to the ability to customise digital financial instruments to the demands of investors. It uses mutualised standards, protocols, and shared procedures to give network users a single common source of truth. Participants in the business network can now more easily collaborate, manage data, and agree with this technology’s application.

Understanding the designs and performance characterizations of hot interconnects on modern data center and high-performance computing (HPC) clusters is a fruitful research topic in recent years. The rapid and continuous growth of high-bandwidth and low-latency communication requirements for various types of data center and HPC applications (such as big data, deep learning, and microservices) has been pushing the envelope of advanced interconnect designs. We believe this is high time to investigate the performance characterizations of representative hot interconnects with different benchmarks. Hence, this paper presents an extensive survey of state-of-the-art hot interconnects on data center and HPC clusters and the associated representative benchmarks to help the community to better understand modern interconnects. In addition, we characterize these interconnects by the related benchmarks under different application scenarios. We provide our perspectives on benchmarking data center interconnects based on our survey, experiments, and results.

Emerging and future applications rely heavily upon systems consisting of Internet of Things (IoT), edges, data centers, and humans-in-the-loop. Significantly different from warehouse-scale computers that serve independent concurrent user requests, this new class of computer systems directly interacts with the physical world, considering humans an essential part and performing safety-critical and mission-critical operations; their computations have intertwined dependencies between not only adjacent execution loops but also actions or decisions triggered by IoTs, edge, datacenters, or humans-in-the-loop; the systems must first satisfy the accuracy metric in predicting, interpreting, or taking action before meeting the performance goal under different cases.

This article argues we need a paradigm shift to reconstruct the IoTs, edges, data centers, and humans-in-the-loop as a computer rather than a distributed system. We coin a new term, high fusion computers (HFCs), to describe this class of systems. The fusion in the term has two implications: fusing IoTs, edges, data centers, and humans-in-the-loop as a computer, fusing the physical and digital worlds through HFC systems. HFC is a pivotal case of the open-source computer systems initiative. We laid out the challenges, plan, and call for uniting our community’s wisdom and actions to address the HFC challenges. Everything, including the source code, will be publicly available from the project homepage: https://www.computercouncil.org/HFC/.