BenchCouncil Transactions on Benchmarks, Standards and Evaluations最新文献

COVID-19 was one of the deadliest and most infectious illnesses of this century. Research has been done to decrease pandemic deaths and slow down its spread. COVID-19 detection investigations have utilised Chest X-ray (CXR) images with deep learning techniques with its sensitivity in identifying pneumonic alterations. However, CXR images are not publicly available due to users’ privacy concerns, resulting in a challenge to train a highly accurate deep learning model from scratch. Therefore, we proposed CoviDetector, a new semi-supervised approach based on transfer learning and clustering, which displays improved performance and requires less training data. CXR images are given as input to this model, and individuals are categorised into three classes: (1) COVID-19 positive; (2) Viral pneumonia; and (3) Normal. The performance of CoviDetector has been evaluated on four different datasets, achieving over 99% accuracy on them. Additionally, we generate heatmaps utilising Grad-CAM and overlay them on the CXR images to present the highlighted areas that were deciding factors in detecting COVID-19. Finally, we developed an Android app to offer a user-friendly interface. We release the code, datasets and results’ scripts of CoviDetector for reproducibility purposes; they are available at: https://github.com/dasanik2001/CoviDetector

Sponsored and organized by the International Open Benchmark Council (BenchCouncil), the IC conference is to provide a pioneering technology map through searching and advancing state-of-the-art and state-of-the-practice in processors, systems, algorithms, and applications for machine learning, deep learning, spiking neural network and other AI techniques across multidisciplinary and interdisciplinary areas. IC 2023 invites manuscripts describing original work in the above areas and topics. All accepted papers will be presented at the IC 2023 conference and published by Springer CCIS (Indexed by EI). The IC conferences have been successfully held for two series from 2019 to 2022 and attracted plenty of paper submissions and participants. IC 2023 will be held on December 4-6, 2023 in Sanya and invites manuscripts describing original work in processors, systems, algorithms, and applications for AI techniques across multidisciplinary and interdisciplinary areas. The conference website is https://www.benchcouncil.org/ic2023/.

Important Dates: Paper Submission: July 31, 2023, at 11:59 PM AoE Notification: September 30, 2023, at 11:59 PM AoE Final Papers Due: October 31, 2023, at 11:59 PM AoE Conference Date: December 4-6, 2023 Submission Site: https://ic2023.hotcrp.com/

BenchCouncil Transactions on Benchmarks, Standards and Evaluations (TBench) is an open-access journal dedicated to advancing the field of benchmarks, data sets, standards, evaluations and optimizations. This journal is a peer-reviewed, subsidized open-access journal where The International Open Benchmark Council (BenchCouncil) pays the open-access fee. Authors do not have to pay any open-access publication fee. However, at least one of the authors must register BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench) (https://www.benchcouncil.org/bench/) and present their work. It seeks a fast-track publication with an average turnaround time of one month.

We invite submissions covering a wide range of topics from various disciplines, with a particular emphasis on interdisciplinary research. Whether it pertains to computers, AI, medicine, education, finance, business, psychology, or other social disciplines, all relevant contributions are welcome.

At TBench, we prioritize the reproducibility of research. We strongly encourage authors to ensure that their articles are prepared for open-source or artifact evaluation before submission. The journal website is https://www.benchcouncil.org/tbench.

Artificial Intelligence (AI)-based ChatGPT developed by OpenAI is now widely accepted in several fields, including education. Students can learn about ideas and theories by using this technology while generating content with it. ChatGPT is built on State of the Art (SOA), like Deep Learning (DL), Natural Language Processing (NLP), and Machine Learning (ML), an extrapolation of a class of ML-NLP models known as Large Language Model (LLMs). It may be used to automate test and assignment grading, giving instructors more time to concentrate on instruction. This technology can be utilised to customise learning for kids, enabling them to focus more intently on the subject matter and critical thinking ChatGPT is an excellent tool for language lessons since it can translate text from one language to another. It may provide lists of vocabulary terms and meanings, assisting students in developing their language proficiency with resources. Personalised learning opportunities are one of ChatGPT’s significant applications in the classroom. This might include creating educational resources and content tailored to a student’s unique interests, skills, and learning goals. This paper discusses the need for ChatGPT and the significant features of ChatGPT in the education system. Further, it identifies and discusses the significant applications of ChatGPT in education. Using ChatGPT, educators may design lessons and instructional materials specific to each student’s requirements and skills based on current trends. Students may work at their speed and concentrate on the areas where they need the most support, resulting in a more effective and efficient learning environment. Both instructors and students may profit significantly from using ChatGPT in the classroom. Instructors may save time on numerous duties by using this technology. In future, ChatGPT will become a powerful tool for enhancing students’ and teachers’ experience.

Hybrid Transactional/Analytical Processing (HTAP) databases are designed to execute real-time analytics and provide performance isolation for online transactions and analytical queries. Real-time analytics emphasize analyzing the fresh data generated by online transactions. And performance isolation depicts the performance interference between concurrently executing online transactions and analytical queries. However, HTAP databases are extreme lack micro-benchmarks to accurately measure data freshness. Despite the abundance of HTAP databases and benchmarks, there needs to be more thorough research on the performance isolation and real-time analytics capabilities of HTAP databases. This paper focuses on the critical designs of mainstream HTAP databases and the state-of-the-art and state-of-the-practice HTAP benchmarks. First, we systematically introduce the advanced technologies adopted by HTAP databases for real-time analytics and performance isolation capabilities. Then, we summarize the pros and cons of the state-of-the-art and state-of-the-practice HTAP benchmarks. Next, we design and implement a micro-benchmark for HTAP databases, which can precisely control the rate of fresh data generation and the granularity of fresh data access. Finally, we devise experiments to evaluate the performance isolation and real-time analytics capabilities of the state-of-the-art HTAP database. In our continued pursuit of transparency and community collaboration, we will soon make available our comprehensive specifications, meticulously crafted source code, and significant results for public access at https://www.benchcouncil.org/mOLxPBench.

Generative Pretrained Transformer, often known as GPT, is an innovative kind of Artificial Intelligence (AI) which can produce writing that seems to have been written by a person. OpenAI created this AI language model called ChatGPT. It is built using the GPT architecture and is trained on a large corpus of text data to respond to natural language inquiries that resemble a person’s requirements. This technology has lots of applications in healthcare. The need for accurate and current data is one of the major obstacles to adopting ChatGPT in healthcare. GPT must have access to precise and up-to-date medical data to provide trustworthy suggestions and treatment options. It might be accomplished by ensuring that the data used by GPT is received from reliable sources and that the data is updated regularly. Since sensitive medical information would be involved, it will also be crucial to consider privacy and security issues while utilising GPT in the healthcare industry. This paper briefs about ChatGPT and its need for healthcare, its significant Work Flow Dimensions and typical features of ChatGPT for the Healthcare domain. Finally, it identified and discussed significant applications of ChatGPT for healthcare. ChatGPT can comprehend the conversational context and provide contextually appropriate replies. Its effectiveness as a conversational AI tool makes it useful for chatbots, virtual assistants, and other applications. However, we see many limitations in medical ethics, data interpretation, accountability and other issues related to the privacy. Regarding specialised tasks like text creation, language translation, text categorisation, text summarisation, and creating conversation systems, ChatGPT has been pre-trained on a large corpus of text data, and somewhat satisfactory results can be expected. Moreover, it can also be utilised for various Natural Language Processing (NLP) activities, including sentiment analysis, part-of-speech tagging, and named entity identification.

Spiking Neural Networks (SNNs) show great potential for solving Artificial Intelligence (AI) applications. At the preliminary stage of SNNs, benchmarks are essential for evaluating and optimizing SNN algorithms, software, and hardware toward AI scenarios. However, a majority of SNN benchmarks focus on evaluating SNN for brain science, which has distinct neural network architectures and targets. Even though there have several benchmarks evaluating SNN for AI, they only focus on a single stage of training and inference or a processing fragment of a whole stage without accuracy information. Thus, the existing SNN benchmarks lack an end-to-end perspective that not only covers both training and inference but also provides a whole training process to a target accuracy level.

This paper presents SNNBench—the first end-to-end AI-oriented SNN benchmark covering the processing stages of training and inference and containing the accuracy information. Focusing on two typical AI applications: image classification and speech recognition, we provide nine workloads that consider the typical characteristics of SNN, i.e., the dynamics of spiking neurons, and AI, i.e., learning paradigms including supervised and unsupervised learning, learning rules like backpropagation, connection types like fully connected, and accuracy. The evaluations of SNNBench on both CPU and GPU show its effectiveness. The specifications, source code, and results will be publicly available from https://www.benchcouncil.org/SNNBench.

The Reserve Bank of India (RBI) has recently launched the country’s first pilot project for the digital currency known as the digital rupee or e-Rupee (e^₹). The launch of the digital rupee represents a significant advancement in the “Digital India” revolution. It will be a fantastic opportunity for India since it might make conducting business easier while enhancing the security and resilience of the overall payments system. Digital currency attempts to rapidly progress monetary policy to disrupt physical money, lower the cost of financial transactions, and reshape how the money will circulate. Although the effects of digital currency cannot be foreseen, it is extremely important to thoroughly research digital currency and its effects on the operational stage. The development of a digital currency infrastructure has some challenges in terms of performance, scalability, and different usage scenarios. The article clarifies what e^₹ is. How does it work? What makes it different from cryptocurrencies? What are the major challenges and prospects for it in India?

Learning-based malware detection systems (LB-MDS) play a crucial role in defending computer systems from malicious attacks. Nevertheless, these systems can be vulnerable to various attacks, which can have significant consequences. Software obfuscation techniques can be used to modify the features of malware, thereby avoiding its classification as malicious by LB-MDS. However, existing portable executable (PE) malware datasets primarily use a single obfuscation technique, which LB-MDS has already learned, leading to a loss of their robustness evaluation ability. Therefore, creating a dataset with diverse features that were not observed during LB-MDS training has become the main challenge in evaluating the robustness of LB-MDS.

We propose a obfuscation dataset ERMDS that solves the problem of evaluating the robustness of LB-MDS by generating malwares with diverse features. When designing this dataset, we created three types of obfuscation spaces, corresponding to binary obfuscation, source code obfuscation, and packing obfuscation. Each obfuscation space has multiple obfuscation techniques, each with different parameters. The obfuscation techniques in these three obfuscation spaces can be used in combination and can be reused. This enables us to theoretically obtain an infinite number of obfuscation combinations, thereby creating malwares with a diverse range of features that have not been captured by LB-MDS.

To assess the effectiveness of the ERMDS obfuscation dataset, we create an instance of the obfuscation dataset called ERMDS-X. By utilizing this dataset, we conducted an evaluation of the robustness of two LB-MDS models, namely MalConv and EMBER, as well as six commercial antivirus software products, which are anonymized as AV1-AV6. The results of our experiments showed that ERMDS-X effectively reveals the limitations in the robustness of existing LB-MDS models, leading to an average accuracy reduction of 20% in LB-MDS and 32% in commercial antivirus software. We conducted a comprehensive analysis of the factors that contributed to the observed accuracy decline in both LB-MDS and commercial antivirus software. We have released the ERMDS-X dataset as an open-source resource, available on GitHub at https://github.com/lcjia94/ERMDS.