Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137762
Y. Sit, Cho-Li Wang, F. Lau
Cluster-based web server is a popular solution to meet the demand of the ever-growing web traffic. However existing approaches suffer from several limitations to achieve this. Dispatcher-based systems either can achieve only coarse-grained load balancing or would introduce heavy load to the dispatcher Mechanisms like cooperative caching consume much network resources when transferring large cache objects. In this paper, we present a new network support mechanism, called Socket Cloning (SC), in which an opened socket can be migrated efficiently between cluster nodes. With SC, the processing of HTTP requests can be moved to the node that has a cached copy of the requested document, thus bypassing any object transfer between peer servers. A prototype has been implemented and tests show that SC incurs less overhead than all the mentioned approaches. In trace-driven benchmark tests, our system outperforms these approaches by more than 30% with a cluster of twelve web server nodes.
{"title":"Socket cloning for cluster-based web servers","authors":"Y. Sit, Cho-Li Wang, F. Lau","doi":"10.1109/CLUSTR.2002.1137762","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137762","url":null,"abstract":"Cluster-based web server is a popular solution to meet the demand of the ever-growing web traffic. However existing approaches suffer from several limitations to achieve this. Dispatcher-based systems either can achieve only coarse-grained load balancing or would introduce heavy load to the dispatcher Mechanisms like cooperative caching consume much network resources when transferring large cache objects. In this paper, we present a new network support mechanism, called Socket Cloning (SC), in which an opened socket can be migrated efficiently between cluster nodes. With SC, the processing of HTTP requests can be moved to the node that has a cached copy of the requested document, thus bypassing any object transfer between peer servers. A prototype has been implemented and tests show that SC incurs less overhead than all the mentioned approaches. In trace-driven benchmark tests, our system outperforms these approaches by more than 30% with a cluster of twelve web server nodes.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85018853","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137748
K. Pedretti, R. Brightwell, Joshua Williams
In this paper, we describe additions and modifications to the Computational Plant (Cplant/sup /spl trade//) system software to support multi-processor compute nodes and to support heterogeneous node types. We describe how these capabilities have been incorporated into our scalable runtime system and how these changes affect the interface seen by end users and application developers. We also discuss several important operating system and networking issues that can directly impact application performance. We present some initial performance metrics that indicate how our current implementation scales when multiple processes are running on a single node.
{"title":"Cplant/sup /spl trade// runtime system support for multi-processor and heterogeneous compute nodes","authors":"K. Pedretti, R. Brightwell, Joshua Williams","doi":"10.1109/CLUSTR.2002.1137748","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137748","url":null,"abstract":"In this paper, we describe additions and modifications to the Computational Plant (Cplant/sup /spl trade//) system software to support multi-processor compute nodes and to support heterogeneous node types. We describe how these capabilities have been incorporated into our scalable runtime system and how these changes affect the interface seen by end users and application developers. We also discuss several important operating system and networking issues that can directly impact application performance. We present some initial performance metrics that indicate how our current implementation scales when multiple processes are running on a single node.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"82950919","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137771
M. Janjua, M. M. Yasin, Ch. Falak Sher, K. Awan, I. Hassan
CEJVM is a cluster enabled Java Virtual Machine, which executes in a distributed fashion among collaborating nodes of a dedicated cluster. It extends Java's multithreading mechanism to the parallel computing paradigm by transparent migration of independent application threads modeled in master worker paradigm. The goal is to obtain improved performance for computationally-intensive multi-threaded Java programs without modifying traditional JVM code, Java language or compiler implementation. Deploying a master worker relationship among the nodes in the cluster, CEJVM monitors, packs, transports and resurrects the Java threads on "Master JVM" and helper "Worker JVMs". Currently, we have created a prototype that runs pure Java applications on the local Ethernet based cluster of Win2K computers. We have achieved appreciable speedup for an ANN training program written in Java (with independent threads) on CEJVM without any modification to the program's source code.
{"title":"CEJVM: \"cluster enabled Java Virtual Machine\"","authors":"M. Janjua, M. M. Yasin, Ch. Falak Sher, K. Awan, I. Hassan","doi":"10.1109/CLUSTR.2002.1137771","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137771","url":null,"abstract":"CEJVM is a cluster enabled Java Virtual Machine, which executes in a distributed fashion among collaborating nodes of a dedicated cluster. It extends Java's multithreading mechanism to the parallel computing paradigm by transparent migration of independent application threads modeled in master worker paradigm. The goal is to obtain improved performance for computationally-intensive multi-threaded Java programs without modifying traditional JVM code, Java language or compiler implementation. Deploying a master worker relationship among the nodes in the cluster, CEJVM monitors, packs, transports and resurrects the Java threads on \"Master JVM\" and helper \"Worker JVMs\". Currently, we have created a prototype that runs pure Java applications on the local Ethernet based cluster of Win2K computers. We have achieved appreciable speedup for an ANN training program written in Java (with independent threads) on CEJVM without any modification to the program's source code.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87738134","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137731
Qianfeng Zhang, C. Keppitiyagama, Alan S. Wagner
We present work that extends our previous Myrinet port for LAM/MPI, MPI-NP, with collective communication primitives on the NIC. This work is another step in our experiment of making the NIC MPI aware. We believe that an MPI aware control program on the NIC can deliver a richer set of performance enhancements, not just restricted to better bandwidth/latency, to MPI applications. MPI collective communication involves considerable interactions between the communication subsystems of the nodes that are not of any direct interest to the application. By migrating these talkative components to the Myrinet network interface card we allow this dialog between the nodes to happen with minimum latency. We explore the advantage of supporting several MPI collective communication routines on the NIC. These include MPI /spl I.bar/Bcast (), MPI/spl I.bar/Barrier and MPI/spl I.bar/Comm/spl I.bar/Create ().
{"title":"Supporting MPI collective communication on network processors","authors":"Qianfeng Zhang, C. Keppitiyagama, Alan S. Wagner","doi":"10.1109/CLUSTR.2002.1137731","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137731","url":null,"abstract":"We present work that extends our previous Myrinet port for LAM/MPI, MPI-NP, with collective communication primitives on the NIC. This work is another step in our experiment of making the NIC MPI aware. We believe that an MPI aware control program on the NIC can deliver a richer set of performance enhancements, not just restricted to better bandwidth/latency, to MPI applications. MPI collective communication involves considerable interactions between the communication subsystems of the nodes that are not of any direct interest to the application. By migrating these talkative components to the Myrinet network interface card we allow this dialog between the nodes to happen with minimum latency. We explore the advantage of supporting several MPI collective communication routines on the NIC. These include MPI /spl I.bar/Bcast (), MPI/spl I.bar/Barrier and MPI/spl I.bar/Comm/spl I.bar/Create ().","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87070219","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137755
Ming Li, Wenchao Tao, Dani Goldberg, I. Hsu, Y. Tamir
We describe the communication infrastructure (CI) for our fault-tolerant cluster middleware, which is optimized for two classes of communication: for the applications and for the cluster management middleware. This CI was designed for portability and for efficient operation on top of modern user-level message passing mechanisms. We present a functional fault model for the CI and show how platform-specific faults map to this fault model. Based on this fault model, we have developed a fault injection scheme that is integrated with the CI and is thus portable across different communication technologies. We have used fault injection to validate and evaluate the implementation of the CI itself as well as the cluster management middleware in the presence of communication faults.
{"title":"Design and validation of portable communication infrastructure for fault-tolerant cluster middleware","authors":"Ming Li, Wenchao Tao, Dani Goldberg, I. Hsu, Y. Tamir","doi":"10.1109/CLUSTR.2002.1137755","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137755","url":null,"abstract":"We describe the communication infrastructure (CI) for our fault-tolerant cluster middleware, which is optimized for two classes of communication: for the applications and for the cluster management middleware. This CI was designed for portability and for efficient operation on top of modern user-level message passing mechanisms. We present a functional fault model for the CI and show how platform-specific faults map to this fault model. Based on this fault model, we have developed a fault injection scheme that is integrated with the CI and is thus portable across different communication technologies. We have used fault injection to validate and evaluate the implementation of the CI itself as well as the cluster management middleware in the presence of communication faults.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76930907","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137749
J. Navarro, R. Evard, Daniel Nurmi, N. Desai
Systems administrators of large clusters often need to perform the same administrative task hundreds or thousands of times. Administrators have traditionally performed some time-consuming tasks, such as operating system installation, configuration, and maintenance, manually. By combining network services such as DHCP, TFTP, FTP, HTTP, and NFS with remote hardware control and scripted installation, configuration, and maintenance techniques, cluster administrators can automate these administrative tasks. Scalable cluster administration addresses this challenge: What hardware and software design techniques can cluster builders use to automate cluster administration on very large clusters? We describe the approach used in the Mathematics and Computer Science Division of Argonne National Laboratory on Chiba City I, a 314-node Linux cluster; and we analyze the scalability, flexibility, performance and reliability benefits and limitations from that approach.
{"title":"Scalable cluster administration - Chiba City I approach and lessons learned","authors":"J. Navarro, R. Evard, Daniel Nurmi, N. Desai","doi":"10.1109/CLUSTR.2002.1137749","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137749","url":null,"abstract":"Systems administrators of large clusters often need to perform the same administrative task hundreds or thousands of times. Administrators have traditionally performed some time-consuming tasks, such as operating system installation, configuration, and maintenance, manually. By combining network services such as DHCP, TFTP, FTP, HTTP, and NFS with remote hardware control and scripted installation, configuration, and maintenance techniques, cluster administrators can automate these administrative tasks. Scalable cluster administration addresses this challenge: What hardware and software design techniques can cluster builders use to automate cluster administration on very large clusters? We describe the approach used in the Mathematics and Computer Science Division of Argonne National Laboratory on Chiba City I, a 314-node Linux cluster; and we analyze the scalability, flexibility, performance and reliability benefits and limitations from that approach.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76857516","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137777
W. Ligon
Parallel I/O remains a critical problem for cluster computing. A significant number of important applications need high performance parallel I/O and most cluster systems provide enough hardware to deliver the required performance. System software for achieving the desired goals remains in the research and development stage. A number of parallel file systems have achieved remarkable goals in one or more of several key areas related to parallel I/O, but there is still great reluctance to commit to any file system currently available. This is mostly due to the fact that these file systems do not address enough issues at once in a package that is robust enough for widespread use. Critical goals in the development of an operation parallel file system for clusters include: high performance with scalability; reliability/fault tolerance; flexible and efficient integration with parallel codes; portability. These issues give rise to problems with interfaces and semantics, in addition to specific technical problems such as distributed locking, caching, and redundancy. The next generation of parallel file systems must look beyond traditional interfaces, semantics, and implementation methods in order achieve the desired goals. Of equal importance is the issue of knowing to what extent a given file system achieves these goals. Given that no file system is likely to address all of these goals equally well, it is important to be able to measure a given file system's utility in these areas through benchmarking or other evaluation methods. We explore a few of these issues and include specific examples and a case study of the PVFS V2 team's approach to these issues.
{"title":"Research directions in parallel I/O for clusters","authors":"W. Ligon","doi":"10.1109/CLUSTR.2002.1137777","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137777","url":null,"abstract":"Parallel I/O remains a critical problem for cluster computing. A significant number of important applications need high performance parallel I/O and most cluster systems provide enough hardware to deliver the required performance. System software for achieving the desired goals remains in the research and development stage. A number of parallel file systems have achieved remarkable goals in one or more of several key areas related to parallel I/O, but there is still great reluctance to commit to any file system currently available. This is mostly due to the fact that these file systems do not address enough issues at once in a package that is robust enough for widespread use. Critical goals in the development of an operation parallel file system for clusters include: high performance with scalability; reliability/fault tolerance; flexible and efficient integration with parallel codes; portability. These issues give rise to problems with interfaces and semantics, in addition to specific technical problems such as distributed locking, caching, and redundancy. The next generation of parallel file systems must look beyond traditional interfaces, semantics, and implementation methods in order achieve the desired goals. Of equal importance is the issue of knowing to what extent a given file system achieves these goals. Given that no file system is likely to address all of these goals equally well, it is important to be able to measure a given file system's utility in these areas through benchmarking or other evaluation methods. We explore a few of these issues and include specific examples and a case study of the PVFS V2 team's approach to these issues.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81702709","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137761
K. H. Yum, Eun Jung Kim, C. Das, Mazin S. Yousif, J. Duato
Admission and congestion control mechanisms are integral parts of any Quality of Service (QoS) design for networks that support integrated traffic. In this paper we propose an-admission control algorithm and a congestion control algorithm for clusters, which are increasingly being used in a diverse set of applications that require QoS guarantees. The uniqueness of our approach is that we develop these algorithms for wormhole-switched networks. We use QoS-capable wormhole routers and QoS-capable network interface cards (NICs), referred to as Host Channel Adapters (HCAs) in InfiniBand/spl trade/ Architecture (IBA), to evaluate the effectiveness of these algorithms. The admission control is applied at the HCAs and the routers, while the congestion control is deployed only at the HCAs. Simulation results indicate that the admission and congestion control algorithms are quite effective in delivering the assured performance. The proposed credit-based congestion control algorithm is simple and practical in that it relies on hardware already available in the HCA to regulate traffic injection.
{"title":"Integrated admission and congestion control for QoS support in clusters","authors":"K. H. Yum, Eun Jung Kim, C. Das, Mazin S. Yousif, J. Duato","doi":"10.1109/CLUSTR.2002.1137761","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137761","url":null,"abstract":"Admission and congestion control mechanisms are integral parts of any Quality of Service (QoS) design for networks that support integrated traffic. In this paper we propose an-admission control algorithm and a congestion control algorithm for clusters, which are increasingly being used in a diverse set of applications that require QoS guarantees. The uniqueness of our approach is that we develop these algorithms for wormhole-switched networks. We use QoS-capable wormhole routers and QoS-capable network interface cards (NICs), referred to as Host Channel Adapters (HCAs) in InfiniBand/spl trade/ Architecture (IBA), to evaluate the effectiveness of these algorithms. The admission control is applied at the HCAs and the routers, while the congestion control is deployed only at the HCAs. Simulation results indicate that the admission and congestion control algorithms are quite effective in delivering the assured performance. The proposed credit-based congestion control algorithm is simple and practical in that it relies on hardware already available in the HCA to regulate traffic injection.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85309481","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137746
D. Turner, Xuehua Chen
In a Linux cluster, as in any multiprocessor system, the inter-processor communication rate is the major limiting factor to its general usefulness. This research is geared toward improving the communication performance by identifying where the inefficiencies lie and trying to understand their cause. The NetPIPE utility is being used to compare the latency and throughput of all current message-passing libraries and the native software layers they run upon for a variety of hardware configurations.
{"title":"Protocol-dependent message-passing performance on Linux clusters","authors":"D. Turner, Xuehua Chen","doi":"10.1109/CLUSTR.2002.1137746","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137746","url":null,"abstract":"In a Linux cluster, as in any multiprocessor system, the inter-processor communication rate is the major limiting factor to its general usefulness. This research is geared toward improving the communication performance by identifying where the inefficiencies lie and trying to understand their cause. The NetPIPE utility is being used to compare the latency and throughput of all current message-passing libraries and the native software layers they run upon for a variety of hardware configurations.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85481978","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2002-09-23DOI: 10.1109/CLUSTR.2002.1137790
M. Pourzandi, I. Haddad, C. Levert, Miroslaw Zakrzewski, M. Dagenais
Traditionally the telecom industry has used clusters to meet its carrier-class requirements of high availability, reliability, and scalability, while relying on cost-effective hardware and software. Efficient cluster security is now an essential requirement and has not yet been addressed in a coherent fashion on clustered systems. This paper presents an approach for distributed security architecture that supports advanced security mechanisms for current and future security needs, targeted for carrier-class application servers running on clustered systems.
{"title":"A new architecture for secure carrier-class clusters","authors":"M. Pourzandi, I. Haddad, C. Levert, Miroslaw Zakrzewski, M. Dagenais","doi":"10.1109/CLUSTR.2002.1137790","DOIUrl":"https://doi.org/10.1109/CLUSTR.2002.1137790","url":null,"abstract":"Traditionally the telecom industry has used clusters to meet its carrier-class requirements of high availability, reliability, and scalability, while relying on cost-effective hardware and software. Efficient cluster security is now an essential requirement and has not yet been addressed in a coherent fashion on clustered systems. This paper presents an approach for distributed security architecture that supports advanced security mechanisms for current and future security needs, targeted for carrier-class application servers running on clustered systems.","PeriodicalId":92128,"journal":{"name":"Proceedings. IEEE International Conference on Cluster Computing","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2002-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"83879695","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：