Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00034
Ítalo Santos, Júlio César C. Filho, S. Souza
[Context:] Mobile devices have become increasingly popular, and mobile applications should guarantee a very high level of reliability and quality. Mobile application testing needs to consider several unique requirements that distinguish it from conventional software testing. [Objective:] Our study aims to establish an overview of the testing practices conducted in mobile companies, to identify weaknesses that can be improved to make the testing activity more effective. [Method:] The survey questions were carefully designed using the Goal/Question/Metric method to provide relevant information to the questions raised in our study. [Results and Conclusions:] Our study outlines that native applications are more common. The testing level more performed is the system test and the positions that perform testing levels and objectives are described. Practices related to testing technique selection in the context of mobile applications are highlighted. In the context of this study, Cucumber and Selenium are the testing tools most used to automate testing activity. Some mobile testing characteristics were outlined to understand how the testing in mobile applications run on different devices, how testers deal with the diversity of operating systems that are constantly updated and whether tests are unified to testing a mobile app that runs in different platforms. Furthermore, we report the main challenges faced by testers during the validation of the mobile app.
{"title":"A survey on the practices of mobile application testing","authors":"Ítalo Santos, Júlio César C. Filho, S. Souza","doi":"10.1109/CLEI52000.2020.00034","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00034","url":null,"abstract":"[Context:] Mobile devices have become increasingly popular, and mobile applications should guarantee a very high level of reliability and quality. Mobile application testing needs to consider several unique requirements that distinguish it from conventional software testing. [Objective:] Our study aims to establish an overview of the testing practices conducted in mobile companies, to identify weaknesses that can be improved to make the testing activity more effective. [Method:] The survey questions were carefully designed using the Goal/Question/Metric method to provide relevant information to the questions raised in our study. [Results and Conclusions:] Our study outlines that native applications are more common. The testing level more performed is the system test and the positions that perform testing levels and objectives are described. Practices related to testing technique selection in the context of mobile applications are highlighted. In the context of this study, Cucumber and Selenium are the testing tools most used to automate testing activity. Some mobile testing characteristics were outlined to understand how the testing in mobile applications run on different devices, how testers deal with the diversity of operating systems that are constantly updated and whether tests are unified to testing a mobile app that runs in different platforms. Furthermore, we report the main challenges faced by testers during the validation of the mobile app.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"93 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116756703","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00049
Mónica Fatecha Burian, Patricia Fauvety, Nathalie Aquino, Magalí González, Daniel Romero, L. Cernuzzi, J. Paniagua, Ronald Chenu
The state of the sidewalks in Asunción (Paraguay) is far from being optimal. There are many problems related to obstructions, necessary repairs, lack of ramps, unevenness of surface, among others. In addition, the Municipality of Asunción does not have automated mechanisms to know their updated status. In this work we propose SmartMoving, a mobile application that collects information on the state of the sidewalks, with the help of citizens, and recommends pedestrian paths with less obstacles. The application can be especially useful for people with reduced mobility, as well as for the Municipality of Asunción. This type of application is based on citizen participation, since it receive data from them, and therefore requires a particularly friendly user experience, adapted to users and their daily context. Therefore, the need for a participatory and user-centered design, as a basis for the development of the application. Therefore, in this article we present the user-centered design process that has been followed for the development of SmartMoving, involving users with reduced mobility.
{"title":"Design of SmartMoving, an Application for Pedestrians with Reduced Mobility","authors":"Mónica Fatecha Burian, Patricia Fauvety, Nathalie Aquino, Magalí González, Daniel Romero, L. Cernuzzi, J. Paniagua, Ronald Chenu","doi":"10.1109/CLEI52000.2020.00049","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00049","url":null,"abstract":"The state of the sidewalks in Asunción (Paraguay) is far from being optimal. There are many problems related to obstructions, necessary repairs, lack of ramps, unevenness of surface, among others. In addition, the Municipality of Asunción does not have automated mechanisms to know their updated status. In this work we propose SmartMoving, a mobile application that collects information on the state of the sidewalks, with the help of citizens, and recommends pedestrian paths with less obstacles. The application can be especially useful for people with reduced mobility, as well as for the Municipality of Asunción. This type of application is based on citizen participation, since it receive data from them, and therefore requires a particularly friendly user experience, adapted to users and their daily context. Therefore, the need for a participatory and user-centered design, as a basis for the development of the application. Therefore, in this article we present the user-centered design process that has been followed for the development of SmartMoving, involving users with reduced mobility.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"137 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122557184","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Applications for Interactive Digital Television (IDTV) are becoming increasingly popular. Users no longer interact only with software applications on computers or srnartphones, but also through a television set. Therefore, developers must address efficiency and effectiveness aspects when creating those applications to provide the user with a satisfying interaction experience. Thus, this paper proposes an Application Usability Model for IDTV based on two approaches: the quality of the software product and the quality in the use of the software product. Additionally, the model mainly draws upon the ISO/IEC 25010 standard in which a set of usability characteristics are defined. These characteristics have been divided into subcharacteristics of the IDTV to quantify each of their attributes according to several metrics, and with the obtained values, identify the usability problems that might exist in software applications. For instance, to have a better idea of how the evaluation should be performed, two examples are presented that employ the proposed usability model on applications designed for IDTV from different domains.
{"title":"Evaluating the Usability of Interactive Digital Television Applications","authors":"Priscila Cedillo, Cristina Sánchez-Zhunio, Xavier Garnica-Bautista, Edwin Cabrera, Elizabeth Sari, Xavier Maita-Tepan","doi":"10.1109/CLEI52000.2020.00026","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00026","url":null,"abstract":"Applications for Interactive Digital Television (IDTV) are becoming increasingly popular. Users no longer interact only with software applications on computers or srnartphones, but also through a television set. Therefore, developers must address efficiency and effectiveness aspects when creating those applications to provide the user with a satisfying interaction experience. Thus, this paper proposes an Application Usability Model for IDTV based on two approaches: the quality of the software product and the quality in the use of the software product. Additionally, the model mainly draws upon the ISO/IEC 25010 standard in which a set of usability characteristics are defined. These characteristics have been divided into subcharacteristics of the IDTV to quantify each of their attributes according to several metrics, and with the obtained values, identify the usability problems that might exist in software applications. For instance, to have a better idea of how the evaluation should be performed, two examples are presented that employ the proposed usability model on applications designed for IDTV from different domains.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122897955","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00012
J. Aguilar, Francisco Díaz, J. Altamiranda, E. Montoya
In this work is proposed a parameter adaptive system for emerging serious games, which allows the emerging of properties in a game, in order to adapt it to the educational context in which it is being used. Previous articles have proposed an engine for emerging serious games based on the ant colony optimization algorithm, which allows the initial emergence of a serious game. In this paper, another component of the game adaptive sub-system of the engine is specified, which allows its dynamic adaptation while the game is running, in this case, in order to adapt its properties to the educational context-domain. Particularly, the property emergence system is based on the use of cultural algorithms, which establishes a learning process to modify its parameters. This system is tested in the context of a smart classroom.
{"title":"An Emerging Serious Game Engine with a Parameter Adaptive System based on an evolutionary approach","authors":"J. Aguilar, Francisco Díaz, J. Altamiranda, E. Montoya","doi":"10.1109/CLEI52000.2020.00012","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00012","url":null,"abstract":"In this work is proposed a parameter adaptive system for emerging serious games, which allows the emerging of properties in a game, in order to adapt it to the educational context in which it is being used. Previous articles have proposed an engine for emerging serious games based on the ant colony optimization algorithm, which allows the initial emergence of a serious game. In this paper, another component of the game adaptive sub-system of the engine is specified, which allows its dynamic adaptation while the game is running, in this case, in order to adapt its properties to the educational context-domain. Particularly, the property emergence system is based on the use of cultural algorithms, which establishes a learning process to modify its parameters. This system is tested in the context of a smart classroom.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121417846","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00068
Sivana Hamer, Christian Quesada-López, Alexandra Martínez, Marcelo Jenkins
Many courses in the software engineering area are centered around team-based project development. Evaluating these projects is a challenge due to the difficulty of measuring individual student contributions versus team contributions. The adoption of distributed version control systems like Git enables the measurement of students' and teams' contributions to the project. In this work, we analyze the contributions within five software development projects from undergraduate courses that used project-based learning. For this, we generate visualizations of aggregated Git metrics using inequality indexes and inter-decile ratios, which offer insights into the practices and processes followed by students and teams throughout the project development. This approach allowed us to identify both inequality among students' contributions and development processes with a non-steady pace, rendering a useful feedback tool for instructors and students during the development of the project. Further studies can be conducted to assess the complexity and value of students' contributions by analyzing their source code commits and other software artifacts.
{"title":"Measuring students' contributions in software development projects using Git metrics","authors":"Sivana Hamer, Christian Quesada-López, Alexandra Martínez, Marcelo Jenkins","doi":"10.1109/CLEI52000.2020.00068","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00068","url":null,"abstract":"Many courses in the software engineering area are centered around team-based project development. Evaluating these projects is a challenge due to the difficulty of measuring individual student contributions versus team contributions. The adoption of distributed version control systems like Git enables the measurement of students' and teams' contributions to the project. In this work, we analyze the contributions within five software development projects from undergraduate courses that used project-based learning. For this, we generate visualizations of aggregated Git metrics using inequality indexes and inter-decile ratios, which offer insights into the practices and processes followed by students and teams throughout the project development. This approach allowed us to identify both inequality among students' contributions and development processes with a non-steady pace, rendering a useful feedback tool for instructors and students during the development of the project. Further studies can be conducted to assess the complexity and value of students' contributions by analyzing their source code commits and other software artifacts.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125778532","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00048
A. Lugo, Nathalie Aquino, Magalí González, L. Cernuzzi, Ronald Chenu
A low average number of people per private vehicle and inappropriate road infrastructure results in heavy traffic that wastes space, time and money for the people involved. To optimize these resources, it is intended to promote carpooling between people who share the same destination, for example, colleagues at work or students at a university. This paper presents UCarpooling, a matching system for commuting between people of a same institution. UCarpooling is aimed at optimizing the number of passengers in vehicles during routine trips to and from work or study. The difference with respect to other similar proposals is that UCarpooling takes into account logistical details (place of departure, time of entry, etc.) and personal traits (if you smoke, what genres of music you listen to, etc.) as variables to calculate the percentage compatibility that different people have to carry out a carpool. A simulation of the use of UCarpooling in a university in Asunción, Paraguay, yields favorable data reaching the conclusion that its adoption is quite beneficial for the institution that adopts it, the people who use it, and the cities where it is adopted.
{"title":"Ucarpooling: Decongesting Traffic through Carpooling using Automatic Pairings","authors":"A. Lugo, Nathalie Aquino, Magalí González, L. Cernuzzi, Ronald Chenu","doi":"10.1109/CLEI52000.2020.00048","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00048","url":null,"abstract":"A low average number of people per private vehicle and inappropriate road infrastructure results in heavy traffic that wastes space, time and money for the people involved. To optimize these resources, it is intended to promote carpooling between people who share the same destination, for example, colleagues at work or students at a university. This paper presents UCarpooling, a matching system for commuting between people of a same institution. UCarpooling is aimed at optimizing the number of passengers in vehicles during routine trips to and from work or study. The difference with respect to other similar proposals is that UCarpooling takes into account logistical details (place of departure, time of entry, etc.) and personal traits (if you smoke, what genres of music you listen to, etc.) as variables to calculate the percentage compatibility that different people have to carry out a carpool. A simulation of the use of UCarpooling in a university in Asunción, Paraguay, yields favorable data reaching the conclusion that its adoption is quite beneficial for the institution that adopts it, the people who use it, and the cities where it is adopted.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"91 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130916794","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00015
Romina Valdez, Khevin Roig, Diego Pinto, José Colbes
One of the most important unsolved problems in the area of Computational Biology is the prediction of protein structures. A key element in this problem is the prediction of contacts in a protein from its amino acid sequence, since it provides fundamental information for the determination of its three-dimensional structure. Due to the attention devoted to this subproblem, especially in the last decade, there are a large number of methods in the literature that obtain very good results; but there is still a considerable room for improvement. In the 13th edition of the Critical Assessment of protein Structure Prediction (CASP), a notable progress has been achieved in this area due to the use of deep learning and deep convolutional residual neural networks in state-of-the-art methods; in addition to the use of additional information from other predictions, such as solvent accessibility, conformation of the secondary structure, etc. The present work analyzes the performance of the most outstanding CASP13 methods, considering a larger test set (483 proteins) with proteins of four different classes according to SCOP. The results were evaluated using the CASP metrics. The analysis indicates that most of the selected methods have an accuracy above 90% for the test set used; SPOT-Contact being the best prediction method in general, and at least one of the best in each of the SCOP classes. The test cases and implementations made for the evaluation of results are publicly available.
{"title":"Performance Analysis of Protein Contact Prediction Algorithms","authors":"Romina Valdez, Khevin Roig, Diego Pinto, José Colbes","doi":"10.1109/CLEI52000.2020.00015","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00015","url":null,"abstract":"One of the most important unsolved problems in the area of Computational Biology is the prediction of protein structures. A key element in this problem is the prediction of contacts in a protein from its amino acid sequence, since it provides fundamental information for the determination of its three-dimensional structure. Due to the attention devoted to this subproblem, especially in the last decade, there are a large number of methods in the literature that obtain very good results; but there is still a considerable room for improvement. In the 13th edition of the Critical Assessment of protein Structure Prediction (CASP), a notable progress has been achieved in this area due to the use of deep learning and deep convolutional residual neural networks in state-of-the-art methods; in addition to the use of additional information from other predictions, such as solvent accessibility, conformation of the secondary structure, etc. The present work analyzes the performance of the most outstanding CASP13 methods, considering a larger test set (483 proteins) with proteins of four different classes according to SCOP. The results were evaluated using the CASP metrics. The analysis indicates that most of the selected methods have an accuracy above 90% for the test set used; SPOT-Contact being the best prediction method in general, and at least one of the best in each of the SCOP classes. The test cases and implementations made for the evaluation of results are publicly available.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121255135","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00021
Jorge A. Montes, F. Giraldo
This paper presents the redesign of the academic follow-up process established in the Colombia' Ministry of National Education (MEN) Guide No. 34, by the application of the Business Process Management (BPM) model, to improve its quality indicators within the Academic Management area in a public educational institution in the department of Quindío (Colombia). This case is presented as a feasible project that proposes a solution to the need to restructure the way in which academic processes are interpreted and managed within the country's public educational institutions due to the lack of specification, understanding of current processes and components. that integrate the institutional academic management. The methodology incorporates the process improvement techniques proposed by Dumas, M., et al. The redesign of the academic monitoring process, and the specification of the new key performance indicators or KPIs for each redesigned sub-process, promoted the quantification of the positive impact of the application of the BPM model in the redesign of the academic follow-up process and the improvement of its quality indicators within the field of academic management.
{"title":"Application of a BPM-Based Approach in the Redesign of the Academic Follow-up Process of a Public High School in Colombia","authors":"Jorge A. Montes, F. Giraldo","doi":"10.1109/CLEI52000.2020.00021","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00021","url":null,"abstract":"This paper presents the redesign of the academic follow-up process established in the Colombia' Ministry of National Education (MEN) Guide No. 34, by the application of the Business Process Management (BPM) model, to improve its quality indicators within the Academic Management area in a public educational institution in the department of Quindío (Colombia). This case is presented as a feasible project that proposes a solution to the need to restructure the way in which academic processes are interpreted and managed within the country's public educational institutions due to the lack of specification, understanding of current processes and components. that integrate the institutional academic management. The methodology incorporates the process improvement techniques proposed by Dumas, M., et al. The redesign of the academic monitoring process, and the specification of the new key performance indicators or KPIs for each redesigned sub-process, promoted the quantification of the positive impact of the application of the BPM model in the redesign of the academic follow-up process and the improvement of its quality indicators within the field of academic management.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126802288","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00011
J. Andrade, T. Silva, R. J. F. Junior, J. Maia, G. Campos
This paper describes and evaluates a Mean-Shift-based (MS) approach to an instance of the Cooperative Target Observation (CTO) problem domain. A performance comparison is presented with a k-means-based approach to the baseline implementation published to the CTO problem. Inspired by the idea of modeling the problem for urban centers in which the movement of targets is restricted to the streets and roads, we also evaluate the effect of the movement of the targets being restricted to a rectangular grid on the relative performance of the algorithms. We conclude that the MS-based approach is superior to the k-means-based approach and that the target motion restricted to a grid improves both algorithms' performance but does not change its relative positions.
{"title":"Cooperative Target Observation using Density-based Clustering with Self-tuning and a New Grid Environment","authors":"J. Andrade, T. Silva, R. J. F. Junior, J. Maia, G. Campos","doi":"10.1109/CLEI52000.2020.00011","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00011","url":null,"abstract":"This paper describes and evaluates a Mean-Shift-based (MS) approach to an instance of the Cooperative Target Observation (CTO) problem domain. A performance comparison is presented with a k-means-based approach to the baseline implementation published to the CTO problem. Inspired by the idea of modeling the problem for urban centers in which the movement of targets is restricted to the streets and roads, we also evaluate the effect of the movement of the targets being restricted to a rectangular grid on the relative performance of the algorithms. We conclude that the MS-based approach is superior to the k-means-based approach and that the target motion restricted to a grid improves both algorithms' performance but does not change its relative positions.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131525016","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-01DOI: 10.1109/CLEI52000.2020.00036
Jorge M. Ram'irez, Marcos Báez, F. Casati, L. Cernuzzi, B. Benatallah
This paper reports on the challenges and lessons we learned while running controlled experiments in crowdsourcing platforms. Crowdsourcing is becoming an attractive technique to engage a diverse and large pool of subjects in experimental research, allowing researchers to achieve levels of scale and completion times that would otherwise not be feasible in lab settings. However, the scale and flexibility comes at the cost of multiple and sometimes unknown sources of bias and confounding factors that arise from technical limitations of crowdsourcing platforms and from the challenges of running controlled experiments in the “wild”. In this paper, we take our experience in running systematic evaluations of task design as a motivating example to explore, describe, and quantify the potential impact of running uncontrolled crowdsourcing experiments and derive possible coping strategies. Among the challenges identified, we can mention sampling bias, controlling the assignment of subjects to experimental conditions, learning effects, and reliability of crowdsourcing results. According to our empirical studies, the impact of potential biases and confounding factors can amount to a 38% loss in the utility of the data collected in uncontrolled settings; and it can significantly change the outcome of experiments. These issues ultimately inspired us to implement CrowdHub, a system that sits on top of major crowdsourcing platforms and allows researchers and practitioners to run controlled crowdsourcing projects.
{"title":"Challenges and strategies for running controlled crowdsourcing experiments","authors":"Jorge M. Ram'irez, Marcos Báez, F. Casati, L. Cernuzzi, B. Benatallah","doi":"10.1109/CLEI52000.2020.00036","DOIUrl":"https://doi.org/10.1109/CLEI52000.2020.00036","url":null,"abstract":"This paper reports on the challenges and lessons we learned while running controlled experiments in crowdsourcing platforms. Crowdsourcing is becoming an attractive technique to engage a diverse and large pool of subjects in experimental research, allowing researchers to achieve levels of scale and completion times that would otherwise not be feasible in lab settings. However, the scale and flexibility comes at the cost of multiple and sometimes unknown sources of bias and confounding factors that arise from technical limitations of crowdsourcing platforms and from the challenges of running controlled experiments in the “wild”. In this paper, we take our experience in running systematic evaluations of task design as a motivating example to explore, describe, and quantify the potential impact of running uncontrolled crowdsourcing experiments and derive possible coping strategies. Among the challenges identified, we can mention sampling bias, controlling the assignment of subjects to experimental conditions, learning effects, and reliability of crowdsourcing results. According to our empirical studies, the impact of potential biases and confounding factors can amount to a 38% loss in the utility of the data collected in uncontrolled settings; and it can significantly change the outcome of experiments. These issues ultimately inspired us to implement CrowdHub, a system that sits on top of major crowdsourcing platforms and allows researchers and practitioners to run controlled crowdsourcing projects.","PeriodicalId":413655,"journal":{"name":"2020 XLVI Latin American Computing Conference (CLEI)","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132850244","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：