Pub Date : 2020-01-02DOI: 10.1080/03057267.2020.1735757
X. Zhai, Y. Yin, J. Pellegrino, Kevin C. Haudek, Lehong Shi
ABSTRACT Machine learning (ML) is an emergent computerised technology that relies on algorithms built by ‘learning’ from training data rather than ‘instruction’, which holds great potential to revolutionise science assessment. This study systematically reviewed 49 articles regarding ML-based science assessment through a triangle framework with technical, validity, and pedagogical features on three vertices. We found that a majority of the studies focused on the validity vertex, as compared to the other two vertices. The existing studies primarily involve text recognition, classification, and scoring with an emphasis on constructing scientific explanations, with a vast range of human-machine agreement measures. To achieve the agreement measures, most of the studies employed a cross-validation method, rather than self- or split-validation. ML allows complex assessments to be used by teachers without the burden of human scoring, saving both time and cost. Most studies used supervised ML, which relies on extraction of attributes from student work that was first coded by humans to achieve automaticity, rather than semi- or unsupervised ML. We found that 24 studies were explicitly embedded in science learning activities, such as scientific inquiry and argumentation, to provide feedback or learning guidance. This study identifies existing research gaps and suggests that all three vertices of the ML triangle should be addressed in future assessment studies, with an emphasis on the pedagogy and technology features.
{"title":"Applying machine learning in science assessment: a systematic review","authors":"X. Zhai, Y. Yin, J. Pellegrino, Kevin C. Haudek, Lehong Shi","doi":"10.1080/03057267.2020.1735757","DOIUrl":"https://doi.org/10.1080/03057267.2020.1735757","url":null,"abstract":"ABSTRACT Machine learning (ML) is an emergent computerised technology that relies on algorithms built by ‘learning’ from training data rather than ‘instruction’, which holds great potential to revolutionise science assessment. This study systematically reviewed 49 articles regarding ML-based science assessment through a triangle framework with technical, validity, and pedagogical features on three vertices. We found that a majority of the studies focused on the validity vertex, as compared to the other two vertices. The existing studies primarily involve text recognition, classification, and scoring with an emphasis on constructing scientific explanations, with a vast range of human-machine agreement measures. To achieve the agreement measures, most of the studies employed a cross-validation method, rather than self- or split-validation. ML allows complex assessments to be used by teachers without the burden of human scoring, saving both time and cost. Most studies used supervised ML, which relies on extraction of attributes from student work that was first coded by humans to achieve automaticity, rather than semi- or unsupervised ML. We found that 24 studies were explicitly embedded in science learning activities, such as scientific inquiry and argumentation, to provide feedback or learning guidance. This study identifies existing research gaps and suggests that all three vertices of the ML triangle should be addressed in future assessment studies, with an emphasis on the pedagogy and technology features.","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2020-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2020.1735757","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43534576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-01-02DOI: 10.1080/03057267.2020.1735758
D. Holzberger, Sarah Reinhold, O. Lüdtke, T. Seidel
ABSTRACT In the present meta-analysis, we examine how secondary school characteristics – such as schools’ academic press, school climate, material resources, personnel resources, classroom climate, instructional practices, out-of-school activities, and socioeconomic status (SES) composition – provide opportunities for students to engage in science and maths, and how these matter with regard to students’ cognitive and motivational-affective outcomes. The meta-analysis includes 71 (international) articles from large-scale studies with a total of 3,960,281 students, 260,390 schools, and 285 effect sizes that were transformed to correlation coefficients. Multilevel meta-analyses were performed. Results identified a number of school variables that can be regarded as relevant for making a difference in student outcomes and at the same time be influenced by education. These refer to school variables such schools’ academic press, classroom climate, instructional practices, and out-of-school activities. Moreover, SES composition was significantly related to student outcomes. Material and personnel resources as well as school climate yielded a close to zero effect. No differences were found between cognitive and motivational-affective outcome variables or between science and maths. The results point to the most promising school characteristics for promoting students’ outcomes and emphasise schools’ potential for students’ engagement in science and maths.
{"title":"A meta-analysis on the relationship between school characteristics and student outcomes in science and maths – evidence from large-scale studies","authors":"D. Holzberger, Sarah Reinhold, O. Lüdtke, T. Seidel","doi":"10.1080/03057267.2020.1735758","DOIUrl":"https://doi.org/10.1080/03057267.2020.1735758","url":null,"abstract":"ABSTRACT In the present meta-analysis, we examine how secondary school characteristics – such as schools’ academic press, school climate, material resources, personnel resources, classroom climate, instructional practices, out-of-school activities, and socioeconomic status (SES) composition – provide opportunities for students to engage in science and maths, and how these matter with regard to students’ cognitive and motivational-affective outcomes. The meta-analysis includes 71 (international) articles from large-scale studies with a total of 3,960,281 students, 260,390 schools, and 285 effect sizes that were transformed to correlation coefficients. Multilevel meta-analyses were performed. Results identified a number of school variables that can be regarded as relevant for making a difference in student outcomes and at the same time be influenced by education. These refer to school variables such schools’ academic press, classroom climate, instructional practices, and out-of-school activities. Moreover, SES composition was significantly related to student outcomes. Material and personnel resources as well as school climate yielded a close to zero effect. No differences were found between cognitive and motivational-affective outcome variables or between science and maths. The results point to the most promising school characteristics for promoting students’ outcomes and emphasise schools’ potential for students’ engagement in science and maths.","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2020-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2020.1735758","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43093030","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-01-02DOI: 10.1080/03057267.2020.1733183
Nigel C. Skinner
{"title":"Teaching biology in schools: global research, issues and trends","authors":"Nigel C. Skinner","doi":"10.1080/03057267.2020.1733183","DOIUrl":"https://doi.org/10.1080/03057267.2020.1733183","url":null,"abstract":"","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2020-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2020.1733183","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43099147","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
ABSTRACT In this study, we reviewed 76 journal articles on employing drawing assessment as a research tool in science education. Findings from the systematic review suggest four justifications for using drawing as a type of research tool, including assessment via drawing as (a) an alternative method considering young participants’ verbal or writing abilities, and affective or economic reasons, (b) a unique method that can reveal aspects not easily measured by other methods, (c) a major method that reflects characteristics of science subjects, and (d) a formative assessment to diagnose students’ ideas to benefit their learning. Furthermore, five research trends of studies using drawing as assessment tools are identified, including: (a) students’ conceptions of scientists from the Draw-a-Scientist-Test (DAST) and evolving studies, (b) students’ understanding or mental models of science concepts, (c) participants’ conceptions of science learning or teaching, (d) students’ inquiry abilities and modelling skills via drawing, and (e) technology to support drawing. For each trend, we synthesised and commented on the current findings. A framework conceptualising phases and issues when designing research and instruments employing drawing assessments is proposed. The review provides insights into the design and future direction of research employing drawing assessments in science education.
{"title":"A systematic review of trends and findings in research employing drawing assessment in science education","authors":"Hsin‐Yi Chang, Tzung-Jin Lin, Min‐Hsien Lee, Silvia Wen‐Yu Lee, Tzu-Chiang Lin, Aik-Ling Tan, Chin-Chung Tsai","doi":"10.1080/03057267.2020.1735822","DOIUrl":"https://doi.org/10.1080/03057267.2020.1735822","url":null,"abstract":"ABSTRACT In this study, we reviewed 76 journal articles on employing drawing assessment as a research tool in science education. Findings from the systematic review suggest four justifications for using drawing as a type of research tool, including assessment via drawing as (a) an alternative method considering young participants’ verbal or writing abilities, and affective or economic reasons, (b) a unique method that can reveal aspects not easily measured by other methods, (c) a major method that reflects characteristics of science subjects, and (d) a formative assessment to diagnose students’ ideas to benefit their learning. Furthermore, five research trends of studies using drawing as assessment tools are identified, including: (a) students’ conceptions of scientists from the Draw-a-Scientist-Test (DAST) and evolving studies, (b) students’ understanding or mental models of science concepts, (c) participants’ conceptions of science learning or teaching, (d) students’ inquiry abilities and modelling skills via drawing, and (e) technology to support drawing. For each trend, we synthesised and commented on the current findings. A framework conceptualising phases and issues when designing research and instruments employing drawing assessments is proposed. The review provides insights into the design and future direction of research employing drawing assessments in science education.","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2020-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2020.1735822","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42803425","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-12-11DOI: 10.1080/03057267.2019.1694792
Juan Quílez
This study consists of a categorisation of the vocabulary of chemistry, focusing on the terminological difficulties student face when learning this subject. Therefore, this classification intends t...
本研究包括化学词汇的分类,重点是学生在学习这门学科时面临的术语困难。因此,这种分类意在…
{"title":"A categorisation of the terminological sources of student difficulties when learning chemistry","authors":"Juan Quílez","doi":"10.1080/03057267.2019.1694792","DOIUrl":"https://doi.org/10.1080/03057267.2019.1694792","url":null,"abstract":"This study consists of a categorisation of the vocabulary of chemistry, focusing on the terminological difficulties student face when learning this subject. Therefore, this classification intends t...","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2019-12-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2019.1694792","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41798868","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-07-03DOI: 10.1080/03057267.2020.1735756
Guopeng Fu, A. Clarke
Agency and structure postulate a dialectic relationship: agents’ actions shape and are shaped by social structure in a spiral and dynamic manner. Empirical studies in pre-collegiate science educati...
{"title":"Moving beyond the agency-structure dialectic in pre-collegiate science education: positionality, engagement, and emergence","authors":"Guopeng Fu, A. Clarke","doi":"10.1080/03057267.2020.1735756","DOIUrl":"https://doi.org/10.1080/03057267.2020.1735756","url":null,"abstract":"Agency and structure postulate a dialectic relationship: agents’ actions shape and are shaped by social structure in a spiral and dynamic manner. Empirical studies in pre-collegiate science educati...","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2019-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2020.1735756","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46471812","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-07-03DOI: 10.1080/03057267.2019.1722420
Martin Riopel, Lucian Nenciovici, Patrice Potvin, Pierre Chastenay, Patrick Charland, J. Sarrasin, Steve Masson
Serious games have become increasingly available to educators. Empirical studies and meta-analyses have examined their impact on learning achievement. However, natural sciences could have a special...
{"title":"Impact of serious games on science learning achievement compared with more conventional instruction: an overview and a meta-analysis","authors":"Martin Riopel, Lucian Nenciovici, Patrice Potvin, Pierre Chastenay, Patrick Charland, J. Sarrasin, Steve Masson","doi":"10.1080/03057267.2019.1722420","DOIUrl":"https://doi.org/10.1080/03057267.2019.1722420","url":null,"abstract":"Serious games have become increasingly available to educators. Empirical studies and meta-analyses have examined their impact on learning achievement. However, natural sciences could have a special...","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2019-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2019.1722420","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42520556","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-01-02DOI: 10.1080/03057267.2019.1645533
Lilith Rüschenpöhler, S. Markic
ABSTRACT This article gives an overview of the current state of science and technology self-concept research. Following a defined selection process, we analysed 74 peer-reviewed journal articles published from 1998 to 2017, which are indexed in the ERIC database and that deal with science and technology self-concepts (STSC) of school children and adolescents. In our analysis, we focus on the theoretical foundations, measurement instruments, and main findings from this area. (i) Theoretical foundations: today’s research on STSC is mainly based on the Shavelson and Marsh models of self-concept, i.e. it follows the tradition of educational psychology. (ii) Measurement instruments: a number of established and validated measurement instruments are available. However, the existing methodological resources should be employed more rigorously. (iii) Main findings: Some findings are well documented, such as the positive relation with achievement, the gender gap, and the fact that students of non-dominant ethnic groups tend to have lower STSCs. Recommendations: in order to gain a deeper understanding of these phenomena, it could be fruitful to further elaborate connections with science identity research and to enrich STSC research with qualitative data.
{"title":"Self-concept research in science and technology education – theoretical foundation, measurement instruments, and main findings","authors":"Lilith Rüschenpöhler, S. Markic","doi":"10.1080/03057267.2019.1645533","DOIUrl":"https://doi.org/10.1080/03057267.2019.1645533","url":null,"abstract":"ABSTRACT This article gives an overview of the current state of science and technology self-concept research. Following a defined selection process, we analysed 74 peer-reviewed journal articles published from 1998 to 2017, which are indexed in the ERIC database and that deal with science and technology self-concepts (STSC) of school children and adolescents. In our analysis, we focus on the theoretical foundations, measurement instruments, and main findings from this area. (i) Theoretical foundations: today’s research on STSC is mainly based on the Shavelson and Marsh models of self-concept, i.e. it follows the tradition of educational psychology. (ii) Measurement instruments: a number of established and validated measurement instruments are available. However, the existing methodological resources should be employed more rigorously. (iii) Main findings: Some findings are well documented, such as the positive relation with achievement, the gender gap, and the fact that students of non-dominant ethnic groups tend to have lower STSCs. Recommendations: in order to gain a deeper understanding of these phenomena, it could be fruitful to further elaborate connections with science identity research and to enrich STSC research with qualitative data.","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2019-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2019.1645533","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47459551","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-01-02DOI: 10.1080/03057267.2019.1658058
K. Taber
ABSTRACT Experimental studies are often employed to test the effectiveness of teaching innovations such as new pedagogy, curriculum, or learning resources. This article offers guidance on good practice in developing research designs, and in drawing conclusions from published reports. Random control trials potentially support the use of statistical inference, but face a number of potential threats to validity. Research in educational contexts often employs quasi-experiments or natural experiments rather than true experiments, and these types of designs raise additional questions about the equivalence between experimental and control groups and the potential influence of confounding variables. Where it is impractical for experimental studies to employ samples that fully reflect diverse populations, generalisation is limited. Series of small-scale replication studies may be useful here, especially if these are conceptualised as being akin to multiple case studies, and complemented by qualitative studies. Control conditions for experimental studies need to be carefully selected to provide the most appropriate test for a particular intervention, and considering the interests of all participants. Control groups in studies that replicate innovations that have been widely shown to be effective in other settings should experience teaching conditions that reflect good practice and meet expected teaching standards in the research context.
{"title":"Experimental research into teaching innovations: responding to methodological and ethical challenges","authors":"K. Taber","doi":"10.1080/03057267.2019.1658058","DOIUrl":"https://doi.org/10.1080/03057267.2019.1658058","url":null,"abstract":"ABSTRACT Experimental studies are often employed to test the effectiveness of teaching innovations such as new pedagogy, curriculum, or learning resources. This article offers guidance on good practice in developing research designs, and in drawing conclusions from published reports. Random control trials potentially support the use of statistical inference, but face a number of potential threats to validity. Research in educational contexts often employs quasi-experiments or natural experiments rather than true experiments, and these types of designs raise additional questions about the equivalence between experimental and control groups and the potential influence of confounding variables. Where it is impractical for experimental studies to employ samples that fully reflect diverse populations, generalisation is limited. Series of small-scale replication studies may be useful here, especially if these are conceptualised as being akin to multiple case studies, and complemented by qualitative studies. Control conditions for experimental studies need to be carefully selected to provide the most appropriate test for a particular intervention, and considering the interests of all participants. Control groups in studies that replicate innovations that have been widely shown to be effective in other settings should experience teaching conditions that reflect good practice and meet expected teaching standards in the research context.","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2019-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2019.1658058","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46487984","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2019-01-02DOI: 10.1080/03057267.2019.1658059
Jaana Herranen, M. Aksela
ABSTRACT Students’ questions have an important function in science learning, and in inquiry-based approaches. Inquiry teaching in which the students’ own questions are used is promising, but a holistic view of the research and practice is lacking. A systematic review was conducted on 30 articles, both research report articles as well as descriptive and evaluative report articles on the use of students’ questions as a starting point for inquiry-based science education. The review was carried out using deductive content analysis. This review concentrates especially on the formulation and utilisation of students’ questions in inquiry, the benefits of the reported approaches, and the roles that the teacher and the students take during student-question-based inquiry approaches. This review shows that there are various ways to obtain and use students’ questions. The results also highlight the teacher’s important role in inquiry teaching already in the planning phase. Moreover, the teacher’s role is affected by how much emphasise is placed on science learning, and how much value is put on the students’ questions. Finally, a model for student-question-based inquiry (the SQBI-model) for science education is presented. This model should be acknowledged also in the teacher education.
{"title":"Student-question-based inquiry in science education","authors":"Jaana Herranen, M. Aksela","doi":"10.1080/03057267.2019.1658059","DOIUrl":"https://doi.org/10.1080/03057267.2019.1658059","url":null,"abstract":"ABSTRACT Students’ questions have an important function in science learning, and in inquiry-based approaches. Inquiry teaching in which the students’ own questions are used is promising, but a holistic view of the research and practice is lacking. A systematic review was conducted on 30 articles, both research report articles as well as descriptive and evaluative report articles on the use of students’ questions as a starting point for inquiry-based science education. The review was carried out using deductive content analysis. This review concentrates especially on the formulation and utilisation of students’ questions in inquiry, the benefits of the reported approaches, and the roles that the teacher and the students take during student-question-based inquiry approaches. This review shows that there are various ways to obtain and use students’ questions. The results also highlight the teacher’s important role in inquiry teaching already in the planning phase. Moreover, the teacher’s role is affected by how much emphasise is placed on science learning, and how much value is put on the students’ questions. Finally, a model for student-question-based inquiry (the SQBI-model) for science education is presented. This model should be acknowledged also in the teacher education.","PeriodicalId":49262,"journal":{"name":"Studies in Science Education","volume":null,"pages":null},"PeriodicalIF":4.9,"publicationDate":"2019-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/03057267.2019.1658059","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49338369","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}