Educational outcomes from traditionally underrepresented groups are generally worse than for their more advantaged peers. This problem is typically known as the awarding gap (we use the term awarding gap over ‘attainment gap’ as attainment places the responsibility on students to attain at equal levels) and continues to pose a challenge for educational systems across the world. While Learning Analytics (LA) dashboards help identify patterns contributing to the awarding gap, they often lack stakeholder involvement, offering very little support to institutional Equality, Diversity and Inclusion (EDI) leads or educators to pinpoint and address these gaps. This paper introduces an innovative EDI LA dashboard, co-created with diverse stakeholders. Rigorously evaluated, the dashboard provides fine-grained insights and course-level analysis, empowering institutions to effectively address awarding gaps and contribute to a diverse and inclusive higher education landscape.
Practitioners notes
What is already known about this topic
- Traditionally underrepresented groups face educational disparities, commonly known as the awarding gap.
- Underachievement is a complex multi-dimensional problem and cannot be solely attributable to individual student deficiencies.
- LA dashboards targeting this specific problem are often not public, there is little research about them, and are frequently designed with little involvement of educational stakeholders.
What this paper adds
- Pioneers the introduction of a dashboard specifically designed to address the awarding gap problem.
- Emphasises the significant data needs of educational stakeholders in tackling awarding gaps.
- Expands the design dimensions of Learning Analytics (LA) by introducing a specific design approach rooted in established user experience (UX) design methods.
Implications for practice and/or policy
- Insights from this study will guide practitioners, designers, and developers in creating AI-based educational systems to effectively target the awarding gap problem.
Large language models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM-based applications to understand and generate natural language can potentially improve instructional effectiveness and learning outcomes, but the integration of LLMs in education technology has renewed concerns over algorithmic bias, which may exacerbate educational inequalities. Building on prior work that mapped the traditional machine learning life cycle, we provide a framework of the LLM life cycle from the initial development of LLMs to customizing pre-trained models for various applications in educational settings. We explain each step in the LLM life cycle and identify potential sources of bias that may arise in the context of education. We discuss why current measures of bias from traditional machine learning fail to transfer to LLM-generated text (eg, tutoring conversations) because text encodings are high-dimensional, there can be multiple correct responses, and tailoring responses may be pedagogically desirable rather than unfair. The proposed framework clarifies the complex nature of bias in LLM applications and provides practical guidance for their evaluation to promote educational equity.
Practitioner notes
What is already known about this topic
- The life cycle of traditional machine learning (ML) applications which focus on predicting labels is well understood.
- Biases are known to enter in traditional ML applications at various points in the life cycle, and methods to measure and mitigate these biases have been developed and tested.
- Large language models (LLMs) and other forms of generative artificial intelligence (GenAI) are increasingly adopted in education technologies (EdTech), but current evaluation approaches are not specific to the domain of education.
What this paper adds
- A holistic perspective of the LLM life cycle with domain-specific examples in education to highlight opportunities and challenges for incorporating natural language understanding (NLU) and natural language generation (NLG) into EdTech.
- Potential sources of bias are identified in each step of the LLM life cycle and discussed in the context of education.
- A framework for understanding where to expect potential harms of LLMs for
A key goal of educational institutions around the world is to provide inclusive, equitable quality education and lifelong learning opportunities for all learners. Achieving this requires contextualized approaches to accommodate diverse global values and promote learning opportunities that best meet the needs and goals of all learners as individuals and members of different communities. Advances in learning analytics (LA), natural language processes (NLP), and artificial intelligence (AI), especially generative AI technologies, offer potential to aid educational decision making by supporting analytic insights and personalized recommendations. However, these technologies also raise serious risks for reinforcing or exacerbating existing inequalities; these dangers arise from multiple factors including biases represented in training datasets, the technologies' abilities to take autonomous decisions, and processes for tool development that do not centre the needs and concerns of historically marginalized groups. To ensure that Educational Decision Support Systems (EDSS), particularly AI-powered ones, are equipped to promote equity, they must be created and evaluated holistically, considering their potential for both targeted and systemic impacts on all learners, especially members of historically marginalized groups. Adopting a socio-technical and cultural perspective is crucial for designing, deploying, and evaluating AI-EDSS that truly advance educational equity and inclusion. This editorial introduces the contributions of five papers for the special section on advancing equity and inclusion in educational practices with AI-EDSS. These papers focus on (i) a review of biases in large language models (LLMs) applications offers practical guidelines for their evaluation to promote educational equity, (ii) techniques to mitigate disparities across countries and languages in LLMs representation of educationally relevant knowledge, (iii) implementing equitable and intersectionality-aware machine learning applications in education, (iv) introducing a LA dashboard that aims to promote institutional equality, diversity, and inclusion, and (v) vulnerable student digital well-being in AI-EDSS. Together, these contributions underscore the importance of an interdisciplinary approach in developing and utilizing AI-EDSS to not only foster a more inclusive and equitable educational landscape worldwide but also reveal a critical need for a broader contextualization of equity that incorporates the socio-technical questions of what kinds of decisions AI is being used to support, for what purposes, and whose goals are prioritized in this process.
Despite the growing interest in using multimodal data to analyse students' actions in Computers-Supported Collaborative Learning (CSCL) settings, studying teacher's orchestration load in such settings remains overlooked. The notion of classroom orchestration, and orchestration load, offer a lens to study the implications of increasingly complex technology-supported learning environments on teacher performance. A combination of multimodal data may aid in understanding teachers' orchestration actions and, as a result, gain insights regarding the orchestration load teachers perceive in scripted CSCL situations. Studying teacher orchestration load in CSCL helps understand the workload teachers experience while facilitating student collaboration and assists in informing design decisions for teacher supporting tools. In this paper, we collect and analyse data from different modalities (i.e. electrodermal activity, observation notes, log data, dashboard screen recordings and responses to self-reported questionnaires) to study teachers' orchestration load in scripted CSCL. A tool called PyramidApp was used to deploy CSCL activities and a teacher-facing dashboard was used to facilitate teachers in managing collaboration in real time. The findings of the study show the potential of multimodal data analysis in investigating and estimating the orchestration load experienced by teachers in scripted CSCL activities. Study findings further demonstrate factors emerging from multimodal data such as task type, activity duration, and number of students influenced teachers' orchestration load.
Multimodal learning analytics (MMLA) offers the potential to provide evidence-based insights into complex learning phenomena such as collaborative learning. Yet, few MMLA applications have closed the learning analytics loop by being evaluated in real-world educational settings. This study evaluates the effectiveness of an MMLA solution in enhancing feedback and reflection within a complex and highly dynamic collaborative learning environment. A two-year longitudinal study was conducted with 399 students and 17 teachers, utilising an MMLA system in reflective debriefings in the context of healthcare education. We analysed the survey data of 74 students and 11 teachers regarding their perceptions of the MMLA system. We applied the Evaluation Framework for Learning Analytics, augmented by complexity, accuracy and trust measures, to assess both teachers' and students' perspectives. The findings illustrated that teachers and students both had generally positive perceptions of the MMLA solution. Teachers found the MMLA solution helpful in facilitating feedback provision and reflection during debriefing sessions. Similarly, students found the MMLA solution effective in providing clarity on the data collected, stimulating reflection on their learning behaviours, and prompting considerations for adaptation in their learning behaviours. However, the complexity of the MMLA solution and the need for qualitative measures of communication emerged as areas for improvement. Additionally, the study highlighted the importance of data accuracy, transparency, and privacy protection to maintain user trust. The findings provide valuable contributions to advancing our understanding of the use of MMLA in supporting feedback and reflection practices in intricate collaborative learning while identifying avenues for further research and improvement. We also provided several insights and practical recommendations for successful MMLA implementation in authentic learning contexts.
Practitioner notes
What is currently known about this topic
- Multimodal learning analytics (MMLA) seeks to generate data-informed insights about learners' metacognitive and emotional states as well as their learning behaviours, by utilising intricate physical and physiological signals.
- MMLA has not only pioneered novel data analytic methods but also aspired to complete the learning analytics loop by crafting innovative, tangible solutions that relay these insights to the concerned stakeholders.
- A prominent directi
扫码关注我们
求助内容：
应助结果提醒方式：
Book学术文献互助群
群 号：481959085
京公网安备 11010802042870号
京ICP备2023020795号-1