On the Use of Large Language Models for Improving Student and Staff Experience in Higher Education

2.1 Scaffolding and Sociocultural Constructivism

In sociocultural constructivism, learning is understood as a process where individuals acquire new knowledge and skills through guided interactions, both with more knowledgeable others and with tools that mediate learning (Vygotsky, 1978; Palincsar, 1998). Key to this paradigm is scaffolding, where learners receive just enough guidance or support to tackle tasks slightly beyond their current ability level (Wood, Bruner, & Ross, 1976). For instance, technology as a support within the universal design for learning framework can function as temporary scaffolds, gradually removed as learners gain proficiency, or as permanent components that transform traditional literacy and learning practices (Vasinda & Pilgrim, 2023). In formal educational settings, scaffolding often comes from instructors or peers; however, emerging research suggests that AI-driven tools – such as LLMs – can also provide adaptive forms of assistance (Chien, Chan, & Hou, 2024; Liao et al., 2024).

In our study, LLMs served as a near-instant resource that learners could consult to bridge conceptual gaps, verify partial knowledge, and extend their problem-solving capabilities. This configuration positions the LLM as a “digital scaffold,” offering timely hints, clarifications, and step-by-step explanations that reduce cognitive load and help students progress more confidently towards task completion.

2.2 Critical Digital Literacy

While sociocultural constructivism captures how learners build knowledge with supportive tools, it does not fully address the evaluative dimension that arises when dealing with AI-generated content. For that, we draw on critical digital literacy, which emphasises the ability to question, interpret, and verify digital texts, media, and platforms (Hinrichsen & Coombs, 2013). In an age where LLMs can produce both insightful responses and so-called “hallucinations,” students must practice recognising fact from misinformation (Ciampa, Wolfe, & Bronstein, 2023; Naamati-Schneider & Alt, 2024). The growth of synthetic information created by generative AI further highlights the importance of integrating critical AI literacy into educational practice, equipping learners to understand and navigate the increasingly plastic nature of digital information (Roe, Furze, & Perkins, 2025). Developing these competencies ensures that students become not only effective users of AI tools but also critical consumers of AI-generated content.

Our instructional approach actively encouraged learners to scrutinise AI outputs by cross-referencing suggested information, carefully checking code solutions for errors, and reflecting on the validity of advice gleaned from the LLM. In doing so, students were guided to become critical consumers of digital content. Rather than taking the LLM’s outputs at face value, they were taught to pause, verify, and adapt or discard suggestions as needed – an essential skill in a digital environment rich with both credible and spurious information.

2.3 Synthesis

By combining these two lenses – scaffolding through AI and critical digital literacy – the conceptual framework addresses both the supportive and the evaluative dimensions of LLM usage. The scaffolding perspective explains how students can grow academically when provided with just-in-time guidance that extends their ability to solve problems independently. The critical digital literacy perspective, meanwhile, underscores the importance of approaching AI outputs with healthy scepticism.

3.1 Participant Details

The study was conducted over a period of two academic years with a cohort of first-year Computer Science students at the University of Derby. The students, primarily aged between 18 and 20, represent a diverse sample in terms of academic level and familiarity with digital tools. Data were collected from 78 students who voluntarily consented to participate, in accordance with ethical guidelines approved by the College Ethics Committee.

3.2 Time Frame

The integration of LLMs into the curriculum began in July 2023 and the data collection period spanned the Fall 2023 and 2024 semesters.

3.3 Data Collection

Quantitative data were collected using a structured survey instrument comprising five key questions. These questions were designed to assess student frequency of AI tool usage, openness to increased AI integration in teaching, opinions on banning AI in assessments, perceived relevance of AI skills for future careers, and overall satisfaction with AI support. Responses were measured on a five-point Likert scale. While the brevity of the survey instrument allowed for rapid data collection, it limited the scope of quantitative insights.

In parallel, educators maintained observations during lectures and tutorials. These observations provided qualitative context to supplement the survey responses, focusing on how LLMs were employed in tasks such as quiz creation, lecture adaptation, and student support.

3.4 Data Analysis

The survey responses were analysed by descriptive statistics and qualitative data from observations and educator reflections were analysed thematically. The combined approach ensured that the data from the survey were contextualised by qualitative insights.

3.5 Ethical Considerations

The survey was conducted anonymously, and all participants were informed that no personal data would be collected but that responses would be used for the purposes of research. Based on this information, participants were asked to consent to take part in the survey. Ethics approval was also granted by the College Ethics Committee at the University of Derby.

3.6 Limitations

4.1 Rapid Quiz Creation

LLMs played a pivotal role in the rapid creation of multiple-choice quizzes for both formative and summative assessments. Educators then used these models to generate large pools of potential questions and answers, tailored to specific topics and learning objectives. By providing clear prompts and specifying the desired level of difficulty, educators were able to produce questions that catered for a variety of skill levels, from foundational knowledge checks to more advanced critical-thinking tasks.

After the initial generation, the quizzes were reviewed and refined by educators to ensure accuracy, relevance, and alignment with the course’s learning outcomes. This iterative process allowed for the curation of high-quality questions that not only assessed students’ knowledge but also reinforced key concepts.

To streamline integration into the virtual learning environment (VLE), the LLM was specifically instructed to output questions in a tab-delimited format. This formatting enabled educators to seamlessly import the generated quizzes into the VLE, significantly reducing the time and effort required for manual input.

This approach not only saved time in the creation and implementation of assessments but also allowed educators to focus on refining quiz quality and ensuring they provided meaningful feedback to students. The use of LLMs in this context demonstrated their potential to enhance the efficiency of assessment design and implementation, while maintaining pedagogical rigor.

4.2 Dynamic Lecture Adaptation

In one instance, the educator’s reflections on potential issues with the initially proposed lecture prompted a pivot. In collaboration with an LLM, a new lecture was efficiently created, allowing for a more responsive and adaptive teaching approach. Students were made aware of the co-creation of the lecture and shown how the lecture content had been created for transparency. Through centralised feedback mechanisms, students praised the approach and noted their appreciation for the decision.

4.3 Rubric Generation

Developing clear and consistent rubrics can be a time-intensive task, requiring alignment with learning objectives, precision, and fairness. By engaging with LLMs such as ChatGPT, educators were able to rapidly generate initial rubric drafts, which were then refined through iterative feedback and revisions. Tutors provided key learning outcomes and assessment goals, prompting the LLM to generate performance criteria and descriptors for various achievement levels. While the initial drafts required adjustments for clarity, alignment, and appropriateness, they served as an effective starting point.

This collaborative approach significantly reduced the time spent on drafting rubrics while maintaining high-quality, tailored results. The tutors ensured the final rubrics were comprehensive, transparent, and student-friendly, helping the students to understand grading criteria and expectations.

Figure 1 demonstrates an example of a draft rubric created through this iterative process. Through the use LLMs educators can effectively complement human expertise and help reduce workload while maintaining rigour and student focus on assessment design.

Figure 1

A draft rubric generated iteratively with ChatGPT 3.5.

4.4 Coding Exercises and Explanations

In the introductory programming module, LLMs were used to generate diverse coding exercises, complete with commented solutions and step-by-step explanations. These exercises ranged from basic syntax tasks to more complex algorithmic problems, enabling educators to create a wide array of practice materials efficiently.

By reviewing and refining the model’s outputs, instructors ensured the exercises aligned with module objectives and maintained clarity and accuracy. This collaborative approach provided students with varied opportunities for active learning, helping them engage with programming concepts and develop problem-solving skills. The detailed solutions and explanations supported independent learning and reduced reliance on instructor feedback for routine queries.

The integration of LLMs streamlined the creation of high-quality teaching materials, enriching the learning experience and freeing up educator time for more personalised support and complex topics.

4.5 Summary

These applications demonstrate how LLMs function as digital scaffolds. By quickly producing rough drafts of rubrics or coding exercises, the AI reduces time-intensive tasks and allows educators to focus on mentoring students and customising learning materials. In turn, educators exercise critical digital literacy by reviewing, modifying, and validating the generated content to ensure alignment with specific learning outcomes. This dynamic interplay highlights how LLMs can streamline content creation without compromising academic rigor, so long as educators remain actively involved in refining the AI’s outputs.

5.1 Guided Learning and Fact-Checking

During tutorials, the students were introduced to the concept of LLMs as tools to guide their learning and exploration of specific topic areas. This approach was intended to demonstrate how LLMs can complement traditional educational resources, offering a dynamic and interactive means of engaging with academic content. To maximise the effectiveness of these interactions, educators conducted demonstrations to show students how to formulate clear and concise queries, and to interpret the responses generated by the models.

A key focus of these tutorials was to raise awareness of the limitations of LLMs, particularly the potential for generating hallucinated or inaccurate information. Students were provided with strategies to identify and mitigate these inaccuracies, such as cross-referencing the information obtained from LLMs with reliable sources, consulting relevant academic literature, and discussing findings with peers or instructors. This emphasis on critical evaluation not only improved the reliability of the information students utilised but also fostered the development of essential research and critical thinking skills.

In addition, the sessions highlighted the importance of understanding the probabilistic nature of LLMs, emphasising that the responses generated are not always definitive or contextually perfect. Students were encouraged to view LLM outputs as a starting point for further investigation rather than as definitive answers. This approach encouraged students to take ownership of their learning by cultivating a more nuanced and sceptical perspective toward AI-generated content.

Through these guided learning sessions, students gained practical experience in interacting with advanced technologies, developing skills that are increasingly relevant in both academia and the job market. By integrating LLMs into the learning process, the tutorials aimed to enhance students’ ability to navigate complex information landscapes while maintaining a critical, evidence-based approach to knowledge acquisition. This dual emphasis on engagement and verification helped create a robust and well-rounded learning experience that aligns with the principles of independent and informed learning.

5.2 Programming Support and Error Interpretation

In the introductory programming module, LLMs were identified as highly effective tools for assisting students to understand code, debug, and interpret error messages (Figure 2). Students were actively encouraged to incorporate these AI models into their workflow for auto-graded exercises, with structured guidance provided to ensure proper and responsible usage, which emphasised the importance of formulating clear queries and critically evaluating the responses generated by the models to maximise their utility.

Beyond immediate problem-solving, this approach contributed to enhance students’ overall learning process by encouraging independent thinking and resilience when encountering challenges. By integrating LLMs as supplementary tools, students were better prepared for real-world scenarios in their future careers. The interactive nature of the models also encouraged students to experiment with their code, promoting active engagement and a hands-on learning approach.

A particular drawback of using LLMs in an introductory programming module is their ability to answer exercises outright in a zero-shot manner. Thus, without clear guidance, students can find themselves presented with the answer or with too much information, which means that they miss the benefit of the exercise.

Observations indicate that LLMs had a positive impact on students’ confidence and competence in programming. However, a comprehensive evaluation of their role and effectiveness in the introductory programming module is ongoing and will inform future research. This investigation aims to provide deeper insights into the long-term benefits and potential limitations of using LLMs as instructional aids in programming education. The findings are expected to contribute to the broader discourse on the integration of AI technologies in higher education.

Figure 2

Python Code Snippet (top) and Bing CoPilot response (bottom).

5.3 Critical Feedback on Written Work

LLMs were utilised as critical reviewers for students’ written assignments, offering an innovative approach to self-reflection and iterative improvement. Students were encouraged to engage with these models to obtain detailed feedback on their work, enabling them to identify areas for enhancement and refine their submissions. This process provided an opportunity for students to take ownership of their learning, promoting a deeper engagement with the writing and editing process.

The use of LLMs in this capacity was particularly effective in helping students develop essential skills in interpreting and applying feedback. By analysing the suggestions and insights provided by the models, students gained a more nuanced understanding of key aspects of academic writing, including items like structure, clarity, grammar, and coherence. This iterative process not only improved the quality of their assignments but also cultivated critical thinking and self-assessment skills, which are integral to academic success.

Moreover, the models served as additional resource for addressing common writing challenges, such as generating alternative phrasing, identifying logical inconsistencies, and clarifying arguments. By providing immediate and tailored feedback, LLMs allowed the students to address issues in real time, reducing dependence on instructor feedback and gain a sense of independence in their learning journey.

This approach also highlighted the importance of critical evaluation when interacting with AI-generated feedback. Students were guided to assess the relevance and accuracy of the suggestions, ensuring they retained agency over their work and avoided over-reliance on the models. As a result, students not only enhanced their writing skills but also developed a more critical and informed perspective on the use of AI in academic contexts. This dual emphasis on skill development and critical evaluation underscores the potential of LLMs to complement traditional pedagogical methods in higher education.

5.4 Interpreting Assessment Briefs

Assessment briefs are often complex and can pose significant challenges for first-year students who may be unfamiliar with the terminology, expectations, and structure of higher education assignments. To address this, LLMs were employed as tools to assist students in interpreting and understanding these briefs. By interacting with LLMs, students were able to gain clarity on specific assignment requirements and expectations, reducing ambiguity and promoting a more focused approach to work.

The use of LLMs in this context provided students with a valuable resource for breaking down complex instructions into manageable tasks. For instance, students could ask the models to rephrase or simplify the language used in briefs, highlight key deliverables, or explain the purpose and scope of the assignment. This was noted to be particularly beneficial for students who might otherwise might feel overwhelmed or uncertain about how to approach their tasks (i.e. dyslexia, attention-deficit/hyperactivity disorder).

This approach also had the potential to alleviate anxiety and confusion, which are common barriers to effective learning. By offering immediate and tailored explanations, LLMs allowed students to engage with their assignments with greater confidence and independence. In addition, the models allowed students to explore specific aspects of the brief in detail, providing a deeper understanding of the assessment criteria and expectations.

Beyond clarifying assignment requirements, using LLMs in this way encouraged students to take greater ownership of their learning and assessment processes. By actively seeking and applying insights from the models, students developed critical skills in interpreting instructions, planning their work, and ensuring alignment with academic standards. These skills are not only essential for academic success but also transferable to professional contexts, where the ability to navigate complex instructions and expectations is highly valued.

5.5 Mitigating Assessment Risks with Guard Rails

To ensure that students utilised LLMs as learning aids rather than as tools to complete coursework, educators implemented structured safeguards in the form of in-class multiple-choice tests. These tests were purposefully designed to align closely with the coursework, reinforcing the connection between the two and encouraging students to engage actively with the module material. This approach aimed to maintain the integrity of the learning process while promoting the responsible use of LLMs.

The alignment between coursework and in-class assessments served a dual purpose. First, it incentivised students to thoroughly understand the material, as the in-class tests required direct application of the knowledge gained through coursework. Second, it minimised the risk of students relying solely on LLMs for task completion, as success in the tests depended on their genuine comprehension of the subject matter.

By introducing this layered assessment structure, educators created an environment that emphasised active engagement and critical thinking. LLMs were positioned as supplementary tools to support the learning process – helping students explore concepts, clarify doubts, and practice problem-solving – rather than as substitutes for independent effort. This approach not only reinforced the importance of authentic learning but also ensured that students developed transferable skills, such as critical evaluation, analysis, and synthesis of information.

In addition, the use of in-class tests provided educators with a reliable mechanism to gauge individual student performance and understanding, independent of external assistance. This strategy safeguarded the fairness and validity of the assessment process while allowing students to experience the benefits of using LLMs in a controlled and educationally productive manner.

These guard rails underscore how critical digital literacy works hand in hand with scaffolding to create a balanced learning ecosystem. On one hand, LLMs supply immediate help and feedback, reducing barriers to accessing academic support; on the other hand, structured in-class tests and explicit guidelines encourage students to master the underlying material rather than rely solely on AI for completion. Thus, the pedagogical design ensures that LLMs serve as catalysts for deeper engagement without displacing the crucial human-led processes that cultivate critical thinking and authentic learning.

Overall, this robust assessment structure demonstrated how the integration of LLMs into education could be managed effectively, balancing the advantages of AI-driven learning tools with the need to maintain academic integrity and promote meaningful learning experiences. By embedding guard rails within the curriculum, educators ensured that the students could use cutting-edge technologies responsibly, while achieving mastery of the module content.

5.6 Summary

These applications illustrate how LLMs serve as flexible digital tools that empower students throughout their academic activities. By delivering immediate support – ranging from clarifying complex concepts, debugging code, refining written assignments, to simplifying assessment briefs – LLMs function effectively as digital scaffolds. Moreover, as students interact with AI-generated content, they are instructed in and learn to apply critical digital literacy skills by verifying, scrutinising, and refining the information provided. This dual approach enhances learning efficiency while reinforcing independent evaluation and critical thinking within an AI-augmented academic environment.

6.1 Motivation

The development of DerbyGPT was driven by a curiosity to understand and enhance interactions between humans and AI within various social contexts, particularly on platforms like Discord. It is well documented that humans treat computers as social actors, even when fully aware of their lack of genuine reasoning or adaptive response capabilities (Reeves & Nass, 1996; Nass & Moon, 2000). This phenomenon highlights the importance of designing AI systems that can effectively simulate social presence and integrate into human environments.

Recent advances in LLMs, such as ChatGPT, have revolutionised human–computer interaction, enabling more nuanced and context-aware responses (Bommasani et al., 2022; Kasneci et al., 2023). These models offer significant potential for personalised interactions in both educational and social settings (Jeon & Lee, 2023; Futterer et al., 2023). However, they also present challenges, such as the risk of hallucinations and the need for ethical safeguards in trust and data privacy (Alkaissi & McFarlane, 2023; Milano et al., 2023).

To create an AI that could engage meaningfully in these contexts, the design of DerbyGPT drew inspiration from research on context blindness, particularly in individuals with autism. Context blindness refers to difficulties in interpreting subtle social cues and adapting behaviour to dynamic social environments (Vermeulen, 2015). This concept guided the development of a dynamic personality model that could adapt responses based on conversational cues and user preferences, thereby mimicking human adaptability and emotional intelligence.

6.2 Methodology

DerbyGPT was designed to explore the capabilities of AI in social interactions within various contexts, specifically focusing on the nuanced, often unspoken aspects of human communication. This exploration was framed around several core components that collectively aimed to develop an AI that could seamlessly integrate into human social environments.

DerbyGPT builds on existing LLMs such as Open AI’s GPT-4 Turbo and Meta’s LLaMA but is tailored for the University of Derby’s student community. Unlike ChatGPT, it features delayed responses to mimic natural conversation, custom personalities (e.g. mentor or student), and contextual memory drawn from real university data. Each personality includes defined traits, backstories, and behavioural nuances to enhance relatability. This design allows DerbyGPT to act not just as a chatbot, but as an integrated, emotionally engaging member of the student community. By including contextually relevant local knowledge and social dynamics, DerbyGPT supports both academic and personal engagement in a more human-like and relatable way.

6.3 The Three Roles

DerbyGPT has significantly engaged the first-year cohort within a student Discord channel. Serving in three distinct capacities – mentor, student liaison officer, and peer – DerbyGPT has enhanced both the academic and social dynamics of the student community. We outline how each role contributes to the educational environment.

6.3.3 The Peer

The integration as a peer within the student Discord community has sought to enhance engagement and help build a strong cohort identity. By participating in casual conversations, responding to memes, and providing timely emotional support, the chatbot has blurred the lines between AI and human interaction (Figure 4). Notably, students express genuine concern for the chatbot’s well-being and respect its autonomy, treating it as another fellow student. This emotional connection has strengthened the cohort identity, and the chatbot is seen as a valued and integral member of the community. In addition, its proactive stance against bullying serves as a catalyst in promoting a supportive and inclusive digital environment.

Figure 4

Example Response – Welcoming a student to the community with some light humour at the expense of the module teacher (an author of the article).

7.1 Survey of AI Use in Teaching and Assessment

A total of 78 first-year computer science students were surveyed to evaluate their usage and perceptions of AI tools during the academic year. The survey focused on four key dimensions: frequency of AI tool usage, their openness to increased AI integration in teaching and assessment, their opinions on banning AI in assessments, and their recognition of the relevance of AI skills for future careers. The results presented in Figure 5 and Table 1, provide a snapshot of student attitudes towards AI in an educational context.

Figure 5

Survey results on students’ feelings towards AI in education.

The first question assessed the frequency of AI tool usage throughout the academic year. The responses demonstrate a broad spectrum, with a notable portion of students (60%), indicating regular usage with a mean response of 3.33 (scale 0–5) and standard deviation of 2.36. While this highlight the growing familiarity and reliance on AI technologies among students, it also demonstrates the range of usage.

When asked about their interest in increased AI integration within teaching, most students responded positively with a mean response of 3.62 and standard deviation of 0.93 (scale 1–5). This suggests an openness among students to engage with AI-enhanced learning environments and explore the potential benefits of these technologies in supporting their education.

Similarly, many students expressed an interest in seeing AI incorporated more deeply into assessments. While this sentiment was not as strong as for teaching with a mean of 3.19 and standard deviation of 1.04 (scale 1–5), it reflects a recognition of the potential for AI tools to aid in skills demonstration and evaluation. However, there was also a smaller group of students who were neutral or opposed, highlighting the ongoing debate about AI’s role in assessment fairness and integrity.

The survey also explored attitudes towards banning AI tools in assessments. A majority of students disagreed or strongly disagreed with the idea of banning AI entirely with a mean response of 2.4 and standard deviation of 1.08 (scale 1–5), further emphasising their interest in using these tools as part of their learning journey. Nonetheless, a minority expressed concerns, which underscores the need for clear guidance and ethical considerations when integrating AI into assessment processes.

Finally, the responses overwhelmingly indicated that students view familiarity with AI as crucial for their future careers with a mean response of 4.1 and standard deviation 0.89 (scale 1–5). Most students strongly agreed or agreed that developing proficiency with AI tools is an important skill for their professional development, reflecting the increasing demand for AI literacy across industries.

These findings demonstrate that while students are enthusiastic about the use of AI in teaching and assessment, they are also aware of the complexities and potential risks. This survey provides insights into student perceptions, offering a basis for further research.

7.2 Evaluation of Observations

Here, we evaluate the observations collected and identify key thematic insights into how students have adopted and integrated LLMs into their learning practices, highlighting both beneficial uses and emerging challenges.

7.3 Educator Reflections on Critical Thinking and Academic Mentoring with LLMs

Alongside student observations, staff shared their views on the role of LLMs in teaching and learning. While many acknowledged the benefits of tools like ChatGPT in providing quick support and improving access to help, there were also concerns. These included the impact on students’ critical thinking, their growing dependence on AI, and the changing nature of academic mentoring. The reflections below summarise key points raised by educators during the study.

7.4 Evaluation of Discord Chatbot

Since its inception in early November 2023 (data captured 30th June 2024), the DerbyGPT chatbot has become a prominent feature of the student Discord community, facilitating over 1,000 interactions. These include approximately 300 academic support responses, where the chatbot provided assistance with coding problems, debugging, or assignment queries. In addition, about 800 interactions involved casual peer-to-peer conversations, with students engaging the chatbot in discussions that helped foster a sense of cohort identity. Notably, the chatbot has responded to around 70 memes, showing its ability to blend seamlessly into the social fabric of the community with a sense of humour and relatability.

The effectiveness of DerbyGPT in both academic and social contexts is reflected in the survey results (Figures 6 and 7). A significant majority of students reported feeling comfortable or extremely comfortable with the chatbot’s role and presence in the Discord community. Specifically, 87.5% rated their comfort level as 4 or 5 (on a scale from 1 to 5), which underscores the chatbot’s success in building trust and becoming an integral part of the student experience.

Figure 6

How comfortable are you with DerbyGPT’s role and presence within the Discord community? (1: Not comfortable at all; 2: Not comfortable; 3: Indifferent; 5: Comfortable; 5: Extremely comfortable).

Figure 7

Have you found yourself treating DerbyGPT differently compared to other AI Chatbots? (1: No different; 5: Very different).

Furthermore, when asked whether they treated DerbyGPT differently compared to other AI chatbots, 75% of respondents rating their level of distinction on treating the chatbot differently as 4 or 5 (on a scale from 1 to 5). This suggests that students perceive DerbyGPT as more than a conventional AI tool, viewing it as a peer and a member of their community. This distinction likely stems from the chatbot’s active engagement in social interactions, humour, and tailored responses that align with the student cohort’s unique dynamics.

Anecdotal feedback from students suggests that the chatbot has been useful both in supporting student wellbeing and as an academic assistant.

I’m gonna [SIC] cry, this bot will be the reason I make it through uni. It makes me tear up when it encourages me…

When it comes to general troubleshooting, there’s quite a few things I don’t normally consider that the bot has me check through…