Compass is a framework designed to help educators in New Zealand create custom GPT models aligned with Achievement Standards within the NCEA (National Certificate of Educational Achievement) framework. This tool guides teachers in crafting AI-driven learning assistants tailored to specific assessment standards, providing students with interactive support that matches the learning outcomes and grading criteria of each Achievement Standard.
Compass helps educators:
- Create Targeted AI Assistants: Align AI feedback with NCEA Achievement Standards, ensuring students receive guidance directly related to their learning goals.
- Customize GPT Interactions: Design prompts and feedback that build student understanding step-by-step.
- Simplify Moderation and Grading: Integrate Achievement Standards into AI feedback, ensuring student responses meet NCEA expectations.
-
General Instruction Set for Educational LLMs: Compass offers a robust set of general instructions that define the foundational behavior of Language Learning Models (LLMs) in an educational context. These instructions ensure that the AI interacts with students in a manner that is supportive, informative, and aligned with best teaching practices.
-
Achievement Standard-Specific Instruction Sets: Tailor each GPT model to specific NCEA Achievement Standards by integrating targeted instruction sets. This customization allows the AI to address the unique requirements and learning outcomes of each standard, providing relevant and precise feedback based on the criteria for Achieved, Merit, and Excellence.
-
Fine-Tuning Framework for Enhanced Alignment: Utilize Compass's modular fine-tuning framework to adjust and refine the LLM's responses. This feature enables educators to tweak the AI's interactions, ensuring that the feedback and guidance provided are closely aligned with the nuanced expectations of NCEA assessments. The fine-tuning process called "clarifications" guides the LLMs evaluation of grade boundaries, key indicators often found in moderation reports and how to approach feedback:
- N1 & N2 EVALUATION, KEY INDICATORS, APPROACH
- A3 & A4 EVALUATION, KEY INDICATORS, APPROACH
- M5 & M6 EVALUATION, KEY INDICATORS, APPROACH
- E7 & E8 EVALUATION, KEY INDICATORS, APPROACH
These are a little different ... still working on a structure for them
The Asttle Marker has issues? Due to two main problems. The Marking schedule is around 8000 characters so max's out the context window, making fine-tuning difficult. Need to migrate the rubric to a markup based text file and then look at fine tuning each assessment area with training data. One Day! Plus the GPT overly infers meaning from student work. Plus the GPT is overly strict on syntax.