GenAI: Design

This section on Design is for where you have a longer time frame and it is possible to design the assessment from scratch or fully redesign the existing assessment.

As GenAI tools continue to advance, it is becoming increasingly difficult to detect instances of GenAI misuse in assessments and detection will become increasingly difficult, even impossible. Dawson (2021) argues that if we set restrictions on technologies that are not feasible to detect, the integrity and validity of assessment is likely to be reduced, not enhanced. Therefore, attempting to create assessments that GenAI tools cannot complete, or where we prohibit use, is not a sustainable approach to assessing. Instead, assessments should provide opportunities for students to utilise GenAI tools to support their knowledge and skill development (Rudolph, et al., 2023).

When redesigning assessments, make students aware of how GenAI can and cannot inform the ways in which they respond to the task. This approach will instruct them in the utilisation of current tools and technologies demanded in the workplace. This will also remove the difficulty of having to identify prohibited GenAI use in assessment (Dobrin, 2023).

If there is a long time frame, for example, designing a new unit design, or where changes to an existing unit can be made in UCMS, this allows opportunity to rethink assessment in response to GenAI, including assessment type and Unit Learning Outcomes (ULOs). The modified ULOs where appropriate could also be designed to incorporate components of local or personal contexts and foster critical thinking and creativity. When you are incorporating ULO revisions as part of assessment redesign, you will need to ensure that the revision of the ULOs maintains academic rigour and relevance as well as alignment with the Course Learning Outcomes relevant to the level of study for the unit. Here is an example:

Original ULO across an undergraduate course (level 1, 2 or 3)

Revised ULO for a Level 1 unit (1000)

Revised ULO for a Level 2 or 3 unit (2000 or 3000)

Discuss the effects of climate change on the environment.

Discuss the effects of climate change on the environment broadly and using recent local examples.

Explanation:

We've expanded the original ULO to include not only an understanding of climate change effects generally but also a discussion of recent local examples. The core knowledge of comprehending the effects of climate change on the environment remains unaltered, ensuring that foundational knowledge remains intact. But, by encouraging discussions of climate change within students' immediate surroundings, we not only enhance engagement and relevance but also introduce complexity that makes it challenging for students to accomplish this ULO using GenAI.

The change to the ULO allows for the marking criteria to include the quality of discussion on the recent local examples. 

The changes are still consistent with an undergraduate foundational coursework unit (Level 1).

Critically discuss the effects of climate change on the environment broadly and using recent local contexts.

Explanation:

This revised ULO emphasises the higher-order requirement for the discussion. It challenges students to consider climate change effects in a deeper, critical manner generally and in recent local contexts. The focus is shifted towards higher-order cognitive skills, such as critical thinking, less likely to be automated by GenAI. In addition, students need to critically engage with recent local events and discuss these, something that GenAI would be less likely capable of doing well.

The changes to the ULO also provide scope for higher weighting in the marking criteria for the ‘human’ components: critical thinking and current local contexts.

The ULO reflects the level of study required for discipline-building or integrated coursework units (Level 2 or 3).



Once your ULOs are ready, you can refer to the tables presented in the following section and select the cognitive level that aligns with the ULOs being measured and identify potential assessment task types. You can then consider the mitigation strategies you need to implement to address GenAI-related risks specific to your chosen cognitive level.

Designing assessment for GenAI resilience

Please refer to the following tables that are categorised based on the cognition level to be achieved (which should align to the ULOs being measured by the assessment).

Each table contains information on the suitable assessment types that could be used to measure the achievement of the cognition level, the conditions that may lead to AI breaches, and the potential mitigation strategies. This resource is adapted from University of Wollongong (2023) and UNSW (2023).

Understand - Demonstrate knowledge and comprehension

Suitable assessment typesConditions that may lead to Academic Integrity breachesPotential mitigation strategies

Quiz

Short-answer questions

Summary writing

Identifying main ideas

Checklists

Glossary development

Reusing briefs/questions without modification from one term to another.

Questions assessing lower order thinking skills (Bearman et al., 2023).

  1. Consider these quiz settings: 
    • Create a question pool with randomised questions or prompts to discourage students from easily sharing GenAI-generated responses
    • Shuffle answer choice order
    • Set reasonable time limits
    • Display one question at a time 
  2. Refresh questions at regular intervals.
  3. Present questions demanding advanced cognitive abilities for responses. For instance, rather than asking, "Which of the 2 options are most probable?," use advance prompt, "Select 4 out of 6 options that best justify the likelihood of the given scenario". While ChatGPT is capable of providing insightful responses and analysis, there might be limitations in handling complex scenarios that require deeply nuanced analysis, particularly in specialised or technical subjects.
  4. Avoid questions that require factual recall. Consider questions that require higher order thinking skills like analysis, synthesis and evaluation.
  5. Refer to specific or local information used in class discussions, lectures and other unit resources (Mulder et al., 2023).
  6. Introduce ambiguity or multiple valid interpretations.
  7. Provide explicit instruction on citation practices, making sure students understand the importance of giving credit to the original sources, even when using GenAI-generated content.

For more information: 7 tips to make your quizzes AI-resilient this Spring - LX at UTS

Apply - Transfer knowledge to practical situations

Suitable assessment typesConditions that may lead to Academic Integrity breachesPotential mitigation strategies

Case study

Short written responses

Problem-solving scenarios

Simulations

Role-playing

Calculations

Project proposal

Research proposal

Workbook

Handouts

Annotated bibliography

Group work

Task that requires responses to common question or problems which can be addressed through simple searching or derived via GenAI (Bearman et al., 2023).

Use of paraphrasing tools by students that limit plagiarism detectability.

  1. Craft assessment prompts that present complex and unique scenarios, requiring students to think deeply and creatively. These scenarios should be difficult for GenAI tools to replicate effectively.
  2. Use tasks that involves students positioning their work in their own life, work or community, while also providing them with opportunities for choice in how they portray their ideas (Bearman et al., 2023). 
  3. Emphasise the elements of the task that are less likely to be completed by the Chatbot, for example, analysis of case studies or scenarios requiring students to refer specifically to materials presented or discussed in classes (Mulder et al., 2023). 
  4. If your task involves a process-oriented approach, ask students to show their working and explain the rationale of how they reached their conclusions. Placing more emphasis on process rather than the final product makes it difficult for students to outsource the task (Mulder et al., 2023).

Analyse - Think critically and make judgements

Suitable assessment typesConditions that may lead to Academic Integrity breachesPotential mitigation strategies

Essay

Blog/articles

Discussion board

Journal

Literature review

Project report

Report

Wiki

Project-based assessments

Debates

Video presentations

Industry live and static pitches/briefs

Task that centres around producing a single output or response.

Use of paraphrasing tools by students that limit plagiarism detectability.

Not using assessment formats that require critical thinking, personal insights and analysis makes it easier for students to rely solely on GenAI-generated content.

  1. Engage students in discussions about the ethical implications of using GenAI tools for assessments, fostering a deeper understanding of responsible GenAI use.
  2. Craft assessment prompts that present complex and unique scenarios, requiring students to think deeply and creatively. These scenarios should be difficult for GenAI tools to replicate effectively.
  3. Include prompts that require personal reflections or real-world experiences, making it difficult for GenAI tools to generate suitable responses (Mulder et al., 2023).
  4. Ask students to refer to highly specific or local information used in class discussions, lectures and other unit resources (Mulder et al., 2023).
  5. Consider assessment types that are not text-based, e.g. oral presentations, multi-modal submission, face-to-face assessments (interview, viva voce, practical exam).

Evaluate - Synthesise information, generate original ideas and design new solutions

Suitable assessment typesConditions that may lead to Academic Integrity breachesPotential mitigation strategies

Portfolio activities

Presentations/poster presentations

Reflections

Digital storytelling

Reflective journals

Self-assessments/reviews

Pitching ideas for change

Reviewing ideas for change

Debates

Ethical dilemmas

Critical review

Oral assessments

Single output-focused task where students use GenAI (such as DALL-E, Midjourney) without permission to produce the artefact.

Use of paraphrasing tools by students that limit plagiarism detectability.

Not using assessment formats that require critical thinking, personal insights and analysis makes it easier for students to rely solely on GenAI-generated content.

  1. Frame assessment questions within a specific context or real-world application that makes it challenging for GenAI-generated content to fit appropriately.
  2. Assign collaborative projects, where ULOs permit, that necessitate teamwork, discussions and brainstorming, making it difficult for students to rely solely on GenAI-generated content.
  3. Include prompts that require personal reflections or real-world experiences, making it difficult for GenAI tools to generate suitable responses (Mulder et al., 2023).
  4. Ask students to submit drafts or outlines of their responses in Discussion forum activities, before the final assessment. This provides insight into their thought processes and prevents last-minute GenAI-generated content.
  5. Include questions that require students to consider ethical, social or cultural implications of their proposed solutions, which are more difficult for GenAI to address comprehensively.

Create - Perform procedures and showcase techniques

Suitable assessment typesConditions that may lead to Academic Integrity breachesPotential mitigation strategies

Hand-on lab experiments

Physical demonstration

Role play or simulation

Lab report

Observation of real or simulated professional practice

Students might use pre-generated instructions or procedures obtained from GenAI tools or other sources, passing them off as their own work.

Students might use GenAI tools or other external applications to guide their technique execution, bypassing the need for genuine skill demonstration.

  1. Conduct in-person or virtual live assessments where students demonstrate procedures in real-time. This ensures that they are actively performing the techniques.
  2. Request students to record videos of themselves performing the procedures. This allows for a visual assessment of their techniques and skills.
  3. Alongside the procedure demonstration, ask students to provide detailed step-by-step explanations of the process. This showcases their understanding and knowledge of the techniques (Mulder et al., 2023).
  4. When possible, conduct assessments using actual equipment or tools that students need to physically manipulate.
  5. Set time limits for performing procedures, preventing students from relying on GenAI tools to generate lengthy and detailed responses.
  6. After performing the procedure, ask students to reflect on their experience, challenges faced, and lessons learned. GenAI tools struggle to replicate genuine personal reflections.
  7. Design dedicated practical exam sessions where students are assessed based on their hands-on skills and ability to execute procedures correctly.

Quick tips for assessment design and GenAI

To minimise risk to academic rigour and academic integrity:

Consider using assessment tasks such as:

In the assessment task instructions:

In the rubric:

Synchronous oral presentations where there is the opportunity for Q&A

Ask students to connect their responses to module content and materials, resources and readings.

Where possible, increase weighting for criteria requiring higher-order cognitive skills.

Multi-modal submission 

Use current events and local issues.

Where possible, minimise marks for content that can be readily achieved to an acceptable standard through GenAI. Have a go yourself to see what outputs can be generated.

Face-to-face/ oral interactive assessment, e.g. interview, viva voce, practicals

Require students to use personal examples and experiences where there is evidence of the experience, for example, a critical reflection on placement journal.

Where possible, increase the weighting for criteria requiring input from personalised experiences and local recent events that can be substantiated.

Innovative industry-based assessments where GenAI use is part of industry practice

Purposefully use GenAI as a tool for completing aspects of the assessment, for example:

  • Focus on fact-checking. Have students generate GenAI responses to a prompt and then ask them to work through each claim and statement for accuracy and authenticity with scholarly sources, emphasising critical thinking and information literacy.
  • Focus on the students explaining and evaluating the processes involved in generating a response, not the outputs themselves, what the limitations and benefits are, and what ‘human’ factors need to be added.

Where possible, allocate e-marks for process-based responses rather than just the output, i.e. the grade descriptors for the relevant criteria should reflect the student's explanation and rationalisation of the processes involved, not just the end product.   

An important note: With any assessment design, there needs to be clear alignment with the ULOs being measured. This includes alignment of the marking criteria to the ULOs. Grading needs to reflect the achievement of the ULOs being assessed.

Once you have considered the Design aspects of your assessment, continue with the rest of the assessment design process by clicking on Analyse, Act, Inform, Educate, Check and Evaluate.



Bearman, M., Ajjawi, R., Boud, D., Tai, J. & Dawson, P. (2023). CRADLE Suggests… assessment and genAI. Centre for Research in Assessment and Digital Learning, Deakin University, Melbourne, Australia. doi:10.6084/m9.figshare.22494178

Dawson, P. (2021). Defending assessment security in a digital world: preventing e-cheating and supporting academic integrity in higher education. Routledge.

Dobrin, S. I. (2023). Talking about generative AI: a guide for educators. Broadview Press. https://sites.broadviewpress.com/ai/talking/

Mulder, R., Baik, C., & Ryan, T. (2023). Rethinking Assessment in Response to AI. Melbourne Centre for the Study of Higher Education. https://melbourne-cshe.unimelb.edu.au/__data/assets/pdf_file/0004/4712062/Assessment-Guide_Web_Final.pdf

Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessment in higher education? Journal of Applied Learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.9

University of New South Wales (UNSW). (2023, March 12). Aligning assessment with outcomes. Teaching. https://www.teaching.unsw.edu.au/aligning-assessment-learning-outcomes

University of Wollongong. (2023, March 13). Generative artificial intelligence and assessment security. Learning, Teaching and Curriculum. https://ltc.uow.edu.au/hub/article/gen-AI-and-assessment-security#:~:text=Three%20approaches%20that%20subject%20coordinators%20might%20like%20to,issues%20related%20to%20the%20use%20of%20genAI%3B%20or


Note: Given the rapidly evolving nature of GenAI technologies and largely opinion-based and low-level evidence on emerging practices for use in higher education, this resource represents the status quo at the time of writing (August 2023). As changes to policies and technology develop, and evidence for best practice emerges, practice recommendations outlined here are likely to continue to change and develop.