GenAI: Analyse

If there is insufficient time to completely redesign an assessment, it is important to analyse and test the existing assessment to assess its susceptibility to GenAI. Follow the steps below to assess and test how ‘GenAI-resilient’ your assessment is.

Step 1: Review your task type

Review your assessment task type to see if the task is likely to be susceptible to academic integrity breaches. Use the current GenAI Practice Guide: Signals of and How to Assess (scu.edu.au)s (scu.edu.au) which includes a checklist of signals of potential unacceptable use of GenAI, to see if you think it is susceptible.

Tasks that are unlikely to be GenAI-resilient include those that:

• depend on older scholarly sources (>12 months old); this can be dependent on the GenAI tool
• require a single submission only with no feedback on drafts (i.e. producing a single output task)
• require lower-order thinking skills
• require recall of knowledge and skills only
• focus on assessment output only as opposed to the processes involved.

Step 2: Test your task

Test your assessment task using GenAI tools.

• Add your assessment instructions to the GenAI tool and see what the output is.
• Progress the output by providing additional prompts for refinement and then see what the output is.

GenAI is a risk

If, upon reviewing your assessment type and the output generated by GenAI tools, you determine that the current task is susceptible to being satisfactorily completed by a GenAI tool, note the following.

• It is essential to make necessary modifications to the task instructions to mitigate the risk of academic misconduct (see next step – Act – below for further information on this) and clearly set the limitations for the use of GenAI in the task.
• Think about how you could use GenAI as part of the process to complete the assessment task – so you are assessing the process and not the product
• If academic integrity cannot be appropriately mitigated through modifications, according to SCU policy, you have the option of prohibiting the use of GenAI for the task. You need to communicate the prohibition of GenAI to students; ideally, this should be done in the assessment brief.

Within the assessment brief, this statement should be chosen (from the two options given):

GenAI is not a risk

If you have determined the current task is designed in a way that GenAI tools cannot complete it to a satisfactory standard, include a statement below in the assessment brief that defines the limit of use of GenAI tools. It is essential to teach students how to use these technologies correctly and efficiently, to ensure that their learning process is meaningful and efficient (Villasenor, 2023, as cited in Yu, 2023).

For example, let students know they can use GenAI tools, such as ChatGPT, to assist with research, ask questions to guide their research and gain clarity on their understanding of concepts. However, they can’t use it to write their assessment. The following is an example from the assessment brief template (it can be modified as deemed appropriate for defining the limits for the assessment task you have set).

Remember that you will need to be able to determine if they have breached the limits you have set.

Within the assessment brief, this statement should be chosen (from the two options given):

Your GenAI limitations will be written after this statement.

NOTE: Examples of limitations that have been set in assessment tasks can be found under Act – Option 2.

Step 3: Review your rubric

When analysing assessment briefs for GenAI implications, criterion weighting should also be considered. In the context of marking criteria that are explicitly linked to the ULOs being assessed, consider what might be easily achieved by GenAI compared to higher-order cognitive tasks that may be less susceptible to GenAI use.

Look at the criterion weighting in the rubric.

• Is too much weight given to lower-order thinking skills (look for verbs like describe, explain, identify, outline)?
• Can demonstration of knowledge be applied and contextualised, rewarding the elements of application and critical thinking?
• Is there scope within the ULOs to weight the higher-order and contextual elements higher than others?

Reward higher-order thinking skills: Assigning appropriate weight to criteria that assess higher-order thinking skills encourages students to think deeply and critically analyse information, and can provide opportunity for application of knowledge to real-world scenarios (Bearman et al., 2023).

Note: Under the SCU Assessment, Teaching and Learning Procedures, all graded assessment tasks must be assessed using clear, explicit criteria aligned to the Unit Learning Outcomes. Hence, setting grading criteria around GenAI use where it is not explicit to the ULOs should be avoided.

Once you have considered the Analyse aspects of your assessment, continue with the rest of the assessment design process by clicking on Design, Act, Inform, Educate, Check and Evaluate.

Note: Given the rapidly evolving nature of GenAI technologies and largely opinion-based and low-level evidence on emerging practices for use in higher education, this resource represents the status quo at the time of writing (August 2023). As changes to policies and technology develop and evidence for best practice emerges, practice recommendations as outlined here are likely to continue to change and develop. 

Return to GenAI in Learning and Teaching Page