The infidelity of rubrics for assessing online discussions
Brief summary of a very interesting paper:
Elliott, B (2010). A review of rubrics for assessing online discussions. International Computer Assisted Assessment Conference. 20th & 21st July 2010. University of Southampton. [17 pages, PDF format]
- Educational benefits of asynchronous online discussion include: integration; elaboration; communication of outside experiences and material; self-reflection; experience of technologies; time management.
- Drawbacks include: disembodiment and absence of cues; volume of messages; different demands from face-to-face, and so differently inhibiting.
- Are assessment processes up to the job? Sadler’s idea of ‘fidelity’ e.g. effort as an ‘input variable’ which therefore shouldn’t fall within the definition of academic achievement
- Online collaborative work environments have new affordances. Worries about the ongoing relevance of the assessment process.
- The study: literature review yielded 20 rubrics; these were examined with respect to type of rubric, scoring, and type of criteria used within the rubric; little consistency of terminology or expression – 128 separate rubric criteria were identified, reduced to 33, and 10 categories
- The most commonly occurring criteria related to:
2. academic discourse
4. learning objectives
5. critical thinking.
- Fidelity? More than half of the rubrics made no reference to the learning objectives. Two of the above criteria relate to non-academic competencies, such as participation (the most common category)
- Conclusions (caution due to small sample): the majority of rubrics for assessing online discussion exhibit low fidelity; none took account of students’ final level of understanding; none took account of the unique potential of the online environment.
Express rubric as criteria
Include holistic assessment of learners final level of understanding or competency
Make criteria valid measures of course objectives
Accordingly, criteria should not reward effort or participation
Free of bias
Recognise unique affordances of online writing
A few observations – we have an Employability Strategy now, and with it a growing recognition of the kinds of ‘non-technical skills’ (a.k.a. ‘life skills’ or ‘soft skills’) which Sadler has contentiously called “non-achievements” in academic terms. That said, even if these were enshrined in the learning objectives, the balance in the rubrics identified by Elliott would still be awry in favour of these ‘soft skills’ and at the expense of credit for meeting academic learning objectives.
This is another part of the ongoing double standard (laden phrase, but intended in the most neutral sense) with which society in general tends to approach online environments. We need to talk about this.