Learning Technology jottings at Goldsmiths

Thoughts and deeds

The infidelity of rubrics for assessing online discussions

leave a comment »

Brief summary of a very interesting paper:

Elliott, B (2010). A review of rubrics for assessing online discussions. International Computer Assisted Assessment Conference. 20th & 21st July 2010. University of Southampton. [17 pages, PDF format]

  • Educational benefits of asynchronous online discussion include: integration; elaboration; communication of outside experiences and material; self-reflection; experience of technologies; time management.
  • Drawbacks include: disembodiment and absence of cues; volume of messages; different demands from face-to-face, and so differently inhibiting.
  • Are assessment processes up to the job? Sadler’s idea of ‘fidelity’ e.g. effort as an ‘input variable’ which therefore  shouldn’t fall within the definition of academic achievement
  • Online collaborative work environments have new affordances. Worries about the ongoing relevance of the assessment process.
  • The study: literature review yielded 20 rubrics; these were examined with respect to type of rubric, scoring, and type of criteria used within the rubric; little consistency of terminology or expression – 128 separate rubric criteria were identified, reduced to 33, and 10 categories
  • The most commonly occurring criteria related to:
    1. participation
    2. academic discourse
    3. etiquette
    4. learning objectives
    5. critical thinking.
  • Fidelity? More than half of the rubrics made no reference to the learning objectives. Two of the above criteria relate to non-academic competencies, such as participation (the most common category)
  • Conclusions (caution due to small sample): the majority of rubrics for assessing online discussion exhibit low fidelity; none took account of students’ final level of understanding; none took account of the unique potential of the online environment.
  • Recommendations:
    Express rubric as criteria
    Include holistic assessment of learners final level of understanding or competency
    Make criteria valid measures of course objectives
    Accordingly, criteria should not reward effort or participation
    Free of bias
    Recognise unique affordances of online writing

A few observations – we have an Employability Strategy now, and  with it a growing recognition of the kinds of ‘non-technical skills’ (a.k.a. ‘life skills’ or ‘soft skills’) which Sadler has contentiously called “non-achievements” in academic terms. That said, even if these were enshrined in the learning objectives, the balance in the rubrics identified by Elliott would still be awry in favour of these ‘soft skills’ and at the expense of credit for meeting academic learning objectives.

This is another part of the ongoing double standard (laden phrase, but intended in the most neutral sense) with which society in general tends to approach online environments. We need to talk about this.

Written by Mira Vogel

January 11, 2011 at 14:40

Posted in assessment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: