Archive for April 2009
Update (6 May 09) – the presentations are now available.
This event was held (in the Institute of Education’s Centre for Excellence in Work-Based Learning for Educational Professionals on 28th April 09) to disseminate and discuss the findings from the Formative E-Assessment project (FEASST) project.
FEASST was funded by JISC. The project elicited cases of real-world assessment practice from academics and abstracted these into patterns – a state somewhere between anecdote and grand theory which other tutors could take use in their own practice (it did not seek to make recommendations). The project report is available, and the presentations will soon be (hopefully linked from the project site).
I came to this event with feedback foremost in mind. The crux of designing formative assessment is designing for good feedback – feedback which is position and made in such a way that it can change learning for the better. Across the sector (and including Goldsmiths) the respondents to the National Student Survey tell us they want more and better feedback on the work they submit. From the Times Higher:
Graham Gibbs, visiting professor at Oxford Brookes University, says assessment is teachers’ main lever “to change the way students study and get them to put effort into the right things”.
Ever since the National Student Survey was launched in 2005, students have consistently given the lowest scores to the assessment and feedback they receive. The National Union of Students says this means there is some way to go before the sector does indeed “get it right”.
“1. Should be for learning, not simply of learning.”
Throughout the day, this view of assessment was confirmed.
On the meaning of ‘formative’
It’s very easy to set assignments and collect learners’ work, but much more of a challenge to do something with it.
Assessment for learning requires designs which afford moments of contingency and scope for modification in such a way that the gap between what the learner knows and what they need to know can be closed; which focus on each learner’s trajectory; and the output of which is in some way measurable and comparable. If there is no scope for feeding back into and modifying the learning experience, then the assessment can’t be said to be formative.
The project report and presentations provide some definitions – it was felt to be important to nail down what formative means, here. Black and Wiliam (2009) conceptualise formative assessment as five key strategies
- Engineering effective classroom discussion, questions, and learning tasks that elicit evidence of learning;
- Providing feedback that moves learners forward
- Clarifying and sharing learning intentions and criteria for success
- Activating students as owners of their learning
- Activating students as resources for one another
Roles – tutors, learners and peer learners
Diana Laurillard’s Conversational Framework, which underpinned the concept of formative assessment in the project, includes the roles of tutor, learner and peer learner.
We can think about feedback in terms of tutors’ pedagogical development, or in terms of peer learners’ metacognitive development, or in terms of an automated system.
What emerged strongly was the push towards peer learning and peer assessment, and towards automation. But to what extent can peers, who are themselves learners, fulfill the criteria outlined above by Black and Wiliam? The Soft-Scaffolding case study may provide insights.
What also emerged strongly was the importance of embodying sound disciplinary and pedagogical understanding so that feedback can be grounded in the difference between what a given student knows, what they are required to know, and facilitate a progression in this direction. Giving good feedback is rarely intuitive. To put this another way, some studies have found that it can negatively affect learning. If too much, or too early, or too frequent feedback is given, students can grope towards the correct (or favoured) response but may not be able to improve on this performance in subsequent challenges. Poor feedback may fossilise error. Some formulations of feedback can discourage students.
Wiliams mentioned that in schools the speed of learning for the children with the best teachers is up to four times that of the worst teachers.
There was a question about capturing the tacit knowledge experts bring to bear on their marking and feedback. Wiliams mentioned ‘judgement policy capture’ (if I heard this correctly) which involves analysing experts in their marking of e.g. submitted work which is boring but whose grammar and syntax are perfect, and the other way round, and see how much difference this makes.
On the role of the ‘e’ in e-assessment
And what does the ‘e’ mean here – for Goldsmiths? The FEASST project identified the following attributes of the ‘e’:
- speed – timeliness of response, allowing the next iteration of problem solving (particularly in objective assessments where feedback can be automated) to begin more quickly.
- storage capacity – eg of feedback snippets, illustrations
- processing – automation, adaptivity to individual learners, scalability
I’d also add:
- contextualising feedback – one example of this is the ability, in most word processing software, to comment directly onto work in such a way that they appear in context rather than disjointedly at the end or in a separate document.
- categorisation – related to storage, the ability of tutors to excerpt, highlight or ‘tag’ examples of phenomena (e.g. pitfalls, strong analysis) in a given piece of work, or across submissions in a given assignment
- searchability – students can search comments as well as tutors searching for illustrations and examples
However, beyond a delivery mechanism, the potential of e-formative assessment is uncertain.
Assessment case studies and patterns incorporating ‘e’
FEASST collected cases of real-world assessment practice from academics and abstracted these into patterns – a state somewhere between anecdote and grand theory, intended for other tutors to use in their own practice. A number of the presentations illustrated the power of the ‘e’. The ‘Try Once Refine Once’ pattern was based on the case of Spanish language learners translating 40 assessments. Without the automation, tutors would have been burdened with marking 10,000 sentences during (if I heard correctly) a single term. The Feedback On Feedback pattern was based on the the Open University’s Open Mentor – a system which captured tutor feedback and graphically represented it as different types (positive, negative, questions, answers). The Narrative Spaces pattern was intended to promote the exploration of mathematical ideas in (seemingly antithetical) narrative form, and it was straightforward that online environments afforded this narrative construction in different media. The Audiofiles case was a project which had investigated the effects of providing feedback in the form of audiofiles returned to students via the VLE and found some effects – including that comments were richer, longer, more personalised and more emphatic (on audio feedback see also this study).
So, in some case the ‘e’ enabled scaling up of assessment enabling valuable opportunities for practice with high quality feedback which minimised the risk of learners simply fossilising errors through repetition. In other cases, it provided environments for externalising thinking in different media. And in other cases the focus was on the tutor’s pedagogical development, using sophisticated natural language recognition to analyse the tutors’ feedback comments and represent them in categories along with suggestions about how to shift from less to more effective types.
To summarise the ‘e’ has far more to offer than right, wrong and grade.
Below are ideas and findings from several of the presentations. The project literature review (presented by Daly) was particularly rich in feedback findings. Of particular relevance to non-objective, qualitative feedback is the work of Shute (2008).
In his presentation, Wiliams made mention to a literature review on feedback which had identified c. 5,000 research studies. 131 of these studies studied the effects of feedback. In around 50 of these cases, feedback made learners perform worse. So a good question to ask about feedback is what kind of responses in a learner does feedback trigger? Feedback needs to anatomise quality; needs to focus not on the individual but on the task.
It’s often the case that feedback comes too late to be relevant – particularly to those learners who are focussed on their summative assessments (the ones “that count”). Feedback needs to be a medical rather than a post-mortem – looking through the windscreen rather than through the rear-view mirror.
Some theory about feedback from Wiliams’ presentation. Grades – even high grades – don’t communicate what is good about a piece of work so that this can be repeated later, nor what could be improved. Receiving a grade alone exascerbates the normative effects to do with self-image in the proportion of learners who regard ability as fixed – such learners experience an impulse to compare their own grade with that of their peers, and to evaluate themselves comparatively in this way.
The key, then, is to promote a view of learning as incremental and to do so by providing feedback which fosters a cognitive engagement in learning – it causes thinking. This leads to activation of attention along the growth pathway rather than the well-being pathway – it promotes mastery orientation rather than performance orientation. Read Boekaerts on self-regulated learning the difference between the growth pathway and well-being pathway in learning.
As outlined in the project report, good feedback needs to:
- alert learners to areas of weakness
- diagnose the causes and dynamics of the weaknesses
- make suggestions about opportunities to improve learning
- address socio-emotive factors in communicating the above
Looking back at this post, I’ve omitted some of the day – most notably Diana Laurillard’s mapping of the patterns into her extended Conversational Framework. This presentation is definitely worth following up.
Some e-assessment links from the FEASST project:
- Presentations from an earlier event (while we await the presentations from this one)
- The project report
Black and Wiliam (2009) Developing the theory of formative assessment. Educational assessment, evaluation and accountability;21(1)5-31.
Boekaerts, Zeidner and Pintrich (2000) Handbook of self-regulation – research, theory and application. Elsevier.
Shute (2008) Focus on formative feedback. Review of Educational Research; 78: 153-189.
It’s generally agreed that, as the boundaries between teachers and learners, and between learning and living, become less defined, and as more emphasis is placed on social learning, that certain qualities are increasingly important:
- Integration is important for cross-referencing, for sharing and for aggregating and reaggregating in different ways for different contexts
- Learner control is important – in order to work optimally learners should be able to choose the environments they work in, although institutions should ideally provide and support instances of environments which fulfil core learning requirements. (It kind of goes without saying that the same goes for teachers.)
At Goldsmiths we use Moodle, an open source Virtual Learning Environment i.e. a teacher-controlled environment which despite many opportunities for interaction and learner control, can feel somewhat at odds with the idea of portfolio learning, social networking and project work. Moodle has been thin on tools for content creation, collaboration and sharing – tools and environments for these activities have developed independently and can be integrated if there is the will and resource (IT programming skills) in an institution.
Google is a leading web-based global advertising company responsible for some of the most innovative communication, information, authoring and hosting tools for individual users. Google Search is a famous world-beater but Google is also the creator of Google Apps (including its Education Edition), a free suite of online office tools (document authoring, presentation, calendar, chat with or without audio and video, email, sites etc) based in the so-called cloud – i.e. hosted and served rather than locally installed and therefore available from any web connection regardless of location. Google Apps occupies spot 12 and 15 respectively for learners and e-learning professionals in Jane Hart’s top learning tools for 2009.
Google Apps and Moodle are now integrating. This means that:
“From a teacher’s perspective, this provides an easy way to assign students to collaborative tasks without having to worry about the students having different operating systems or incompatible software or being unable to access an online system.”
More at Moodle Rooms (set up a user account to view).
This is probably going to take off in a big way.
Wired has a piece on Masaki Fujihata’s Pocket Film Festival.
“A good pocket film means recording personal happenings, or someone’s personal way of observing the world,” he says. “Normally people try to imitate the big Hollywood films but to make a really good short pocket film make sure [you] start collecting many images. A month or so later you can start to edit them and see what you did.”
Fujihata invited two winners from the 2007 festival, Daisuke Kobayashi and Toru Oyama, along to show me the techniques. I brought along an INQ1 from 3 – a phone whose USP is that it’s wired specifically for uploading content straight onto the internet.
BankArt NYK, an art space on Yokohama’s quayside, is their chosen set. The only limitations they are given are to keep the scenes in the shoot to no more than five. Simplicity is key. It is that method that helped them claim the Grand Prix for their street film 720/24 – as in 720 minutes in 24 hours – two years ago.
They use time as the theme for this masterclass too. “It is now 12.08,” Toru declares as Daisuke, bedecked in fur flying hat and goatee beard, pinches the cameraphone between thumb and forefinger and spins it about like the hands on a clock. As each minute passes, the camera will turn to signify the passing of time. This method, using images from their native Tokyo and showing council estates, playgrounds, subway stations and other urban imagery, is what won them the big prize.
In BankArt NYK they home in on the minutiae of the concrete mausoleum. Pieces of art, the LED lift indicator, the lift doors closing and even a bus parked on the dockside for variety.
They never shoot anything more than a few metres away. That range is the camera’s best operating environment. “The good point of making a movie with a mobile phone is that the people making it and the people watching it are very near each other,” they advise. “Normally, if you’re using a regular camera then the distance between the viewers and the cameraman is very far away.” They make the most of each other’s movement and Daisuke includes Toru in his clips.
Toru gives me another tip: keep the clips short and simple. Some pocket film-makers shoot with the phones only for the imagery to be viewed on a cinema screen. Not these guys. What they see through the phone’s screen is exactly how and what they want their audience to see.
I put up an opportunistic, entirely anonymous survey of staff and students on our VLE (learn.gold) shortly after the snow closures back on Feb 2nd / 3rd 2009.
The text of the question:
“During the recent college closure due to bad weather, did you use any of the following communication technologies to study with, or facilitate the study of, others?
By ‘study’ we mean learning, thinking and responding together, rather than organisational/operational stuff like rescheduling or submitting work. Please select N/A if you didn’t use the technologies for studying with others (which is, of course, perfectly fine by us – but we’d be interested to know).
- 1 = slight use
- 5 = heavy use
- N/A = non-use”
There were 86 respondents.
|Type of technology||Average scale value / number of N/As|
|Video conferencing (eg Skype video)||1.7 / 75|
|Online telephony without video (eg Skype calls)||2.0 / 71|
|Instant messaging aka ‘chat’ (eg GoogleChat, Skype or Facebook chat)||3.6 / 47|
|‘Shared desktop’ (Skype or MSN Messenger)||3.2 / 64|
|Editing documents collaboratively (Google docs, wiki)||2.4 / 72|
|Discussion on blog(-type) posts or bulletin boards (Facebook, blogs)||3.0 / 48|
|3.4 / 22|
|Landline or mobile phonecalls||3.4 / 33|
|Text messaging (SMS)||3.8 / 28|
A bit of discussion
As somebody remarked in the Comments field (see below)
“Why did anyone want to communicate any more than to say, ‘Uni is shut today’? You should have been outside playing in the snow!!”
Quite. But those who did learn with others on those days primarily used text-messaging, phonecalls and email, and used them moderately. Somewhat fewer used web chat moderately. Still fewer used blog or bulletin board discussion moderately, and even fewer made moderate use of shared desktops. A tiny minority used video conferencing and online telephony slightly.
Other technology / comments
- did anyone study that day?
- I found it helpful to know I didn’t need to struggle in. Got on with some work at home.
- I talked to my roommates – face to face, the good old fashioned way
- I trudged to Goldsmiths on foot through the blizzard–only to find the college closed; it would have been nice if the History dept had had a phone round saying not to come, like other colleges didhad
- Photo messaging
- pigeon post
- the communication from university was dreadful. did not hear untill 10.30 that it was closed… i start my journey at 6.30am!!!!
- There is not enough room in here to fully communicate my rage on this issue. There is not enough room in the world to fully communicate my despair at the constant amateurish failures of Goldsmiths
- Turns out i don’t study with others at all.
- Why did anyone want to communicate any more than to say, ‘Uni is shut today’? You should have been outside playing in the snow!!
Conclusions / limitations
So there we go. At Goldsmiths, email and phones are still the go-to technologies for learning together when it’s impossible to get together.
Unless… unless the more intrepid technology users didn’t stop by our Virtual Learning Environment to take our survey. This is possible – perhaps likely – but hard to ascertain.
Parker, J (2009) Academics’ virtual identities. Teaching in Higher Education;14(2):221-224(4).
This short discussion paper moves from the mesmeric force of Socrates, which led to his persecution and eventual execution as an intellectual paedophile, through the svengali-like power-play of some academic teaching, touching on F.R. Leavis and Mamet’s Oleanna. It ends with the dissolution – in the form of the Web 2.0 eroticisation of the academic’s offer of teaching – of this master-disciple model of pupilage in academia:
“… there are two Facebook groups in my friend’s name. One, with 40 worldwide members, declares him to be the greatest historian of the modern era. The other speculates about his sexual proclivities and predilections and contains rival claims about why the student poster is more likely to make it with him than the previous respondent. This doesn’t seem a big deal – the discourse, specifics and tone is that of teenage girls throughout Web 2.0 – as if talking to girlfriends, they speculate and appropriate the sex object whether he is a Hollywood or television star (brought within range or fantasy range at least, to any reader of the London Lite and its ilk). A boy seen and fancied in a club or bar or . . . a lecturer.
But . . . teachers all go into the teaching situations projecting an identity: professional, enthusiastic, caustic, evangelical or whatever. We do not ‘simply’ transmit disciplinary material; we project to and enthuse the next generation; we accept the role of disciplinary representative and trainer/coach/tutor to those who will move into other work and those who we hope to follow in our footsteps. (Some questionnaire-based research projects report that the single most important factor in ‘excellent’ teaching is enthusiasm and charisma.) So, what’s new?
Answer: Web 2.0. As academics, we control our public image via our websites, Research Assessment returns and dustjacket biographies. In previous times, if we were talked about in the student union, pub or campus coffee bar, we were frankly validated. But, RateMyProfessors – which in addition to ‘easiness’, ‘helpfulness’, ‘clarity’ and ‘rater interest’ offers a chilli ‘hot’ symbol – and Facebook offer another identity, promulgated between current but also to future students. (There are now Facebook groups for those expecting to go to university or college the following year: by the time ‘Freshers Week’ comes they will have been in communication and forming a group dynamic for anything up to nine months . . . ) We now have an uncontrolled, de-regulated, continuously commented on and modulated virtual identity.
Does it matter? Perhaps we should be flattered that we are the target of paparazzi-type attention? Students have always gossiped and speculated about their teachers, perhaps in an attempt to reverse the power politics of the teaching situation. Pity those who do not have an adulatory Facebook group started for them by their students . . . .
Yes, it does matter. Every teaching situation is at base an offering of the self, of one’s passions, vitality, engagement with and formation by the discipline’s epistemology and sometimes, importantly, ontology. If nothing vital is on offer, then the students should surely, rather, stay at home and read authoritative public academic documents. Teaching is an offer, and that that offer can publicly and popularly be interpreted as sexual is deeply problematic.”
Just thought I’d note this because it was so interesting.
I agreed to be interviewed by a Design undergraduate about death in online social networks. We spent an hour together in Loafers – I was so glad I did it because it really opened my eyes to the hierarchy between online and offline lives. There is currently no online network of support for people who lose online friends or lovers. Often the news is broken online in ways which starkly differ from the ways in which it would be ethical to break it offline – indeed, with a brutality which demonstrates the contempt in which online relationships are held. The needs which are recognised and indeed uncontroversial in the bereaved of the offline world are almost totally overlooked in the online world. One day a profile is existing, curtailed, and the next – after relatives have sent a scanned death certificate to Facebook – it is brutally expunged. This student was addressing these things. I was lucky enough to see some of her storyboards.
What really hit me was the idea learning technologists tend to espouse – that technologies are transformative – didn’t make sense here. Learning technologists are fond of telling the old story of cinema. When the cine camera was first invented, what did people use it for? Why, they filmed theatrical productions – same three unities, same proscenium arch, a single fixed camera angle. It was only a long time later that cinematography emerged – the different scenes, the angles, the split screen, the shifts of focus. Learning technologists encourage academics to think transformatively about the intersection of technologies and their discipline.
But in the case of death online? What does technology mean here? And how does one even raise the subject? “Transform the practice of your grief for the online environment, people!” No.
Bring on the designers.
(Technical notes – we brought laptops for web access. Easily recorded the interview to Audacity using my laptop’s integrated mic. Ask CELT for the podcasting How To Guide, have a look at the Podcasting for Pedagogical Purposes How To Guides, or see the pages on recording on the Oxford / Cambridge / Open University growing STEEPLE wiki)