User:Markvanharmelen/Nicol

David Nicol Technology-supported assessment

focused intro

Background/Definitions

Assessment is a complex process in formal learning contexts. It involves (i) devising learning tasks to sample the learning expected from participation in a course or programme of study (i.e. knowledge, skills and attitudes) (ii) assessing performance on these tasks and interpreting the results, and (iii) using the information acquired to enhance further learning (e.g. feedback) and/or to produce a grade.

This entry discusses the ways in which technology might support these different stages in the assessment cycle (i-iii): task design – assessment/interpretation - feedback/grading. It offers both the pedagogical background and some examples of tools and practices. It focuses both on scenarios where assessment has summative purposes (to arrive at a mark or grade) and where it has formative purposes (to provide feedback to support learning and development).

In the sections that follow the potential offered by technology is examined in the light of the following assumptions.

1. Effective assessment assumes a theory of learning and a model of cognition. In order for assessment to be a valid measure of learning for formative or summative purposes those involved must have a reasonable understanding of how disciplinary knowledge and skills are acquired, and of the trajectory of their development (Pelligrino, Chudowsky and Glaser, R, 2001).

2. If the purpose of higher education is not just to acquire knowledge, skills and understanding in the discipline but also to develop in students the ability to monitor, evaluate and regulate their own learning, then there must be a shift from teacher regulation to learner regulation over the course of an undergraduate degree (Nicol and Macfarlane-Dick, 2006). To achieve this, students must be actively involved in the processes of assessment, in the different components of the assessment cycle (i-iii). In other words, assessment is a partnership, which depends as much on what the student does as what we do as teachers.

3. Peer interaction and collaborative learning can enhance the development of learner self-regulation. Peer processes can help attenuate the teacher’s voice and strengthen the students’ voice; they often result in students scaffolding each other’s learning and they also develop in students the ability to make assessment judgements about the work of others (e.g. where peers critique or assess each other’s work), skills which are often transferred when students turn to regulating their own work.

Assessment is said to drive student learning: it can provide the motivation for learning (e.g. through the awarding of marks and grades) but it also enables learning to take place through the provision of feedback. Feedback comes from different sources not just the teacher: it is also generated by students as a by-product of engagement in learning and assessment tasks and it can be generated more systematically through organised reflection and self-assessment activities. Feedback is also produced during group activities where peers give each other informal feedback; it can also be systematically generated through peer critiquing activities. Teachers play a key role in delivering feedback and in orchestrating self and peer feedback opportunities using a range of technologies and tools. This wiki examines how technology might support these different forms of feedback as well as marking and grading.

Findings from research

When part of a well-defined pedagogical approach, technology can support any stage in the task design – assessment/interpretation – feedback/ marking cycle.

Task Design

Technology can support task design for assessment in a variety of ways. It can support the presentation of assessment tasks to students and it often enables more flexibility in the timing of assessments. Increased clarity of task goals and greater flexibility in timing both give students more control over their learning and assessment thus enhancing opportunities for self-regulation. Technology can make it easier for teachers to monitor and track learner progress (e.g. through the recording of student activities) and to tailor assessments to individual student needs (e.g. through adaptive testing). Also, using Web 2.0 formats such as wikis, blogs, online discussions, social software and virtual worlds (e.g. Second Life) it is possible to assess and support the development of a much wider range of knowledge, skills and attitudes than in the past. Some examples of the possibilities are provided below.

Many tools are available to support the administration of assessment processes making them more efficient and less time consuming for academic staff. Virtual Learning Environments (e.g. Blackboard, Moodle) can make it easier to present assessment tasks to students (e.g. to publish task requirements, the criteria to be used in assessment and the timings for submissions) and to track and record student progress (e.g. automatic time logging of activities and assignment submissions).

Tools for objective testing, within virtual learning environments and within dedicated assessment engines (e.g. Question Mark Perception) allow teachers to orchestrate frequent assessment testing (e.g. online objective testing) which can be used both to offer flexibility in the time and place of assessment and/or to encourage students to spend more ‘time on task’ out of class. Time on task has been correlated with enhanced student learning (Chickering and Gamson, 2001) with many studies showing that frequent objective testing enhances the performance of students in final exams (e.g. Haigh, 2007)

Computer-supported assessment also makes it possible to enrich and make assessment tasks more authentic, for example, incorporating multimedia presentations and to enable better alignment of tests to students levels of understanding (Conole and Warburton, 2005). For instance, as a test of their understanding, students learning a foreign language might watch a recorded video of current affairs programme and answer objective questions relating to the content (see, Nicol, in press). Adaptive testing involves modifying the nature of the test based on the responses the student has made to earlier tests. Although modifying interactions in this way can be achieved in paper tests, this is far more efficient in computer-supported environments. Some researchers have also attempted to use computer programs in specific disciplinary domains (e.g. mathematics) to automatically generate multiple variations of the same class of problem types (Bennett, 1999).

As well as presenting tasks, the recording features within computer-mediated learning environments are valuable for assessment purposes as they enable tutors to present milestones for complex tasks with timed release and alerts (e.g. project work), to monitor student activity and achievement and to take action to support those students who fall behind or are in difficulty. The milestones and records of activity also enable students to monitor and reflect on their own progress and that of their group. For example, SPARK (the Self and Peer Assessment Resource Kit), peer support software for group working, provides records of student work online as they engage in collaborative group tasks (see, Freeman and McKenzie, 2002).

Many Web 2.0 tools such as wikis, blogs , e-portfolios and discussion boards also allow the monitoring of learning as it is progressing. Some of these tools, by providing a record of ongoing interactions, also have the potential to facilitate the assessment of soft skills (e.g. group communication amongst students) that are difficult to assess in traditional settings. They also create some new assessment possibilities such as assessing a student’s contributions to a class wiki or assessing individual contributions to an online discussions or a student's reflective capabilities as exemplified through blogging.

Assessment and Interpretation

Moving on through the assessment cycle, technology can play a role in supporting the assessment (measurement) of knowledge, skills and attitudes and the interpretation of assessment data. Most teachers wish to ensure that assessment is valid and actually samples the knowledge, skills and attitudes that students are expected to develop through the course or programme of study or that are required in professional practice. It is also important that the assessment data can be used to support learning not just to certify achievement. Technology offers great promise in both these areas: it can help increase the validity of assessment tasks; it can also help teachers to chart students’ developing understanding; and it can help support the development of meta-cognitive capabilities important for the transfer of learning to new problems and contexts.

Assessments are more likely to be valid and productive measures of student development if they are underpinned by a model of learning that both explains and charts the trajectory of learner development from novice to expert in the discipline (Pelligrino, Chudowsky and Glaser, 2001). Models enable us to not only test current levels of understanding and skills in relation to levels or standards but also to design subsequent experiences to stimulate learner progression.

One of the most important differences between experts and novices is how they perceive the relationships between disciplinary concepts. When they solve complex problems, experts access internal networks of inter-related concepts. They are known to possess tightly organised and inter-related conceptual schemas, which they draw on to solve such problems. A number of technological tools have been designed to support the assessment of knowledge organisation in conceptual domains. For example, concept-mapping software has been developed to enable individuals or groups of learners to create graphical representations of how they perceive relationships amongst concepts. This helps students externalise their understanding and tutors to identify students’ current state of development. Software has also been developed that helps teachers to assess the concept maps that are produced by students (O’Neil and Klien, 1997).

In a similar vein, intelligent tutoring programs have been developed that can directly compare novice against expert performance in complex cognitive domains such as physics (e.g. Mastering Physics, Mastering Chemistry) with this information used to provide ongoing feedback (http://www.masteringphysics.com/site/index.html) . Such programs carry out a fine-grained analysis of performance while the student is engaged in problem solving tasks, including open-ended tasks, comparing ongoing performance against the correct problem solving moves. Using a databank of known conceptual difficulties derived from analysis of problem-solving by real students, the program is able to provide hints, tips, ask questions or give instructions to move students forward. Mastering physics has been developed drawing on the performance of thousands of students across the world so all known conceptual difficulties have been mapped out. Similar work has been carried out in other science subjects such as chemistry, biology, astronomy (see, http://www.masteringphysics.com/site/product/other-products.html ) although there is much less of this kind of work in arts and social science subjects, disciplines where there is no one right answer.

Some technology-enhanced assessments also involve the use of sophisticated modelling and simulation environments to capture complex problem-solving and decision-making. For example, students might engage in a business game ( see, http://www.economicsnetwork.ac.uk/teaching/simulations/principlesofmicroeconomics.htm) underpinned by a real world model or they might generate and test hypotheses within a virtual model of an engineering plant (e.g. Davies, 2002).

Intelligent tutoring systems and simulations are examples of dynamic assessment where the software allows fine-grained analysis of student responses as they occur, based on an underlying model. Simpler types of dynamic assessment are possible which do not rely on the development of sophisticated software. One example is electronic voting software (EVS) which can be used to test students on a regular basis in class with the results used by the teacher to modify dynamically the path of instruction within the teaching session or across sessions (Nicol and Boyle, 2003: Boyle and Nicol, 2003).

Using EVS the teacher explains an important concept (e.g. force in mechanics or cost-uncertainty in economics) then presents students with a multiple choice test of their understanding. The students respond with handsets and the software computes the class responses within a few seconds and displays the results to the class (e.g. a bar chart showing numbers selecting each alternative). If many students respond with a wrong answer, the teacher can trigger peer dialogue (i.e. ask students to discuss their answers with peers), provide feedback directly or re-teach the concept. Although the software only tallies the student responses, it enables the teacher to orchestrate some quite sophisticated feedback activities (peer and teacher) based on current levels of understanding. Banks (2006) provides many examples of EVS use across many disciplinary contexts.

Feedback and Grading

A key goal in higher education is to shift the balance of responsibility, over the course of the undergraduate degree, from one in which the teacher regulates student learning towards scenarios where students learn to take control of and manage their own learning (learner regulation). Assessment processes are the fulcrum for this shift of responsibility. Nicol and Milligan (2006), drawing on the earlier paper by Nicol and Macfarlane-Dick (2006), have taken a pedagogically driven approach to formative assessment supported by technology and tried to show how different tools might support the development of learner self-regulation. They identified seven principles of good feedback practice based on research and explained how technology might support the implementation of each principle in blended learning contexts.

Good feedback practice:

1. helps clarify what good performance is (goals, criteria, standards);

2. facilitates the development of reflection and self-assessment in learning;

3. delivers high quality feedback that helps learners self-correct;

4. encourage interaction and dialogue around learning (peer and teacher-student);

5. encourages positive motivational beliefs and self-esteem;

6. provides opportunities to close the gap between current and desired performance;

7. provides information to teachers that can be used to help shape their teaching.

Principle 1: ‘helps clarify what good performance is (goals, criteria, expected standards)’.

Sadler (1989) identified three conditions necessary for students to be able to regulate their own learning and to benefit from teacher feedback: they must possess a concept of the goal or standard being aimed for in a learning or assessment task; they must be able to compare current performance against that goal or standard; and they must be able to take action to close the gap between current and desired performance. Feedback provides information about current performance relative to goals but if students engage in tasks with incorrect or weak conceptions goals then feedback is less likely to connect (Hounsell, 1997). More importantly, if goals are unclear then students will not be able to assess their own performance (self0-regulate) or the performance of others. However, there is considerable research evidence showing that there are often mismatches between teachers’ and students’ conceptions of goals and of assessment criteria and standards (Hounsell, 1997: Norton, 1990, Nicol and Macfarlane-Dick, 2006).

One commonplace way of clarifying task requirements (goals/criteria/standards) in e-learning contexts is to publish online descriptions of assessment criteria and/or standards for different levels of achievement. However, many studies have shown that it is difficult to make expectations explicit through such written documentation (Rust, Price and O’Donovan, 2003). Most criteria for complex tasks are difficult to articulate and are often ‘tacit’ and unarticulated even in the mind of the teacher (Yorke, 2003). Hence there is a need for complement online criteria. One approach is to create an online discussion space and encourage students to spend time before beginning the task to discuss the requirements and criteria (Palloff and Pratt, 2005).

Another approach that has proved particularly powerful has been to provide students with ‘exemplars’ of performance (Orsmond, Merry and Reiling, 2002). In an online context, exemplars are easily made available to students for consultation, for example, within a repository or virtual learning environment (VLE). However, it might be more effective to supplement this approach with additional activities that encourage students to interact with, and externalise, criteria and standards. For instance, groups of students might be required, before carrying out an assignment, to examine two exemplars of a completed task (e.g. a good and a poor essay) and to post within an online discussion board their reasons why one is better than the other including the criteria that formed the basis of their judgement. The teacher might then clarify any areas of misunderstanding or mismatches in conceptions and publish online a criterion sheet that draws on this student-generated discussion. In problem-solving disciplines another approach might be to provide an online simulations demonstrating how an expert works through a problem-solving task.

Principle 2: ‘facilitates the development of reflection and self-assessment in learning’

One of the most effective ways to develop self-regulation in students is to provide them with many opportunities to practise regulating aspects of their own learning (Pintrich, 1995). In the online context, there are many ways of organising for reflection and self-assessment and many tools to support these processes. The most common practice is to create and administer online objective tests and quizzes that can be used by students to assess their understanding of a topic or area of study (Bull and McKenna, 2004). While such tests do provide for a degree of self-regulation this is low level as students are not involved in identifying goals for self-assessment and might often guess the answer in some types of test (e.g. MCQs).

One way of enhancing reflection in multiple choice tests is to introduce confidence based marking (CBM). Gardner-Medwin (2006) has used CBM with multiple-choice tests with medical students. In CBM students not only select an answer but they rate their confidence in the answer and the mark awarded is based on whether the answer is correct and the confidence rating on a one to three scale. Gardner-Medwin maintains that this procedure encourages students to think deeply their own knowledge and about whether they have a reliable reason for choosing the answer. The value of CBM is that encourages deep reflection but does not require the teacher to analyse student reasoning and therefore involves no extra time. It is therefore surprising that few assessment engines currently have CBM capability. A more powerful approach to self-regulation using multiple choice tests would be to have students in pairs create their own tests and to discuss them online in groups (see, Nicol, 2007). In this scenario, students would not only be involved in setting goals (the question) but also in identifying responses (answers) that meet, and don't meet, those goals. Arthur (2006) has described how this kind of approach might work in accountancy (see, http://www.itl.usyd.edu.au/Synergy/article.cfm?articleID=283). From a similar perspective, Ellaway et al (2006) has shown how the power of a medical simulation, used to test students medical knowledge and skills, could be significantly enhanced by asking students in groups to create their own simulation using the software (Labyrinth) and then to defend their constructions in presentations to teaching staff.

Another way to directly involve students in monitoring and regulating their own learning is through portfolios. The construction of a portfolio requires that students reflect on their achievements and select work that meets defined standards. In addition, students might be asked to write a reflective essay or keep a reflective journal in relation to their learning. Many educationalists have been experimenting with e-portfolios to support formative and summative assessment (e.g. Cotterill et al, 2006).

Principle 3: 'delivers high quality feedback information that helps learners self-correct'

In online contexts much work has been carried out to assist teachers in giving feedback to students. Denton (2001) has developed an electronic feedback system that enables teachers to construct feedback reports to students. These reports contain general comments, standard comments (about specific elements of the work) and personal comments (to the individual). Standard feedback comments represent a time saving feature of this system; they can be selected by number or from a drop down list and attached to student work. Denton reports that teachers claim they can give higher quality feedback and that the system saves time.

Whitlock, Watt, Raw and Moreale (2004) have developed a pedagogically driven tool that supports tutors with marking of electronically submitted assignments. The tool, an open source mentoring application called Open Mentor, checks the quality of tutor feedback comments against a model of good commenting which is tied to the level of marks that students are awarded for written work. The tool helps tutors balance socio-emotive and cognitive support within their commenting and is intended to help students become more reflective learners. Whitlock (2006) has developed and tested this tool across a range of disciplines and shown its effectiveness.

Principle 4: 'encourages interaction and dialogue around learning (peer and teacher-student)'.

Many researchers maintain that dialogue is an essential component of effective feedback in higher education (Laurillard, 2002). Yet mass higher education has reduced opportunities for individualised teacher-student dialogue. This is an area where new pedagogical approaches supported by technology offer significant potential. For example, a number of researchers have shown how feedback dialogue in essay tasks can be enhanced using software systems that support peer commenting and assessment (e.g. Davies, 2000: Bostock, 2001; Sung et al, 2005). Using these systems students are able to get a rich array of comments from their peers which they can then use to make improvements before submission. Some of these systems enable students to contribute to the marking of each other’s essay. Involving peers in commenting and in marking not only enhances dialogue without increasing teachers time but it also gives students more control over their own learning. Indeed, giving students opportunities to participate in assessment from the marker perspective is central to developing in students the capacity to regulate their own learning.

Another approach to increasing dialogue around assessment tasks is to have students engage in a group tasks where they are encouraged, as part of the task, to give each other feedback. This was the approach adopted by Baxter (2006) in a first year psychology course (see also Nicol, in press). Baxter required his students in groups (of 6 or 7 students) to write six 800-word essays online over the first year. Each group participated in a closed forum in WebCT where they discussed the assignments, shared their work and gave each other feedback before posting the final essay. Each essay task was preceded by structured activities: the students answered questions about important concepts relevant to the essay task and read background texts. On completion of an essay, students were given two kinds of teacher feedback: general comments about class performance and access to good essay examples selected from the groups’ submissions. Students could assess their own performance against these model answers.

Baxter reported that the students worked harder, produced essays of a much higher quality than past student cohorts and that they scaffolded each other’s learning: this was reflected in better essay marks in the final exam as indicated by a controlled comparison with the previous year. One notable feature of this study was that all student activities were recorded online: this enabled Baxter to monitor and coordinate the work of 85 groups and orchestrate teacher and rich peer feedback. This would have been almost impossible to achieve without the technology without a substantial increase in teaching resources.

Principle 5: 'encourages positive motivational beliefs and self-esteem’.

Self-regulation, motivation and self-esteem are inextricably linked. The more motivated students are and the higher their level of self-esteem, the more likely it is that they will persist and regulate their own learning. Students’ motivation is determined by whether they perceive that their own needs are being met, by whether they see value in what they are doing and by whether they believe they have the ability to succeed with reasonable effort (Meece et al, 2006). Rather than being fixed or completely determined by the environment, motivation is ‘constructed’ by students based on their appraisal of the teaching and learning context (Paris and Turner, 1994). This means that teachers can have an influence on motivation.

Each of the feedback principles described in this section can be implemented in ways that are motivational: for example, in ways that increase the students’ sense of control over their learning (e.g. through self and peer feedback) or that give students experiences of success (e.g. through opportunities for testing and retesting). Ways in which technology might support these processes have also been discussed: for example, greater control over learning can be given to students by enabling more flexibility in assessment requirements with online tests available at times and places to suit learner needs or by providing opportunities for students to manage, record and evaluate their own learning through e-portfolios. Using simulations and intelligent tutoring systems can also be motivational when they are based on real-life systems (authentic) and when the feedback allows students to see what progress they are making towards their own goals in a dynamic and immediate way.

Many writers have noted that Web 2.0 technology has the greatest potential to enhance learning and motivation in HE. Most young people today have accounts on social networking sites (e.g. myspace, facebook) where they construct and organise knowledge, share content, comment on each other’s productions (i.e. provide feedback) and develop networks of friends. The Web 2.0 tools used to carry out these activities include blogs, podcasts, wikis, virtual worlds etc. Some recent commentators have argued that these social networking activities, while mostly carried out in support of informal learning, suggest new models for learning in formal contexts such as higher education. This would involve tapping into the habits of learners accustomed to social networking, where knowledge is co-constructed and sharing and revising are common-place. Some higher education institutions have been trying to capitalise on these developments.

Principle 6: 'provides opportunities to close the gap between current and desired performance’.

Boud (2000) maintains that 'The only way to tell if learning results from feedback is for students to make some kind of response to complete the feedback loop (Sadler, 1989). This is one of the most often forgotten aspects of formative assessment. Unless students are able to use the feedback to produce improved work, through for example, re-doing the same assignment, neither they nor those giving the feedback will know that it has been effective.' (Boud, 2000, p158)

If students are to become better at regulating and managing their own learning they must develop the ability to make use of the feedback they receive. Yet in HE students rarely have the opportunity to directly apply feedback especially in the case of planned assignments. Invariably they move on to the next assessment task soon after feedback is received. While not all work can be re-submitted, many writers argue that re-submissions should play a more prominent role in learning (Boud, 2000). Also, greater emphasis needs to be given to providing feedback on work in progress (e.g. essay structures, plans for reports, sketches etc.) and to engage students in planning strategies for improvement.

Many technology tools can help create environments that support feedback use. Workflow management tools in virtual learning environments allow students and staff to keep records of work at different stages, tracking tools in virtual learning environments and software such as TURNITIN enable changes made to assignments after feedback easier to identify. E-portfolios enable students to keep versions of their work together with associated feedback. Online objective tests can be organised so that students have opportunities to re-take the same tests and simulations and intelligent tutoring systems allow students to get dynamic feedback during the act of production of an answer.

Principle 7: 'provides information to teachers that can be used to help shape the teaching'.

Good feedback practice is often discussed in terms of providing good information to the students about learning. However, it is just as important that teachers have good information about student progress in learning so that they can tailor their teaching in relation to their students' needs (Yorke, 2003: Black and Wiliam, 1978). This requires the creation of learning environments that provide rich and ongoing data about student learning.

Frequent assessment tasks, especially diagnostic tests, can help teachers generate cumulative information about students’ levels of understanding and skill so that they can adapt their teaching accordingly. This is one of the key ideas behind the use of electronic voting systems (Nicol and Boyle, 2003; Boyle and Nicol, 2003). With EVS teachers can gain regular feedback information about student learning within large classes by using short test-feedback cycles (see Principle 4). The teacher receives feedback on areas of student difficulty signalled by the spread of responses to multiple-choice tests and through listening to students’ explanations of concepts during the class-wide discussions. It is also possible with electronic voting systems to analyse the data kept on the computer about students responses to tests to identify specific areas of conceptual difficulty that recur from year to year. All this information can be use to shape subsequent teaching.

Online assessment tools can also provide invaluable quantitative and qualitative information to the teacher about student learning. These tools normally have an inbuilt reporting functionality. Two types of information are common to online assessment systems – class reporting and individual reporting. At class level, the teacher can identify questions that posed problems across a large cohort of students. From an analysis of student responses, areas of conceptual difficulty in the subject matter or poor questions can be identified and corrective action taken. At the individual level, the problem-solving strategies used by specific students can be unpacked and feedback targeted to that cohort. (Ashton et al, 2004). Online discussions also provide information about student learning. Monitoring and analysing these discussions enables teachers to identify areas of conceptual difficulty and to take remedial action by introducing new tasks or by suggesting further readings.

Marking and Grading

Some of the ways in which technology can support marking and grading have already been discussed in other sections of this wiki. In dynamic assessments using intelligent tutoring systems and simulations the measurement of performance will often be built in. Students will often receive feedback as a profile of correct and error responses as well as qualitative feedback about alternative approaches and about how to make performance improvements. Where objective tests are administered online the measurement of performance and the management of statistical data about individual and cohort performance can also be automated through appropriate programming of software. In relation to more open-ended tasks that require evaluative judgements of a more qualitative nature technology can really only aid and support marking and grading processes. Current research on computational methods for scoring essays highlights the difficulty of automating the marking of qualitative assessments (e.g. Landauer, Laham, Foltz, 2003).

Valentini, Neri and Cuchiarelli, (2003) provide a comparison of ten systems that have been used to support automated assessment of free text essay answers. These range from low level systems that analyse surface features of the text (e.g style of surface linguistic features) to those that carry out semantic analyses using corpus-based approaches to model building (e.g. analysing the deep discourse structure). These authors conclude their review by noting that ‘the most common problem…[in]… automated essay grading is the absence of a gold standard to calibrate human marks and of a clear set of rules for selecting master texts’. In other words, the difficulty that markers have in agreeing on what is a good essay answer and the difficulty they have in defining why an essay is good makes it difficult to program software to carry out this task. Greatest progress has been made where there are large samples of essays, all written on the same topic and already graded by humans that can be used to program the software. In such cases, high levels of correlation between the software and the human marker have been shown. But these considerations make this approach of limited value in HE generally.

Automated free text marking is a developing area with the potential for significant savings in staff time. However, there is still much work to be done before this approach becomes mainstream in HE. More fruitful is research directed towards the more tractable problem of supporting marking and grading where the system scores for different components of an essay (e.g. the structure of the argument) which helps the teacher to mark while but at the same time supports the teacher in providing feedback to students.

Implications for practice

Given that technology can support almost any stage of the assessment cycle (task design -assessment/interpretation and feedback/grading) it is important to identify where benefits can best be leveraged. A key issue for academic staff and their institutions is time and resources. Some software requires a large financial investment (simulation software, databanks of objective tests) and individual institutions will not have the resources to develop these. In disciplines where the curriculum is similar across institutions (e.g. medicine) publishers could make investments in assessment applications or consortia of institutions might develop and share these resources.

Any use of technology must be underpinned by clear objectives. Practitioners must ask themselves whether the technology being applied is intended to enhance student learning through feedback or to support marking and grading. These are not necessarily in conflict but care needs to be taken with assessment design and use of technology if the goal is to ensure that these objectives are not in opposition. One example of this might be where the assessment regime encourages students to focus their efforts on passing the test rather than on developing deep understanding of subject matter.

In this Wiki a set of principles have been proposed to guide feedback practices. These principles have been tested at module level across many disciplines within the Scottish Funding Council supported Reengineering Assessment Practices (REAP) project (see http://www.reap.ac.uk) which involved supporting their implementation using a range of technologies. Many HE institutions have incorporated these principles into their assessment policy documents and as a framework to support enhancement. The REAP project identified that the ways in which the principles are implemented differed depending on the discipline, the type of student (e.g. full-time, part-time, distance) and the teaching and learning context. For this reason, it is recommended that a ‘tight-loose’ approach to implementation be adopted (Thompson and Wiliam): while teachers should try to maintain fidelity to the pedagogy (the educational intent) behind each assessment principle (tight) the techniques of implementation should be tailored and adapted to suit the specific teaching and learning context (loose).

A key idea behind all the feedback principles is that the more active the students are and the more responsibility that they have in the implementation of a principle, the more empowering the educational experience. For example, a teacher might ‘clarify what good performance is’ (principle 1) for an essay writing task by providing students in advance of the assignment with a list of printed criteria. Alternatively, the teacher might organise a session where students are required to examine some example essays (e.g. produced by a previous student cohort) to identify which is better and why. The second approach would usually be more empowering than the first because the students would be more actively engaged in constructing, internalising and owning the assessment criteria. It is recommended, therefore, that in formulating applications consideration is always given to how responsibility might be shared with students so that they are active participants in assessment processes.

Success factors

The success of any technology can be evaluated in terms of the extent to which it supports and enhances student learning and the extent to which it maximises the return on investment for the programme of study and in relation to institutional objectives. There is a paucity of literature on how to carry out a systematic cost-benefit analysis of investments in technology to support teaching, learning and assessment in HE, although there some promising approaches are being developed (e.g. Laurillard, 2007: Nicol and Coen, 2003). In principle, the application of technology to assessment might be evaluated through input measures: for example, one might ask whether the technology application leads to a better use of staff time? Evaluations might focus on output measures such as ratings by students of their learning experience as happens in the UK National Student Survey, by analysis of achievement gains in examinations or through an analysis of retention statistics. However it is important to note that these measures will be confounded by other changes that are associated with technology application (e.g. changes in the methods of assessment, in the way the teacher interacts). From the teacher perspective, success of technology applications might be evaluated in relation to process measures such as the effectiveness of the technology in resolving real teaching and learning issues such as the need to personalise feedback to different kinds of learner or to address the demands of formative feedback when class sizes are large.

There is a need for more focused evaluations of technology application in HE. Otherwise there is a danger that technology will only raise the cost of teaching and learning without any real benefit. Also, some convincing evidence that the application of technology has benefit is necessary to convince institutions that there will be a return on their investment and to convince academic staff that change is worth the effort.

Issues and future directions

Future developments in technology-supported assessment are likely to occur at a number of levels. Firstly, big steps forward might will be realised by just making it easier to use existing technological tools so as to make them better suited for assessment purposes. Most assessment engines and virtual learning enviornments do not support confidence-based marking, tools to support peer critiquing activities are under-developed as are tools that support essay marking and feedback online or that link audio files (for audio feedback) seemlessly to text assignments. Tools such as wikis and blogs are not at present not seemlessly integrated into virtual learning environments or within other university systems. At the next level up, more work might be directed to the development of intelligent tools and simulations that model learning processes and that could provide intelligent feedback to students on their developing understanding. However this is an area that will require significant investment and collaborative development.

Looking further in the future there are many possibilities. Some researchers are looking at ways of supporting formative assessment using mobile devices. For example, Professor Richard Kimbell has shown how mobile devices might allow students to reflect on and capture their own learning as it happens using pictures, audio and video files with all these files being downloaded from a personal digital assistant to seemlessly create a multimedia record of learning (i.e. a portfolio). This is one example of innovative thinking.

In all areas of assessment, progress will depend more on how we conceptualise learning and assessment processes pedagogically than on the actual tools we use. Educators who beleive that learning is a collaborative endeavour and that students and teachers should be involved in the co-creation of course content will be looking for different technologies from those who believe that learning is about aquiring received knowledge. For the former, Web 2.0 tools to support the student generation of content and social networking will be a focus for future development as will new methods of assessing student contributions and collaboration.

Conclusion

This entry focuses on how technology can support assessment practices in higher education. It has been shown that technology can help teachers present and construct assessment tasks, make valid judgements of the student progress in learning, facilitate the provision of feedback and support the production and delivery of marks. Some principles of assessment have been proposed that might be used to help shift the balance away from thinking about assessment as involving only the act of judging student achievement (assessment of learning) to assessment as a process that actually supports long term learning and development (assessment for learning). This shift of thinking is necessary not only to ensure that teacher time spent on assessment yields maximum benefit in relation to what is learned but also to ensure that when students leave university they are able to continue learning without the need for teacher supervision.

References

Arthur, N (2006) Using student-generated assessment items to enhance teamwork, feedback and the learning process, Synergy: Supporting the scholarship of teaching and learning at the University of Sydney, 24, 21-23. Published by Institute for Teaching and Learning, University of Sydney. Accessed January 2008 at http://www.itl.usyd.edu.au/Synergy/article.cfm?articleID=283

Ashton, H.S., Beevers, C.E., Schofield, D.K. and Youngson, M.A. (2004), ‘Informative reports – experiences from the Pass-IT project’, paper presented at 8th International Computer Assisted Assessment Conference, Loughborough University, UK. Online. Available at: <http://sd.lboro.ac.uk/caanew/pastConferences/2004/proceedings/ashton.pdf>

Baxter, J. (2007), A case study of online collaborative work in a large first year Psychology class, From the REAP International Online Conference: Assessment Design for Learner Responsibility, May 29-31, http://ewds.strath.ac.uk/REAP07

Black, P. and Wiliam, D. (1998), Assessment and classroom learning, Assessment in Education, 5(1), 7-74.

Bennett, R.E. (1999), Using new technology to improve assessment, Educational Measurement: Issues and Practice, 18(3), 5-12

Bostock, S. J. (2004), Motivation and electronic assessment, in S Alexander and A Irons (eds) Effective Learning and Teaching in Computing, London: RoutledgeFalmer.

Boud, D. (2000), Sustainable assessment: rethinking assessment for the learning society, Studies in Continuing Education, 22(2), 151-167.

Boyle, J.T. and Nicol, D. J. (2003), Using classroom communication systems to support interaction and discussion in large class settings, Association for Learning Technology Journal, 11(3), 43-57.

Bull, J. and McKenna, C. (2004), Blueprint for computer-assisted assessment, London: RoutledgeFalmer.

Conole, G and Warburton, B (2005), A review of computer-assisted assessment, ALT-J Research in Learning Technology, 13(1), 17-31

Conole, G. and Danson, M (2004), Special Issue: Computer assisted assessment, ALT-J, Research in Learning Technology.

Cotterill, S., Bradley, P and Hammond, G.(2006), ePortfolios: supporting assessment in complex educational environments, in Bryan, C. and Clegg, K. Innovative Assessment in Higher Education, London: Routledge, Taylor and Francis Group.

Davies, P. (2003), Closing the communication loop on computerised peer-assessment of essays, Association for Learning Technology Journal, 11(1), 41-54

Davies, C.H.J. (2002), Student engagement with simulations: a case study, Computers and Education, 39, 271-282.

Freeman, M.A. and McKenzie, J. (2002) Implementing and evaluating SPARK, a confidential web-based template for self and peer assessment of student teamwork: benefits of evaluation across different subjects, British Journal of Educational Technology, 33(5), 553-572.

Haigh, M (2007), Sustaining learning through assessment: an evaluation of the value of a weekly class quiz, Assessment and Evaluation in Higher Education, 32(4), 457-474

Hounsell, D. (1997) Contrasting conceptions of essay-writing, in: F. Marton, D. Hounsell and N. Entwistle (eds) The Experience of Learning Edinburgh: Scottish Academic Press.

Landauer, T.K., Laham, D. and Foltz, P. (2003) Automatic essay assessment, Assessement in Education: Principles, Policy and Practice, 10(3), 295-308.

Meece, J.L., Anderman, E.M. and Anderman, L.H. (2006), Classroom goal structure, student motivation and achievement, Annual Review of Psychology, 57, 487-503.

Nicol, D, (in press), Assessment for Learner Self-regulation: Enhancing achievement in the first year using learning technologies, Assessment and Evaluation in Higher Education

Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education, 31(1), 53-64.

Nicol, D.J. and Boyle, J.T. (2003) Peer Instruction versus class-wide discussion in large classes: a comparison of two interaction methods in the wired classroom, Studies in Higher Education, 28(4), 457-473.

Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199-218.

Nicol, D. J. & Milligan, C. (2006), Rethinking technology-supported assessment in terms of the seven principles of good feedback practice, in Bryan, C. and Clegg, K. Innovative Assessment in Higher Education, London: Routledge, Taylor and Francis Group.

Norton, L. S. (1990) Essay writing: what really counts?, Higher Education, 20(4), 411-42.

Orsmond, P., Merry, S. and Reiling, K. (2002) The use of formative feedback when using student derived marking criteria in peer and self-assessment, Assessment and Evaluation in Higher Education, 27(4), 309-323.

Palloff, R.M. and Pratt, K. (2005), Collaborating online: learning together in community, San Francisco: Jossey-Bass.

Paris, S. G. and Turner, J. C. (1994), Situated Motivation' in P. R. Pintrich, D. R. Brown, and Weinstein, C. E. (eds) Student Motivation, Cognition and Learning. Hillsdale, New Jersey: Lawrence Erlbaum Associates.

Pelligrino, J.W., Chudowsky, N. & Glaser, R (2001) Knowing what students know: the science and design of educational assessment, National Academy Press.

Rust, C., Price, M. and O’Donovan, B. (2003) Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education, 28(2), 147-164.

Sadler, D.R. (1989) Formative assessment and the design of instructional systems, Instructional Science, 18(2), 119-144.

Valenti, S., Neri, F. and Alessandro, C. (2003) An overview of current research on automated essay grading, Journal of Information Technology Education, 2, 319330. White, B.Y. and Friederiksen, J.R. (1998) Inquiry, modeling and metacognition: making science accessible to all students, Cognition and Instruction, 16(1), 3-118

Whitlock, D (2006) Electronic assessment: marking, monitoring and mediating learning. International Journal of Learning Technology, 2(2/3), 264-276.

Whitlock, D., Watt, S.N.K., Raw, Y and Moreale, E (2004) Analysing tutor feedback to students: first steps to constructing an electronic mentoring system, ALT-J, 1(3), 31-42.

Yorke, M (2003) Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice, Higher Education, 45(4), 477-501.