About The CIT-eA Project
This guide contains the outputs of the CIT-eA e-assessment project. It contains:
- A toolkit – to support those implementing e-assessment
- Case Studies – based on the experience of the project
- Collaborative frameworks – to support e-assessment
- Outline proposals for a national e-assessment service – to support the development of e-assessment in the FE sector
The aims of the project were ambitious:
| “The aims of the project are ambitious” |
- Explore and identify the barriers to the adoption of e-assessment and identify workable solutions.
- Develop resources, tools and products that will improve the operational efficiency and effectiveness of providers
- Create processes, to enable improved uptake of existing e-assessment options as well as drive future development.
You can find out more about the background and rationale for the project by consulting the ‘Background to the CIT-eA Project’ at the back of this guide.
Benefits
There are a great number of benefits to be gained by adopting e-assessment; in the ‘Getting Started’ section of this guide we list some of them. There is potential to be able to improve the speed, quality and consistency of assessment as well as feedback provided to students. It can also play a vital part in overcoming the pressing problems of teaching greater numbers of student from more academically diverse backgrounds with limited resources. These are big claims to make and the remit of our project included ‘explore and identify the barriers to adopting e-assessment and identify workable solutions’. This is what we hope you will find in this guide.
Problem Areas
E-assessment is a subset of the wider field of e-learning, in 2004 a group of prominent researchers[1] observed:
“The current situation can be best described as high-level ambitions with poor implementation.”
In the 10 years or so since, much has changed and learning technology is now firmly on the senior management agenda. As the Jisc BOLT project has observed, the main challenge is now to make the organisational and cultural changes that are needed, and that needs top-down management drive and engagement as well as bottom up innovation and creativity. This is a tricky problem, the Jisc BOLT project describes it like this:
“In most organisations the ability to accept and embrace technology in learning and teaching requires a major cultural change. Staff are typically fearful of change and methods need to be applied to try and overcome these barriers”
This kind of problem is sometimes described as a ‘wicked design problem[2]’, meaning that it’s hard to solve effectively because there are lots of conflicting ideas, values and interest groups involved. The Jisc BOLT project provides good advice for dealing with these issues and the internal politics of an organisation.
| “a major cultural change” |
Approaches
Taking the above ‘Problem Space’ as our starting point we have approached the task in a way that looks at ‘how things work’ and what needs to change to make the best use of the available technology. This means taking a critical approach to both the claims made for the technology and the way things currently work in our current college systems. In this situation we take it as self-evident that there are two main ingredients in developing realistic and sustainable solutions to adopting e-assessment:
Systematic methods – understanding how the component pieces of the Scottish FE system work (nationally, regionally, internally in the college, students, employers, political and economic factors). This approach has been recognised as essential in recent studies:
“[e-Assessment] touches on many aspects of institutional practice and is a matter of importance for staff (and hence their representative professional bodies) in many different roles: managerial, learning and teaching, learning support, IT and administration.”
Jisc Educational Management of Assessment (EMA) Landscape Report p.6
Creative interventions – understanding the ‘systemic’ characteristics of the problem is a sound foundation for creativity. The trick is to develop actions that can take these into account and make a positive change in a particular local context. This means listening and negotiating to make step-by-step progress that can generate lasting change and be the basis for further development.
Scope
The scope of the project was determined by operating within the environment of qualifications, which are developed and regulated by the Scottish Qualifications Authority (SQA), and offered mostly in colleges. Although our focus is on the Scottish further education college sector, much of this guide will apply equally to other sectors such as higher education, community based learning and work-based learning etc.
We focused on assessments of Higher National Units, drawn from the qualifications area for Business related subjects. In this context the learning outcomes, assessment criteria, evidence requirements and conditions for assessment are specified in the ‘unit descriptors’. These also provide guidance about assessment methods and assessment instruments and what evidence is needed to show achievement by students. In addition, SQA often provides ‘exemplar assessments’ – where sample instruments of assessment and supporting materials are supplied for use by a centre. The colleges’ delivery of these SQA units (usually as part of larger subject programmes) is subject to a number of internal and external quality management procedures, which we shall discuss later in this guide.
Towards a Solution
The outputs of our project are intended to contribute to and support the growing community of those involved in designing, developing and supporting e-assessment in Scottish education and beyond.
| “the words ‘Creative’ and ‘Systematic’ … are important” |
The inclusion of the words Creative’ and ‘Systematic’ in the subtitle of this guide are important indicators of the qualities we think that are needed to make progress in adopting e-assessment generally. You can find out more about our background thinking in the ‘About the Project’ section at the rear of this guide. In this guide we argue that in order to make progress it is essential to understand the context in which you are working in terms of the technology, students and institution etc. and the limitations that these impose. The mind-set we seek to develop in the reader is similar to that required in engineering – an enquiring, analytical, systematic and problem-solving attitude. It understands that while problems may seem similar the actual context may require very different and often ingenious solutions – that’s the creative bit. Solutions are arrived at after understanding how the different ‘systems’ interact (students, lecturers, college processes and procedures and IT infrastructure, the SQA, local working cultures etc.).
Below we describe how the project outputs help work towards finding practical and sustainable solutions:
Toolkit – this is comprised of the numbered sections in this guide and is based on the ADDIE model of systematic instructional design[3] – Analyse, Design, Develop, Implement, and Evaluate. It provides guidance and practical tools for those involved in the actual creation of e-assessments and it encourages the development of critical analysis skills as the foundation for effective problem solving.
Case Studies – showing the development of some real-life e-assessment solutions from start to end.
Collaborative Framework Proposals – discussing ways that internal and external collaboration can support the adoption of e-assessment (within colleges, with other colleges, with employers, with the private sector etc.)
Outline proposals for a national e-assessment service – in an era of continuing financial pressure on education these proposals build on the collaborative framework in order to find ways of making the most of what we already have.
[1] Integrated E-Learning: implications for pedagogy, technology and organisation, Jochems W., van Merriënboer J. and Koper R (2004) London: Routledge and Falmer
[2] See http://en.wikipedia.org/wiki/Wicked_problem
[3]The Bolt Jisc funded BOLT project from Border college has a nice introduction to the ADDIE model at this link: http://www.boltlanding.whitecreativecompany.co.uk/elearningstart/instructional-design/