This guide has been produced as part of a 16 month e-assessment project carried out between 2014 – 15 and steps through the stages of implementation and critically analyses the issues involved. The intention is that that this guide will play a part in supporting the growing community of those involved in designing, developing and supporting e-assessment. The project was called Creating Innovative Technology – enhanced Assessments and used the acronym CIT-eA and was been led by the City of Glasgow College in a diverse partnership comprised of the following participants:

• City of Glasgow College
• Scottish Qualifications Authority (SQA)
• College Development Network (CDN)
• Student Participation in Quality Scotland (SPARQS)
• Edinburgh College
• Borders College
• Ayrshire College
• Jisc
• Colleges E-Assessment Group (CEAG)
• Walter Patterson Consultancy

CIT-eA Project Aims

The general aims of the project were ambitious and included:

1. Explore and identify the barriers to the adoption of e-assessment in the college sector in Scotland through practical work and by adopting a partnership approach identify workable solutions. Create processes, to enable improved uptake of existing e-assessment options as well as drive future development.
2. Develop resources, tools, products and processes that will improve the operational efficiency and effectiveness of providers.

CIT-eA Project Objectives:

The specific objectives are to create:

1. A toolkit of resources to enable greater flexibility and efficiency in delivery, as well as redefining the learner experience by incorporating more authentic and valid assessment approaches to improve learner engagement and employability
2. Collaborative Frameworks for implementation, demonstrating how an educational institution can work with internal and external stakeholders to overcome barriers to e-assessment and drive innovation.
3. A case study with HN Business
4. Identify Processes, which facilitate a move towards a national shared e-Assessment service approach, reflecting the recommendations of the McClelland Review (2011) for the public sector in Scotland.

Purpose, scope and audiences

This guide aims to provide practical guidance to those involved in changing their existing assessment practices from ‘traditional’ paper-based and face-to-face models to ones that make more use of technology. Public education systems in Scotland, the UK and elsewhere are all still in the process of making the transition into the digital realm. Our aim in producing this guide is to provide realistic and practical help in making the transition and to assist those involved to systematically evaluate their own working contexts in order to develop creative and effective solutions.

The audiences for this guide include:

• Lecturers and learning technologists[1] and support staff in the Scottish Further Education sector
• Those responsible for developing and guiding institutional strategy,
• National policy and funding developers and administrators.

The scope of the project was determined by operating within the environment of qualifications, which are developed and regulated by the Scottish Qualifications Authority (SQA), and offered mostly in colleges. We focused on assessments of Higher National Units, drawn from the qualifications area for Business related subjects. In this context the learning outcomes, assessment criteria, evidence requirements and conditions for assessment are specified in the ‘unit descriptors’. These also provide guidance about assessment methods and assessment instruments, which would be suitable for generating evidence to show achievement by students. In addition, SQA often provides ‘exemplar assessments’ – where sample instruments of assessment and supporting materials are supplied for use by a centre. The colleges’ delivery of these SQA units (usually as part of larger subject programmes) is subject to a number of internal and external quality management procedures.

Although our focus is on the Scottish further education college sector, much of this guide will apply equally to higher education, community based learning and work-based learning etc. The inclusion of the words Creative’ and ‘Systematic’ in the subtitle of this guide are important indicators of the qualities we think that are needed to make progress in adopting e-assessment generally.

An important observation to make here, that we shall pick up later in this guide, is that small changes in assessment practice using technology can produce big benefits and changes. A key element of our approach is that we are focusing on the ‘systemic’ nature of the changes needed to support the effective use of technology in assessment. So, in addition to exploring the technical aspects of e-assessment we also examine the wider contextual factors that are critical to a successful implementation. This approach is based on evidence from a range of research initiatives indicating that skills shortages, lack of time and institutional incoherence are the major obstacles in this area. These, together with the considerable commercial and political hype surrounding the use of technology in education require a more considered and holistic approach, rather than focusing on narrow technical matters.

In the process of undertaking the project it became clear to us that changes to assessment practice by using technology tends to have what we have come to call a ‘ripple effect’. A change in one part of the system affects other parts and to make a change in one place means making changes elsewhere as well – that’s how systems work. Changes here can spread out to the practices of teachers, student learning, the design of courses and institutional processes. In retrospect, this is hardly surprising as assessment is at the ‘business end’ of our public educational institutions and national systems – so any change here is bound to have wider effects. In this guide we have tried to capture these wider connections and how they work. This has also made us realise that adopting e-assessment can be an important and effective component of any change strategies to improve educational provision.

Approach

In developing this guide and completing the project work we have based our approach on two fundamental observations:

1. Educational outcomes are strongly affected by contextual factors like subject matter, students, teachers, institutional cultures, resources etc. Education is also subject to constantly changing agendas set by powerful political and economic forces[2]
2. Formal education, the kind that happens at colleges and universities, is a ‘workplace’ that employs teachers and support staff and a ‘business’ that consumes money in exchange for providing a service to students or ‘customers’, who will then use the ‘products’ of their education in society i.e. the knowledge, skills, aptitudes and – crucially – the certificates that signify they have reached the stated standard.

It is this ‘certificated’ aspect of formal education that is important to understand and how this can affect the implementation of e-assessment. The certificate is what gives formal education a currency or value in society, it indicates not just that the learner has reached a certain level of skills and knowledge but also that institutional and national quality systems stand behind the certificate to assure that level has in fact been achieved. Thus, any changes to assessment methods (with or without technology) goes right to the heart of formal education in our society and acts as a highly efficient ‘lightning conductor’ to reveal vital underlying factors (attitudes, skills, educational philosophy, and personal values etc.) that are normally invisible.

Taking this as our starting point we have employed a number of well-established concepts to help shape our work. We briefly list the main ones below to give the reader an idea of ‘where we are coming from’.

Concepts

Creativity: an important factor in designing and developing solutions to difficult and complex problems. It is also an important component of being a good teacher – the ability to be able to use a solid foundation of experience and knowledge to improvise and adapt to reach a positive outcome.

Systems Theory: stresses that it is important to understand how different parts of an institution relate to each other, and to external systems like employers and the Scottish Qualifications Agency (SQA) that oversees quality in the delivery of SQA qualifications in the colleges. This means analysing what may affect our proposed changes, and these will often be non-technical factors, as we have found in our case studies.

Socio-Technical Design: Is especially useful in understanding that success in adopting a new technology in a workplace is strongly affected by non-technical matters such as working cultures and management styles.

Action Research: A general problem solving methodology (there are many varieties) that includes cycles of analysis, activity and reflections to solve particular problems often carried out in participation with others affected by the same problem. In the course of the research the understanding of the problem may change and novel solutions may be found that were not visible at the outset of the process. This requires an enquiring, responsive and creative management style, one that has a lot in common with engineering methods and we can see here a link back to the need for creativity as well.

Learning Design: The work of Diana Laurillard and others in the field of ‘Learning Design’ has strongly influenced. In this connection our simple e-assessment design template is in fact a learning ‘design pattern’ generator[3]. This technique is borrowed from the world of art and design practice – especially architecture, this all sounds a lot fancier than it really is! The template encourages teachers to record in a ‘lightly structured’ manner the main details of the problem they are trying to solve, the main contextual factors involved and the proposed solution. Broadly using a structure like this:

• Name for the pattern
• Description of the problem/activity
• Context
• Actions and elements that play a role in coming to a solution
• Solution, itself expressed succinctly in terms of activities and resources etc.

[1] ‘learning technologist’ is a term that covers a multitude of roles and skills in education – in this context we are thinking specifically of those people involved in supporting teachers in their use of assessment related tools in VLEs and e-portfolios etc.

[2] This situation is very similar to that faced by our public healthcare systems.

[3] Educational design and networked learning: Patterns, pattern languages and design practice by Peter Goodyear gives a wide-ranging overview of this research area.