User:Pbacsich/benchmarking e-learning

From Wikipedia, the free encyclopedia

This page is for the moment static and has been transferred to Wikipedia .--Paul Bacsich 13:22, 9 September 2006 (UTC)

This page is a guide to the main systems worldwide for benchmarking e-learning. It is arranged in alphabetical order of name of system, with some additional sections on History and Themes.

Contents

[edit] History

Benchmarking e-learning appears to have arisen largely since 2000 but is rapidly growing in amount of work. For a more general history of e-learning see History of virtual learning environments


[edit] ACODE

ACODE is the eponymously named benchmarking scheme under development by the Australasian Council on Open, Distance and E-Learning, whose web site is at http://www.acode.edu.au/.

Development of this started in 2004 as a pilot project. There are some fragmentary references to it on the web including ACODE press releases of 18-19 May 2006 and 16-17 November 2005. A potential ACODE scheme is mentioned in a presentation by Paul Bacsich in November 2005 at the University of Sydney, significant because Bacsich is a benchmarking analyst also cited in one of the ACODE press releases.

In another intriguing hint, an ACODE statement of January 2006 notes that in "December 2005: Dr Stephen Marshall (Victoria University of Wellington) surveyed members on how well supported students are with regard to IT support and access to helpdesk". Analysis of his institutional affiliation and email address makes it clear that this is the Dr Stephen Marshall the author of the e-Learning Maturity Model methodology for benchmarking e-learning.

[edit] BENVIC

BENVIC is a methodology for benchmarking e-learning developed under an EU project, also called BENVIC (in full, Benchmarking of Virtual Campuses) in the era 1999-2001. There is a project web site still at http://www.benvic.odl.org/ – but it has not been updated since February 2002. The BENVIC consortium was led by UOC, the Open University of Catalonia and had a strong set of partners (including University College London in the UK). However, for various reasons including retirement of key staff the work does not seem to have continued – or least web searches indicate that follow-up work is not evident.

The BENVIC system has eight core meta-indicators. These are:

  • Learner Services
  • Learning Delivery
  • Learning Development
  • Teaching Capability
  • Evaluation
  • Accessibility
  • Technical Capability
  • Institutional Capability

All of them, with the exception of Accessibility, are the kind of top-level groupings that one finds in other methodologies. UK readers should also note that Accessibility does not mean only the narrow sense of SENDA but also covers many aspects of Widening Participation.

Each of these eight meta-indicators is associated with a range of assessment measurements (indicators) which enables BENVIC users to carry out an initial benchmarking diagnostic. The assessment measurements are comprised of three types:

  • structural measurements
  • practice measurements
  • performance measurements.

There are in total 72 structural and practice indicators – which is rather more than in many systems but less than some others. Whether or not this is an issue depends crucially on how difficult it is to score these indicators and whether they are all "compulsory".

Indicators (other than the performance ones which are metrically based, i.e. numeric) are scored on a scale of 0-2. Many other systems use a scale of 1-5, but one could use a natural mapping of 0 to 1, 1 to 3, and 2 to 5.

There is a more detailed analyis of BENVIC on the web.

[edit] CHIRON

CHIRON is an EU-funded project (under the Leonardo programme) whose aim is "to develop reference material presenting and analysing research outcomes, experiments and best practice solutions for new forms of e-learning, based on integration of broadband web-, digital TV- and mobile technologies for ubiquitous applications in the sector of non-formal and informal life-long learning". There is a CHIRON web site at http://semioweb.msh-paris.fr/chiron/. They tend to use the phrase "u-learning" rather than "e-learning", where "u" denotes "ubiquity",

As part of this brief, CHIRON appears to be developing a benchmarking methodology. This is specifically referred to in Work Package 7.

There are 11 criteria, divided into a total of 216 indicators. The criteria are as follows:

  • 01 Goals and Objectives of the course (12 indicators)
  • 02 Institutional Support (14 indicators)
  • 03 Course Development (50 indicators)
  • 04 Course Structure (12 indicators)
  • 05 Course Content (25 indicators)
  • 06 Teaching/Learning (19 indicators)
  • 07 Student Support (18 indicators)
  • 08 Faculty Support (4 indicators)
  • 09 Evaluation and Assessment (24 indicators)
  • 10 Accessibility (26 indicators)
  • 11 Language (12 indicators)

Most of the indicators are best described as specific and rather detailed e-learning standards and guidelines (for example on house style, usability, etc). The remaining few are drawn from a range of sources, including from the Quality on the Line criteria developed in the late 1990s by the Institute for Higher Education Policy in the US.

[edit] ELTI

ELTI is the name of one of the methodologies being trialled in the Higher Education Academy Benchmarking Pilot, by three universities:

  • University of Bristol
  • University of Hertfordshire
  • University of Wales Institute Cardiff.

The version of ELTI on which the trials were originally based is the JISC version held at the JISC site http://www.jisc.ac.uk/index.cfm?name=project_elti. The document ELTI Audit Tools is the most directly relevant to what is commonly accepted as benchmarking, especially section 1 on so-called "Institutional Factors"; however, some at least of the pilot sites took a broader view.

[edit] eMM (Maturity Model)

The phrase "eMM" is the commonly used abbreviation for the longer phrase "e-learning Maturity Model". This is the name of one of the methodologies being trialled in the Higher Education Academy Benchmarking Pilot, by the University of Manchester.

Full details of eMM can be found at the master site http://www.utdc.vuw.ac.nz/research/emm/index.shtml.

Version 2 of eMM has changed considerably from the Version 1 of 2003, as noted at http://www.utdc.vuw.ac.nz/research/emm/VersionTwo.shtml.

There is also now a blog, at http://artemis.utdc.vuw.ac.nz:8000/pebble/tags/eMM.

Note that the eMM and associated documentation is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.

[edit] E-xcellence

E-xcellence (pronounced “E-excellence”) is an EU-funded project run by EADTU (the European Association of Distance Teaching Universities) with the assistance of 12 other partners. It has a project web site at http://www.eadtu.nl/e%2Dxcellence/.

E-xcellence started in January 2005 and is due to conclude in November 2006, with a major launch at the EADTU conference in Tallinn, Estonia, 23-24 November 2006. Originally E-xcellence was not envisaged as a benchmarking methodology but as a quality monitoring tool, but at arbout a year into the project there was a shift in emphasis and benchmarking is now one of the aims envisaged for E-xcellence. In fact there are three orientations of the methodology:

  • Assessment tool (at both institutional and programme level) (i.e. benchmarking)
  • Quality improvement tool (internal quality care system)
  • Accreditation tool for accreditation

Expected outcomes include:

  • List of criteria for "good" e-learning (i.e. setting standards of excellence and indicators for validation)
  • Manual on good practices (a web-based guide)
  • Quality assurance system (internal validation based on the standard of excellence)
  • Reports on pilots which will test the validation approach
  • Establishment and training of a visitation team both for quality assurance and for accreditation (to be seen as distinct procedures)

There is little public information so far but it is believed that the basis of the E-xcellene benchmarking methodology is as follows:

  1. Based on criteria, with (at the time of writing) 20 threshold criteria and a further 30 excellence criteria
  2. Each criterion is a bundle of indicators
  3. Criteria are not yet scored, but this may be added by the time of the final release
  4. The threshold" level can be determined by a self-audit, but the "excellence" level requires in addition a visit from an expert team.

[edit] IQAT

IQAT - pronounced "eye-cat" - is a newish benchmarking and quality enhancement methodology developed by Hezel Associates, a well-known firm of e-learning consultants, in conjunction with a number of university partners. Unlike most benchmarking methodologies it even has its own web site. The methodology was formally launched in June 2006.

It describes itself as "a web-based tool to track and benchmark institutional data systematically across time and among peer institutions".

The work is being done in partnership with NUTN, the National University Telecommunications Network, and with sponsorship from Cisco Systems. NUTN has for some time had a major interest in quality and more recently benchmarking, as demonstrated for example by the topics and speakers at their 2006 conference. In particular there was a launch presentationof IQAT. One of the leading organisations in NUTN is Michigan State University, who have a leading role in organising an upcoming (October 2006) conference on quality in Beijing, China, entitled the First International Forum on Online Education: Quality Assurance, with a range of quality and benchmarking experts as speakers.

However, probably since it is a commercial and proprietary tool, little further public information appears to be available on IQAT. There are some resources and a glossary on the IQAT site but these appear in the main to be about distance learning not about benchmarking it, with the exception of a report entitled The Status of Benchmarking at Higher Education Institutions (in the US).

[edit] MASSIVE

MASSIVE is an EU-funded project coordinated by the University of Granada - the web site is at http://cevug.ugr.es/massive/.

The prime aim of the project is to design a model of support services for European universities engaged in e-learning. This is being done in a cooperative way involving a large network of organisations, the members of the MASSIVE consortium, tothether with stakeholders not directly participating in the project, who liaise via a Strategic Advisory Committee. The university members of MASSIVE include:

  • Universidad de Barcelona
  • University of Bergen
  • Budapest University of Technology and Economics
  • University of Edinburgh
  • EuroPACE and its member universities

Student involvement is handled via ESIB, the National Unions of Students in Europe.

A key outcome of MASSIVE is to a promote a peer review evaluation approach, based on models widely tested in the university partners. Via Peer Review Visits, those in charge of the best support services practices will help each university to refne and improve their support services for e-learning. At this point the project becomes very similar to a benchmarking project.

[edit] MIT90s

The MIT90s framework has been used by the University of Strathclyde (one of the 12 institutions in the Higher Education Academy Benchmarking Pilot) to structure its approach to benchmarking e-learning.

The framework was developed by Michael S Scott Morton as part of the work of the "MIT90s" initiative which flourished at MIT in the early 1990s. Michael S Scott Morton is now Professor Emeritus at MIT in the Sloan School of Management. The work is cited under various names: it is correctly entitled The Corporation of the 1990s: Information Technology and Organizational Transformation, edited by Michael S Scott Morton with an introduction by Lester Thurow (Oxford University Press, USA), published February 1991, ISBN: 0195063589.

There has been some confusion over the correct name of the initiative and the framework. Readers will find "MIT 90s", "MIT90" and even "MITs 90" in various references.

The MIT90s framework has been central to a number of JISC and related studies (including from DfES) on adoption and maturity. In UK post-16 e-learning terms, probably the most successful use of this is in the RAIIE study led by David Nicol of the University of Strathclyde, funded by JISC during the period April 2003 to May 2004, which produced a final report A Framework for Managing the Risks of e-Learning Investment. There is also a substantial strand of work in Australia associated with the names of Philip Yetton, Anne Forster, Sandra Wills and others, some of it funded by DETYA. This also used the concept of strategic aligment described below.

The MIT90s initiative developed several companion pieces of work o which two are from Venkatraman: transformation levels and strategic alignment. Professor N Venkat Venkatrama is now Professor of Management at Boston University School of Management (and also a visiting professor at London Business School) but was at the Sloan School of Management at MIT during the period of relevance. His work on IT-Induced business reconfiguration is Chapter 5 (pages 122-158) of the main book referred to above.

The Venkatraman thesis is that business use of IT passes through five levels, differing in both the degree of business transformation and in the range (and amount) of potential benefits. The levels are:

  • Localised exploitation
  • Internal integration
  • Business process redesign
  • Business network redesign
  • Business scope redefinition.

Levels 1 and 2 are called evolutionary levels – levels 3, 4, and 5 are called revolutionary levels.

This has been applied to educational systems, especially in the schools sector, by Becta and DfES.

In passing, it is interesting that this is one of the first situations where a 5-point scale has been used in a situation akin to benchmarking.

The MIT90s framework could have relevance to e-benchmarking frameworks. In particular, the latest version (2.0) of the Pick & Mix methodology uses the MIT90s framework for tagging its criteria in the Pick&Mix 2.0 release of which a beta description is now available.

It is an interesting question as to whether the MIT90s framework could have wider relevance to an e-benchmarking framework.

[edit] OBHE

OBHE is an eponymous benchmarking methodology run by OBHE, the Observatory on Borderless Higher Education. The Observatory is a joint initiative of ACU, the Association of Commonwealth Universities and Universities UK, the association of all UK universities. The Observatory is now 100% funded by subscriptions and consultancy, and has over 130 institutional subscribers from more than twenty countries. It offers a wide range of services of which benchmarking is one. Within the benchmarking offering is a range of sub-offerings, of which one is deployed for the Higher Education Academy clients.

The OBHE methodology is a collaborative benchmarking methodology where a group of institutions get together and jointly agree relevant areasa of interest (in this case, within the e-learning space) and in a later phase, look for good practices. A succinct description of the variant of their methodology used for the Higher Education Academy Benchmarking Pilot is here.

The methodology goes back to benchmarking work done for European and Commonwealth universities in the 1990s. A related study to the Higher Education Academy Pilot is the work done for OECD to produce the report E-learning in Tertiary Education: Where Do We Stand? As reported in the Proceedings of the OECD Conference on Post-Secondary E-learning, "This was based on a detailed, qualitative survey of current e-learning practice in 19 institutions of higher education in North America, South America, Europe and the Asia-Pacific region. The information gathered through this survey was complemented by quantitative data collected by OBHE from the 500 members of the Association of Commonwealth Universities and of Universities UK."

[edit] Pick&Mix

Pick&Mix (in the past called "Pick & Mix" with spaces) is the name of one of the methodologies being trialled in the Higher Education Academy Benchmarking Pilot, by three universities:

  • University of Chester
  • University of Leicester
  • Staffordshire University

The version of Pick&Mix on which the trials were originally based is version 1.0, first described in a public domain document from the ALT-C 2005 conference and refined slightly to version 1.1 for the Higher Education Academy. During the pilot, after the criterion-setting phase, this was updated to version 1.2.

The release being offered to HEIs for Phase 1 is version 2.0. A beta version of this is described here.

Version 2.0 has benefited substantially from input from the pilot user group of the Universities of Chester, Leicester and Staffordshire, who share in the moral rights of authorship. Released versions of Pick&Mix are in future to be put into the public domain via a Creative Commons license (as was version 1.0 and a summarised literature search).

Pick&Mix was first developed in 2005 after an extensive literature search to suit the needs of Manchester Business School for a comparative methodology for benchmarking e-learning, and a beta version of 1.0 used for a study of 12 comparable institutions to Manchester Business School. The full study is and remains confidential to MBS but a presentation of the highlights was made at the University of Sydney in November 2005.

Further information on Pick&Mix including a range of presentations and papers, and material on related methodologies, within the "critical success factors" tradition of benchmarking can be found here.

The recognised abbreviation for Pick&Mix (e.g. as used in tables) is "PnM", although "P&M" is sometimes seen also.

From now on this is still in preparation.

[edit] Vocabulary

There are a number of terms that are used in benchmarking in general and in benmarking e-learning in particular.