Mechanisms of Technology Implementation and Evaluation

Alan Jacobs, SCC
Chair

Jamie Cavalier, DIST
Group Support

Maria Harper-Marinick, DIST
Margaret Hogan, CGCC
Patricia Honzay, DIST
Irwin Noyes, SCC
Ann Oehmke, CGCC
Toni Rodriguez, GWCC
Karen Schwalm, GCC
Other participants in this Ocotillo Group were: 
  Toni Rodriguez, Karen Schwalm, Cynthia Leshin, 
  Maria Harper-Marinick, Naomi Story, and Patricia 
  Honzay.

Preface

There are two early motivations for this Ocotillo group: To improve the support for technology innovations and to learn how (and when) to expand successful projects into department- or college- or District-wide implementations.

Our group, Mechanisms of Technology Implementation and Evaluation, was given the charge to "discuss and define the mechanisms for infusing, or not infusing, technology after "it" has been tested or developed into the MCCCD." Our task, then, is to describe the processes of technology implementation in MCCCD. But our task does not end with description; inherent in our charge is a decision: to infuse or not to infuse. Part of our name, after all, is Evaluation.

Evaluation of innovations seems like a pretty risky concept. Some would argue that too close an evaluation of innovations can have a chilling effect, to the extent that fewer (and tamer) innovations are attempted. Others will counter that we need to learn from each others' mistakes as well as from successes. How would we differentiate successes from failures unless evaluations were performed?

This Ocotillo group contends that we can build on each others' successes and learn from each others' failures. We recognize that evaluations might work well in an atmosphere of trust and support, where it might be disastrous in a hostile environment.

In this report we intend to present a description of technology infusion as it occurs within the District. In this description we hope to present a vocabulary and a schema for talking about technology infusion. (In fact, both the vocabulary and schema apply to many more kinds of infusion than just technology infusion.) And with the vocabulary and the schema, we hope to provide innovators and managers with a common way to think about particular innovations and what those innovations need in order to succeed.

It is, then, with both fear and trembling, that we present the following report. Not because it may provide leaders with a common language for encouraging the success of innovations, but because that same language will also be used to say "no" to the further infusion of certain innovations. In addition to providing a context for an individual to improve the chances for the success of his/her own innovation, it also provides a context for an individual innovator to say "no" to the further infusion of his/her own idea.

In submitting this report, we want to encourage innovation. We believe that more innovations will be more successful if all the parties concerned have a clearer understanding of the process of infusion: that there may be several stages, that the challenges in each stage are different, and that different outcomes may be reasonably expected from the different stages.

Our goal is to learn to support different innovations well, differently.

Two Cultures

Instructional decisions regarding technology (actually, most instructional decisions) take place at the intersection of two different cultures. One is the culture of community and consensus.

The other is that of independent, professional contractor. Most notably evidenced by the large numbers of part-time faculty, but it is also seen in the myriad of small and large instructional judgments that are made by all faculty . The instructor may decide to change the emphasis in the standard course outline. The instructor may decide to include a technology component in a course, as a way to solve an instructional problem that has occurred, only to choose a different solution the following semester.

Given that individual faculty judgment is a strong aspect of the culture, the schema for technology infusion must not ignore the individual. Given that a college is also a community, the schema must also reflect the striving for a sense of consistency and commonality.

The co-existence of these two cultures implies that we might consider two different schema for describing technology infusion.

From an individual innovator's point of view, one first gets an idea, experiments with it to learn more about it, tries it out on a small scale and, if successful, tries it on a larger scale. During this time the idea may be revised or abandoned if it isn't working. On the other hand, new vistas may appear from initial, tentative uses. In fact, small scale implementations may reveal profound side effects which encourage or discourage further work.

From the organization's point of view - the department or college or District - an innovation generally catches on sporadically over time. Many individuals are first interested in learning about the idea, and later in testing it. Much later the idea may be in routine use at some locations and, at the same time, other individuals are just getting the idea and wanting to experiment.

Since technology is continually changing, we will always be asking the questions: "What technology should we use? Where? When?" Therefore, we really should come to grips with how we make those decisions. We should employ procedures and methodologies that encourage us to ask appropriate questions, and to avoid inappropriate ones, so that we can support different innovations well.

Schema I: From Idea to Reality

We propose the following schema as a guide in understanding the development of ideas for the application of technology to instruction from the individual's point of view. This schema is intended to be a classification schema. It may be used to develop methodologies and procedures for evaluating technology decisions.

In this schema there are five zones. The first is the zone of Getting the Idea. Sometimes this can happen in a reflective state, but it most often occurs in contact with others. In MCCCD, Ocotillo has provided one forum for sparking ideas. In any case, an idea hits home, sparks further ideas, and leads the individual to want to learn more.

The second zone is Learn More about the idea through exploring, reading, research, etc. In this zone one just wants to learn about a given technology. Perhaps no instructional problem is identified at this point. The goal is just to sit behind the wheel and see where it takes you. In some cases the experiment may be to take a technology developed for one purpose and see if it can be put to other uses. In this zone, a person explores the limits of the technology and gets a feel for its potential uses. If the technology is being used elsewhere, the fastest way to learn may be to combine individual exploration with a close examination of its current use.

Zone three is the Small Scale Implementation. After playing with the technology, a potential use may be identified. The small scale implementation is a live test of that potential use. This test may be as modest as a single assignment in a course, or as extensive as a theme around which a course is organized. However, the small scale implementation rarely involves more than a single instructor in a single course. Small scale implementations tend to be idiosyncratic and contain high levels of personal involvement and time commitment. In this zone, the instructor often measures success by student outcomes, as well as by the ease with which it fits into the rest of the course structure.

Zone four is the Large Scale Implementation. Having experienced success in the previous zone, we're ready to involve other instructors in the innovation. This implementation contains an entirely different set of risks than the small implementation did. The other faculty may not be true believers, nor even familiar with the idea at all. Success in zone four depends on resolving issues about appropriate training of those involved, suitable standardizations, and support needs. Success depends on the proper organization of materials, keeping to a common timetable, and establishing procedures so that the idea can be self-sustaining. Success in zone four is more difficult to measure. Student outcomes and faculty perceptions are important, but so are an evaluation of the procedures, standards, and budget.

In Zone five the instructor and others are routinely using the idea. It is now self-sustaining and supported by the normal operational budget. Faculty and students are comfortable in the regular use of the idea. In fact, they expect to be using it.

Faculty and other initiators within each zone revise and rethink the idea in the context of current experience. Even in the idea stage, a person is already customizing it to their own purposes. Some projects never develop beyond the small scale implementation, but are continually revised and improved at that level.

Figure A. An Overview of Schema I

Instructions:
  1. Identify the zone that best fits your innovation, by checking which sets of questions best applies.
  2. Questions for Zones below you identified zone are inappropriate, or a best premature.
  3. The questions associated with the decision symbol are posed prior to entering the next zone.
  4. Questions may be asked and answered by individual innovators, as well as by chairs/deans/manager/funders. 

ZONE 1
Getting the idea

FROM:  
Each other, Lodestar, Conferences, Internet, 
Publications, Ocotillo, etc.

What are some possible instructional uses?
What do you hope to learn?
Who else is doing this?
Why is this idea important to explore?
Does it lead? or follow?
Does the idea have a future?
Given the choice of to Experiment or to do Research, 
or to Observe others, which method is most justified 
in this case?
Is there a problem which you think this might solve?

Go to the next zone?

ZONE 2
Learn more about the idea

What do you already know about this?
What experience have you had with this?
What do you hope to accomplish?
Who else has done this?
What if it is successful? Then what?
Who else might also do this? Are they involved? How?
What problem do you hope to solve with this?

What are the questions being addressed by the project?

What can I do in this medium?
What is this good for?
How does this work?
How might it serve learning?

Go to the next zone?

ZONE 3
Implement the idea small scale

What do you already know about this?
What's the plan of implementation?
How long will it take?
How do you define success?
What are the costs/benefits?
How will it be maintained?
What organizational hurdles do you expect?
What problem do you expect to solve with this?

What are the questions being addressed by the project?

What will it take to make this idea work?
In what context can it work?
What else needs to be in place?
What is the impact on support staff?
Does it fit the target student audience?
What is the impact on student learning?

Go to the next zone?

ZONE 4
Implement the idea LARGE scale

Who will be responsible operationally?
What's the process of renewal and reevaluation?
What are the costs/benefits?
How will on-going training be handled?

What are the questions being addressed by the project?

It's worth doing.  How can we get all the parts to 
work together?
Are there worthwhile results?
Any side effects?
What support really is necessary?

Go to the next zone?

ZONE 5
The idea is now in routine use

What are the questions being addressed by the project?

How can we improve it?

Refocus and refine

Note:  The questions below are intended to be a 
guide.  Explore the questions.  You may not be able to 
answer all the questions; nor are there right and 
wrong, or expected, answers.

What results do we expect from this?
Personal/professional growth
A report on future possibilities
Some inspiration for others
To demonstrate a commitment to innovation

Who could we look to for support?
Colleagues/peers
Dept./Div. chairs
Ocotillo

Comments?
It's okay to re-invent the wheel
Benefits accrue to the organization in any case
There is "strategic" play
There may be no 'problem' to solve
If it doesn't work, so what?  Take a risk!

What results do we expect from this?
An evaluation of the project
To learn the costs and benefits of the idea
That others will learn from this experience
That not all projects will develop further
To learn its organizational impact
What is the impact on other faculty?
What is the impact on subsequent courses?
What is the impact on facilities?
What is the impact on connectivity?
To learn what support/training is necessary
How has the idea changed?
Were there unexpected results?
What needs to be in place for this to be a success?
Is it replicable?

Who could we look to for support?
Colleagues/peers
Dept./Div. chairs
Technology support
District
Ocotillo

Comments?
We don't want to re-invent the wheel; we do want to 
improve the wheel.
Focus on "strategic" implementations; build for the 
future.
The results need to be widely available.
It's OK to make mistakes and make changes.
The project will evolve during this zone.
To make some decisions your really do need to try 
things out.

What results do we expect from this?
Success
A replicable model
To dove $
To fold it into on-going operations
To develop an operational plan

Who could we look to for support?
College President
Appropriate Deans
Instructional Council
Dept./Div. chairs
Upper level District Administration
District:  MCLI, ITS
Technology leadership
Colleagues/peers
Ocotillo

Comments?
We assume the wheel is going to help us in our basic 
mission.
Training is an important factor.
CBAM is invoked.

What results do we expect from this?
Results
Efforts to fine-tune it

Who could we look to for support?
College President
Appropriate Deans
Instructional Council
Dept./Div. chairs
District:  MCLI, ITS
Technology leadership
Colleagues/peers
Ocotillo

Schema II: Adopting the Innovation

Substantial changes happened in zone four of schema I. The challenges for success in that zone had more to do with involving, training, and coordinating the work of others, than directly with the innovation itself. In many respects it moved out of individual decision and control to a more community effort. In fact, preceding the move to zone four, there was probably a group decision to attempt the large scale implementation. For this reason, the focus on the individual is inadequate in zone four; we need a schema which can give us insight into the movement of the idea through a community. Concerns Based Adoption Model (CBAM) is a schema which has been used for precisely this purpose. The purpose is to understand the process by which an innovation moves through an organization, in the context of a large scale implementation. CBAM is a comprehensive and complex system, developed and tested over many years. It provides both a theory and a framework for understanding the dynamics of successful implementation of any innovation in an organization.

Those who are familiar with CBAM can use its concepts and vocabulary to shed light on some of the issues that arise in zones one-three of schema I, but its main focus is: Now that we've decided that innovation X is valuable, what do we need to do to get X into routine use?

An individual may define and redefine a project many times during the early stages as its salient features become clearer and as it meets the reality of student use. These salient features, however, are difficult to change once in zone four, Large Scale Implementation. In fact it is crucial to success in zone four that all participants have a clear understanding of the "expectations during the initial implementation phases." (Taking Charge of Change, Hord, et al, Association for Supervision and Curriculum Development, Alexandria VA, 1987, ISBN: 0-87120-144-5). CBAM uses the concept of Innovation Configuration to clarify and communicate the variety of ways the innovation can be implemented successfully, and it clarifies the critical components of the innovation. During the implementation, the innovation configuration can be used as an evaluation guide. It can be used both to promote the success of the innovation and also to address the question of how well the innovation has been implemented in terms of its own description of success.

CBAM is based on several assumptions about change:

  1. Change is a process, not an event.
  2. Change is accomplished by individuals.
  3. Change is a highly personal experience.
  4. Change involves developmental growth.
  5. Change is best understood in operational terms.
  6. The focus of facilitation should be on individuals, innovations and the context.

The focus on individuals becomes apparent when looking at two different dimensions of CBAM: Stages of Concern and Levels of Use. CBAM identifies seven Stages of Concern:

  • These Stages of Concern can be used by the leaders of the innovation, as well as by the participants. They can be used to identify small successes and failures, to plot the progress of the large scale implementation, and to determine intervention strategies to improve the chances of success.

    CBAM includes another dimension of describing the process of adoption of the innovation: Levels of Use. There are 7 levels of use:

    According to the CBAM model, people tend to move sequentially through the levels of use from 0 to IVA. At that time most remain in routine use, while others may move "back" or "forward." During the infusion of the innovation, leaders will offer different kinds of information and workshops as people move through different levels of use.

    CBAM works as a tool for guiding the infusion of an innovation, once the decision has been made to infuse.

    Limitations

    1. Schema I looks too linear. This linearity makes the inadvertent judgment that to be further along the scale is better. Reality is much more recursive than this schema indicates.
    2. Schema I leaves the distinction between small-scale and large-scale implementation rather ambiguous. Until the schema is tested against more real innovations, this is a limitation. However, it may turn out to be an advantage in terms of flexibility.

    Recommendations

    Many thanks to Margaret Hogan and Irwin Noyes for presenting a distillation of the CBAM model to us. And to Ann Oehmke for leading us in CBAM's game Making Change. And to Brent Bartel for making arrangements and taking notes.

    Index

    Ocotillo Report '93
    back next
    © 1993, Maricopa Center for Learning and Instruction (MCLI) Maricopa County Community College District, Arizona