Evaluation

What is the goal of evaluation?

Teachers of research ethics have a professional obligation as academics and as researchers to aspire to the highest quality of educational programs. As a minimum, it is often necessary to rely on anecdotal resports of best practices; however, the best use of all of our resources will occur only with seeking evidence-based approaches that align pedagogical and curricular goals with measurable outcomes. In short, we should be evaluating the outcomes of our teaching programs.

Evaluation of training programs is essential to:

• Determine whether training goals are being met.
• Identify areas for improvement in program format and content.
• Provide quantitative evidence of success to administrators, faculty, and funding agencies.
• Demonstrate to trainees that the training is important and taken seriously.

What are the basic principles for effective evaluation?

Proper conduct of evaluation studies is governed by many principles that are not necessarily familiar to experienced researchers. The following are examples of some excellent, and accessible starting points for designing and executing studies that can help us to understand the impact of research ethics training programs upon those for whom the training was intended:

• Basic Guides: http://gsociology.icaap.org/methods/basicguides.html
• Evaluation Research: http://faculty.chass.ncsu.edu/garson/PA765/evaluation.htm
• Scale Construction: http://www.stat-help.com/scale_20100316.pdf

What should be evaluated?

An implicit assumption of evaluation is that it is possible to measure evidence for success in achieving goals of training in research ethics. This view is often assumed, but much more research is needed to assure that this is in fact the case. Mechanisms of feedback can consist of written evaluation forms, but verbal feedback can be at least as valuable. Verbal evaluation may consist of unsolicited comments, but can and should be explicitly requested in conversations with trainees or others, or as a topic of discussion at one or more points during (or after) the course of a training program. Evaluation instruments can be used to assess virtually all aspects of a training program including content, format, the trainees, the instructor, and the program itself. Examples of topics to be considered by the evaluation include:

• Course content
  1. Relevance (Is the material covered relevant to the trainees?)
  2. Currency (Is the information up to date?)
  3. Completeness (Have important topics been left out?)
• Course format
  1. Frequency and duration of meetings
  2. Cases (Is the use of cases effective?)
  3. Lectures (Are the lectures useful?)
  4. Discussions (Is the time spent in discussion useful?)
• Trainees
  1. Increased knowledge
  2. Improved decision-making skills
  3. Increased awareness
  4. Ability to identify practical applications of what has been learned
• Instructor
  1. Knowledgeable
  2. Effective
• Program
  1. Number of participating trainees
  2. Institutional awareness of program
  3. Institutional support for program

Who should do the evaluation?

Evaluation is not solely a matter of assaying student satisfaction using a form distributed at the end of a course. The process of evaluation should include feedback from:

• Trainees (What are their perceptions of program effectiveness? What have they learned?)
• Peers of the instructor
• The instructor (self-evaluation)