CHAPTER 3

CHAPTER 3

METHODOLOGY

    This chapter describes the procedures that the writer undertook to conduct this research study. The purpose of this study was to identify the essential components of curricular learning communities in higher education. A panel of experts participated in a four-round Delphi process designed to identify these essential features. Included in this chapter is a discussion of the research method utilized and its application to the current study, a discussion of how the study participants were identified and selected, an explanation of how the survey instruments were designed and implemented, and a description of the data analysis process. The writer used a modified Delphi process to first elicit and then rate the importance of characteristics of curricular learning communities in higher education.

Introduction of Research Method

The Delphi Technique:

    The Delphi technique was invented by Olaf Helmer and Norman Dalkey of the Rand Corporation in 1953 for the purpose of addressing a specific military problem (Helmer, 1983, p. 134). The object of the Delphi method is to obtain a reliable response to a problem or question from a group of experts. This is done by giving individuals in the group a series of questionnaires (or interviews) that reiterate the same questions while providing group feedback from previous rounds (Helmer, 1983, p. 135). Delphi experiments tend to produce convergence of opinion – not just toward the mean but toward the true value (Helmer, p. 153). The Delphi technique is based on the qualities of "anonymity, statistical analysis, and feedback" (Armstrong, 2000). In a Delphi study, the participants do not interact with one another, their responses are anonymous, the group results are given in terms of means, medians, or standard deviations, and participants are given the opportunity to reconsider their response after receiving the group feedback. Although originally developed and used by Rand as a forecasting methodology, the Delphi technique is used today in business, education, and the social sciences for a number of applications, including management decision making, policy evaluation, program planning, and prioritization of issues or actions (Delbecq, Van de Ven, & Gustafson, 1975; Dunham, 1998).

Delphi Advantages:

When faced with an issue where the best information obtainable is the judgment of knowledgeable individuals, and where the most knowledgeable group reports a wide diversity of answers, the old rule that two heads are better than one, or more practically, several heads are better than one, turns out to be well founded. (Dalkey, 1972, p. 4).

    Central to the Delphi philosophy is the old adage, two heads are better than one. Generally, a group response will come closer to the truth than that of any one individual. The Delphi technique offers the advantage of group response without the attendant disadvantages sometimes experienced with group problem solving or decision-making.

    According to Helmer, one of the advantages to the Delphi technique is that the expert participants are more likely to generate reasoned, independent, and well-considered opinions in the absence of exposure to the "persuasively stated opinions of others" (1983, p. 135). Because the experts do not ever participate in a face-to-face discussion, there is no danger of one or more individuals’ opinions being swayed by a more dominant or more experienced individual. Dalkey points out two other advantages that arise from the absence of face-to-face discussion: 1) irrelevant or biasing communication based on group interests rather than the problem at hand is eliminated and, 2) peer pressure or pressure to conform to the group is not a factor (1972, pp 19-20).

    Another advantage of the Delphi technique is its efficiency and flexibility, especially in light of modern communication technologies such as e-mail and the Internet. Experts may be drawn from a wide geographic area, and the participants’ commitment in terms of time and money invested is minimal. Delphi studies, though traditionally conducted in four rounds, can be done with more or fewer rounds, depending on the information needs (Delbecq et al., 1975). Debecq also notes the advantage of the usual Delphi procedure of obtaining ideas in writing, as the act of writing forces participants to contemplate the subject thoughtfully and tends to produce a high volume of ideas (p.34).

    Finally, the Delphi method has been shown to be an effective way to conduct research when the responses being sought are value judgments rather than factual information. Although it is more difficult to assess the "correctness" of value judgments, it is generally agreed upon that value judgments are not all equal but can in fact be more "right" or more "wrong." To test the value of using Delphi procedures in obtaining non-factual data, Dalkey and Rourke (1972) conducted Delphi experiments in which university students were asked about the objectives of higher education. They determined from the outcome of these experiments that Delphi procedures are "appropriate for generating and assessing value material" (p.57). Linstone and Turoff (1975) agree that Delphi is particularly useful for studies that call for subjective judgment rather than precise statistical analysis.

Delphi Limitations:

    Delbecq, Van de Ven, and Gustafson (1972) point out that Delphi should not be used when any of the following three critical conditions are not present: adequate time, participant skill in written communication, and high participant motivation (p. 84). They estimate that a minimum of 45 days is required to carry out a Delphi study. Participants must be knowledgeable and able to clearly communicate their ideas. A high degree of motivation is needed to offset the tendency for participant dropout as the study progresses. Because there is no direct contact between participants, those who are not highly motivated and interested in the subject at hand may feel isolated or detached from the process.

    Linstone and Turoff (1975) also note that participant dropout can be a problem in Delphi studies. They suggest that failure to allow participants to adequately contribute their own perspectives to the problem or issue at hand or failure to properly recognize or reward participants can negatively impact results. Other common pitfalls they mention are the use of unclear evaluation scales and poor techniques in interpreting results.

    Lang (1998) discusses the problem of bias in Delphi studies that can occur from poorly worded or leading questions or selective interpretation of the results. He suggests that this problem can be overcome by using a facilitation team or an unbiased facilitator who has no stake in the results of the study.

Selection of the Delphi Method:

    Linstone and Turoff suggest that the Delphi procedure should be considered for research problems when one or more of the following conditions apply:

· The problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on a collective basis

· The individuals needed to contribute to the examination of a broad or complex problem have no history of adequate communication and may represent diverse backgrounds with respect to experience or expertise.

· More individuals are needed than can effectively interact in a face-to-face exchange.

· Time and cost make frequent group meetings infeasible.

· The efficiency of face-to-face meetings can be increased by a supplemental group communication process

· Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured

· The heterogeneity of the participants must be preserved to assure validity of the results, i.e., avoidance of domination by quantity or by strength of personality (bandwagon effect") (1975, p.4)

    The Delphi method is well suited to the current research problem, that is, identification of the essential features of curricular learning communities. First, the question to be posed is subjective and calls for value judgment – the results cannot be analyzed in a strictly statistical manner. Second, since higher education professionals who are experts in the field of learning communities are widely scattered geographically, this method would allow for input from highly qualified individuals without the need for travel and with a minimal commitment of time on the part of these individuals. Finally, the participants, as higher education professionals, would be very likely to be able to communicate their ideas in writing and, as learning community practitioners and researchers, would be motivated by their own existing commitment to the topic.

    Delphi has been a frequent methodology choice for educational studies. An on-line search of doctoral dissertations since 1980 revealed that 119 studies have been conducted on higher education topics (UMI Proquest Digital Dissertations, 2001). Some specific examples of Delphi studies involving higher education include Wicklein’s 1993 study to identify critical issues and problems in technology education, for which he assembled a panel of 25 technology education professionals from 15 states (Wicklein, 1993), Lane’s 1989 study evaluating distance education courses (Lane, 1989), Lindstrom’s 1994 study to determine appropriate requirements for programs providing alternative approaches to teaching licensure (Lindstrom, 1994), and Lendman’s 1994 study to identify the core curricular components that promote critical thinking in post-secondary students (Lendman, 1995).

Selection of Participants

Panel Qualifications:

    The success of a Delphi study is largely dependent on the quality of the participants. According to Delbecq, Van de Ven, and Gustafson, potential respondents should meet four criteria.

It is unrealistic to expect effective participation unless respondents: (1) feel personally involved in the problem of concern to the decision makers; (2) have pertinent information to share; (3) are motivated to include the Delphi task in their schedule of competing tasks; and (4) feel that the aggregation of judgments of a respondent panel will include information which they too value and to which they would not otherwise have access (1975, pp 87-88).

    The panelists used for this study were all experienced practitioners and/or researchers of curricular learning communities in higher education. All are associated with higher education institutions and are now or have recently been involved in learning community programs. Many have contributed to the existing literature on learning communities.

    To identify potential participants, the writer enlisted the assistance of Jean MacGregor, Director of the Washington Center for Undergraduate Education and leader of the National Learning Communities Project. This project, funded by a $1.1 million grant from the Pew Charitable Trusts, aims to "continue to strengthen learning community efforts in the next four years (2000-2003) through a series of interrelated activities" (National Learning Communities Project web page, 2001). In conjunction with this project, a group of fifty learning community experts were identified and named as Learning Community "fellows." With assistance from MacGregor, the writer selected thirty-two persons from the list of fellows and requested their participation in the study (Appendix B). Twenty-two people responded positively, and seventeen of those persons actually participated through the final round of the Delphi survey.

Panel Size:

    The panel size of seventeen fits within the guidelines recommended for Delphi studies. Helmer and Dalkey used a panel of seven experts in their original Delphi experiment in 1953 (Helmer, 1983). Turoff (1975) suggests a panel size of anywhere from ten to fifty participants (p.86). A Delphi panel that consists of a homogeneous group, such as a group of experts from the same general discipline area, need only involve ten to fifteen people (Delbecq et. al, 1975). Wicklein (2000) used a panel of 25 experts in his study. Dalkey, Brown, and Cochrane found that error decreased rapidly as the group size increased from one to about thirteen; further small decreases in error continued to a size of about 25 people, at which point the error rate stabilized (1969). Based on these findings, they continued their experiments using groups of fifteen to twenty people.

Instrument Design and Implementation

Round One: Initial Survey:

    Helmer (1983) suggests that it may be necessary to begin a Delphi study with an open-ended question designed to help define and identify potential subject matter to be included in subsequent questionnaires. The first round in the current study (see Appendix C) consisted of a brief survey, designed to collect some demographic data on the participants, and one open-ended question:

In your opinion and based on your experience with learning communities, what are the essential characteristics of a learning community program? Consider factors such as the curriculum, the teaching and learning methods, the way the program is structured, and the environment or climate of the program. Please list 5 – 10 features.

The survey was sent out to participants as a Word attachment via electronic mail. Respondents were asked to return the survey by e-mail, fax, or postal mail.

Round Two: Questionnaire One:

    A list of 79 features was compiled from the information obtained in the initial survey. Obvious repetitions were eliminated, though items that were similar but not exactly the same were maintained. Items were sorted into four categories: Curricular Features, Pedagogical Features, Structural Features, and Environmental Features. Participants were asked to rate each feature on a Likert-type scale, identifying each feature as an "essential" (5), "very important" (4), "moderately important" (3), "slightly important" (2), or "not important"(1) characteristic of a curricular learning community (see Appendix D).

Round Three: Questionnaire Two:

    Questionnaire Two listed only the features that had received a mean rating of 4.0 or higher in the previous round. Once again the items were placed into the four categories of Curricular, Pedagogical, Structural, and Environmental. The mean score for each item was indicated on the questionnaire by an "x" placed on the scale in the appropriate place. Panelists were asked to reconsider these remaining items and rate them once again on the same five-point scale used previously. At the end of this questionnaire, panelists were asked to list three to five features that they believed would be most essential to include when creating a curricular learning community (see Appendix E).

Round Four: Questionnaire Three:

    The third and final questionnaire listed the forty features that received the highest rating (determined by mean and mode) on the previous questionnaire. Panelists were given the following information: ranking of the items from first and second questionnaire, mean score of the items from both rounds, and the number of times each item was selected as one of the three to five most important items. In this questionnaire, the items were not listed by category, but instead each item was tied to a potential theme or themes that might be connected to that feature (see Appendix F).

    In this round, panelists were asked to assign a total of 100 value points to the forty items. At the end of this questionnaire, participants were asked to answer the following open-ended question:

The number of curricular learning communities in higher education has increased greatly in the last two decades. Do you believe this expansion will continue over the next decade? Why or why not?

Data Analysis

    Responses to questionnaires 1 and 2 were analyzed to determine the group rankings of the items. Mean score, mode, standard deviation, and range were computed for each of the items. Votes and points assigned to items by participants in questionnaires 2 and 3 were tallied. Levels of agreement were determined for each item by calculating the percentage of panelist that rated an item as "essential" or "very important." Responses to the open-ended question on the final questionnaire were analyzed qualitatively.

Summary of Data Collection and Analysis Procedures:

    The following table outlines the four-round Delphi procedure that was followed in this study:

TABLE THREE

Summary of Delphi Process

Round 1

Round 2

Round 3

Round 4

Date mailed out:

Date due back:

11/20/2000

12/01/2000

12/05/2000

12/15/2000

1/03/2001

1/12/2001

1/19/2001

1/29/2001

Instrument

Open-ended survey

Questionnaire 1

Questionnaire 2

Questionnaire 3

Data collected

Demographic information and list of 5 – 10 essential characteristics of curricular learning communities in higher education.

Level of importance (on a five point scale) for each identified feature.

Revised level of importance (on a five point scale) and three to five most essential features.

Assessment of relative importance of the remaining features by assigning points to each feature for a total of 100 points and response to open-ended inquiry.

Data Analysis

Compile list of features

Prepare Questionnaire 1 using compiled list.

Compute mean, mode, standard deviation, range, and level of agreement for each item.

Prepare Questionnaire 2 using only items receiving highest ratings and including mean.

Compute mean, mode, standard deviation, range, and level of agreement for each of the features assessed.

Tally votes. Prepare Questionnaire 3 using only items receiving highest rating and including mean, rating, and number of votes received.

Tally number of points received for each item. Analyze changes in importance for each item. Draw conclusions based on results.

	Round 1	Round 2	Round 3	Round 4
Date mailed out: Date due back:	11/20/2000 12/01/2000	12/05/2000 12/15/2000	1/03/2001 1/12/2001	1/19/2001 1/29/2001
Instrument	Open-ended survey	Questionnaire 1	Questionnaire 2	Questionnaire 3
Data collected	Demographic information and list of 5 – 10 essential characteristics of curricular learning communities in higher education.	Level of importance (on a five point scale) for each identified feature.	Revised level of importance (on a five point scale) and three to five most essential features.	Assessment of relative importance of the remaining features by assigning points to each feature for a total of 100 points and response to open-ended inquiry.
Data Analysis	Compile list of features Prepare Questionnaire 1 using compiled list.	Compute mean, mode, standard deviation, range, and level of agreement for each item. Prepare Questionnaire 2 using only items receiving highest ratings and including mean.	Compute mean, mode, standard deviation, range, and level of agreement for each of the features assessed. Tally votes. Prepare Questionnaire 3 using only items receiving highest rating and including mean, rating, and number of votes received.	Tally number of points received for each item. Analyze changes in importance for each item. Draw conclusions based on results.