Personal tools
  •  
You are here: Home Education Research for the Classroom Teacher unit-7 Formative Evaluation: International Teacher Education Program (ITEP) Assignment

Formative Evaluation: International Teacher Education Program (ITEP) Assignment

Document Actions
  • Content View
  • Bookmarks
  • CourseFeed
Formative Evaluation Activity :: Ed-Net Masters Program Assignment :: Formative Analysis Assignment :: ITEP Assignment

Introduction

The following exercises deal with the training and retention of teachers in a remote area of the world. ( Map 1 , Map 2 ) The exercises lead to the setting up of a formative evaluation design using qualitative methods. The simulation is organized in the following steps:

Orientation: To begin the simulation please read the following exerpt from a letter to members of an online class about the International Teacher Education Program (ITEP). The ITEP Program is operated by the Church of Jesus Christ of Latter-day Saints and uses full-time missionaries to provide teacher training. Evaluating this program was an assignment Irene and I did while on sabbatical leave in 2005-2006 at Brigham Young University Hawaii.

You should be aware that many schools in the USA and abroad are operated by religious groups and are called parochial schools. The schools served by ITEP are operated by the Mormons. The point of this exercise is to show an example of a formative evaluation, not to highlight a particular religious group or to proselytize in any way. (Frankly, if I had been asked to evaluate a program run in Catholic or Methodist-funded schools or the Koranic schools in Northern Nigeria, as my Father was in 1965, that is what I would be showing.) If you have any reason to believe that viewing a program connected with this religious group would be objectionable to you in any way, you are invited to do the other formative evaluation exercise involving the Ed Net masters degree program. (That is why we have optional assignments.)

Credit for writing the Guided Design script goes to Mrs. Dorothy Little, a doctoral student at USU from Morgan, Utah.

Now, watch a clip from the video, “International Teacher Education Program (ITEP): A Partnership for Teacher Training” by Irene Eastmond, M.Ed. This video is fairly lengthy (48 minutes), so you should just watch enough of it to get a flavor for the kinds of teachers who are teaching, the kinds of students they are working with, and some feel for the classrooms where they work. (You are welcome to watch the entire video for teaching pointers, if you want. This was Irene’s second video production, and I think that she has a good eye for important information).

The following are the steps of the Guided Design process, steps that you have encountered in other exercises. In this case, the objective is to design a formative evaluation for the ITEP program.

  • State the goals of the work
  • Gather information
    • Locate relevant sources
    • Complete fieldwork (out of class activity)
    • Summarize findings
  • Generate possible solutions or recommendations
  • Specify the constraints
  • Select a solution or recommendation
  • Evaluate and revise as necessary, reflect, add information

We recognize these steps as a basic problem solving method. As you follow these steps, emphasis will be upon your creative decision making while completing the exercise. You should work from the perspective of designing a qualitative ethnography. You are encouraged to think creatively as the study unfolds, and to offer the solutions at each stage of the problem that seem best in your judgment or in the judgment of your group. Feel free to review the methods of ethnography in Leedy and Ormrod, p. 137, or online at http://www.prenhall.com/leedy (Select chapter 7 from the drop-down menu at the top left. I recommend looking at the checklists 7-1 and 7-2 "Checklists Planning a Qualitative Study" for a workable set of criteria for evaluating a qualitative study, uncluding the ITEP evaluation.)

The bulletin board feature of WebCT is available to provide feedback to some of your questions. We ask that you post your questions to the board and engage yourself in the threaded discussion. However, you should be prepared to come up with your own solution. If after a few days you do not feel you have received a reasonable answer to your question, feel free to pose the question to the T.A. who will respond or forward it to the instructor.

Step 1: State the Goals of the Work.

The video clip described an interaction between the International Teacher Education Program (ITEP) volunteers who were “outsiders” to the Island culture, whose purpose was to assist with the teacher-training program, and the local Pacific Island teachers who were the intended benefactors of the program.

Prior to coming to Hawaii on sabbatical in 2005, the Eastmonds were unacquainted with the ITEP. In 1996, Nick and his wife Irene had spent a one-year sabbatical in the Republic of South Africa and then afterward, Nick taught an Honors course at USU about it for 4 years entitled “Race and Communication in the U.S. and the New South Africa.” Nick is proficient in French and has published and spoken widely on a post-graduate level about international aspects of education. Irene’s background complements rather than parallels Nick’s. Her strengths are Music, English, Speech Communication, and K-12 education.

After experiencing South Africa the Eastmonds often discussed multicultural issues. At dinner one evening Nick remarked, “It is no wonder that people from the less developed nations can’t make a place for themselves in today’s global economy.” He was thinking about one of his international students who was unable to renew an expired visa and whose remote country was struggling economically. “You know, the survival of smaller countries depends on how well the people can use a global language. But many of them don’t get opportunities to learn outside their own cultures.”

While dishing up steaming food Irene quipped, “We should use distance education to instruct all the people of the world.” Her jaunty mood countered his reflective one.

“If only it were that easy—” then he challenged, “Exactly how do you suggest we use a distance ed program to connect with all cultures?”

“We can’t connect with everyone that way,” Irene admitted more soberly. “A scatter-shot approach won’t work. Even if the technology were universally available, learners can’t learn if the content doesn’t connect with their prior experience and if…”

“…If the content is totally foreign to them?” he finished.

She nodded. Despite her banter, as a multicultural teacher she was quite passionate about ethnic perspectives.

“For example,” he continued, “how do you think a universal language or a single global money system should be taught around the world in smaller nations? How can local teachers qualify themselves to teach it?”

“Local teachers need to be empowered,” she responded quickly. “And of course the content needs to be available to them. They should be free to adapt the new content—to gain ownership of it—from within their cultures.” A missing link became evident as she spoke. Quickly she added, “And to empower multicultural learners we need bridges between the races.”

“Bridges?” Nick inquired with a mouthful of food.

“Bridges. Respect builds bridges. Respect between races—respect for the preservation of cultural roots and for the preservation of native languages.” Irene’s philosophy tumbled out with mounting fervor.

“Learners absolutely need to feel respect,” he agreed.

“And,” Irene added, “in the first fifteen seconds of a conversation—even if you mean well—folks can tell if respect is present, or not present!”

Nick laughed at her animation. She had just recapped his platform for International Instruction. He jotted down a few notes from their conversation, feeling amazed that she could take his philosophy and re-state it so artfully.

The school year of 2005-06 again found the Eastmonds on sabbatical leave from Utah State University. This time, an opportunity arose to evaluate the International Teacher Education Program (ITEP) in the Pacific Islands. It was a program that had been operating for over a decade for the purpose of helping to localize global instruction.

As Nick explained from Hawaii in May, 2006, “[This sabbatical year] we spent one month visiting universities in Europe and attending a conference in South Africa, then three months in Mauritius in the Indian Ocean, prior to coming to Hawaii. We have been in Hawaii since the Jubilee Celebration in October 2005 [where work on the ITEP evaluation commenced].” Nick served as a program evaluator and qualitative researcher, and Irene as researcher and video ethnographer. This was their first opportunity to work together as researchers since 1972 when they completed a needs assessment on the Island of Guam.

The real problem in the Pacific Islands ( Map 1 , Map 2 ) had been to find ways of certifying local people as teachers to teach in the schools in the Pacific Islands. An impressive partnership had been functioning in the area for about 12 years to reach that goal. Nick and Irene were invited by the Dean of the College of Education at BYU Hawaii to conduct an evaluation of ITEP during the final months of their sabbatical year.

Task 1: State the Goals of the Work

As they go for their first formal meeting with the Dean of the College, what is Nick's and Irene's objective?

  • To identify the essential structure of globalized and localized instruction?
  • To gain knowledge about ITEP?
  • To challenge the system?
  • To get money?

Task: Answer the following questions and compare your answer with the Expert's

Is the Eastmond's approach the right way to begin (i.e. will they be biased?)

Task 1: Expert Feedback

The Eastmond’s objective is to gain knowledge about the partners of ITEP and how each interacts with the rest. They lack knowledge about how the people of the Pacific Islands experience ITEP in their nations. They also lack knowledge about the complex workings and policies of ITEP. Because the Eastmonds both belong to the religious group that sponsors ITEP, and because their personal insights seem to parallel the basic goals and ideologies of ITEP, they agree that a strong critical framework is necessary to guide their questions and to help them recognize biases.

They do feel passionate about the pending evaluation and are qualified to begin gathering information. Funding for the project is not a major factor in their motivation to proceed, since they will be reimbursed for expenses only. The evaluation project does not add to their income. If exact trouble spots can be found in the ITEP operation they ultimately want to identify them so that improvements can be made. They decide the best approach to doing this study is qualitative, using ethnography and Eisner’s method of educational criticism. They decide to increase validity of the research by videotaping certain parts of it.

Step 2: Gather Information

The Eastmonds wanted to find existing research on ITEP. They had access to the university library in Hawaii. Because of the emergent nature of qualitative research, they wanted most of all to mingle with members of the current ITEP partnership and then to identify key informants who could provide valuable insights and could lead them to other helpful individuals.
Locate relevant sources:

Task: Think of yourself as a qualitative researcher. You are an outside visitor to the ITEP operation. What sources would you consult, and in what order? Enter your sources below and compare with the expert.

Task 2: Expert Feedback

The Eastmonds went looking for background sources on ITEP. A primary source for their information will be with other people connected to ITEP: the Dean of the College, the current ITEP Coordinator couple, and other faculty at BYU – Hawaii.

In addition, they will look for written sources of information: studies, handbooks, and the ITEP website.

Task 3: Reading for Background Information

As a starter, have a look at the ITEP web site at: http://soe.byuh.edu/itep/index.htm

You can read about the history of the program, see the Handbook used to orient mission couples, as well as the Photo Album of pictures on the four islands.

Nick was looking through an orientation booklet for senior missionaries when Irene handed him the report she had finished reading. "This paper presents a really good background for the work," she said.

Read and respond to the report, A Schools-University Partnership across Five Countries in the South Pacific (Bailey, Mitchell, and Winstead, 2002). Keep the following questions in mind as you prepare your half-page response to this reading. (1) What is the stated problem? (2) Who are the primary beneficiaries of the ITEP partnership? (3) Who are the administrators? (4) What are the issues faced by ITEP? (5) As a researcher/evaluator, what is your perspective? What are your biases?

Task: Now write a personal response to the Bailey, et. al. article. Your response should be approximately 200 words in length (a half page).

Task 4: Information Gathering

Task: Do the Following:

Once you have identified all of the interacting parts of ITEP then you can start to put things into context for your inquiry, as Nick and Irene did. By interviewing the coordinators and others, and by reading relevant materials, Eastmonds identified the interacting parts of ITEP as:

  • ITEP Coordinators: One couple stationed in Hawaii whose organizational skills and communications keep the program running
  • Principals in participating schools in the remote Pacific Islands Area
  • Teachers in the participating schools who are seeking to further their education
  • Teachers in the area who are chosen to receive scholarships to attend BYU-H or BYU-Provo:
    • IWES-ED (The International Work Experience Scholarship in Education)
    • ILDS (The International Leadership Development Scholarship)
  • Sponsors of the participating schools
  • CES (Church Education System), which directly sponsors the Island schools
  • BYU-H, which provides teacher training and certification
  • The Church Missionary Program, which provides qualified senior couple missionaries

Task: Answer the following questions and compare your answer with the Expert's

What research questions would provide the Eastmonds with the information they need for evaluating ITEP? List three or four Questions

Task 4: Expert Feedback

These basic questions seemed most relevant to the researchers:

  • What is ITEP?
  • What elements are working well and what elements are not?
  • Are there other dimensions for expansion or improvement of ITEP that should be considered?
  • What ways should it develop in the future, given a long-range perspective?

Task 5: Research Design

What tools should the Eastmonds use to collect the data? How should they frame their questions to obtain the information they need for reporting an evaluation of the ITEP? How will they organize, code, and analyze the collected field notes and videotapes? Ideas for designing ethnography may be found in chapter 7 of the Leedy and Ormrod text or online at http://www.ResearchNavigator.com ($15. subscription required) or http://wps.prenhall.com/chet_leedy_practical_8/0,9599,1569809-,00.html .

Task: Answer the following questions and compare your answer with the Expert's

Task 5: Expert Feedback

Tools that Nick and Irene chose for the research include Elliot Eisner’s 5 points of educational criticism. This will provide a useful framework for difficult, probing questions about school environment. Data will be collected from focus groups with teachers, interviews with ITEP couples and school administrators, and “participant observation”, i.e. participating in activities with these couples for approximately one week on each island. For summarizing the data, a procedure labeled the “constant comparative method” (Glaser and Strauss, 1985) was selected to facilitate examining large amounts of field notes at each stage. This should help to prevent the accumulation of an unmanageable assortment of data toward the end of the study.

Refining the Information:

“Look at how one senior couple can affect the program for better or for worse,” noted Irene as she was studying differences between the personalities of incoming and outgoing missionary couples. “It must be a huge responsibility to screen prospective missionaries as to their qualifications!”

“And to transfer them seamlessly without disrupting the services,” added Nick.

Irene suggested, “You know, we should compare a senior missionary couple on the Islands to a professor on the BYU campus, since both are expected to fill the role of instructors and advisors to students.”

“Compare them from what perspectives?” prompted Nick. “The student’s? The ITEP couple’s? The visiting professors’?”

“Maybe from all perspectives, but especially from our perspectives. We are the researchers… and we are ‘outsiders,’” she stated. “One of the questions we should ask is how the ITEP classes on the islands compare with on-campus classes in Hawaii…”

“…and we should find out how attractive ITEP looks to teachers on the Islands,” added Nick. “The Islands provide more of a cultural milieu than on-campus. Actually, it’s up to the missionaries to provide a bridge between cultures—more than would be expected on campus. They must provide a kind of bridge between the Island’s culture and that of North America.”

“Hmm. A bridge, huh? Didn’t I tell you?” she laughed mischievously.

“Seems like you did…”

“You know,” she continued more thoughtfully, “couple missionaries in the South Pacific represent cultural entities of their own—on each separate island!”

Nick scratched his head at her perceptiveness. “I agree!” he said, “and I wonder what the couples themselves think of the experience. What improvements would they recommend in the program? The couples are certainly central to the effectiveness of the program.”

Irene added, “And Island teachers are central to its purpose.”

Task 6: From what sources will possible recommendations emerge?

Nick and Irene have their questions and sub-questions in mind. They've selected tools for organizing and framing their findings, and their conversations are helping to define the work. What perspectives will they employ as they write up the study?

  • Pacific Island youth
  • Pacific Island teachers
  • University students on campus in Hawaii
  • Senior couple missionaries in the remote islands
  • Visiting professors from BYU-H
  • All of the above

Task: Answer the following questions and compare your answer with the Expert's

What perspectives do you feel might reveal the weakest links in the ITEP function?

Task 6: Expert Feedback

The best opportunity to access qualitative information about ITEP will probably come from the perspectives of senior couple missionaries and from the perspectives of Island teachers. This information will be supplemented from the perspectives of principals, the ITEP coordinating couple in Hawaii, visiting professors, and the partners of ITEP (CES, the Church Missionary Program, and BYU-H). Nick’s and Irene’s first person voices will be strongly evident in their writings, as is the case with most qualitative research.

Task 7: Generate possible solutions or recommendations:

Note: This step asks you to suggest possible ways that an evaluation of ITEP could take place.

Task: Answer the following questions and compare your answer with the Expert's

The big question underlying this step is whether it will be necessary to visit each of these islands as part of the evaluation, since travel is both expensive and time consuming. You might suggest some ways that the program could be evaluated with or without a visit to the islands.

  • List 1: Options with a visit required
  • List 2: Options where a visit would not be necessary

Task 7: Expert Feedback

List 1: Options with a visit required

  • An ethnographic study, including site visits, observation of classes, interviews with important people, and some focus groups.
  • A cost-effectiveness study, where the results of student achievement on the national exams would be compared with the scores of students attending the government schools. Obtaining the necessary data would be difficult to do without going onsite.

List 2: Options not requiring a visit

  • A questionnaire study, similar to that conducted by Bailey, Mitchell and Winstead, (2000), using questions that allow for open-ended responses, from teachers, administrators, and ITEP couples.
  • A follow-up study of couples who have served in ITEP in the past 5 years, using telephone interviews with e-mail follow-up, documenting their perceptions and suggestions.
  • An evaluation of only the activities occurring on Hawaii, to include interviewing scholarship recipients, program administrators, critique of the web site, and observing the orientation of the ITEP couples.

In conducting any evaluation, it would be well to keep the concerns of the participants in this program. Here is a partial listing of concerns to keep in mind.

(1) Concerns of the teachers who live and teach on a remote island:

  • Providing for my family by teaching
  • Increasing educational credentials, thus qualifying for a pay raise
  • Increasing educational expertise
  • Limited accessibility to technology and the Internet
  • Limited travel opportunities

Task 8: Specify the Constraints

Task: Answer the following questions and compare your answer with the Expert's

Make a list of the constraints you feel are important to the ITEP evaluation.

Task 8: Expert Feedback

Time: The sabbatical was half over. Nick and Irene had six months to work part-time on the evaluation, with six weeks of that set aside for intensive fieldwork. So, if the Eastmonds are to be involved, this study needs to be something that will take no longer than 6 months time.

Money: Funding was available to provide travel, communication, and a few supplies to supplement their existing video, audio, and computer equipment. Food and lodging during the six-weeks of field work would be provided chiefly by the senior missionary couples on the Islands.

Other constraints were established with the help of chosen tools and methodology:

  • Focus on the needs and performance of Island teachers and their advisors, the couple missionaries
  • Supporting information coming from the other ITEP partners
  • Theoretical framework (Eisner’s Educational Criticism)
  • Steps of Ethnography
  • Constant-Comparative interpretation of data

Task 9: Select a solution or recommendation:

Assignment: Read the proposal for the Formative Evaluation . This was the plan submitted to authorize the study. (Note: The actual expenses for travel were at least 60% higher than those expected. Even though there was no payment made to the Eastmonds, the study cost the program a substantial sum of money).

Now, watch the Evaluation Video (18 minutes). This video was Irene's first effort to produce an evaluation video, and both she and the client were quite satisfied with the result.

Executive Summary of the ITEP report (Eastmond and Eastmond, 2006), An Evaluation of the International Teacher Education Program (ITEP) of the Church Education System (CES) of the Church of Jesus Christ of Latter-day Saints and Brigham Young University, Hawaii School of Education. This statement describes the major findings of the study. The entire study is made available for your use if you want to see it. It will give you an example of a detailed qualitative report and what that approach has to offer. Note the Appendices, each with additional information.

For a closer examination of one couple's experience with ITEP (actually, this is a revision of Appendix A of the report), read the article by Nick entitled Up Close and Personal : A Case Study of an ITEP Missionary Couple in the International Teacher Education Program of the Church Education System (CES), 2006. As you read, consider the audience for which the paper is written. Pay particular attention to the perspectives of the senior couple missionaries, the concerns that they must face, and their possible motives for volunteering in the ITEP.

Task:In the space below, describe the steps you would take to carry out a formative evaluation of ITEP. Determine whether a trip to the islands is really necessary. Describe the ways you would collect and analyze the data.

Task 10: Evaluate and revise as necessary, reflect, add information

Link to the ITEP Final Report

This is an opportunity to think over what you have seen with this formative evaluation project. Here are some questions you might want to react to:

  • What were the strengths and weaknesses of evaluating ITEP using a qualitative approach? How would the results have been different if a more quantitative approach had been used?
  • Could a valid study have been done without visiting all the islands? Aside from disappointing the Eastmonds, what would have been the drawback of taking this approach?
  • What did you learn about ITEP at the end of the study that was not apparent to you in the early stages of learning about it?
  • What suggestions do you have for improving the evaluation done in this fashion?

Write your personal response to these products of the formative evaluation. Think about:

  • How the authors used perspective (voice) to report controversial or sensitive information
  • How the qualitative data was organized and triangulated
  • How the written report addressed the needs of the client (the ITEP partners)

Task: Organize and write your concise response of about 400 words (one page). You will be expected to demonstrate your understanding of the qualitative method (Leedy and Ormrod, chapter 7-8) as it applies to this report.

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . factpetersen. (2007, August 01). Formative Evaluation: International Teacher Education Program (ITEP) Assignment. Retrieved January 08, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/education/research-for-the-classroom-teacher/unit-7/formative-evaluation-international-teacher-education-program-itep-assignment.html. This work is licensed under a Creative Commons License Creative Commons License