This week my colleague, Bill Johnston, and I wrapped up a project we’ve been working on for the last year and a half. We finished the final report and gave a formal presentation to our client, the Department of Corrections. We had been charged with evaluating a program that aims to reduce the recidivism (re-offending) rate for Native American youth in three Minnesota counties.
The client hopes to reduce recidivism by providing the youth with mental health and chemical dependency services, as well as opportunities to participate in Peacemaking Circles. Peacemaking Circles, based on the indigenous Circle process, bring people together to connect in a common purpose, without judgment. In this case, the youth who has committed the offense, his or her family, the victim(s) of the offense and other stakeholders participate in the Circle.
Bill and I understand that in conducting an evaluation in Native American communities, we cannot use the same processes that we might typically use. Early in the project we met with the Tribal Council. We told them what we hoped to do; we asked how they wanted to be involved, and we followed through. We had hoped that this meeting and follow up would help facilitate data gathering from community members who had participated in the Peacemaking Circles, but that didn’t happen. The youth and their families did not want to be interviewed – either by us or by our representatives: community members who worked with them on a regular basis. We ended up gathering secondary research via the Peacemaking Circle facilitators, but I wondered what we could have done differently.
To that end, Bill and I chatted over happy hour with an evaluation colleague of ours, Mary Ellen Murphy. Mary Ellen is a long-term evaluator who has done some study of evaluation in Native American communities. During the course of our conversation, she suggested that I take a look at a YouTube video that had made an impact on her own perspective: Strengths Enhancing Evaluation Research (http://www.youtube.com/watch?v=GEubejt8oUg). The video details the year that two University of Hawaii researchers spent working with two `aina (land & culture)-based programs on the Hawaiian island of Oahu.
I can’t do the video justice in a short blog post, but here are the four key Strengths Enhancing Evaluation Research (SEER) steps that the researchers used:
Engagement: This involves meeting with program stakeholders, asking to be guests and hopefully being invited to develop a relationship with the community. This step does not involve asking questions about the program. It’s about introducing yourself, finding out what they want to know, what they want to do, and what their goals and expectations are. This can take two or three sessions.
Data Collection: The researchers then collaborated with the program community toward data collection. The stakeholders let them know who would be involved. Their process included interviews, but I was particularly intrigued by their use of Learning Circles. In Learning Circles the emphasis is on human beings sitting together and learning from each other (how they do their program, the kind of concepts that were important to them, etc.) They focused on collecting stories. The researchers videotaped the interviews and Listening Circles for note taking purposes.
Data Analysis: Once data collection was done, they reviewed the tapes, recorded the stories and created charts in which they documented the main issues. They also did situational mapping where they looked at the human and non-human elements of the work. They then drew out themes and important processes and took them back to the communities to complete and enhance the information. They worked with the stakeholders to determine what had evolved from the data and how would they like to use it. What were the issues that came up as strengths and what are the areas that need more support from funders?
Customized Evaluation Tool Design: The researchers then worked with the program stakeholders to develop customized tools. In this particular case, they developed a multimedia evaluation tool that will allow anyone who comes to the website to see how program involvement fosters the connection between its participants and the land (a goal of the program).
I found this videotape fascinating and enlightening. Ultimately, though, the question that the three of us asked ourselves during our happy hour discussion remains. Can we (a white woman, a white man and an African American woman) do effective evaluation work in Native American communities? The researchers highlighted in the video were indigenous. Can it be done if we partner closely with Native American community members? I’d be interested in hearing your perspective.
