Research Ethics Training of Trainers: Developing Capacity of Research Ethics Training of Trainers: Developing Capacity of Bolivian Health Science and Civil Society Leaders Bolivian Health Science and Civil Society Leaders

: Research and research ethics (RE) capacity is a key element for addressing health priorities of low - and middle-income countries (LMICs). With support from a NIH/FIC Research Ethics Education and Curriculum Development grant, a RE Training of Trainers (TOT) was implemented in Bolivia. The Steering Committee, including representatives from four Bolivian universities and PAHO, developed the RE TOT that included face-to-face, online and practicum components. Twenty trainees including faculty, researchers and community leaders participated. Pre/Posttest results demonstrated significant increases in overall RE knowledge (13.1% improvement, p-value < 0.0001). An evaluation demonstrated participants valued participatory learning strategies and the flexibility of the online component. TOT participants during the practicum component delivered RE workshops to their university and civil society communities in four regions (n= 3,700 people). The goals of the grant were accomplished through the development of a Steering Committee and implementation of the TOT course. Next steps include the design and implementation of a master’s level research ethics education program in Bolivia.


Introduction
Supporting local researchers and institutions in developing countries to conduct high quality health research is a critical step towards improving the health of the population.Increasing the capacity of academics, researchers and health professionals to strengthen research infrastructure will facilitate meeting the Sustainable Development Goals (SDGs) (1).The health-related SDGs address issues of poverty, agriculture, education, gender, water and sanitation, energy, consumption, economic growth, inequality, housing, climate change, and sustainable development.
These issues are central to the Bolivian constitution and the National Development Plan (NDP) that aim to improve the lives of all Bolivians with a particular focus on marginalized indigenous peoples (2).
In Bolivia, recent political, social, and economic advances have increased interest in strengthening the national research agenda and in developing sustainable approaches to supporting research(3) however; research capacity has been historically limited by insufficient funding and support.In 2013, the Swedish International Development Agency (SIDA, provided approximately 25 million USD (216 MSEK) for the 2013-2017 period to strengthen research capacity in Bolivia through education, funding of research projects, and support to university administration (4).The national conversation among scientists suggests that the time is right to develop educational programs to improve the ability of Bolivian researchers to conduct research that incorporates ethical practices with the goal of improving health for all (3).
Few academics, researchers and health professionals have received formal training in research ethics (RE) and there are significant challenges to the consistent application of RE practices in the conduct of research (5,6).A study assessing the status of RE practice in Bolivia reported a lack of knowledge on how to protect research subjects and an absence of functioning research ethics committees (RECs), insufficient RE infrastructural support and RE training (3).These findings highlight the importance of developing a systematic approach to RE education and support for strengthening institutional policies and practices.
To address these needs, the Universidad Mayor de San Andrés (UMSA) in La Paz, Bolivia, in collaboration with Touro University California (TUC), applied for an International Research Ethics Education and Curriculum Development Award (R25) funded by the United States' National Institutes of Health/Fogarty International Center (NIH/FIC) in June 2012 (7) (8,9).

Study Aims
The principle aim of this paper is to evaluate the Bolivian TOT Research Ethics Course implemented during July-September, 2014 as a part of Bolivian RE Education Planning Grant.We will focus on a description of the collaborative process of course development and implementation including curriculum design, participant recruitment and course implementation strategies.We also assess participant evaluation of the course utilizing self-administered participant satisfaction questionnaires administered after the face-to-face and online course components and change in RE ethics knowledge and skills through administration of Pre/Posttest.We also describe participants' efforts to disseminate RE educational activities in each of their home universities and regional communities.Finally we summarize lessons learned and implications for future RE educational programs in Bolivia.

Steering Committee
To guide development of the project and develop

Adult Learning Methods
The development of the research ethics cur-riculum was informed by adult learning educational principles including using interactive learning methods, emphasizing skills relevant to "real world" environments, and including participants actively in the learning and assessment process (10).Teaching methods, such as lectures, modular instruction, role-plays, and small group discussions were used to adapt to the different learning styles of the participants (11)(12)(13)(14).By applying adult learning theories, the course leaders prioritized helping learners lay the foundations for a career of life-long development in the RE field (15).The curriculum emphasized an integration of the unique backgrounds of the different types of participants which may enhance the richness of activities such as the role-playing practices and other group activities (10).The TOT program incorporated a blended educational design that included dynamic didactic components along with a flexible online component which has been suggested to be an effective learning approach for adult professionals (16).Researchers in Bolivia have reported in a recent needs assessment that of all course styles, they would rank a hybrid course that included face-to-face and distance learning components as most preferred (3).

Hybrid Course Design
The TOT research ethics course was a hybrid of bioethics and RE educational activities and materials delivered face-to-face utilizing interactive learning strategies and online utilizing distance learning methodologies, followed by a supervised practicum.The three day face-to-face component was followed by the 10 week self-paced online component.After completion of the didactic learning components, the practicum was conducted over 3 months.Each of the eight RE modules had a face-to-face component and an online component.An additional eight CITI modules were included in the online component.
The face-to-face component utilized interactive learning strategies with significant focus on discussion of theoretical content and group activities.Each module was prepared by the faculty selected to teach that module and included dynamic group work, case studies, and round table discussions.The online modules were selfpaced with homework and assigned activities with ongoing feedback from the instructors.An Academic Course Coordinator with expertise in online instruction provided technical support and supervision.

Education Practicum
Upon completion of the training, participants were encouraged to implement educational RE activities in their local communities and institutions.Participants strengthened their knowledge of research ethics and gained important teaching skills among diverse groups of health science students, faculty, community members, healthcare professionals, leaders of civil organizations, and Research Ethic Committee members.

TOT Faculty
National and international health science leaders from the Ministry of Health, WHO, universities and community organizations were selected to teach the modules.The majority of faculty represented Bolivian universities and health organizations.The faculty that taught the face-to-face modules also taught the online modules with the exception of Dr. Fernando Lolas, an international RE expert, who taught the Global Research Enterprise and Publications module.

Participant Recruitment
The Steering Committee developed the TOT participant recruitment plan.Four university faculty and one civil society representative were purposively selected from each of the four participating regional universities/states for a total of 20 invited participants.

-Completion of Online Modules
Data regarding completion of the Online Modules was used to measure participant success in the online course component.Participation in virtual chats, online case studies and homework assignments were used to assess completion of the modules.
-Satisfaction and Attitudes Questionnaires The anonymous, self-administered Participant Satisfaction and Attitudes Questionnaires were administered after the face-to-face course and online components.This data was used to assess the quality and usefulness of the teaching methods and materials.The questionnaires were identical for each component except for one section of five questions that related to the specific course delivery methods.The satisfaction and attitudes questionnaires were also adapted from the FHI course (17).The satisfaction and attitudes questionnaires included 24 questions assessing participants' opinions on the relevance of course topics, previous RE training, attitudes related to RE principles and practices, and satisfaction with course delivery methods.Participants also responded to open-ended questions focused on participants' perceptions and attitudes related to the course.

TOT Pre/Posttests
Nineteen participants completed pretests and thirteen participants completed posttests after they finished both the face to face and online components.The six participants who did not complete the Posttest were not different from the other participants in any demographic characteristic measured.
Figure 1 demonstrates the difference (13.1%) between the mean Pretest score (67.36%) and the mean Posttest score (80.46%) for the 13 participants that completed both the pre and posttest.A paired t-test yields a significant p-value < 0.0001.The area where most improvement was seen was Research Ethics Fundamentals that included topics such as the Nuremberg Code, Declaration of Helsinki and the Belmont Report.

Completion of Online Modules
Of the 19 participants, 16 (84%) completed at least seven modules.There was no difference in module completion between the university faculty and community member participants.

Satisfaction and Attitudes Questionnaires
Twenty six people evaluated the face-to-face training component (including 19 TOT participants and 7 Steering Committee members and course faculty) and 14 completed the online component questionnaire.The majority of respondents (52%) had attended short trainings (less than one week) in bioethics and/or research ethics.Five (19%) of the respondents had not received any bioethics or RE training and six respondents (23%) had attended a longer than one week training.Among those completing the online training, most respondents (64%) reported that the instructions for using the virtual platform were clear and they received sufficient support from the Course Coordinator (86%), however, many stated (57%) that only some of the teachers provided sufficient academic support for the online course component.
As part of the qualitative data analysis process concerning positive and negative course aspects, open-ended survey responses were compiled, reviewed, and coded using "grounded theory" inductive procedures for analyzing qualitative data as described by Glaser and Strauss (18).Similar words/phrases were grouped together and a constant comparison approach was applied by two researchers.The open-ended questions allowed for more than one response and all responses were categorized.In response to the question asking for positive aspects of the in person training, there were 81 separate responses which were categorized into four themes: Participant characteristics, Pedagogy, New knowledge/information, and Curriculum/teachers.Other open-ended questions relating to positive and negative aspects of the face to face and online component were compiled, coded and analyzed using this same process.The themes and numbers of responses for each theme are presented in Table 2.

Positive Course Aspects
Many respondents highlighted the positive aspects of sharing experiences with multidisciplinary colleagues from different organizations and regions across Bolivia (33 responses).Participants appreciated the "different points of view" of their colleagues representing different academic disciplines and social sectors.One respondent commented that the "diversity of the participants allowed us to know their experiences.he/she had when reading the new (research ethics) terms and felt exasperated when trying to understand the material."There were critiques about some aspects of the online teaching methods especially the lack of interaction in the virtual forums and chat discussions.Many respondents felt that there was too much work in the virtual course and that there was not enough time to complete all the requirements.A few participants had some technical problems with the online platform stating they could not "technically manage" the virtual course modules.Some respondents felt that the bibliography and course documents were insufficient or not readily available.
Recommendations for the online course component were also highlighted.Several respondents recommended that more time be given for the online course.One respondent suggested including more activities where participants would review research project proposals utilizing ethical guidelines but realized more time would be needed for these types of practical experiences.Several respondents recommended solutions to the online course technical and platform issues such as developing a course manual for participants with clear instructions for accessing and using the online course and changing to a more user friendly learning platform, for example Moodle.

Potential Impact of the Training
After completing the face-to-face training, 92% of respondents indicated they would make changes in their research activities at their institution or in their communities based on the new information from the course.Respondents planned to present RE workshops for community groups focused on how human rights relates to research and the informed consent process.One respondent planned to return to his university and encourage the administration and faculty to create a REC and another respondent planned to move forward to recruit community representatives to serve on his institution's REC.
After the online training, 100% of the respondents indicated they would make changes in their research activities at their institution or communities based on the new information they learned in the TOT course.They recommended teaching health science students the importance of RE and research design for their graduate thesis projects, creating RECs and inviting community members as participants, training university faculty, improving attitudes related to RE among community and academic leaders, and increasing multidisciplinary participation in all university health science teaching and research.

Practicum -Dissemination
During the 3 month practicum, TOT participants provided RE presentations and workshops to their university, civil society organizations and a range of community groups.Over 3,700 people including medical residents, community organizations and human rights groups, in four different states participated in these RE dissemination workshops.A manuscript summarizing these outreach activities is currently in development.

Discussion
The

Study Limitations
The results of the TOT course may not be generalizable to future RE master's level courses which would require a significantly increased commitment of time and effort by the participants.The RE curriculum for a future course will also present more in depth RE information than was appropriate for the relatively brief TOT training reviewed for this paper.Although the types of individuals recruited for a more in depth master's level course may be similar to those recruited for the TOT, our results may not be generalizable to future more intensive RE training efforts in Bolivia.

Challenges of TOT Model
To use virtual modules and education platforms, technology skills are required which could limit the participation of some students and the ability to monitor student learning.The lack of face-toface contact and facilitator interaction can diminish the possibility of support and mentoring provided by instructor and limit in depth discussions or clarifications of the concepts.

Lessons Learned
An important aspect of the TOT course development was the essential role of the Steering Committee in developing and overseeing all aspects of the design and implementation of the course.The Steering Committee meetings were an opportunity not only to collaboratively accomplish significant work tasks necessary for successful implementation of the TOT but also to forge professional relationships that will be essential for developing future research and RE efforts in Bolivia.
TOT participants emphasized the importance of building teams, utilizing interactive learning strategies and providing flexibility through incorporating online learning.Results from the review of NIH/FIC funded RE programs also emphasized the importance of utilizing multiple pedagogical approaches to adapt to the learning needs of the trainees (14).Including leaders from religious organizations and human rights groups as members of the regional teams was cited as strengths of the TOT course.Community members added important new perspectives to course discussions and group activities.Community member participants were able to complete the face-to-face and online modules with similar participation rates and similar test scores assessing knowledge gained as the university faculty participants.

Adult Learning Methods
TOT participants highlighted the value of the interactive group learning activities which of-

Research Ethics Dissemination
Participants demonstrated significant commitment to extending RE related knowledge and practices learned in the TOT program to their regional communities as evidenced from the significant dissemination efforts conducted in the TOT practicum period.University faculty used materials from the bibliography provided by the TOT course to develop lectures for undergraduate and graduate health science students.Community members conducted workshops to raise awareness of the importance of RE principles for research within their organizations and civil society groups.The TOT curriculum development and course implementation ignited an enthusiastic response among participants representing faculty, researchers and civil society leaders to inspire and educate a range of health science professionals and community members about the importance of bioethics and research ethics in Bolivia.

Research Ethics Collaboration
From the results of the Pre/Posttests, Completion of Online Modules, and Satisfaction and Attitudes questionnaires, it was clear that course content related to international RE standards was new to many of the course participants.For this reason, principles and perspectives from PAHO, CITI, CIEB, TUC and the Bolivian scientists were helpful to understand international RE is-sues.Broadening RE education partnerships to include international RE expert groups, public health sector, community-based organizations and PAHO will also strengthen the capacity of future efforts to implement a master's level RE educational program in Bolivia.A mixture of national and international faculty and educational resources will be important for developing innovative and effective approaches to improving RE knowledge and practices throughout Bolivia.

Next steps
It is essential to develop and implement a master's level RE course to significantly improve the RE capacity in Bolivia.Teams of experts are needed in each department/region to provide educational support for health scientists and researchers in the public and private sector.These individuals can provide mentorship for researchers and REC members.Changing research related practices in universities, hospitals and health clinics will require a concerted effort that will be best sustained not through trained individuals working in isolation in universities but through teams of individuals who are trained and motivated to strengthen RE practices and policies in their institutions and communities.Lolas and Rodriquez (2012) present an in depth discussion of the rationale for the importance of RE training in Latin America.They recommend a combination of face-to-face and online learning "within the same cultural community" in consideration for the local/regional nature of ethical considerations related to research (19).They also emphasize the importance of addressing the widening disparity in research resources including financial and educational support for research and RE by providing high quality, culturally appropriate RE training programs within the context of particular countries and regions (19).A central aim of the Bolivian RE TOT program was to support teams of Bolivian academics, researchers and community leaders who could work collaboratively to develop specific RE programs in Bolivia.
This study suggests that a face-to-face and online hybrid course structure utilizing multidisciplinary participatory pedagogies may be an optimal approach to collaborative learning, changing attitudes/perceptions, increasing knowledge, and improving skills related to human subject research in Bolivia.Additional training and mentoring for RE faculty related to virtual teaching methodologies will be required as well as clearer platform guides/ instructions and improved distance learning support technology.The lessons learned and educational implications from the TOT will be important for creating an effective and successful master's level RE education course in Bolivia.

Figure 3
Figure 3 depicts the average scores by content theme.Participants increased knowledge in all areas including Research Ethics Principles, Responsible Conduct of Research, Research Ethics Fundamentals, Special Themes, and Research Supervision.

Table 1 .
Steering Committee members selected four faculty members from different departments in their Medical/Health Science universities who had RE experience.One civil society/community health leader from each region/ state was invited to attend the training representing groups such as Human Rights organizations, the Catholic Church or HIV/AIDS advocate organizations.Modules the course components were not included in the data analysis evaluating the course components reported below.
Participation All those who were selected agreed to attend with 19 participants attending the three day face-to face-TOT training.Eight Steering Committee members and nine additional TOT faculty participated in the face-to-face component of the TOT.Several of these Steering Committee members and faculty chose to participate and completed the online modules.Members of the Steering Committee and faculty who participated in -TOT Pre/Posttests The Pre/Posttest included 20 multiple choice and true/false questions for a total of 44 potential correct answers/points.The Pre/Posttest was adapted from the Family Heath International (FHI) research ethics training curriculum test (17).Participants took the Pre /posttest before the faceto-face component and after completing the online components.The questionnaire measured the participants' knowledge in 5 areas : Research Ethics Principles, Research Ethics Fundamentals, Responsible Conduct of Research, Research Supervision, and Special RE Themes.

Table 2 .
TOT Perceptions and Attitudes *Numbers in parentheses indicate the number of responses for each theme.