: Identifying the

Information Systems (IS) interoperability in Public Administration (PA) is a main goal and a major challenge for PA professionals. Achieving interoperability among IS that are technologically disparate and that exist in different organizational contexts is a complex task, being affected by multiple aspects, not yet satisfactorily known and characterized. The aim of this paper is to unveil the forces that influence IS interoperability initiatives in PA. The inquiry was inspired by Lewin’s Field Theory. The data generation process was based on a Delphi study involving 55 experts from PA, IS/IT industry, and academy. A set of 31 forces were identified and ranked based on the level of importance they assume in IS interoperability initiatives. Thirty eight propositions describing restraining and driving influences were also formulated, as well as 24 propositions about forces’ configuration in the current context of Portuguese PA that represent the specific constellation of forces acting in IS interoperability initiatives in that country. The results of this study provide an understanding of the complex of forces acting in IS interoperability, contributing to improve the study, management, and implementation of these initiatives and, consequently, to the establishment of a PA with more adequate, sustained, and sustainable levels of interoperability.


Introduction
Since the late 90's, the development of e-government and, more specifically, the creation of an electronic Public Administration (PA) have been considered as main concerns and priorities for action by most governments all over the world, being seen as essential vectors to promote and foster a new paradigm of society currently known as the "Information Society".Since then, the quest and the need to deeply transform the way PA operates won the interest of politicians, who started to direct considerable attention to the formulation and implementation of strategies that could lead to the achievement of the intended transformation.
Information Technologies (IT) assume, in this context, an important and leading role, being recognized as a primary instrument to e-government development and to the modernization and transformation of the public sector.The aim and concern of politicians and PA professionals is to harness the potential offered by information technologies to transform the Public Administration operating paradigm, making it more efficient, more effective, more transparent, more citizen centered, and more capable of delivering cross-agency public services in an integrated way, through multiple access channels, 24 hours a day, seven days a week [5] - [6], [65].Achieving this "ideal vision" is not a trivial task.Indeed, working according to this new paradigm imposes a fundamental and not so easily achievable condition to PA agencies that is to have the capacity to work in a collaborative way, what inevitably requires and demands the existence of interoperability capacity among their Information Systems (IS).In this context, IS interoperability emerges as a pertinent topic in the PA area, becoming a key requirement for the creation and development of the "ideal vision" typically associated with the e-government phenomenon [21].
The relevance of interoperability in PA is quite evident considering the numerous government programs, action plans, and policy documents published in recent years at national and European levels, the multiple interoperability strategies, frameworks, and platforms being developed in multiple countries, the huge number of research projects on interoperability issues funded namely by the European Commission under their last framework programs, the creation of observatories on e-government interoperability issues, and the raise in the number of conferences and scientific or practitioner commissions and associations focused on the interoperability thematic.
All these facts suggest the recognition by politicians, PA professionals, European Institutions, and other stakeholders in this process (particularly the IS/IT industry community) of the importance of creating more appropriate interoperability levels within and between PA agencies at local and central government levels, as well as at a supranational level such as the European Union.Despite all the recognition that exists around the need and importance of IS interoperability in PA, and despite all the investments and efforts already made in order to improve the collaboration and data exchange capacity of different government agencies, the level of interoperability between agencies is still not adequate and the problems and difficulties caused by this lack of interoperability are still quite evident.Actually, from a citizen standpoint, government agencies keep operating in a very fragmented way, continuing to compel citizens to have to interact with multiple agencies in order to get their requests satisfied.Delays in service delivery, lack of transparency about the procedures involved in service provision, the need to fill and refill forms containing the same set or subset of data, and the incapacity to obtain detailed information about the status of execution of a particular service through different access channels, are evidences of the lack of interoperability which generally persists in PA.
The two paradoxical observations just reported-on the one hand, the fact that interoperability has been recognized as a need in PA and that some efforts and investments have already been made by agencies to improve their interoperability, and, on the other hand, that despite the efforts undertaken the levels of interoperability have not come up to the level it would be desired and adequate-promptly give rise to the question of what makes interoperability initiatives so complex, so challenging, and so difficult to manage and implement.
Being it because of the huge number of agencies that could be involved; the technological, semantic, organizational, or cultural differences existing among them; or even because of the change that the creation of interoperability conveys to the agencies and to their information systems, the view shared by both practitioners and researchers is that interoperability initiatives are indeed highly complex and challenging efforts.
According to [63], the exact extent and complexity of the challenges surrounding the implementation of IS interoperability initiatives in PA, are not yet, neither in theory nor in practice, properly understood.This opinion is also stressed in the white paper published by CompTIA [11], where it is mentioned that there is still no adequate explanation about the factors or forces that may facilitate or limit interoperability initiatives.
In this context, what seems to be one of the major problems underlying IS interoperability initiatives in PA is the lack of an explicit and systematic understanding about the complex of forces acting on the implementation of these initiatives.This lack of understanding makes those responsible for interoperability initiatives to face successive failures, preventing them from better managing the phenomenon of creating more adequate, more sustained, and more sustainable levels of interoperability.Besides contributing to the perpetuation of the dissatisfaction felt by citizens and other agents interacting with PA agencies, these failures also cause financial losses as well as a Combining this set of key ideas with the definition presented in the draft version of EIF V2.0, in this paper interoperability is defined as the ability of disparate, diverse, and autonomous organizations to interact towards an agreed common goal, through the sharing of information and knowledge between the business processes they support, by means of the exchange of data between their respective information and communication technology systems, while preserving their level of autonomy and without having to make a significant effort.
An additional issue becomes evident from the three definitions presented: that interoperability is a broad construct, going beyond the technical dimension traditionally associated to the term.The definitions clearly highlight the semantic, and, particularly, the procedural and organizational dimensions involved in the interoperability construct.Indeed, the literature review carried out on the e-government interoperability area showed that, instead of defending a monolithic perspective of interoperability, as frequently happens in other areas [49], authors in this area have been increasingly recognizing and stressing the need to consider these three dimensions in the study and implementation of interoperability initiatives [21].
The technical interoperability dimension covers the technical issues of linking computer systems and services.It includes aspects such as open interfaces, interconnection services, data integration and middleware, data presentation and exchange, accessibility, and security services [22].
The semantic interoperability dimension is concerned with ensuring that the precise meaning of exchanged information among computer systems and services is understandable, even though they were not initially developed with the purpose to interoperate.Semantic interoperability enables systems to combine received information with other information resources and to process it in a meaningful manner [22].
The organizational interoperability dimension is concerned with defining business goals, modeling business processes, and bringing about the collaboration of administrations that wish to exchange information and may have different internal structures and processes.Moreover, organizational interoperability aims at addressing the requirements of the user community by making services available, easily identifiable, accessible, and user oriented [22].
Beyond technical, semantic, and organizational issues, some documents, such as [23] and [67], also highlight the prominent role of legal and political issues on e-government interoperability, arguing for their inclusion as new dimensions of interoperability.
A final observation that emerged from the literature review is the importance and pertinence of the governance issue in e-government interoperability.The issue of interoperability governance has been introduced and emphasized in documents such as [22] - [24] and [66], that clearly point out the need and relevance of proceeding to the overall governance of the interoperability.The adoption of a more simplistic approach to interoperability, in which the attention is exclusively focused on aspects of collaboration and on ensuring the ability of "being together" in the provision of public service, should not be considered sufficient.As stressed by [32], the most important aspect in PA agencies interoperability is not that they are able of "being together", but that they are able to find out "the most effective way of being together".Consequently, the attention in the quest for interoperability must be set, not only on collaboration issues, but also on matters of coordination, direction, and control of the whole process.
The way interoperability has been defined in this section, as well as the recognition of its multiple dimensions, will influence and shape the search for the forces acting on the implementation of IS interoperability initiatives in the PA context.

Studies in Information Sharing, Information Integration, G2G, and Interoperability
The literature review carried on in this study, with the aim to find out works that are focused in the identification and understanding of the barriers, problems, success factors, and risk factors involved in IS interoperability initiatives in PA and that could be valuable for this study, revealed that research in this area is still quite limited.Indeed, most of the potentially interesting studies found over the literature review were not focused on the interoperability theme in itself but on some related themes such as interconnection, information sharing, information integration in government, and G2G.The first of these studies was published in 1996 by Sharon Dawes, a researcher of the Center for Technology in Government (CTG), University at Albany.This study became a seminal paper, being cited by most works done in this area.The aim of the study was to understand the opinions and attitudes of state government managers towards the benefits and costs of interagency information sharing.In order to do that, Dawes organized her work in two phases.In the first phase, Dawes performed a literature review to identify the benefits and barriers associated with interagency information sharing.The benefits and barriers identified are shown in Table 2.In the second phase, Dawes conducted a survey, in the state of New York, to evaluate the attitudes and opinions of state government managers towards the benefits and barriers identified in the literature, and to examine the policies and tools used to govern information sharing activities.As a result, Dawes proposed a theoretical model of interagency information sharing.This model argues that, as a consequence of their participation in previous sharing initiatives, agencies have perceptions about the potential benefits and risks involved that will affect their performance in future similar initiatives.As new sharing experiences occur, new benefits and risks will be perceived by participants.These new perceptions form the basis for future expectations and produce lessons for a general policy and management framework that could be utilized in the future.
Some years later, Landsbergen and Wolken [39] extended Dawes' model.While having used the interoperability term in their work, the fundamental contribution of these authors was at the interagency information sharing level, by reviewing and updating Dawes' model of interagency information sharing.Based on literature review, document analysis, case studies, and interviews, Landsbergen and Wolken identified three main benefits of interoperability (effectiveness, efficiency, and responsiveness), as well as 11 barriers that, as shown in Table 3, were classified in four different categories: political, organizational, economic, and technical.Those scholars used the information gathered in the study to expand the theoretical model for interagency information proposed by Dawes.The principal contribution of their model is its emphasis on the need for the existence of an infrastructure to support agencies in information sharing, as well as on the need for legal, managerial, and policy approaches to maintain interagency information sharing.
While keeping the focus in the interagency information sharing, the third study centered its attention in the interconnection between state and local agencies.The fourth study was conducted by Joia.The aim of Joia was to propose a heuristic framework to successfully implement G2G endeavors in Brazil.Based on action research and on two case studies, Joia identified three key success factors (security, organizational culture, and training) to the implementation of G2G processes in PA agencies, as well as nine barriers that could arise during G2G projects and affect its success.Joia clustered the barriers in three main categories, as shown in Table 5.The fifth and sixth studies were authored by a set of researchers working with Dawes at the CTG.Both studies are part of a series of works intended to increase the understanding of interorganizational information integration in public agencies and they represent an effort to model the social and technical processes of interorganizational information integration.
The study by Pardo et al., published in 2004, was focused on the identification of the factors and processes involved in integrating information across levels and agencies in government as well as on the understanding of how IT and social factors interact to influence interorganizational information integration.In this study, authors conceptualize the integration across distributed information sources and organizational boundaries as a complex social process in which technological and organizational artifacts are developed and assembled for the purpose of information use.A list of 32 main components of information integration was presented.The components were clustered in four categories as depicted in Table 6.
Keeping the focus on interorganizational information integration, the study published in 2005 by Gil-García et al. examined the factors that influence the success of selected criminal justice integration initiatives.Based on literature review and six case studies, the researchers identified a list of barriers to information integration, namely Turf and resistance to change, IT and data incompatibility, Organizational diversity and multiple goals, and Environmental and Institutional Complexity.The authors also suggested seven strategies for the implementation of interorganizational information integration initiatives, specifically: Retain autonomy of the agencies, Establish and exercise a governance structure, Secure strategic partnerships, Build on long-range and comprehensive planning, Build understanding of the business process, Secure adequate financial resources, and Obtain and nurture executive leadership and legislative support.The seventh study was conducted by Bekkers in the Netherlands.Bekkers' goals have been to identify critical factors which account for the success of IT-driven back office integration projects and to find out how should back office integration be organized: as a management or as a governance challenge.To achieve these goals, Bekkers analyzed four successful Dutch back office integration projects according to a semi-standardized format, involving (1) an analysis of the involved back offices and their core values, (2) an analysis of the perceived resource dependency between the involved back offices, (3) an analysis of the factors which contributed to the definition of a shared information domain, and (4) an analysis of the circumstances under which a project management and/or process management approach contributes to the effective integration of back offices.Based on the case studies, Bekkers concluded that back office integration should be seen as a governance problem and set out six main lessons learned from back office integration projects, namely: Managing multi-rationality in different arenas; The ongoing recognition of interdependency; It is the content that unifies: trust and external pressure as lubricant; Managing the political agenda; The dynamic allocation of costs and benefits; and The balance between project and process management.
The eighth work resulted from a study on e-government interoperability conducted under the MODINIS program funded by the European Commission.The study, entitled Study on Interoperability at Local and Regional Level, ran for 26 months, from December 2004 to February 2007.The study had four main objectives: (1) To characterize the status of local and regional interoperability in European member states; (2) To identify the key success factors of local and regional interoperability; (3) To identify the key barriers of local and regional interoperability; and (4) To issue recommendations on e-government interoperability.Based on an extensive literature review, on the analysis of case studies that represent good practice projects in European member states, and on the analysis of stakeholders' needs and feedback, the authors identified a list of 39 key factors for e-government interoperability at local and regional level.These factors were organized along four different categories as shown in Table 7.The authors also issued 44 recommendations regarding e-government interoperability, organized in a matrix structure that groups the recommendations along two dimensions, namely the suitable level of action (European Union, National, and Local/Regional level) and the areas where a recommendation should be applied (Legislation, Funding/Financial, Policy/Management, and Technical).
While providing an interesting set of factors, barriers, or issues that could eventually be useful to whom is involved in the promotion and implementation of IS interoperability in PA, these studies have two main drawbacks.
First, only two studies ( [39], [66]) focus on the interoperability theme, and from these, one study ( [39]), while using the term interoperability, defines it in an information sharing perspective as being "essentially a problem of sharing information".Despite the proximity of the themes approached in the other six studies, it is not possible to assume that the factors, barriers, or issues identified in these studies are necessarily the ones that are involved in interoperability initiatives.It is important to understand which of those factors, barriers, or issues are effectively relevant to interoperability initiatives, as well as what new additional factors, barriers, or issues, not identified in the studies, must be considered in interoperability initiatives.The study [66] is the only one specifically focused on interoperability, though its emphasis is placed at the European local and regional levels of government.

The Delphi Method
The Delphi method originated in the 50's as result of a set of studies performed by researchers at the RAND Corporation (Santa Monica, California), who investigated the scientific use of expert opinion in inexact sciences [38].
Most of those studies ran in the context of projects in the area of military defense.Considering the military nature of the projects, the method was kept private for about twelve years [13].Only in 1963, after U.S. Air Force's decision of declassifying this method from the category "reserved for military use", have Dalkey and Helmer published the first paper introducing and explaining the use of the Delphi method [13].Since then, Delphi has been widely adopted in areas such as marketing, education, health, transportation, and engineering [34], [58].In the IS field it is also possible to find numerous examples of its application [3], [7] - [9], [17] - [18], [30] - [31], [35] - [37], [51] - [55], [57], [59], [61] - [62].
Basically, the Delphi method is a group communication technique especially used to achieve consensus of opinion among a group of experts [13].The method has been developed in order to address some of the problems associated to other forms of interaction used on more conventional techniques of group dynamics [29], [45], such as Focus Groups, Nominal Groups, Brainstorming, Fishbowl, Synectics, and Roudtables.
The Delphi method is an iterative and structured process of listing, refining, and aggregating the opinions and perceptions of a group of people, called the expert panel, that could make valuable contributions to the resolution or understanding of a complex topic or problem in order to create a consensual shared vision on the matter under discussion [14].Typically, this iterative process consists of sending a series of successive questionnaires to a panel of experts, until a certain degree of consensus is reached among them.On each new iteration, besides the questionnaire, the experts should also be sent a summary of the results achieved in the previous round [45].
The Delphi method is characterized by six main features: sampling, anonymity, iteration, controlled feedback, consensus, and statistical aggregation of group response.
Contrary to what happens in most techniques, the sampling of participants in a Delphi study is not based on a random process that seeks to generate representative samples of a given population [34], [55].Instead, the sampling process is focused on the identification and selection of individuals that have a deep knowledge in the area under discussion or study [34], [55].This sampling process is usually known as "expert sampling" [46] or "purposive sampling" [16].
The anonymity among participants is another feature of the method.Owing to the use of questionnaires, the process of collecting the experts' views and opinions can take place without the need for simultaneous physical presence of experts [58].Since the questionnaires are answered singly, it is possible to keep the individual responses anonymous.Maintaining the anonymity of answers allows panel members to express their views and perceptions in a more honest and genuine way, without being subject to social pressures that may result from the presence of experts with more dominant personalities or status [29], [38], [58], [69].
Another feature of the Delphi method is its iterative nature.As mentioned before, Delphi is based on a successive and systematic process of questionnaire sending, each of which called "round".By having to successively answer to questionnaires, experts have the opportunity to think over and over again on the topic and to discover and add new and more mature ideas on it.Although in its most conventional form the Delphi method involves the execution of four rounds, this number can vary, being quite usual to find studies with two or three rounds [59].Also in its conventional form, the first round has an unstructured nature starting with one or more open questions that gives experts the opportunity to freely express their opinions and perceptions about the subject under analysis [58].After the first round, experts' responses are gathered and consolidated into a set of items, based on which more structured questionnaires are designed and used in the next rounds of the study [58].
As has been stressed above, at each new round or iteration, besides the questionnaire, a set of feedback information, which summarizes the views highlighted by the group in the previous round, is also sent to experts [58].Feedback submitted may take different forms, varying from a simple statistical summary of the group opinion, to the inclusion of qualitative justifications given by the participants who have shown more dissonant opinions and perspectives [14], [38].The combination of the iteration and feedback characteristics gives participants the opportunity to revise and amend or reaffirm their views and perceptions as the study proceeds [45].
According to the principles underlying the Delphi method, the decision to stop the iterative process of questioning should be determined by the level of consensus achieved among the experts.The recommendation of the method is that at the end of each round the level of consensus among experts be evaluated and a decision made: to proceed to a new iteration, if the consensus level is not yet adequate to end the study, if such level is considered appropriate [45].
Once the study rounds are completed, the final opinion of the group is obtained by aggregating the views expressed by each of the individual experts in the last round [14].
Due to the features described, namely the use of experts, its iterative nature, and the existence of feedback mechanisms, the Delphi method is generally referred to as being particularly suitable for performing studies aimed at exploring and describing concepts, topics, or phenomena insufficiently mature, for which there is yet no established evidence base [3], [45].The advantage of using a Delphi method in such contexts results from the fact that in a Delphi study the process of knowledge gathering and generation is based on the views and perceptions of experts which have practical experience on the study area [38], [45].Additionally, since the Delphi method has an iterative nature, it is possible to include new ideas and discoveries as the study progresses [61], which allows a more thorough exploitation of the phenomenon, something impossible in an non-iterative form of questionnaire.Finally, the fact that the Delphi method is a group technique that has feedback mechanisms, allowing that the participating experts reconsider their initial choices in the light of opinions and perspectives expressed by the remaining group [61], greatly contributes to greater consistency of the results.
Thus, after a careful assessment of the characteristics of the Delphi method and its comparison with other alternative techniques, such as traditional questionnaire or focus groups, and taking into consideration the type and the exploratory nature of the research questions of this study (focused on discovering the relevant forces, on identifying their level of importance, their type of influence, and their current configuration in the Portuguese PA) the decision to perform a Delphi study seemed to be rather feasible and adequate in this research.

Delphi Study Design
Some of the main criticisms found in the literature about the Delphi method are not related to the method itself but with how the method is used (or misused) by researchers.As noted by many authors, most of the Delphi studies reported in the literature neither explain clearly how the study was planned and executed nor give enough information about the decisions and options come to during its design [34], [38], [55].Considering these criticisms, the following paragraphs set out the decisions made for each of the eight aspects pointed out in the literature as important elements in the design of a Delphi study, namely: panel constitution, experts invitation, starting of the first round, predefined list generation, structure of the questionnaire, stopping criterion of rounds, mode of execution, and feedback information.

Panel Constitution
The constitution of the panel is quoted as one of the most critical issues in a Delphi study.Despite its importance, there are not in the literature clear prescriptions of how this process should be conducted.
One of the few studies that provide a more structured and detailed procedure for the constitution of the panel is authored by Okoli and Pawlowski [55].Based on a set of recommendations proposed by Delcebq to form nominal groups, Okoli and Pawlowski defined an iterative procedure to identify the experts to be included in a Delphi panel.This procedure served as inspiration to the constitution of the expert panel used in this study.The aim of the first step was to define the criterion that should be verified by all the individuals that could be included in the panel.Based on the "expert" definition provided in the Dictionary of the Academy of Sciences-"a person who knows from experience, and that acquired a vast knowledge in a particular subject; a specialist in a given activity" [1]-and being the term "specialist" defined, in the same Dictionary, as "a person who understands and knows a particular subject or matter" [1], the inclusion criterion adopted in the study was set as any individual who has been involved in initiatives that aim at promoting and implementing interoperability between information systems in Public Administration.
The second step was the identification of the main searching niches.A niche represents a generic area or context where it is likely to find individuals that may fit the inclusion criteria defined in Step 1.Three main niches were identified: PA agencies, IS/IT industry, and academy.Since the focus of this work is on the study of IS interoperability initiatives in PA, public administration agencies were the primary and core searching niche for the identification of experts to include in the panel.Besides those, professionals from IS/IT industry that are product suppliers or services providers of PA agencies were also considered eligible to participate in the panel, as well as academics (scholars or researchers) that could have been involved in efforts or initiatives related to the promotion of IS interoperability in PA, particularly through their participation in consultancy and advisory services.
The third step intended to identify the names of the individuals in each niche that could be invited to integrate the panel.The search that was undertaken focused on the Portuguese context.The strategies followed to identify the expert names included the searching for names (1) on finished or ongoing cross-agency projects, (2) on commission, units, institutes, and associations whose objectives, functions, and activities are related to the promotion of interoperability in PA, (3) on the participants list of the Annual Meeting of Heads of Information Systems and Informatics of Public Administration, which is a forum for debate on the issues and challenges facing the PA, (4) on the web sites of PA agencies, (5) on IS/IT publications available in the PA community of practice, and ( 6) on personal contact lists.A total number of 101 individuals were identified.
The 101 individuals identified were invited in Step 4. In addition to the invitation, the individuals were also asked to suggest the names of other experts that they thought could be interesting to include in the panel.

Experts Invitation
The second decision made during the design phase was how to invite the members of the panel to participate in the study.
The two most usual alternatives mentioned in the literature are either to send an invitation asking experts to participate in the study or to send the invitation when the study starts, i.e. to merge the invitation and the start of the first round.Pondering carefully both alternatives, the decision made was to send the invitation before the beginning of the first round.This decision stemmed from the consideration that the experts' likelihood to accept the invitations to participate in the study would be bigger, since psychologically it looks easier for a person to accept an invitation to participate in an activity that will only take place later.Moreover, it was also considered that if an expert had previously accepted to participate in the study, he was going to feel compelled to answer when later he receives the announcement of the beginning of the study, thus leading to higher response rates.

Starting of the First Round
The third decision made was how to start the first round of the study.The two possible alternatives were the blank sheet approach (approach used in the conventional Delphi method) [13], [18], or the predefined list approach (variant that has been frequently adopted) [7] - [9], [19], [27], [30], [51] - [52], [54], [59].Despite the advantages usually attributed to the blank sheet approach, namely the inexistence of any initial bias and the non-imposing of restrictions on the options and creativity of the participants, the decision was to adopt the predefined list approach.In fact, although the adoption of a blank sheet approach could seem to be the "ideal desired" approach for this study, it revealed itself, however, as a more risky alternative, not only because it usually requires the execution of a greater number of rounds [59] but also because it calls for an increased effort by participants, causing higher levels of wear that could lead to higher dropout rates.On the contrary, the existence of a predefined list in the first round could generate in participants a sense that the effort they would have to make would be less than if they were asked to identify and indicate the relevant forces by their own initiative.Thus, while not being the "ideal desired", the adoption of a predefined list was considered the "ideal feasible" option to adopt in this study.

Predefined List Generation
The decision to start the Delphi study with a predefined list of forces immediately raised the question of how to generate that list.
There were three alternatives envisaged at this level, namely to generate the list from a literature review, from a group discussion, or from a mixed strategy that starts with a literature review process which is followed by a group

Structure of the Questionnaire
The fifth decision made was how to structure the questionnaire so that it would allow gathering the data needed to answer the four research questions, namely what are the relevant forces, what are their importance levels, which are their current configurations, and which type of influence do they exert in IS interoperability initiatives.The questionnaire was structured in three main phases, the first devoted to assessing the level of importance of each force, the second devoted to assessing the current configuration and the type of influence of each force, and the third dedicated to the identification of new relevant forces.
The level of importance was evaluated through the use of Q-sort.Q-sort is a ranking technique that follows a structured procedure that makes respondents to analyze and evaluate the importance of the various forces conjointly, being the classification assigned to a force the result of a series of comparative judgments of the multiple forces [25], [47], [50].This procedure also makes the respondent to attribute different importance levels to each of the forces, thus preventing experts to focus their responses in the same area of the classification scale, reducing the possibility of collisions and ties among the forces ranks [9].
The evaluation of the configuration of each force and of the type of influence that each force may exert over IS interoperability initiatives was held in an interrelated form.Both elements were evaluated based on the use of dichotomous nominal scales (true/false for the configuration and driving/restraining for the type of influence).For each of the forces a proposition was made stating one possible configuration of the force, being the experts asked, first, to say if they consider the proposition true or false in the current context of Portuguese PA and, second, to say if they consider that the indicated configuration for the force was driving or restraining the implementation of IS interoperability initiatives.
Lastly, the instrument used for identifying new forces was quite simple, corresponding to a free text area where experts were invited to identify and describe new forces that they deemed to be relevant but that were not included in the existing list.
After its design, the questionnaire was pretested by six academics in order to assess its readability as well as the existence of inconsistencies.

Stopping Criterion of Rounds
The sixth decision made regarded the criterion to stop the iterative process of the rounds execution.According to the principles underlying the Delphi method, the stopping criterion of rounds is determined by the existence of consensus among the experts.The recommendation of the method is that at the end of each round the level of consensus be evaluated, and, based on it, a decision be made: to proceed to a new iteration if the level of consensus is not significant or to end the study if the level is considered appropriate [45].Given the iterative nature of a Delphi study, it was decided that the assessment of the level of consensus would include two components: the level of agreement of the experts' opinion in the round and the level of stability of the global panel opinion between rounds.The level of agreement of the experts' opinion assesses the homogeneity or consistency of the opinion expressed by experts in each round [28].The level of stability of the global panel opinion assesses whether the view expressed by the overall panel stabilizes over the rounds [55], [60] - [61].
The assessment of each of these two components of consensus was based on statistical measures.Since there were three different topics asked in the questionnaire-the level of importance of each force, its configuration, and the type of influence it exerts-different measures were used.Table 9 summarizes the statistical tests performed.All of them are nonparametric tests, since the data generated was either of an ordinal type or of a nominal type.The Kendall's W coefficient of concordance was chosen to evaluate the level of agreement among the experts' individual rankings of importance in a given round.This coefficient is widely recognized not only by the quality of results it produces but also by the simplicity of its implementation [55].The value of W varies between 0 (no agreement) and 1 (perfect agreement) [64].A high value of W means that the n experts are judging de level of importance of the m forces in a similar way.Some of the studies that used this statistical measure were [7], [55], and [59].
The Spearman's rho correlation coefficient was the statistical measure used to evaluate the stability of the global panel rank of importance between rounds.This measure is used to determine the extent of association or correlation between two rankings [64].At the end of each round, the n individual rankings were combined to generate the global panel ranking for that round, which was then used, together with the global panel ranking of the previous round to calculate the Spearman's rho coefficient.Two studies that used this statistical measure were [59] and [68].
The assessment of the level of experts' agreement regarding the configuration and the type of influence of each force was based on the use of Binomial tests.The Binomial test compares proportions and is generally used to test the occurrence of one of the two realizations of a dichotomous variable [46].
Lastly, the assessment of stability of the global panel opinion regarding the configuration and the type of influence of the forces was based on the use of McNemar tests.The McNemar test is used to test scores or proportions of dichotomous nominal variables in two-paired samples.This test is particularly suitable for studies of the type "before vs. after", allowing to test the significance of the change of opinion [46].
Besides the level of consensus, measured by the level of agreement of the experts and the level of stability of the overall opinion of the panel between rounds, an additional condition-the maximum number of rounds-was also included in the stopping criterion defined for this study.Although this condition could seem somehow controversial, as it is contradictory to the basic Delphi principle of keeping on the rounds until consensus is reached, several studies were found in the literature where no consensus was reached among experts, having the studies stopped not because of the consensus level achieved but because of the high number of rounds performed [12], [34].
Shortly, in each round of the study, the decision of stopping or not the Delphi iterative process was made based on the calculated values of Kendall's W coefficient of concordance, Spearman's rho coefficient of correlation, Binomial tests, and McNemar tests, provided that the total number of rounds was not greater than three.

Mode of Execution
The seventh decision made during the design phase concerned the mode of execution of the Delphi study.After a careful reflection, the decision was to execute the process via the Web.The use of a Web system seemed to have multiple advantages when compared with the use of the traditional form of mail.The advantages may occur at four levels: process efficiency level, budget level, data quality level, and response facility level.
The process efficiency gains arise, primarily, from the fact that the use of a Web system makes the entire communication process (from sending to receiving) considerably faster than when it is conducted via traditional mail [20] and, secondly, from the fact that the required data processing time is lower since the data is already in a digital format, being not necessary to make its manual insertion in the statistical analysis application [59].
At the budget level, the benefits are evident, due to the considerable costs that are associated to conventional mail sending, including costs of paper, printing, and shipping.While the use of the Web could have costs related to the acquisition and support of the Web system [59], the amount involved in this research project was insignificant, since the applicational and technological Web infrastructure was already available for use.
The advantages at the data quality level stem mainly from the fact that using a Web system makes the experts to follow the researcher intended response procedure.This was particularly relevant in this study since one of the questions used the Q-Sort technique, which involves the realization of a specific sequence of steps.The data quality could also be better because as the data will be in digital format there would be no need to manually insert it, thus minimizing the errors that frequently are associated with the manually insertion of data.
Finally, the use of a Web system is also advantageous to the experts, since the task of responding to a question that uses Q-Sort technique by traditional means poses a series of requirements, such as the need to have a totally clean table where various paper cards could be placed, which would make the process much more complex and tedious to the experts.
It is worth mentioning that the use a Web system to support the Delphi study required some special attention, namely in what regards the monitorization of the system in order to ensure that it is was always available.Another concern was to guarantee that no data was lost.To prevent such occurrence, daily copies of the database used by the Web system were made.

Feedback Information
The eighth decision made concerned to what feedback information should be sent to the experts in each new round of the study.The provision of feedback information is a typical and core feature of the Delphi method.It allows the sharing of visions and perceptions among the experts, thus contributing to generate a group consensual opinion about the subject.The decision made in this study was to include the following elements as feedback information: • Regarding the level of importance -the rank position obtained by the each force, as well as the sum of points it scored, its average, its standard deviation, and its variance Additionally, it was also provided to each expert access to the individual answer he provided in the previous round of the study.Some qualitative indications, such as not satisfactory, satisfactory, and very satisfactory, about the group's level of consensus reached in the previous round were also included.

Delphi Study Execution
This section briefly describes how the Delphi study was executed.

Delphi Study Results
The results presented in this section reflect the views, opinions, and experience of the 40 experts that participated in the third and last round of the Delphi study, of which Along the study, a total of 31 forces relevant to the implementation of IS interoperability initiatives in PA were identified.

Level of Importance of the Forces
The first request made to the experts in the Delphi questionnaire was to rank the set of listed forces according to their level of importance to the promotion and implementation of IS interoperability initiatives in PA.The result of this request is systematized in Table 12.Beyond the rank position of each force, the table also contains additional descriptive statistical measures, such as the average, standard deviation, minimum, maximum, range, and interquartile range that help in the understanding and interpretation of the results. 1 ∆ of Average is the difference between the average value obtained by the force that is in the position rank ri and the average value obtained by the force in the position rank ri-1. 2 SD stands for Standard Deviation, and it is a statistical measure of the dispersion of the values of the observations around of the average value. 3Min is the minimum value attributed to a force (which shows the best ranking position attained by the force). 4Max is the maximum value attributed to a force (which shows the worst ranking position attained by the force). 5Range is the difference between the maximum value (Max ) and the minimum value (Min ) of a force. 6IQR stands for InterQuartile Range, and it is obtained by the difference between de 3rd Quartile (Q3) and the 1st Quartile (Q1).* Force not evaluated in Round 1.This force was suggested by experts in Round 1, being submitted to experts' evaluation only in Rounds 2 and 3.
The first evidence that stands out from the table is the high importance caught by the force Political Will.The importance of this force is notorious, not only by the first rank position it obtained, but also from the frequency of responses that it got.As can be seen from Figure 1, 28 of the 40 experts ranked that force precisely the first rank position, meaning that 28 of the 40 experts considered Political Will as the most important force among the 31 forces under evaluation.Only three experts ranked Political Will out of the top 10 positions.12, it becomes clear that Political Will was the force that gathered the highest level of convergence among experts' opinions.Indeed, though the range of this force is still of 15 positions, it is the lower when compared with the ranges of the other 30 forces.In the same sense, the average value of this force also reveals the great importance attributed to this force.Not only is this value the lowest of all the forces, as the difference between the average value of this force (Political Will) and the next force in the ranking (Involvement and Commitment of Agencies) is of 3.33 points, which corresponds to the largest "jump" found in the table.
Together, these facts show the existence of a strong conviction among the experts on the importance that the force Political Will assumes in the promotion and implementation of IS interoperability initiatives in PA.Actually, a favorable and genuine Political Will, characterized by the existence of a strong sponsorship and commitment on the part of politicians, can be highly beneficial to achieve interoperability in PA, since politicians may (1) instigate and promote interoperability projects; (2) channel funds for such type of initiatives; (3) promote the public debate on fundamental but controversial questions, such as privacy and personal data protection; (4) facilitate the creation or amendment of the required legislation, and ( 6) unblock and make disappear many bureaucracies usually surrounding this type of initiatives.
The support of politicians can also be fundamental to "make agencies to participate" in cross-agency initiatives.Because interoperability initiatives involve the participation of various agencies, such initiatives require the harmonization of multiple and different strategies, processes, people, and technologies.This process of harmonization is not a trivial task, being most of the times difficult to achieve without the intervention of political agents.
The second evidence from the table is the existence of different groups of importance.As can be seen from the Δ of Average column, which contains the difference between the averages of forces in successive ranking positions, there are some considerable "jumps" between some forces, particularly between rank 1 and rank 2, between rank 5 and rank 6 and between rank 30 and rank 31.These "jumps" reveal the existence of four groups of importance.The first group, whose elements may be considered as having extreme importance, includes the Political Will force.The second group, which can be considered as comprising the very important forces, includes forces in rank positions 2, 3, 4, and 5.The third group, containing the important forces, includes the remaining forces of the list, with the exception of the force Constitutional Principles, which, alone, constitutes the fourth group, regarded as the least important.Thus, despite the importance that all forces can take in the course of interoperability initiatives, the five forces belonging to the first and second groups, namely Political Will, Involvement and Commitment of Agencies, National Structure for Interoperability Governance, Standards for Interoperability, and Intra-and Interagency Leadership, should receive special attention.
In fact, it is clearly understandable that the existence of involvement and commitment on the part of all agencies constitutes a fundamental condition to the success of interoperability.Indeed, since, by definition, interoperability always involves the conjunct operation of two or more entities, it will only be achievable if all the parts are actively involved and committed, being the presence of each of the agencies absolutely indispensable.
The importance associated to the existence of a National Structure for Interoperability Governance is also understandable.Considering, as mentioned in section 2.1, that, to achieve e-government interoperability, more important than to find out the way of "being together", is to find out "the most effective way of being together", it seems fundamental to address the interoperability endeavor in a comprehensive, rational, structured, and sustained way.This means that the interoperability must be thought in a global form: it must be ensured the consistency among the policy and the technical strategies of the multiple agencies; it must be promoted the convergence of the agencies' actions in crucial areas such as security and privacy; there must be defined and proposed appropriate funding mechanisms; there must be defined global and rigorous mechanisms for monitoring interoperability progress, and there must be identified and provided shared services common to several agencies in order to avoid the duplication of effort and costs.In other words, it seems therefore important to define and implement a governance model for interoperability focused on the overall direction and control of the multiple collective efforts for interoperability.The setting of central and continuous processes and mechanisms of governance, that allow the establishment of unified policies, consistent architectures, common specifications and standards, and compatible procedures and infrastructures, are critical factors to build a truly interoperable PA.

Configuration and Type of Influence of the Forces
The second request made to experts in the Delphi questionnaire was to state the configuration of each force in the current context of the Portuguese PA as well as the type of influence exerted by each force in the IS interoperability initiatives.As already mentioned, the way found to get the answers to these questions was based on the formulation of a statement describing a possible configuration of each force and on asking experts firstly whether that statement was true or false in the current context of Portuguese PA and secondly how the current configuration indicated could influence (driving or restraining) the implementation of IS interoperability initiatives in PA.Table 13 summarizes the responses given by experts.Considering that an expert evaluated the type of influence based on the answer he gave about the configuration of the force, the percentages concerning the type of influence exerted by each force are presented, in the table, separately for the two possible situations: the situation that corresponds to a true configuration and the situation that corresponds to a false configuration.

Configuration of Forces in Current Context of Portuguese PA
Using the percentages presented in Table 13, it was possible to advance some specific statements about the configuration of each force in the current context of Portuguese PA.The statements proposed were only those to which statistical support was found, i.e. the statements that refer to forces to which the Binomial test produced evidence values (p-values) lesser or equal than 0.05, which means that there is a statistically significant difference between the number of experts who considered the statement presented as true or as false.As can be seen from the Binomial test values available in Appendix C, this criterion (p-value <= 0.05) is verified for 27 forces, thus allowing the formulation of the 27 propositions presented in Table 14.

Type of Influence Exerted by the Forces
The percentages shown in Table 13 also allowed the formulation of statements about the type of influence exerted by each force in the promotion and implementation of IS interoperability initiatives in PA.Again, the statements formulated were only those to which statistical support has been achieved, i.e. the statements that refer to situations to which the Binomial test produced evidence values (p-values) lesser or equal than 0.05, which means that there is a statistically significant difference between the number of experts who considered that the force in the situation described would have a driving or a restraining role.
Since the evaluation of the type of influence was dependent on the evaluation of the configuration, the Binomial tests for the type of influence had to be done twice for each force, performing a total of 62 tests (2 tests for each of the 31 forces).
As can be seen from the Binomial test values available in Appendix E, the criterion p-value <= 0.05 is verified in 38 situations, allowing the formulation of 38 generic propositions, 24 describing situations in which a force has a restraining or limiting influence on IS interoperability initiatives (Table 15) and 14 describing situations in which a force has a driving or facilitating influence on IS interoperability initiatives (Table 16).

RP13
-The Perception that the obtained Net Benefits of an Agency, when participating in an IS interoperability initiative, will be negative or that there is no fairness and balance in the distribution of benefits and costs among the various participating agencies in the initiative is a limiting factor for the implementation of IS interoperability initiatives in PA

Constellation of Forces Acting in Current Context of Portuguese PA
Taking the general results set forth previously about the level of importance of each force (Table 12) and the type of influence that a force may exert (Table 15 and Table 16) and combining them with the specific configuration that each force assumes in the current context of Portuguese PA (Table 14), it was possible to depict the global constellation of forces acting on the phenomenon of promotion and implementation of interoperability between IS in the current context of the Portuguese PA.
The simple and intuitive way found out to gather this information was through the use of a Force Field Diagram (FFD), whose concept was introduced by Kurt Lewin.As the name indicates, a FFD is a graphical representation of the set of forces that act, in a given moment, on an entity (individual, group, or organization), phenomenon, or situation.Schematically, a FFD has the configuration of a "T": on the left side of the vertical axis of the "T" are represented the forces driving or facilitating the occurrence of the phenomenon and in the right side of the axis are represented the forces restraining or limiting the phenomenon.According to the terminology and rules of FFD diagrams construction, each of the 31 forces identified and evaluated by the experts in the Delphi study were represented by an arrow.The arrow direction is determined by the type of influence that the force exerts on IS interoperability initiatives and its length is determined by the position achieved by the item in the ranking of importance (longest arrows correspond to more important forces and shorter arrows to less important forces).The forces were presented along the vertical axis in order of importance.Thus, an arrow that is in a higher position in the vertical axis represents a force that has obtained a top position in the ranking of importance, and that was deemed more important by the panelists.
Each arrow has two text elements associated: one above the arrow which shows the name of the force and the other below the arrow which states the configuration of the force in the actual context of the Portuguese PA.To visually reinforce the type of each force, red color was used to represent restraining forces and green color to represent driving forces.The resulting FFD is depicted in Figure 2.
As may be seen from the Looking to the top five, that contains de forces classified as extreme and very important, it can be seen that three of them (National Structure for Interoperability Governance, Standards for Interoperability, and Intra-and Interagency Leadership) are restraining the promotion and implementation of IS interoperability initiatives in the current context of Portuguese PA.Thus, according to the experts' opinion expressed in this study, these three particular forces should be carefully managed and some forms of intervention should be identified and put in place in order to mitigate the restraining or limiting effects caused by the current configuration that theses forces assume in Portuguese PA in order to be able to achieve the adequate and desirable levels of interoperability.

Conclusion
The work described in this paper constitutes a step forward in the search for a deeper understanding on the complex phenomenon of achieving more adequate, more sustained, and more sustainable levels of interoperability in Public Administration.
The aim of this study was to unveil the myriad of forces that may influence and determine the implementation of IS interoperability initiatives in PA.More specifically, this paper sought to identify the various aspects of the realityforces-acting on IS interoperability initiatives in PA, as well as the level of importance assumed by each of those forces and the type of influence that each force may exert on the interoperability initiatives.
In order to achieve this aim, a Delphi study was performed.Based on the Delphi study, a set of "instruments" was produced, namely: 1.A list of 31 forces relevant to IS interoperability initiatives in PA (Table 12).These forces have different natures-political, legal, organizational, semantic, and technical-evidencing and supporting the arguments about the multidimensionality of the interoperability construct.
2. The ranking of importance of the 31 forces, showing the relative level of importance of each of the forces (Table 12).This ranking brought to evidence the existence of four clusters of forces: the extreme important, the very important, the important, and the least important forces.
3. A set of 24 propositions describing restraining or limiting influences of the forces on the promotion and implementation of interoperability initiatives in PA (Table 15).
4. A set of 14 propositions describing driving or facilitating influences of the forces on the promotion and implementation of interoperability initiatives in PA (Table 16).
Besides these four "instruments", the Delphi study also allowed the formulation of a set of 27 propositions that characterize the configuration assumed by each force in the current context of Portuguese PA (Table 14).Combining and instantiating the previous four "instruments", in the light of these 27 configuration propositions, a FFD was drawn representing the overall constellation of forces acting on the phenomenon of IS interoperability in the specific current context of Portuguese PA (Figure 2).
This study represents a valuable contribution to the implementation and management of IS interoperability initiatives in PA.Indeed, besides alerting the practitioners to the set of aspects that may affect this type of initiatives, the study also offers practitioners new insights, concerning the level of importance assumed by each of those aspects as well as about the type of influence they may exert in interoperability initiatives, that haven't, until now, been covered in the literature.
In a concrete way, PA professionals may easily use the four "instruments" produced in this study to make a diagnostic of their PA readiness status to interoperability achievement that may help them to manage its promotion and implementation.The FFD generated in this study to the specific Portuguese PA is an example of that.Indeed, looking at that FFD (Figure 2), Portuguese PA professionals have an overall perspective of what is relevant, how relevant it is, and how it is affecting the promotion and implementation of IS interoperability in PA in Portugal.In the same sense, this set of "instruments" may be used by any other particular PA to generate its specific FFD.The only thing that this other PA needs to do is to identify its specific set of configuration propositions.Having that set of propositions (a set of propositions similar to the one presented in Table 14), it is possible to generate the specific FFD for that particular PA, and thus to have an overall perspective of the specific constellation of forces acting on it, based on which the practitioners may orientate their actions in order to achieve a constellation of forces that is more favorable to the promotion and implementation of interoperability in PA.Additionally, PA professionals can also use these "instruments" to keep track of how they are progressing towards the creation of a more favorable constellation of forces to interoperability.Indeed, if they repeat the diagnostic periodically and compare the multiple generated FFDs along the time, they will be able to understand how well are they directing their efforts in order to promote and implement IS interoperability in PA.
Besides being beneficial to the community of practice, the results of this study are relevant to the research community, contributing to the enrichment of the body of knowledge in the area of e-government interoperability.
While the inclusion of the quest of the level of importance and of the type of influence of each force constitutes a step forward toward the understanding of the complex of forces acting in interoperability initiatives, this study has however two limitations.Both of these limitations refer to ideas and principles argued by Kurt Lewin, in his Field Theory, that were not contemplated in this study, namely the principle of the interdependence of the forces and the dynamic nature of the forces.
Appendix A: Names and Descriptions of the Forces Submitted in Round 1

Changes in the Statu Quo of Agencies
It refers to the existence of internal changes in agencies, particularly as it relates to their values, beliefs, attitudes, habits, and mode of operation, due to their involvement in interoperability initiatives

Condition of Member State of the European Union
It refers to the fact that Portugal, given its status as member state of the European Union, has some rights, obligations, and commitments, particularly in the areas of Information Society and Electronic Government, whose satisfaction may impact on the promotion and establishment of an appropriate environment for the creation of interoperability between IS in Public Administration

Conflict of Interests
It refers to possible disagreements and disputes between agencies, resulting from the existence of disparate and incompatible motivations, objectives, priorities, security requirements, or data quality requirements, between agencies

Constitutional Principles
It refers to the existence of a set of fundamental principles enshrined in the Constitutional Law, such as the principle of separation of powers and the principle of local autonomy, that can influence the promotion and establishment of a suitable environment for the creation of IS interoperability in PA

Disturbance in the Autonomy, Power, and Prestige of Agencies
It refers to the agencies' feeling that their independence and freedom of action (with respect to design options for IS and decisions about which resources to share, which way to do it, and when to do it) and their own identity and public image may be diminished or impaired as the result of their participation in interoperability initiatives

Domain Ontology for Public Administration
It refers to the existence of a formal representation of the concepts used in the PA domain that clarifies the definitions, meanings, attributes and interrelationships that, as a whole, constitute a global specification of the semantics of that domain

Electronic Signature
It refers to the existence of mechanisms, legally recognized and valid, that grant an individual and exclusive right and that may be used for publicizing the authorship of the documents to which is affixed, in a way that: (a) it uniquely identifies the holder as the author of the document, (b) its affixation to the document depends only on the holder will, and (c) its connection to the document allows the detection of any change of its contents

Experience of Institutional Collaboration
It refers to the existence of any history of relationship and cooperation between agencies that are involved in an interoperability initiative

Financial Resources
It refers to the financial capacity, and to the adequacy of existing funding models used, to support the initiatives of IS interoperability in PA

Human Resources
It refers to the amount of human resources available in public agencies, as well as to its capabilities to manipulate the IT involved and required for the implementation of interoperability between IS in Public Administration

Information Security
It refers to the necessity of ensuring the preservation of confidentiality, integrity, and availability of information gathered, manipulated, and exchanged by the agencies involved in interoperability initiatives

Interagency Trust
It refers to the ability demonstrated by each of the agencies to believe that the agencies with which they will interact and collaborate during an interoperability initiative, exhibit a behavior that is in line with existing expectations with regard to aspects such as honesty, intent, integrity and security

Intra-and Interagency Leadership
It refers to the existence of someone with ability to influence, motivate, engage, guide, and coordinate the various parts involved in the interoperability initiatives, whether at the internal level of each particular agency (intra-agency leadership) or at the global level in order to promote the coordination between the various parts involved in interoperability initiatives (interagency leadership)

Involvement and Commitment of Agencies
It refers to the willingness expressed by the agencies to participate and collaborate in a genuine and interested way in IS interoperability initiatives in PA

Methodological Frameworks for Interoperability
It refers to the existence of contributions from academic, scientific and practical origins, in the form of frameworks, models, or theories, that can be used to guide the efforts to develop interoperability between IS in PA

Monitoring and Reconstruction of Cross-Agency Processes
It refers to the need of having mechanisms to perform the monitoring and controlling of the execution state of a transversal process, as well as to allow the whole reconstruction of the end-to-end process, with regards to operations, people involved, and documents created and exchanged in the overall transactions

National Structure for Interoperability Governance
It refers to the existence of an entity at the national level with specific responsibility to promote and facilitate the creation of an environment conducive to the development of IS interoperability in government, particularly through actions such as the definition of a national strategy for interoperability, the definition of frameworks and guidelines, the harmonization of national efforts and European Commission's recommendations, the creation and provision of infrastructure and common services that can be used by various agencies, and the dissemination and exchange experiences

Net Benefits Perceived by Agencies
It refers to the perception that an agency has about the difference between the benefits it foresees to attain and the costs it estimates to incur when participating in an interoperability initiative, as well as its perception about the existence of a fair and balanced distribution of the benefits and costs by the various agencies involved in that initiative

Partnerships with Private/Public Entities
It refers to the creation and promotion of collaborative relationships with private sector organizations, particularly those working in the field of information technology, as well as universities and public or private institutes

People's Attitude Toward Change
It refers to the posture, evaluation, opposition, or feeling exhibited by PA agency professionals about the changes that can occur due to their participation and involvement in interoperability initiatives The adaptation ofOkoli  and Pawlowski's procedure to the context of this study resulted in the execution of the following five steps: Step 1 -Define inclusion criteria; Step 2 -Define key searching niches; Step 3 -Populate niches with names; Step 4 -Invite experts and request indication of new experts; and Step 5 -Invite new experts.

Finally, in Step 5 ,
the thirty-nine new names suggested in Step 4 were invited to participate in the study.In total, 140 experts were invited (101 identified in Step 4 and 39 in Step 5), of which 89 (63%) belonged to Public Administration agencies, 36 (26%) to IS/IT industry and 15 (11%) to academy.

•
Regarding the configuration and type of influence -the percentages obtained by each force in each of the classification categories available for each of the two evaluated elements (percentage of true and false answers in what concerns the force configuration and percentages of driving and restraining in what concerns the type of influence exerted by the force)

Figure 1 :
Figure 1: Frequency distribution of Political Will Looking to the values in columns SD, Range, and IQR in Table12, it becomes clear that Political Will was the force that gathered the highest level of convergence among experts' opinions.Indeed, though the range of this force is still of 15 positions, it is the lower when compared with the ranges of the other 30 forces.In the same sense, the average value of this force also reveals the great importance attributed to this force.Not only is this value the lowest of all the forces, as the difference between the average value of this force (Political Will) and the next force in the ranking (Involvement and Commitment of Agencies) is of 3.33 points, which corresponds to the largest "jump" found in the table.

CP17-
There are Political and Budgetary Cycles and its occurrence may affect the implementation of IS interoperability initiatives in PA CP18 -There is not enough Transparency in Public Agencies in what concerns the public availability either of their enterprise architecture or of its services quality indicators and level of services CP19 -There is a set of legal provisions concerning the Privacy and Protection of Personal Data CP20 -There is Semantic Incompatibility among the same type of information and concepts of the various public agencies CP21 -There is a set of laws and regulations that determines the Authority and Responsibility of each agency and that legally prevents it from acting outside the power established in those diplomas, as well as prevents it to trust their responsibilities to other agencies CP22 -Portugal is a Member State of the European Union CP23 -There is not Technological Incompatibility among agencies CP24 -There are not Disturbances in the Autonomy, Power, and Prestige of the Agencies resulting from their participation in IS interoperability initiatives CP25 -There are Conflict of Interests among agencies CP26 -There are Changes in the Statu Quo of Agencies, particularly as it relates to their values, beliefs, attitudes, habits, and mode of operation, resulting from their involvement in IS interoperability initiative CP27 -There exist Partnerships with Private/Public Entities

RP14-
The inexistence of an Enterprise Architecture for Public Administration is a limiting factor for the implementation of IS interoperability initiatives in PA RP15 -The lack of adequate levels of Interagency Trust is a limiting factor for the implementation of IS interoperability initiatives in PA RP16 -The inexistence of an Ontology for the Domain of Public Administration is a limiting factor for the implementation of IS interoperability initiatives in PA RP17 -The lack of Institutional Collaboration Experience between agencies is a limiting factor for the implementation of IS interoperability initiatives in PA RP18 -The existence of Political and Budgetary Cycles is a limiting factor for the implementation of IS interoperability initiatives in PA RP19 -The lack of Transparency in Public Agencies in what concerns the public availability either of their enterprise architecture or of their services quality indicators and level of services, is a limiting factor for the implementation of IS interoperability initiatives in PA RP20 -The existence of Semantic Incompatibility between the same type of information and concepts of the various public agencies is a limiting factor for the implementation of IS interoperability initiatives in PA RP21 -The fact that the power and scope of activity of each agency are determined in its organic law and regulations (fact that legally prevents the agencies from acting out of the power established in these legal diplomas and to delegate their responsibilities to other agencies) is a limiting factor for the implementation of IS interoperability initiatives in PA RP22 -The existence of Technological Incompatibility between agencies is a limiting factor for the implementation of IS interoperability initiatives in PA RP23 -The inexistence of Disturbances in the Autonomy, Power and Prestige of the Agencies is a limiting factor for the implementation of IS interoperability initiatives in PA RP24 -The existence of Conflict of Interests between agencies is a limiting factor for the implementation of IS interoperability initiatives in PA

Table 1
systematizes eight studies particularly relevant to the aim of this paper.The studies are listed in chronological_order.

Table 1 :
Characterization of studies on information sharing, information integration, G2G, and interoperability • List of seven benefits and seven risks of interagency information sharing, organized in three categories: Technical, Organizational, and Political • Model for interagency information sharing • List of barriers and recommendations for interoperability • Extended theoretical model of Interagency Information Sharing (expands the model of interagency information sharing proposed by Dawes) • The author found out that local government participation in electronic information sharing with state agencies is determined by agency characteristics, environmental characteristics, and electronic information sharing characteristics • Set of 33 recommendations to increase local agency participation in electronic information sharing initiatives, organized in three categories: agency characteristics, environmental characteristics, and electronic information sharing characteristics • List of three key factors to the implementation of G2G initiatives • List of nine barriers that can arise during G2G projects, organized in three categories: Structural, Human, and Technical • Heuristic model for the implementation of G2G • List of 32 components of information integration, organized in four categories: Social processes, Resources, Organizational artifacts, and Technology artifacts • List of barriers to interagency information integration • Set of seven strategies for interagency information integration • The author found out that back-office integration should be seen as a governance problem • List of six lessons learned about back office integration projects • List of 39 interoperability key factors, organized in four categories: Technical interoperability, Semantic interoperability, Organizational interoperability, and Interoperability governance • List of 44 recommendations regarding interoperability, organized according to the type of stakeholder to whom they are targeted (European Union, National, and Local/Regional level) and to the four areas of application of recommendations (Legislation, Funding/Financial, Policy/Management, and Technical)
Finally, Akbulut proposes a set of 33 recommendations, clustered in those three types of characteristics, which could increase local agency participation in electronic information sharing initiatives.

Table 4 :
[2]racteristics that influence, might influence, or have no influence on local agency participation in electronic information sharing with state agencies[2]

Table 7 :
[66]factors for e-government interoperability at local and regional level[66]

Table 8 :
List of forces

Table 9 :
Statistical tests used to evaluate the level of consensus

Table 10
The list of forces submitted to evaluation contained 28 forces in Round 1, 31 forces in Round 2 (the 28 forces of the predefined list used in Round 1 plus three new forces suggested by experts in Round 1), and 31 forces in Round 3 (the same forces that were submitted in Round 2).Table11contains the names of the three new forces suggested by experts in Round 1.A brief description of each of those forces is provided in Appendix B.Taking into account the value of the statistical tests performed for each of the three topics questioned (level of importance, configuration, and type of influence) the level of consensus in Round 1 and Round 2 were evaluated as not enough.In Round 3 the statistical values calculated revealed the existence of sufficient levels of consensus, thus leading to the study termination.The detailed values of the Binomial and McNemar tests are available in Appendices C, D, E and F.
summarizes the main figures of the study execution.

Table 10 :
Characterization of the rounds

Table 11 :
New forces suggested by experts in round 1

Table 12 :
Ranking of importance of the forces

Table 13 :
Configuration and type of influence of each force

Table 14 :
Specific propositions describing the configuration of each force in the current context of Portuguese PA CP1 -There is a favorable Political Will regarding the promotion and establishment of an appropriate environment for the implementation of IS interoperability initiatives in PA CP2 -There is Involvement and Commitment of the Agencies in the implementation of IS interoperability initiatives in PA CP3 -There is not a National Structure for Interoperability Governance with specific responsibility to promote and facilitate the creation of an appropriate environment for the implementation of IS interoperability initiatives in PA CP4 -There are not Standards for Interoperability agreed upon and used by PA agencies CP5 -There is not leadership, either at the internal level of each agency (intra-agency leadership) or at the overall level (interagency leadership), in the initiatives that aim to create interoperability between information systems in Public Administration CP6 -There are not mechanisms for the Monitoring and Reconstruction of Cross-Agency Processes execution CP7 -There are not Control Structures or Control Mechanisms for the Interoperability Initiatives CP8 -There are not enough Financial Resources neither adequate financial models to support the implementation of IS interoperability initiatives in PA CP9 -There are not Human Resources in sufficient quantity and with the required skills to participate in the implementation of IS interoperability initiatives in PA CP10 -There are Methodological Frameworks of academic, scientific, or practical nature that can be use to guide the implementation of IS interoperability initiatives in PA CP11 -There are mechanisms for Information Security which allow the preservation of confidentiality, integrity, and availability of information gathered, manipulated, and exchanged by agencies participating in IS interoperability initiatives CP12 -There is not a clear definition and a clear recognition of the Responsibilities attributed to each agency in the Execution of Cross-Agency Processes typically involved in IS interoperability initiatives CP13 -There is not an Enterprise Architecture for Public Administration, neither at the global PA level, nor at the level of each of its agencies CP14 -There is not an Ontology for the Domain of Public Administration

Table 15 :
Generic propositions describing restraining influences on IS interoperability initiatives in PA RP1 -The lack of Involvement and Commitment of Agencies participating in IS interoperability initiatives in PA plays a crucial limitation on the implementation of such initiatives RP2 -The inexistence of a National Structure for Interoperability Governance is a limiting factor for the implementation of IS interoperability initiatives in PA RP3 -The lack of Standards for Interoperability is a limiting factor for the implementation of IS interoperability initiatives in PA RP4 -The lack of leadership either at the internal level of each agency (Intra-Agency Leadership) or at the overall level of the initiative (Interagency Leadership) is a limiting factor for the implementation of IS interoperability initiatives in PA RP5 -The absence of mechanisms for Monitoring and Reconstruction of Cross-Agency Processes is a limiting factor for the implementation of IS interoperability initiatives in PA RP6 -The lack of Control Mechanisms for Interoperability Initiatives is a limiting factor for the implementation of such initiatives in PA RP7 -The absence of a favorable People's Attitude Toward Change is a limiting factor for the implementation of IS interoperability initiatives in PA RP8 -Either the lack of Financial Resources or the lack of adequate financial models in PA are limiting factors for the implementation of IS interoperability initiatives in PA RP9 -The lack of Human Resources in sufficient quantity and with adequate skills is a limiting factor for the implementation of IS interoperability initiatives in PA RP10 -The lack of Methodological Frameworks for Interoperability is a limiting factor for the implementation of IS interoperability initiatives in PA RP11 -The absence of mechanisms for Information Security which allow the preservation of confidentiality, integrity, and availability of the information gathered, manipulated, and exchanged by agencies participating in IS interoperability initiatives is a limiting factor for the implementation of such initiatives RP12 -The lack of a clear definition, and of a clear recognition, of the Responsibilities attributed to each agency in the Execution of Cross-Agency Processes is a limiting factor for the implementation of IS interoperability initiatives in PA

Table 16 :
Generic propositions describing driving influences on IS interoperability initiatives in PA DP1 -The existence of a favorable Political Will, regarding the promotion and establishment of an appropriate environment for the implementation of IS interoperability initiatives in PA, is a facilitating factor for these type of initiatives DP2 -The existence of Involvement and Commitment on the part of the Agencies participating in IS interoperability initiatives in PA is a facilitating factor for the implementation of these type of initiatives DP3 -The existence of Standards for Interoperability is a facilitating factor for the implementation of IS interoperability initiatives in PA DP4 -The existence of leadership either at the internal level of each agency (Intra-Agency Leadership) or at the overall level of the initiative (Interagency Leadership) is a facilitating factor for the implementation of IS interoperability initiatives in PA DP5 -The existence of a favorable People's Attitude Toward Change is a facilitating factor for the implementation of IS interoperability initiatives in PA DP6 -The existence of Financial Resources and adequate financial models in PA are facilitating factors for the implementation of IS interoperability initiatives in PA DP7 -The existence of Methodological Frameworks for Interoperability is a facilitating factor for the implementation of IS interoperability initiatives in PA DP8-The existence of mechanisms for Information Security which allow the preservation of confidentiality, integrity and availability of the information gathered, manipulated and exchanged by agencies participating in IS interoperability initiatives is a facilitating factor for the implementation of such initiatives DP9 -The Perception that the obtained Net Benefits of an Agency, when participating in an IS interoperability initiative, will be positive or that there is fairness and balance in the distribution of benefits and costs among the various participating agencies in the initiative is a facilitating factor for the implementation of IS interoperability initiatives in PA DP10 -The existence of adequate levels of Interagency Trust is a facilitating factor for the implementation of IS interoperability initiatives in PA DP11 -The existence of Institutional Collaboration Experience between agencies is a facilitating factor for the implementation of IS interoperability initiatives in PA DP12 -The existence of Electronic Signature mechanisms, which are legally recognized and valid, is a facilitating factor for the implementation of IS interoperability initiatives in PA DP13 -The fact that Portugal is a Member State of the EU is a facilitating factor for the implementation of IS interoperability initiatives in Portuguese PA DP14 -The existence of Partnerships with Private/Public Entities is a facilitating factor for the implementation of IS interoperability initiatives in PA diagram, the forces People's Attitude Toward Change, Net Benefits Perceived by Agencies, Interagency Trust, Privacy and Protection of Personal Data, Technological Incompatibility, Changes in the Statu Quo of Agencies, and Constitutional Principles are not depicted, since there was not statistical significance to allow their representation.It is quite clear from the diagram the existence of a large number of forces restraining or limiting the progress of IS interoperability initiatives in PA.Indeed, from the 24 forces represented in diagram, only 7 forces assume, in the current context of Portuguese PA, a driving or facilitating role, being 17 of them restraining factors of such initiatives.