Using a Method and Tool for Hybrid Ontology Engineering: an Evaluation in the Flemish Research Information Space

We report on the results of the application of a method and tool for ontology construction in the research information domain, held in the context of an open data initiative of Flanders. The method emphasizes the use of natural language descriptions of concepts next to formal descriptions, and uses – for the formal definitions – a fact-oriented formalism grounded in natural language. In this experiment, a group of 36 participants were divided into different groups to build ontologies to establish semantic interoperability between autonomously developed research information systems and to annotate the data of an existing system provided by a public administration. User satisfaction of the tool was measured with the Post-Study System Usability Questionnaire. The result of that survey was that the participants were generally pleased with the platform, with its usefulness scoring best. As for the developed ontologies, their use was demonstrated by the applications developed by the participants. The experiment showed that having a formalism grounded in natural language leverages the ontology construction process for the stakeholders. The experiment also shows that a method needs to take into account the collaborative building of workflows within ontology projects, as not all ontology-engineering projects are alike.


Introduction
For a country or region in the current knowledge economy, it is crucial to have a good overview of its science and technology base to develop an appropriate policy mix of measures to support and stimulate research and innovation.Research information systemsalso called Current Research Information Systems [30] are used by the various actors in the research process and their use ranges from "the documentation of research projects and their results over easing the management of research to simplifying research assessment" [4].Those systems also provide a means for giving that overview to all actors and stakeholders.Companies, research institutions and individual researchers can then profit from the information maintained in such a research information system.The development of research information systems require a clear focus on a bounded set of indicators or items of interest that are agreed upon by the stakeholder group to ensure the proper support of aforementioned research process [4].Different systemseach with different (organizational) contexts and requirementsare developed and maintained autonomously.
In Flanders, the Department of Economy, Science and Innovation of the Flemish Government (from here on called EWI) (Site 4) decided to launch the Flanders Research Information Space program (FRIS) (Site 5) to create a virtual research information space covering all Flemish players in the field of economy, science and innovation.The current version of this portal contains, for instance, mash-ups of data on key entities (such as researcher, organization, project; and their relationships) on a geographical map as shown in Figure 1.Another aim of FRIS is to reduce the current administrative burden for universities as they are confronted with repeatedly reporting the same information in different formats to various institutions.Indeed, if all information would be centralized and accessible in a uniform way, creating services for such reports would greatly facilitate the reporting process.The goal of EWI is thus to gather and centralize the information currently residing in the autonomously developed and maintained research information systems of Flanders.Of course, before data can be centralized, this initiative faces two problems: 1) capturing the semantics of the domain in an ontology and 2) appropriately annotate or commit the heterogeneous data sources to that ontology.Solutions to this problem have already been reported in [15], [46], [47], [54].Next to EWI, also the European Union (EU) launched an initiative to tackle to first problem of capturing the semantics, as the EU also wants to track all research information in Europe.The EU asks all knowledge institutions to report their (EU funded) research activities using the Common European Research Information Format (CERIF) [31], a recommendation to EU-members for the storage and exchange of current research information.While the CERIF model, created with Entity-Relationship (ER) diagrams, allows for an almost unlimited flexibility on roles and classifications used with entities, the actual approach has shown its limitations when it comes to communicating the modeled domain facts to domain experts and end users [46].The learning curve for the domain experts to understand the ER model and translate it back to the conceptual level is quite steep.The problem with CERIF is the meta-model behind this format being a meta-meta-model.For instance take the role of a person being the project coordinator of a project, as shown in Figure 2. The person will be recorded as an instance of cfPerson and the project as an instance of cfProject.Their relationship will be captured as an instance of cpProject_Person.The type (classification) of that relationship, however, will be added as an attribute to that relationship; an instance of cfClassification.One thus needs to agree upon the classification schemes in order to guarantee a group of stakeholders to sufficiently understand the exchanged data.Even though the effort provided a means for structuring the data, in did not leverage the problem of coming to the necessary agreements to ensure meaningful exchange of information.The problem is, in other words, only partly solved.
End user and stakeholder involvement will be key for success.This aspect has been touched upon in the list of five myths about the open data and open government initiative by Janssen et al. in [28]: 1.The publicizing of data will automatically yield benefits.In which the authors stress that not taking into account the user using the data will be counterproductive and thus the threshold for usage of that data should be part of the data provider's policy.
2. All information should be unrestrictively publicized.In which the authors point out some issues such as privacy and information quality as factors for not publishing everything.

3.
It is a matter of simply publishing public data.As stated by the authors: A key issue is not to link from the bottom up, but also to use meta-data in the linking.Meta-data is necessary to overcome barriers like searching, interpretation and so on.

4.
Every constituent can make use of open data. in which the fact that everyone can (or should) be able to use all open data is debunked by the sheer fact that not everyone has the necessary expertise and/or resources to conduct analysis on top of open data.

Open data will result in open government.
Where the authors argue why the first does not necessarily leads to the other.
There first two myths aim to convince data providers to publish their data and the second two myths show that the end users are largely neglected [28].Myths 1 and 3 are interesting to look at as they give the end users a more prominent role.The term meta-data is very ambiguous.On one hand the term can refer to data about data such as provenance information (such as source and ownership) as well as the annotations necessary for one to able to interpret the data, in other words: ontologies.Ontologies are commonly defined as "a [formal] explicit specification of a [shared] conceptualization" [22] and constitute the key resources for realizing a Semantic Web [3].Note that the original definition did not contain mention to words formal and shared, but are commonly accepted to be included by the scientific community.An ontology was what CERIF initiative aimed to provide.
Ontologies are being constructed by a group of human agents for a particular purpose.Wenger coined the term communities of practice as [57]: "groups of people, possibly representing different stake holding enterprises, who share a concern or a passion for something they do and learn how to do it better as they interact regularly.They engage in a process of collective learning in a shared domain of human endeavor".From this definition one can deduce that an ontology-stakeholder group is also a community of practice.This definition puts emphasis on the collective learning from a shared knowledge body that emerges from the interactions between the stakeholders about their shared domain of interest [9].Since ontologies are built for a particular purposefor example, enabling semantic interoperability between heterogeneous information systems or annotating resources such that they become meaningful to a groupand therefore create added value, the stakeholders in an ontology project constitute a knowledge-intensive community of practice [9], which we from here on call a community.The requirements of the ontology project are agreed upon and captured in semantic interoperability requirements, and when the needs of that community are met, then the publication of that data will yield benefits, whatever its shape (in terms of, for instance, efficiency, innovation, participation, and transparency).Ontologies are thus social artifacts as they are resources to be shared and constructed among human stakeholders.Previous work (reported in [15], [46]) focused mainly on the creation of ontologies for the annotation and publication of research information data.This experiment, albeit held in the context of the Flemish public administration, had very few stakeholders that actively participated in the ontologyengineering process.Note that we deem that communities and (application) domains that are the subject of an ontology-engineering project are completely different.However, communities tend to identify themselves with the domain in a setting where there is no other community constructing a specification of that domain.When there is more than one community creating an ontology about the same domain, they must necessarily be described from a different perspective, otherwise the communities should be the same.
In this paper, we present the results of a large ontology-engineering experiment in the research information systems domain in which a particular method and tool were adopted.The aim of this study was to see whether both method and tool were adequate to support the ontology-creation process for this particular domain in a scenario that is as realistic as possible: multiple stakeholders and multiple types of stakeholders.To assess the method and tool we use satisfaction survey and to assess the ontologies.The ontologies themselves were examined by looking to what extent these ontologies supported certain applications the participants had to develop.
This article is organized as follows.In Section 2 we describe a collaborative ontology-engineering method and tool in which the communities of stakeholders are used to: (i) ground all agreement processes that lead to ontology evolution, (ii) the communication using natural language to come to these agreements are part of the ontologies.The method and tool are both called GOSPL, which stands for Grounding Ontologies with Social Processes and natural Language.The combination of parameterizing all agreements with the community and the language used by the community results in what is called hybrid ontologies [40].In Sections 3 and 4, we describe the experiment and our findings on the application of GOSPL.Finally, in Section 5, we give a discussion and conclusion.

Ontology Engineering with the GOSPL Method and Tool
The formal semantics of a (computer-based) system quite simply is the correspondence between this system and some real world as perceived by humans.It is usually given by a formal mapping of the symbols in the system's description to objects in that real world, such that relationships and logical statements in the specification language can be assigned a truth-value depending on whether a certain state of affairs among objects exists in the real world.
As the real world is not accessible inside a computer, if we want to store and reason about semantics the world needs to be replaced by an agreed conceptualization, often in the shape of a formal (mathematical) construct.A computer-based, shared, agreed formal description of a conceptualization is known as an ontology.Ontologies constitute the key resources for realizing a Semantic Web [3].While theoretically ontologies should be perfect renderings of a real world, in practice they evolve as successive (one assumes, ever better) approximations of it [23].
A challenge with (linked) open data (or any other meaningful open data) is that many of the people who collect that data are not trained in ontology development and the need for figuring out more efficient ways for allowing people to more easily create structured data.Ontology construction is therefore not a trivial task; appropriate methods and tools to support the construction thereof in a complex collaborative setting are necessary.Quite a few collaborative ontology-engineering methods exist.A survey of ontology engineering methods and tools was presented in [18], which analyzed CYC [24], [35], Business Semantics Management [9], HCOME [33], DILIGENT [42], DOGMA-MESS [11], the methods proposed by Holsapple et al. [26] and Karapiperis et al. [32], METHONTOLOGY [2], [19], NeOn [21], [48] On-To-Knowledge [43]- [44], OntoEng [1], Ontology 101 [41], the Unified Method [52]- [53] and UPON [12]- [13].For Ontology 101, the collaborative tools for ontology engineering developed by the same groupreported in [50]- [51] were also examined.1. Promotes the natural language descriptions of concepts.Most methods and tools consider these natural language descriptions merely as documentation for the labels in the ontologies.
2. Gives ownership of the ontology to the community, rather than have knowledge engineers be responsible for ontology evolution, or settings in which both knowledge engineers and the stakeholders work together.
3. Formalizes the different social interactions within a community such that agreements lead automatically to ontology evolution.These interactions are supported by the natural language descriptions of concepts, which are exchanged within the community.

Table 1: Comparison of methods and tools for ontology engineering
It is also important to take into account the natural language used by the community when formally describing the concepts.This has been explored in methods such as DOGMA-MESS and Business Semantics Management, which uses formalism for representing ontologies that is grounded in natural language.That formalism can then be implemented in other ontology languages such as the Web Ontology Language (OWL), depending on the ontologyengineering project.The method adopted in this paper is Grounding Ontologies with Social Processes and natural Language (GOSPL) [18].GOSPL has been developed to incorporate all of the abovementioned aspects.
In GOSPL, concepts are both represented formally and informally.Formally by means of lexons [39] and informally by means of glosses (human interpretable natural language descriptions).A lexon is a plausible binary fact-type to be interpreted in a certain community.A fact-type is a generalization of a fact.For instance, [Christophe] knows [Pieter] is a fact, and [Person] knows [Person] is a fact-type.The community is used to disambiguate the concepts the labels refer to.An example of a lexon is Research Domain, Congress, is a, subsumes, Event, which states that, in the Research Domain community, the concept referred to with term Congress plays the role of is a on the concept with term Event, and the concept with term Event plays the role of subsumes on the concept with term Congress.A lexon should ideally result in two meaningful sentencesthat is, meaningful for that community, at leastwhen read in both directions, but assuring this quality is the responsibility of the community.
All lexons are entered in the lexon base.Next to the glossary and the lexon base, there is also the notion of a commitment.A commitment is a selection of lexons of the lexon base with constraints and a set of mappings of application symbols to terms and roles in that selection of lexons.These commitments can be described with languages such as -RIDL [55].GOSPL makes a distinction between community-and application commitments.
The first is a selection of lexons and constraints that would ensure the proper semantic interoperability between information systems represented by stakeholders in a community.The latter is a description of how one such information system related to the concepts inside one ore more community commitments.This is done by creating these mappings while importing all lexons and constraints of a community commitment.One can additionally add application-specific lexons and constraints next to those imported.This could be useful to annotatefor examplethe artificial keys of a database.Figure 3 depicts the different processes in GOSPL which we will explain below.In our method, semantic interoperability requirements are currently described as a set of labels (key terms) and goals in an informal manner.These key terms and goals will help others to identify communities as well as aid in setting the boundaries of such a community.
• Articulation with glosses.Starting with the key term when the semantic interoperability requirements of a community emerges, the community first need to agree on a set of definition to which labels will refer.We argue that the stakeholders first need to align their thoughts on the meaning of a label by means of informal descriptions before using those labels to formally describe the concept they refer to.Truth and validity are relative to the community.We are unable to judge the quality of a gloss with respect to the community.If a community deems a particular gloss to be adequate, the gloss is valid.Jarrar, however, did provide some guidelines on what the characteristics of a good natural language definition are in [29].
• Managing Lexons.Lexons are binary fact types to be interpreted in a certain community.Stakeholders are allowed to enter lexons when at least one of its terms is articulated, which means that the term has been annotated with a gloss.
• Managing Constraints.For semantic interoperability, concepts and relations alone not necessary suffice.As Spyns et al. [45] pointed out, the introduction of more constraints in an ontology reduces reusability, but increases usability.For semantic interoperation to succeed, however, constraint needs to be agreed upon by the stakeholder community.GOSPL focuses on constraints to create reference structures: which are sets of unique, total and identifying attributes.In relational databases, these would correspond with the fields that constitute the primary key.They are useful for constructing URIs for the Semantic Web.
• Managing the Application Commitment.Once the community agrees upon lexons and constraints, they can be used to annotate the existingusually autonomously developed and maintainedinformation systems.
-RIDL language for which the web-based tool provides support.Depending on the purpose of the hybrid ontology, the lexons and constraints can be implemented into another formalism.For the Semantic Web and Open Linked Data, for instance, OWL or Description Logics (DL) can be used as an ontology implementation language.Details of one such implementation in a particular DL dialect was presented in [18].Using these implementations for providing a Resource Description Framework (RDF) dataset or an endpoint to query RDF is also a type of application commitment management, as it also involves the annotation of a particular information system.
All agreement processes for managing communities, semantic interoperability requirements, lexons and constraints mainly occurs within one community and results in versions of community commitments.These social processes are called inter-community processes.
The creation of an application commitment is the responsibility of the stakeholders representing an existing information system.The application commitments themselves, however, provide a source of knowledge for starting or guiding discussions by discoveringfor instanceconstraints that stakeholders have declared next to the ones agreed upon in the community commitments.
The GOSPL method also prescribes social processes that do not necessarily take place within one community, called intra-community processes.These social processes are for supporting agreements that glosses or terms refer to the same concept.Synonyms are agreements that two terms refer to the same concept, and gloss-equivalences are agreements that two descriptions refer to the same concept.Ideally, if two communities agree that two descriptions refer to the same concept, the labels associated with those descriptions should be considered referring to the same concept as well; this is called the glossary consistency principle.
All social processes grounded in an argumentation model called IBIS.IBIS is the most accepted argumentation model and stands for Issue-Based Information System [34], which provides a simple and abstract infrastructure for non-trivial problems that cost a lot to solve in terms of resources such as time and money.
The method is supported by a web-based tool bearing the same name, on which we will elaborate more while describing the experiment.The method and tool presented in this paper are the result of an incremental research methodology where the method and tool were gradually refined and improved.This corresponds to the design science research method that aims to look for effective and efficient solutions for a particular solution, often in a systematic way via iteration in which problems in a previous version of a solution are addressed for a subsequent trial.Below we provide some of the insights gained over time.

•
In 2009, a first experiment was conducted in which a MediaWiki (Site 8) instance was used for building ontologies.The ontologies were created using lexons and the natural language descriptions of the labels in that formalism were used as documentation.The talk pages provided a means for asynchronous discussions within the stakeholder community, but were not structured.Structure, here, means that the types of interactions were not made explicit and needed to be analyzed manually.An analysis of these interactions for detecting different types of users were reported in [10].This experiment also gave us insights of promoting the community as first-class citizen in the ontology and ontology-engineering process; which means that all ontology evolution operators need to be parameterized with the community and community-interactions.As the community consists of human stakeholders and the main tool for humans to interact with each other is natural language, concepts are both described by means of natural language descriptions and a more formal representation suitable for machine processing.The parameterization of all ontology evolution operators with the community's interactions and the two distinct representations for concepts lead to what is called hybrid ontologies [40].In this experiment, 14 participants were divided into 3 groups and were given the assignment to create an ontology for describing CVs.

•
In 2011, based on the work of [56] and our experiences with the early prototype, re-conducted a similar experiment in which the social processes were made explicit.Social processes were also introduced for the different parts of the ontology (for instance by creating, removing or changing a relation in the ontology); allowing one to analyze the different interactions more easily and to have the different interactions result in ontology evolution.The method adopted for this experimentreported in [16] was Business Semantics Management (BSM) [9] and the tool adopted was the Business Semantics Glossary (which was based on XWiki) of which the source code was altered to support the different actions.This experiment showed us that a wiki-oriented applicationwhere one can edit whenever and how he wants, although changes are reversibleshowed to hamper the ontology-engineering process.The BSM method prescribes that the process of creating formal and informal descriptions of concepts can happen asynchronously.However, it was observed that a group cannot (easily) agree on a formal representation of a concept before the group agrees to what a label in that formalization should refer to.In other words: natural language descriptions of concepts should precede the formal descriptions of concept to facilitate the process.This corresponds to the alignment process in complex systems theory where a body of agents needs to have a common ground before performing tasks [25].In this experiment, 43 participants were divided into 9 groups and developed ontologies for enabling semantic interoperability between product vendors and portals for describing Requests for Proposals (RFP), which are descriptions of customer needs upon which vendors can bid with an offer.The goal was to create ontologies expressive enough to allow offers to be matched with RFPs.
• In 2012, a new method was proposed in [17], where the informal descriptions of concepts must precede the formal descriptions and those information descriptions furthermore drive the social interactions.A new prototype was also developed which followed the new method.An evaluation of the tool was given in [6].
The conclusions of the evaluation were that the participants were generally satisfied with the system usefulness, in other words: to what extent the system was capable helping the group in achieving their goal.In this experiment, 43 (different!) participants were divided into 11 groups and developed ontologies for enabling semantic interoperability between autonomously developed information systems in the cultural domain as well as to annotate an existing relational database of an organization containing hundreds of cultural events.
Tool-wise, some issues emerged from the survey, which were taken into account for the new version of the tool, which will be presented in this paper.

Experiment
The experiment ran for thirteen weeks in 2013 and was organized as follows: (i) first, participants had to create groups of 2 to 4 persons and develop their own (research) information system by reverse engineering parts of existing systems.(ii) Then, the participants had to create ontologies to enable semantic interoperability between the autonomously developed information systems and one provided by us; (iii) finally, the participants were asked to annotate their own information system and a database of an existing system.

Step 1: Setting the Stage
The participants were all students in the Open Information Systems course taught at the Vrije Universiteit Brussel, which is part of the MSc in Computer Science curriculum.All participants had prior background in computer science, software-or hardware engineering, information technology management or similar.About half of the participants were Belgian.The others came from others countries in Europe, Afrcia, Asia and South-America.The participants (36 in total) were asked to developin groups of 2 to 4 peoplethe conceptual schema of existing research information systems or information systems related to the domain of research based on an existing application.That conceptual schema was then used to derive a database schema for their system.Each group had 3 weeks to develop their own schemas.Since the vast majority of the participants had a background in computer science or engineering, this did not pose a problem.The different groups, their application and size of the group are shown in Table 2.

Step 2: Ontology Engineering
The common entities are obvious (they include notions such as researcher, research project, and publication), and the participants were asked to create ontologies to enable semantic interoperability between their information systems.The participants had used the method and tool to create these ontologies and were given 8 weeks time.
While the project was running, 24 communities were created.The first was a sandbox community allowing the user to test the functionality of the prototype.There are two publication communities: 4 and 19.After two weeks into the project, the stakeholders inside that community needed an internal workflow to handle matters and went for a tabula rasa approach starting from scratch rather than fixing the issues in the first community.The emergence of communities was as follows: 4 (including the sandbox community) in week 1, 13 in week 2, 3 in week 3, 1 in week 6, 1 in week 7 and two in the final week of the ontology project.The list of communities is given below: As stated earlier, each community member interacts to reach agreements necessary to achieve semantic interoperability and these interactions are supported by the IBIS argumentation model.A part of such a discussion is shown in Figure 4.All discussions leadwhen a consensus is achievedto the execution of ontology evolution operators.Figure 4 depicts the proposal of adding the constraint that each call (for projects) must belong to at least one funding program.In this screenshot, members of the Funding Programs community are discussing whether the constraint that each call has at least one date on which the call was published.Replies need to be annotated according to the IBIS framework, such as an objecting evaluation or an objecting justification.The constraint will only be included in the ontology when the community reaches an agreement that the ontology should so.It is interesting to note that communities tended to adopt two different strategies: (i) either they negotiated as much as possible all necessary constraints at the level of the ontology and possible problems are fixed while annotating the system or (ii) the community opted to develop a light-weight ontology suitable for annotation.An example of the first occurred when a community discussed the need to provide both a date and a time for certain entities.Some application did not keep track of times buttogether with the communitydecided they could commit to this ontology by providing a default value (for example, 23:59:59).Note that truth is relative to the community and that when the community agrees that this compromise is good enough to support their business, then this agreement is valid.An example of the second strategy occurred in the Publication community.Quite a few applications stored information about publications, albeit with different structure, granularity and even level of completeness.Publications have for instance different possible identifiers (such as a DBLP key, Digital Object Identifier (DOI), or ISBN) and some publications do not even have such an identifier (such as it is often the case with white papers or non-published technical papers).Rather than creating different communities, the participants opted to agree on as much relations as possible.The advantage of this approach is a highly reusable ontology, but not as usable; for instance one is not guaranteed to obtain a DOI for each publication.
Table 3 contains the number of interactions (in terms of requests and replies, for instance) for each week in each community between participants over those eight weeks.Interactions include proposing changes, concluding, and interacting by means of votes or replies.The Sandbox Community was left out for this table, as it did not contribute to the ontology-engineering project.After W3, the numbers declined.This was due to the spring break.Concepts that were shared across multiple groups often had higher numbers of total interactions.

Step 3: Applying the Ontologies
To validate the ontologies, the groups were asked to annotate their own information system using the ontology with -RIDL and each group had to also demonstrate the ontology by: (i) annotating an existing research information system (a FRIS database dump was provided) with the OWL implementation of the hybrid ontology and (ii) use popular Semantic Web technologies to develop an application on top of the annotated data of their choice.An OWL implementation of the ontology is provided by the tool.
Groups used R2RML [8] or D2RQ [7] for annotating the existing research information system.The D2RQ platform allows one to create a Linked Data server on top of the existing database as well means for creating an RDF dump.The D2RQ platform also provides a SPARQL platform with no reasoning facilities as it translates SPARQL queries into SQL queries.R2RMLa W3C recommendation since September 2012is a standard for annotating relational databases for which different software solutions emerge.
One of the applications developed by the participants is the detection of conflicts of interest.This is particularly important for funding agencies that need to construct review boards for the evaluation of project proposals.The application depicted in Figure 5 shows the tool went looking for all collaborators on projects and within organization of the researcher named Werner Verhelst.The tool is only limited to the data provided by the public administration, but demonstrates the feasibility of listing all the people that should not be taken into account for an advisory board Publications were not taken into account since the dataset provided by EWI only contained some publications of one knowledge institution.The query to detect paper collaborations, however, can easily be extended.

Evaluation of the GOSPL Tool
Usability is defined by the ISO-9241 standard [27] as the effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in particular environments.Usability is a key factor in making the systems easy to learn and to use.Usability testing has been extensively studied and applied by Lewis [38] at IBM Software Group.Classically, usability tests gather both subjective and objective data coming from realistic use case scenarios, as well as descriptions of the most common problems encountered by the test participants [36].Subjective data reflect the participants' opinions regarding the evaluated system, while objective data reflect the participants' observed performance when using the system.
The focus of the usability of the prototype is user satisfaction, of which the results were reported in [14].A classic way to measure the user satisfaction is via questionnaires such as the After-Scenario Questionnaire -ASQ, the Computer System Usability Questionnaire -CSUQ [36], and the System Usability Scale -SUS [5].However, a common mistake is to rely on questionnaires only while evaluating the user satisfaction.There are alternatives to measuring satisfaction with a questionnaire.One example of such an alternative is the Microsoft Desirability Toolkit.
While the questionnaires are often biased towards positive responding, this tool helps elicit negative comments from participants by having the users choose five words that closely matched their experience with the system and have them explain this choice; the Desirability Toolkit thus invites users to be critical [49].Questionnaires need to contain placeholders for such critiques as well, for example by providing comment sections with each question.
A proven standard and effective instrument to assess the user satisfaction is the Post-Study System Usability Questionnaire (PSSUQ).PSSUQ was developed for scenario-based usability evaluation at IBM [36].The environment used was an enterprise-wide and networked office application suite.A follow up study by IBM [37] was performed in the domain of speech recognition [37] using data from five years of usability studies.The follow up produced similar psychometric properties as the original survey.Fruhling and Lee [20] validate these results of the PSSUQ instrument and assess its adaptability to other domains, such as telemedicine.The reason for choosing PSSUQ for this study is mainly the richness of the provided information, with little effort from the user, and the extensive IBM documentation and experience for the statistics it can provide.PSSUQ originally consisted of 19 questions, each question being a statement about the usability of the system.Participants need to answer each statement using a Likert scale of 7 points, where 1 indicates that the user strongly agrees with the statement whilst 7 indicates that the user strongly disagrees with it.PSSUQ is based on a comprehensive psychometric analysis, providing scales for three sub-factors, namely: system usefulness (SysUse); information quality (InfoQual); and interface quality (IntQual).The short version of PSSUQ (and the most recent one, see Table 4) was used with the purpose of saving study time.The questions correspond to the three sub-factors as follows: (1) System usefulness: the average of items 1 through 6; (2) Information quality: the average of items 7 through 12; (3) Interface quality: the average of items 13 through 15; (4) Overall: the average of items 1 through 16.PSSUQ is used in order to measure the user satisfaction when dealing with GOSPL.An advantage is that besides the 16 items in the test, the test participants can make comments and elaborate on their answers.Based on these comments, conclusions are drawn and recommendations for improving the human-system interaction provided.
Table 4: PSSUQshort version [38] Item Item Text Q01 Overall, I am satisfied with how easy it is to use this system.Q02 It was simple to use this system.Q03 I was able to complete the tasks and scenarios quickly using this system.Q04 I felt comfortable using this system.Q05 It was easy to learn to use this system.Q06 I believe I could become productive quickly using this system.Q07 The system gave error messages that clearly told me how to fix problems.Q08 Whenever I made a mistake using the system, I could recover easily & quickly.Q09 The information provided with this system was clear.Q10 It was easy to find the information I needed.Q11 The information was effective in helping me complete the tasks and scenarios.Q12 The organization of information on the system screens was clear.Q13 The interface of this system was pleasant.Q14 I liked using the interface of this system.Q15 This system has all the functions and capabilities I expect it to have.Q16 Overall, I am satisfied with this system.
The purpose of this study is to assess the user satisfaction with the system.The overall usability testing was carried out both implicitly by analyzing the data logs and the user-system interactions and explicitly, by collecting the user feedback via several questionnaires.The outcome of the experiment highlights three aspects of the evaluation: 1) effectiveness; 2) efficiency and 3) satisfaction [38].The answers of the respondents as well as their activity on the platform are shown in Table 5.On the left of this table we have the answers of each of the respondents on questions Q01 to Q16.On the right we have the number of communities created (Con), discussions started (Pro), interactions in a discussion (Dis), number of votes (Vot), the number of discussions concluded (Con) and total (Tot) of all aforementioned numbers for each respondent.The results delivered by the PSSUQ questionnaire are shown in Table 6.The respondents were overall satisfied using the tool, albeit results show considerable room for improvement.System usefulness scored best.Information quality, on the other hand, scored worse.Table 7 shows averages for each question.From Table 7 it becomes apparent that Q07 turns out the worst, closely followed by Q08.According to [38], one should not be surprised to find this value to be on the negative side when performing a PSSUQ, as it is a difficult task to provide usable error messages throughout a product.Better-thanaverage error messages are achieved when the average for Q07 is equal to or less than the mean of the other items in InfoQual [38].In our case, the average for the other items is 3.41 and our error messages thus score poorly.It is actually no surprise that handling errors and error recoveryassessed in Q08also scores badly.Both Q05 (the system was easy to learn) and Q12 (clearness organization of the information) scored best.We then clustered the data by means of the expectation maximization (EM) algorithmusing the Weka 3 toolkit (Site 11) and default parametersto detect clusters within the 23 respondents.This algorithm assigns a probability distribution to each instance that indicates the probability of belonging to each of the clusters and uses cross validation to determine the number of clusters to create.The EM algorithm found two clusters.The difference between the two clusters is the first (containing 11 of the 23 respondents) providing on average more negative All questions within SysUse, InfoQual and IntQual are obviously related.Using the sample correlation coefficient to compare the values for each question, the correlations that are more important than 0.70 are between (i) Q01 and Q02 (0.72), (ii) Q08 and Q15 (0.73), and (iii) Q13 and Q14 (0.82).The (i) and (iii) are questions within one metric, and (ii) seems to us more of a coincidence than an actual correlation.
The survey also allowed the participants to give comments to each of the questions, which highlighted some problems with the current version of the prototype.Of the 23 respondents, 17 have left comments.The most recurring problem was keeping an overview of the discussions (10 occurrences).Some proposals have been made to tackle this problem: 2 respondents proposed a central notification system, one respondent proposed the ability to follow the actions of a particular user, another proposed an RSS feed per community complementing the overall feed, even though one could filter on channel.Other proposals were: identifying the hottest discussions, offering the changes after last login and even a search function over the whole system.
During the experiment, however, it became apparent that different communities behaved differently.It is clear that no two groups are the same and that even the semantic interoperability requirements or the dependencies between communities might have an influence on the interactions between the users.Users expressed the need for tools that allows groups to establish and agree on workflows within the collaborative platform.In other words, the decision processes need to be promoted as part being part of the community and themselves be the result of social processes, and afterwards be the subject of other social processes.Users established those outside of the system and thus were not traceable for new members joining or others who were absent when these discussions took place.
An example of such a workflow was the (maximum) durations of a discussion.The publication community, for instance, gave at most two days for others to provide their input.

Conclusion
Open data initiatives are a crucial driver for open innovation.Moreover, in a setting with high data quality demands this is not merely a matter of publishing public data [28].Data elements must be approved and their semanticscaptured in an ontologymust be agreed on for people, organizations and society in general to gain benefit from it.
The closer these ontologies are to the language used by the stakeholders, the more usable the ontologies become for the stakeholders to create applications around annotated research information.Earlier work showed that agreements on a natural language description for a concept lead to more stable agreements on formal descriptions of that concept, motivating the need to have agreements on natural language descriptions precede the agreements on formal descriptions.This previous work, however, had very few stakeholders that were actively involved in the ontology construction process.To assess a method and tool for ontology engineering in the research information system domain, a more realistic experiment had to be set up.
In this paper, we reported on the application of a collaborative method and tool for the creation of hybrid ontologies in a setting involving an important number of stakeholders and heterogeneous research information systems that needed to interoperate meaningfully.Hybrid ontologies are ontologies in which the stakeholder community becomes an integral part of the ontology and ontology-engineering process as well as the natural language definitions of concept, which are called glosses.In this experiment, 36 participants were divided into groups to develop a conceptual model for their own research information system and needed to collaborate as a whole to create an ontology to annotate both their own system as well as the data of an existing system provided by a public administration.The usefulness of the ontologies was demonstrated by the prototype applications the participants were able to develop.
The user satisfaction of the tool was measured with a usability study, which showed that the participants were generally pleased with the system's usefulness.Problems identified in this survey are the overview of the currently hottest discussions and the need for support for workflow management.Communities require workflows to be established to manage the interactions.These workflows vary depending on the community (its members, for instance) and even the ontology project (with respect to, for instance, its complexity, stakeholder group size, and the heterogeneity of the stakeholder group).One of the key problems in applying (hybrid) ontologies is thus scalability in the engineering process.The people that have the necessary domain expertise to define these policies, rules, and business terms, as well as the owners of the physical data systems that have to knowledge to define the mappings to latter, are usually part of different organizations of organization units.A big challenge here is to determine how to group the right people in communities and assigning the right roles and responsibilities to them.

Figure 1 :
Figure 1: Part of the information on the Large Hadron Collider on the FRIS website

Figure 2 :
Figure 2: Part of the CERIF model in a ER diagram

Figure 3 :
Figure 3: The processes of GOSPL.Through iteration, the communities, ontologies, and applications co-evolve • Managing Communities and Semantic Interoperability Requirements.Communities emerge around semantic interoperability problems, which they describe by means of semantic interoperability requirements.In our method, semantic interoperability requirements are currently described as a set of labels (key terms) and goals in an informal manner.These key terms and goals will help others to identify communities as well as aid in setting the boundaries of such a community.

Figure 4 :
Figure 4: Part of a discussion on the GOSPL platform for hybrid ontology engineering

Figure 5 :
Figure 5: Application to detect conflicts of interest Table 1 provides a simplified comparison table of methods and tools for ontology engineering presented in [18].In this table, Y stands for yes, N stands for no and P stands for Partial.Other labels are explained in the rows where they occur.Note that black cells mean that values for this aspect are not applicable.The comparison in this table shows us that what lacks is a method that:

Table 2 :
The different groups, their application and size in the ontology engineering experiment

Table 3 :
The number of interactions in the ontology-engineering experiment

Table 5 :
Answers of the respondents and their activity on the platform in numbers

Table 6 :
Summative user satisfaction of all respondents

Table 7 :
Minimum, maximum and average scores for each question by the participants