SciELO - Scientific Electronic Library Online

 
vol.8 suppl.1ALTEC y la Gestión Tecnológica en Iberoamérica: Gestión de Proyectos, Conocimiento e Innovación Social y SustentableMaturity and Performance in Information Technology Project Management índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

Links relacionados

Compartir


Journal of technology management & innovation

versión On-line ISSN 0718-2724

Journal of Technology Management & Innovation vol.8  supl.1 Santiago feb. 2013

http://dx.doi.org/10.4067/S0718-27242013000300002 

 

Performance Evaluation of IT Projects - The Shenhar and Dvir Model

 

Renato de Oliveira Moraes1, Fernando José Barbin Laurindo2

Universidade de São Paulo (USP). Departamento de Engenharia de Produção. Avenida Prof. Luciano Gualberto, Travessa 3 N°380. E-mail: 1remo@usp.br, 2fjblau@usp.br


Abstract

This paper presents a scale for the ex-post evaluation of IT projects based on the definition of multidimensional performance of projects proposed by Shenhar and Dvir (2009). From the five dimensions proposed by these authors, a questionnaire was designed to assess the importance of various performance criteria in the ex-post evaluation of projects and IT in different organizations. The responses to the questions in the questionnaires were analyzed using factor analysis and Cronbach's alpha, which measures the scale's internal reliability.

The scales presented in this article are consistent with the evaluation criteria of the model ex-post evaluation of IT projects presented by Moraes and Laurindo (2010), and they are articulated with assessment procedures described by these authors. Accordingly, this paper contributes to the construction of a set of references for an ex-post evaluation of IT projects.

Keywords: performance of IT projects; performance scale; management of IT projects.


 

Introduction

The success rate of IT projects, according to the Standish Group's CHAOS Report, is approximately 16%, while approximately 31% of projects are abandoned or canceled and other projects are completed without achieving their original goals with respect to cost, time and/or technical performance. Although the most common causes are known, organizations that wish to increase their success rate regarding such projects must clearly identify the factors that most affect their IT projects. An ex-post evaluation of projects, however, is an alternative to this identification.

This paper presents a scale for the ex-post evaluation of IT projects based on the definition of the multidimensional performance of projects proposed by Shenhar and Dvir (2009). From the five dimensions proposed by these authors, a questionnaire was designed to assess the importance of various performance criteria in an ex-post evaluation of projects and IT in different organizations. The responses to questions on the questionnaires were analyzed using factor analysis and Cronbach's alpha, which measures the internal reliability of a scale.

The scales presented in this article are consistent with the evaluation criteria of the model ex-post evaluation of IT projects presented by Moraes and Laurindo (2010), and they are articulated with assessment procedures described by these authors. Accordingly, this paper contributes to the construction of a set of references for the ex-post evaluation of IT projects.

Theoretical Review

An incomplete vision of project performance is directly related to fulfilling the original goals of time, cost and quality. Therefore, the work of Baker, Murphy and Fisher (1983), which showed that broader performance criteria are used by professionals, plays an important role in various projects. They proposed the concept of perceived success when they observed in their study that projects that did not meet their original goals of cost, schedule and quality were not necessarily perceived as failed projects by the people involved in the development of the projects. Thus, a project's success is linked to the perception of those involved (stakeholders) regarding the performance of the project.

Figure 1- Model of Project of Success. Adapted from Pinto and Slevin (1986)

Table 1: Dimensions of success Pinto and Slevin (1986).

Pinto and Slevin (1986) provide a definition of project performance that considers both internal factors, which include cost, time and quality (compliance with technical specifications), and external factors, which include use, satisfaction and effectiveness.

While the internal factors are more closely related to and controlled by the manager and are thus not directly affected by the customers and users, the external factors, in contrast, are more directly related to customer behavior (Figure 1).

Table 1 provides a breakdown of the internal and external factors with examples of evaluation criteria.

The authors emphasize that the relative importance of each of the two factors (internal and external) varies with time. While internal factors, which are more subject to influence and control from the project team, are more important in the early stages of the project, while external factors, which are more specific to customer perception, assume greater importance from the time of implementation (Figure 2).

Lim and Mohamed (1999) also recognize the importance of the perception of success, noting that the perception of success is not necessarily the same for all actors involved in a project. For example, these authors separate vision performance into two categories: macro and micro. From the macro perspective, the success of the project can only be obtained in its operational phase, that is, the use of the product as generated by the project. Thus, success depends primarily on users. From the micro perspective, the success of the project depends on the tasks and milestones. Thus, this division - micro and macro - results in the evaluations of process and product, respectively, which is a view of product and process that is shared by other authors.

Cooke-Davis (2000) addresses two separate concepts. The first concept, the project's success, is measured by the degree of achievement of the project's overall objectives. For example, one project aims to generate, through the launch of a more modern product, increased market share, or to develop expertise in specific technologies, among other goals. The second concept is the successful management of the project, for which measurement is based on indicators of meeting deadlines, budgets and quality standards established for the project.

Baccarini (1999) also uses two distinct concepts of performance: success of project management (process view) and product success (product view). The success of the process is linked to the classical aspects of performance (time, cost and quality technical specifications), stakeholder satisfaction and development, and quality management process. This view leads to the following performance criteria:

• anticipate project requirements, meet project needs, use resources efficiently;

• communicate effectively and resolve of cases in a timely manner;

• establish effective coordination of and relationships between stakeholders, engage in teamwork and in participatory and consensual decision making;

• minimize scope changes and eliminate disturbances in the organization (related to work process and culture);

• complete project with no post-closing problems and identify and solve problems during project execution.

The success of the product is evaluated using the following criteria:

• achieves organizational objectives according to strategic buyer / project sponsor;

• meets needs and purposes of users purposes and is appropriate for use;

• meets needs of other stakeholders of the project product.

Figure 2 - Importance of factors of performance over time. Adapted from Pinto and Slevin (1986)

While recognizing the importance of the last successful product, Baccarini (1999) notes that the success of project management (process) tends to influence (positively) the success of the product. He notes that as the performance evaluation depends on who conducts the evaluation as well as on the time of evaluation, it is important to establish a priori the success criteria that will be used to assess a particular project.

Bjeirmi and Munns (1997) also separate the concepts of successful project management of project success; however, their concepts are not complementary. Successful project management is only part of the success of the project, as illustrated in Figure 3, and the project team is involved only at stages 2, 3 and 4 of the project, while customers will be interested in all stages 1 through 6. Accordingly, the team will naturally be more attentive to the successful completion of stage 4 as it terminates its involvement in the project. Customers (or users), however, will be interested in the final results, which are not apparent until the last stage. The authors suggest that the performance evaluation can, accordingly, be completed using three distinct optical factors:

1. Implementation: considers stages 2 through 4 and is focused on technical project management and implementation.

2. Perceived values: considers the users who will interact during the use of the product as well as during the design stage.

3. Customer satisfaction: considers the closure of the project when the client can examine all the influences and can assess compliance with the overall objectives and benefits. Wateridge (1995) examined over 100 projects to determine the success criteria and constraints used in the design of information technology (IT). The work involved contact with project managers, sponsors, users, systems analysts and support staff, and it required him to submit the views of the contacts on the success of the IT projects. While the author claims to have found broad consensus among stakeholders of IT projects, there is some disagreement regarding the inclusion of meeting deadlines and budgets within the definition of success criteria. The author noted a variation in the criteria used in the performances among projects considered to be successful and those considered to be unsuccessful. For projects deemed to be successful, meeting the established quality specifications and achieving commercial success were considered to be more important by project managers, while with respect to project failures, compliance schedules and budgets were the most cited factors that contributed to a lack of success. Users, in general, are more concerned with ensuring a project's end result.

(*) Third parties include local and national authorities, media, environmental groups, general public, etc.

Figure 3 - Scope of project success and success of project management. Adapted from Munns and Bjeirmi (1997)

It is interesting to note the consistency of this result with that of Baker, Murphy and Fisher (1983), who also found that the factors that affect the perception of success are not (exactly) the same as those that affect the perception of failure. Wateridge (1995) also emphasizes the importance of establishing a priori criteria for evaluating performance among project stakeholders. He recalls that a manager is only able to treat adequately the constraints of project success when a consensus exists among stakeholders on the success criteria applied to the project.

This same discussion is revisited in a later work, whereby Wateridge (1998) identified a set of performance criteria often used in IT projects (Table 2). In that study, the criteria used to evaluate the performance of projects suffered slight modifications (Table 3 and Table 4)

The concept of success used by Dvir et al. (1998) has two dimensions: benefits perceived by consumers and fulfillment of project goals (design). These dimensions also suggest a division of the concept of success, as the benefits perceived by consumers can only be evaluated after the implementation of the product design, unlike compliance with the specifications, which can be evaluated during development and project completion.

In contrast, Shenhar et al. (2001) do not recognize the existence of two distinct concepts of success - success and the success of product design - and instead defend the premise that the relative importance of the dimensions of project success change over time. They identify the following dimensions of success:

• Project efficiency (meeting deadlines and budgets);

• Impact on consumer (customer satisfaction and product quality);

• Success of the business (revenue generation, profit share and other benefits derived by the mother organization);

• Preparation for the future (developing organizational infrastructure and / or technology for the future).

However, the proposal of these authors also recognizes that the evaluation of each dimension cannot be conducted simultaneously, as each dimension has a different background (Figure 4).

The relative importance of each dimension varies with time and with the technological uncertainty. In the very short term, project efficiency is the most important dimension and is also the only dimension that can be measured with reliable accuracy. Using the developed product becomes possible, as does the relevant evaluation of other dimensions (Figures 4 through 6).

Table 2: Three key success criteria (frequency of citation) as perceived by users and project managers. Adapted from Wateridge (1995)

Table 3: Criteria used in projects by professional IT development as observed by Wateridge (1998)

Table 4: Top five success criteria (frequency of citation). Adapted from Wateridge (1998)

Table 5: Criteria for success as cited by users and project managers. Adapted from Wateridge (1998)

Figure 4-Success dimensions versus term. Adapted from Shenhar et al. (2001)

Figure 5 - Relative importance of success dimensions versus time. Adapted from Shenhar et al. (2001)

Figure 6- Relative importance of success dimensions versus technological uncertainty. Adapted from Shenhar et al. (2001)

Table 6: Dimensions of the success of projects, according to Shenhar et al.

In projects with low technological uncertainty, project expectations are more strongly linked to marginal contributions in the development, and efficiency is an important factor. For example, when conducting a product update, the interest is in keeping the product in accordance with the specifications of the market, and this approach is not expected to change the life cycle of the product. When working with a high degree of both innovation and technological uncertainty, organizations become more tolerant toward low levels of project efficiency because there is an expectation that the project can eventually generate internal competence in new and emerging technology.

As noted by the authors previously discussed herein, a variation exists in terms of performance indicators, though there is a certain convergence in relation to the dimensions of project performance. A striking difference between the proposals relates to the discussion on how various concepts are related to performance. While some researchers (LIM and Mohamed, 1999; Cooke-Davies, 2000; Baccarini, 1999; Munns, 1997) refer to two distinct concepts - successful project management (focus on process development) and project success (focus on product of resulting project) -others (SHENHAR et al. 2001; Baker Et Al. 1,983; Pinto and Slevin, 1988) suggest that there is a single element with multidimensional characteristics, in which the relevance of each dimension varies with time.

The present work considers a different premise - a unique concept of performance - because the authors believe that this unique concept provides a more interesting temporal perspective regarding project performance.

Table 7 - Performance criteria of the data collection instrument

Method

From the literature review, a questionnaire was designed to collect data separated into 4 parts:

• Identification of the interviewee

• Identification and characterization of the company

• Characterization of the type of project

• Assessment of the importance of the criteria for evaluating performance

The closed question questionnaire was answered by 37 professionals working on various IT projects. The data were subjected to factor analysis, and the reliability of the scales was assessed by Cronbach's alpha value. Table 2 shows the performance evaluation criteria present in the data collection instrument. The respondent assessed the importance of each criterion in the ex-post evaluation of project performance of the organization (irrelevant, unimportant, medium importance, very important, or essential)

Results

In all, 37 valid questionnaires were obtained. The respondents included managers and project team leaders (22%) and professional development leaders (78%) whose experience in developing information systems ranged from 2 to 15 years. For each of the five dimensions of performance, we performed a factor analysis and determined that the sample size fit the recommendation of Hair et al. (2004) in at least five observations for each variable used in the analysis.

In each factor analysis, a single factor with an eigenvalue greater than 1 was extracted. The higher quality of the service criterion dimension Commercial Success was deleted because this variable was extracted at below 0.5, which increased the explained variation and the value of Cronbach's alpha for this scale. Table 8 presents a summary of the observed results.

In a broad sense, the results suggest that the variables used generate good scale performance of the dimensions of the Shenhar and Dvir model. The sampling adequacy measure (Kaiser-Meyer-Olkin (KMO)) for the first dimension is low because this dimension has only two variables. However, in this dimension, the variation of the variables extracted and the value of Cronbach's alpha can be considered good, according to Hair et al. (2005).

The results strongly suggest that this model can be used in ex-post evaluations of the performance of IT projects, which is consistent with the findings of Moraes (2004), Moraes and Laurindo (2010)

Final Remarks

This paper evaluated the use of the Shenhar and Dvir model (2009) for evaluating the performance of IT projects. Data collected through a closed questionnaire indicated that the variables of the performance dimensions of this model allow for the construction of scales with high internal reliability. Previous works (MORAES, 2004; Moraes And Laurindo, 2010) noted the appropriateness of the first two dimensions of performance -- project efficiency and impact on the customer - in assessing the performance of IT projects. It is worth noting that the characteristics of the ex-post evaluation process result in the perception of the project performance to be affected by the time of the evaluation. Thus, the relative dimensions of the performance model of Shenhar and Dvir, and therefore the scales of each of these dimensions, are important at different time points. In general, these scales help professionals to develop evaluation procedures that go beyond ex-post evaluations in which it is only possible to evaluate the efficiency of the project based on cost and timeliness of the project's completion.

As the sampling procedure was not probabilistic, it is expected that other studies, possibly with larger samples, can confirm the findings presented herein.

Table 8 - Summary of test results

References

BACCARINI, David. The Logical Framework Method for Defining Project Success IN: International Journal of Project Management vol. 30, no. 4, pp 25-32, 1999.         [ Links ]

BAKER, Bruce N., Murphy, David C. E Fisher, Dalmar Factors Affecting Project Success In Cleland, D. I. E King, W. R. Project Management Handbook. New York : Van Nostrand Reinhold, 1988.         [ Links ]

COOKE-DAVIES, T. The real success factors on projects. In International Journal of Project Management vol. 20, pp. 185-190, 2000        [ Links ]

DVIR, D. et all. In search of project classification: a non-universal approach to project success factors. IN Research Policy , no. 27, pp. 915-935, 1998        [ Links ]

HAIR, J. F. et all. Multivariate Data Analysis. Prentice Hall,2005        [ Links ]

LIM, C. S. & Mohamed, M. Z. Criteria of project success: an exploratory re-examination. IN International Journal of Project Management vol. 17, no. 4, pp. 243-248, 1999        [ Links ]

MORAES, R. O. Condicionantes de desempenho dos projetos de software e a influência da maturidade em gestão de projetos. Tese de Doutorado. Faculdade de Economia, Administração e Contabilidade da Universidade de São Paulo. 2004        [ Links ]

MORAES, R. O. ; Laurindo, F. J. B. . Avaliação de resultados de projetos de TI. IN: Anais do XXX Encontro Nacional de Engenharia de Produção. São Carlos, 2010        [ Links ]

MUNNS, A. K. & Bjeirmi, B. F. The role of project management in achieving project success. IN: International Journal of Project Management vol 14 no. 2 pp. 81-87, 1997.         [ Links ]

PINTO, J. K. & Slevin, D. P. Critical Success Factors Across the Project Life Cycle IN: International Journal of Project Management , 1986        [ Links ]

SHENHAR, A. et all Project success: a multidimensional strategic concept. IN Long Range Planning, no. 34, pp. 699-725, 2001        [ Links ]

SHENHAR, A. & Dvir, D. Reinventando Gerenciamento de Projetos. São Paulo: M.Books, 2009        [ Links ]

WATERIDGE, John. IT project: a basis for success. IN International Journal of Project Management vol. 13, no. 3, pp. 169-172, 1995        [ Links ]

Selected February 11, 2013.

Creative Commons License Todo el contenido de esta revista, excepto dónde está identificado, está bajo una Licencia Creative Commons