CC BY-NC-ND
235
Developing an approach to ranking innovative IT projects
Vladimir I. Grekul
E-mail: grekoul@hse.ru
Eugene A. Isaev
E-mail: eisaev@hse.ru
Nina L. Korovkina
E-mail: nkorovkina@hse.ru
Tatiana S. Lisienkova
E-mail: tlisienkova@hse.ru
National Research University Higher School of Economics Address: 20, Myasnitskaya Street, Moscow 101000, Russia
Abstract
Digital transformation is a highly topical task for many companies. Implementation and use of breakthrough technologies are an essential part of this process. Nowadays the terms "innovation" and "information technologies (IT)" are treated as equals insofar as IT is exactly what can provide execution of innovative strategy and the digital transformation of a company's business.
Due to the high speed of IT market growth and the emergence of new technologies, companies usually implement them without justified selection and prioritization, and this leads to the high rate of failed innovative IT projects. Often such projects fail to result in commercially successful products or services by which a company can distinguish itself from competitors to consumers. Still the most widespread approach for evaluation and ranking of innovative IT projects concentrates on the expected financial outcomes without due attention to strategic alignment of a project.
This research suggests an approach for ranking innovative IT projects in big companies. The approach entails complex evaluation of expected results of projects on the strategic, environmental, organizational and technological domains of a company. This approach is based on a modified Tornyatzky—Fleischer IT innovation adoption model.
During the first stage of research, the term and definition of innovation have been discussed as well as features of innovative IT projects. The second stage is dedicated to comparison analysis of evaluation approaches for innovative projects as well as to choosing an IT adoption model for further adaptation. On the third stage approbation of the method developed been carried out in one of the Russian big IT/ internet companies. The results of two-year period of approach approbation have proved its suitability and suggested the prospects for further development.
Key words: innovative IT project; IT innovations; innovation life cycle; innovation adaptation models; innovation adoption models; Tornyatzky—Fleischer TOE model.
Citation: Grekul V.I., Isaev E.A., Korovkina N.L., Lisienkova T.S. (2019) Developing an approach to ranking innovative IT projects. Business Informatics, vol. 13, no 2, pp. 43—58. DOI: 10.17323/1998-0663.2019.2.43.58
Introduction
The scale of digitalization of the economy continues to grow with impressive speed. According to McKinsey's forecast, by 2020 the share of digital business in global GDP will be 34% [1]. Businesses are trying to become digital as quickly as possible partly due to the high rate of emergence and development of breakthrough technologies1, as well as the necessity to be closer to their consumers whose life every day becomes more digital. A special term for such business changes has been introduced; that is "digital transformation" — a fundamental transformation of the company's products and services, as well as its business model using information technologies.
An essential part of digital transformation is the implementation of innovative technologies, for example, for corporations such as Huawei [3], Luxoft [4], Samsung [5], companies in the tourism industry [6], and others. Many members of the business community suggest that today information technologies are most often the source of innovation: new technologies and products based on innovative technologies appear on the IT market at least once every 1—1.5 years [7]. However, not all companies manage to successfully adopt new information technologies. That is proved in the PWC global survey, where experts assessed the digital maturity of companies from different industries and
found out that only 10% of the surveyed companies can be considered as "digital leaders"2; around 50% are "digital innovators", who are characterized by the active implementation of new technologies, but without proper coordination and coherence with the company's strategy, as well as without the expertise and support of top management [8].
Nowadays quite often innovations and information technologies are considered as equals: according to research, it is information technologies that can ensure the implementation of innovative strategies in most modern enterprises during the process of digital transformation [9]. Statistics also show that only 10% of innovative IT projects are completed successfully (which means they have a commercially successful product or service as a result) [10]. Thus, most innovative IT projects are recognized as failures.
In the competitive environment, in order to become a "digital leader" companies strive to implement breakthrough technologies by realization of innovative projects which match the company's strategy and support its realization. This is relates especially to medium and large companies that carry out project activities on their own and where several big projects are often carried out at the same time. For such companies, ranking projects can be the basis of a successful management system. Project ranking is a process that increases the
1 Technologies which create new markets, significantly change values on existing ones by making current products not competitive any more (for example: artificial intelligence, internet of things, blockchain, nano-materials etc.) [2]
2 Companies which by using new digital technologies ensure consistency in its development at all levels of the organization, support executing of strategy, provide end-users with competitive products and services with distinctive features
probability of success of strategic projects and increases the consistency of ongoing projects with strategic goals [11]. Implementation of innovative projects without pre-selection and ranking deprives the company of the coherence in its actions and leads it away from the " digital leader" path towards that of a less mature "digital innovator."
A certain class of innovation adoption models has emerged in international practice to help one understand if an innovation is suitable for a company, but their application is not widespread due to their theoretical approach.
A feature of many innovative projects is that a company begins to benefit from it not immediately after project completion, but only after some time — in the long run. In addition, it should be noted that it is rather difficult to estimate the economic efficiency of such projects in advance [12]. A number of works also suggest the problem of the lack of connection between innovative projects and the company's strategic goals as a common thing for innovation projects and the main reason of the failure of such projects [13, 14]. Thus, it can be stated that today the issues of prioritizing innovative IT projects are not sufficiently developed. As a result, more in-depth studies for developing methods and tools that provide an integrated approach to the ranking of innovative IT projects become essential.
Currently, the most common approach to ranking innovative projects (including in the IT field) remains the analysis of the expected financial results of implementation, for example, analysis of the level of profitability, payback period of projects, etc. [12]. That means that this approach considers only financial outcomes of projects while disregarding their longrun effect on the company's product, strategy, customers and other aspects.
The purpose of this study is to develop an approach of integrated evaluation of the impact of innovative IT projects on the company3.
1. Features of IT innovation
Today the term "innovation" can be applied to most areas of society. Depending on the context, this term has a different meaning. So, in the works of Eurostat [15], Schumpeter [16], Anshina [17], Kozlovskaya et al. [18] innovations are presented as the implementation and use of scientific research results in an enterprise. Santo [19], Medynsky [20] and Bary-sheva [21] present innovation as a process of scientific development and research, without an emphasis on further use. Glukhov et al. [22], Twiss [23], the Organization for Economic Cooperation and Development [24] and Edison et al. [25] distinguish the commercial component of innovation in their definitions — the moment when the R&D results acquire an economic value.
Summarizing the works and articles analyzed allows us to define the following features of innovation:
♦ in many definitions it is described as a process with a sequence of actions (stages);
♦ the creation of a new technology or any other embodiment of the idea does not mean the completion of the innovation process: it also includes commercialization follow-up — marketing, product launch on the market and promotion, stimulation of demand, etc.;
♦ innovation is a result in the form of new products, services, technologies, the idea embodied in real life;
♦ innovative activity requires a certain organizational support — the allocation of the necessary resources, changes in the organizational structure, the creation of new functions, teams, etc.;
3 Preliminary results of the study were presented in the graduate work by Tatiana S. Lisienkova, performed at the HSE Faculty of Business and Management in 2017
❖ recommendations on the collection and analysis of data on innovations [19] introduce the term "diffusion" as a way to spread innovation and bring it to consumers and gain the economic value of innovation in such a way.
In the same way, it is also possible to highlight the features of innovation in IT. The definition of the concept of IT innovation is most often found in individual studies [12, 27, 28, 32]. Summarizing the proposed interpretations, IT innovations can be defined as a set of technical innovations that support information exchange technologies and processes, as a result of which information becomes an important component of the production process, product changes, increasing its added value. One of the tasks of IT innovation is to improve the information flow in the organization and improve the quality of information (its efficiency, relevance to the user, reliability, sufficiency).
Based on analysis of the above works, the following features of innovative IT projects can be identified:
❖ increased uncertainty in terms of results (it is often difficult to predict the outcome of such projects and predict the likelihood of their success);
the difficulty of defining the customer and end user of the future product;
■b general use of new practices (approaches to the development, collection of requirements, software environments, products, etc.);
❖ complexity of prototyping the final result of the project;
❖ high uncertainty in estimating the timing of projects: the initial estimate of dates may change significantly after specifying the requirements, the customer and the end user, the concept of the project result;
❖ difficulty of assessing the midterm results of the project, difficulty of monitoring the project plan implementation;
positive effect on the company's value added: the implementation of innovation can
positively affect the company's operating activities, however, first of all, the innovation should be about improving product or service characteristics for the end user.
2. Approaches to evaluating innovative IT projects
With the increasing importance of innovation and the accumulation of experience in managing innovative projects, the need to form ways to assess the impact of innovation is also growing. In a number of works devoted to the evaluation of innovative project results, the fuzziness of initial requirements and the high probability of significant deviations of actual results from expected ones are indicated as significant problems [26]. The authors of [27] distinguish four groups of approaches for evaluating innovative projects: financial, multi-criteria, approaches based on correlations, and approaches for evaluating project portfolios.
Financial approaches correspond with the classical theory of management accounting, which means the decision-making is based on a comparison of investments and economic benefits presented in monetary units. The most common methods of financial evaluation include the calculation of rate of return, the project payback period, the net present value of the project, and the internal rate of return.
To evaluate the intangible outcomes of information technology implementation, non-financial indicators are needed. The difficulty here lies both in the choice of units for measurement of material and intangible outcomes, and the need to take into account the importance of different criteria. To solve this problem, a multi-criteria approach is applicable, where a set of criteria to cover tangible and intangible outcomes is defined before evaluation. Such criteria are assigned with weights of importance, and the projects themselves are subsequently compared by an integral score. The most comprehensive methods that take into account various criteria include the infor-
mation economy method and the SIESTA method (Strategic Investment Evaluation and Selection Tool Amsterdam) [28].
Approaches based on relative indicators are used to compare projects with each other. These indicators may take into account not only financial performance, but also the number of improved business processes or the number of new products and services of the company. The method of return on management investment (ROM) allows evaluating the maturity and effectiveness of management in company.
Project portfolio approaches allow us to evaluate projects at various organizational levels,
considering not only the characteristics of projects, but also various business indicators, and in some cases even the results of business model changes. This group of approaches questions not only the advisability of investing in a specific project, but also defines important business activities the company wants to improve. Such approaches include the method of Bedel [28], the method of investment mapping and the method of forming an investment portfolio [27].
To select an approach group for the ranking of innovative IT projects, a comparative analysis of the methods described above has been carried out (Table 1). The analysis of the inno-
Table 1.
Comparison of approaches to evaluation of innovative projects4
Аpproach Criteria ^^^^ Financial approaches Multi-criteria approaches Relative indicators approaches Project portfolio evaluation approaches
Approach coverage IT project IT project, company IT project, company IT project, company
Account of the project results' uncertainty + (while calculating the final indicator) + (can be evaluated with a separate criterion) + (while calculating the final indicator) + (while calculating several scenarios)
Account for project timing uncertainty - + - +
Account for project result value added + / - + (can be evaluated with a separate criterion) + / - +
Quantitative evaluation + (financial indicators) + + (financial indicators) +
Qualitative evaluation - + - +
Methodological support + / - + / - + / - + / -
Complexity of approach implementation Financial competence needed Financial competence needed Financial competence needed
Nature of approach results Interval rate scale Ordinal rate scale (ranking) Interval rate scale Interval or ordinal rate scale
4 The sign "+" means that a model gets a positive score for a criterion, "—" means a negative score, "+ / —" means that a model gets appositive score but with some limitations and restrictions
vative IT projects features allowed us to form a list of criteria for comparison of approaches:
1. approach coverage (does the approach use only project information or takes into account the specifics of the organization, the impact of project results on the activities of various departments of the company, the achievement of strategic goals via project results, etc.);
2. the possibility of taking into account the uncertainty of the project results;
3. the possibility of accounting for the uncertainty of the project timing;
4. the ability to take into account the impact of the project's result on the product or service value added;
5. the nature of the criteria for evaluating the approach (qualitative or quantitative);
6. the nature of the results: what type of scale is used to present the evaluation results — nominal, ordinal or interval [29];
7. methodological support: whether there are any recommendations for the collection of information, the calculation of the intangible and tangible benefits, recommended specialists and roles, procedures and evaluation timelines;
8. the complexity of the approach: the complexity of collecting information for the application of the method, the visibility of the results obtained, the need for special competencies of employees.
As the table shows, the group of multi-criteria approaches is of the greatest interest. Such approaches are not limited to financial evaluation, which is important when analyzing innovative IT projects which may result in intangible benefits. Moreover, the multi-criteria approaches are applicable not only at the level of an individual project, but also at higher levels (for example, at the level of organization's strategic goals) by incorporating relevant criteria into the approach. Multi-criteria approaches are more flexible in terms
of taking into account the uncertainty of the project timing. The ordinal rating scale allows comparison and ranking of projects of a different nature, scale, purpose with the same criteria, all of which is important not only for the market or the industry as a whole but for a certain company. However, for evaluation of high-risk innovative projects it is insufficient to have an optional criterion for risk assessment. It is more acceptable to calculate several project scenarios (i.e. worst, best, moderate ones) by analogy with approaches to the project portfolio analysis.
3. Development of an approach to ranking innovative IT projects
With the development of IT and the analysis of innovations in this area, the scientific community has identified a separate area of research into the "adoption" of IT innovations. "Adoption" here means the successful implementation of IT innovation in a company, leading to a qualitative improvement in its operations and performance [28]. In different studies, the factors which influence the adoption or rejection of IT innovations are outlined. The results of such studies are summarized in models, frameworks and recommendations which allow us to evaluate the possibilities of adoption for a particular innovation.
The most widespread IT innovation adoption models are technology adoption models [30], potential implementation of technologies [31], the "Diffusion of Innovations" model [32] and the "Planned Acceptance" framework [33], as well as the technology-organization—environment model (TOE model) [34].
Study of these models allowed us to identify the criteria for comparison analysis, taking into account the limitations of the models, the core objects of models and the results of their use. The criteria developed include:
♦ type of approach;
♦ the size of the company the approach works with;
♦ the phase of the IT innovation life cycle the approach works with;
♦ the organizational level of acceptance evaluation (i.e. single position, department, division, branch etc.);
♦ accounting for IT innovation alignment with corporate strategy;
♦ the of adoption of IT innovation analysis;
♦ the result of approach.
The results of comparison analysis are presented in Table 2.
According to comparison, the Fleischer— Tornyatsky model of the technological, organizational, and environmental context (TOE model) is of the highest interest [34]. This model can be implemented in a company of any size as well as for innovation at any phase of its life cycle. Moreover, evaluation is carried out at the organizational level and considers its activities from different aspects through three contexts. What is more important, only this approach suggests the oppor-
Table 2.
Comparative characteristics of IT innovations adoption models
№ Approach
COMPARISON CRITERIA
Type of approach Size of evaluated company Phase IT innovation life cycle Level of acceptance Account of strategic alignment Object of analysis Result of approach
IT adoption model
Model
Small
Innovation emerging
Individual (evaluated for a certain position)
Internal characteristics of a company
Evaluation of IT innovation adoption for a certain position/role
Diffusion of innovation
Framework
Small, medium, large
Corporate (the entire enterprise)
Internal and external characteristics of a company
Evaluation of IT
innovation adoption for a certain company
Planned acceptance
Framework
Small, medium
Innovation emerging
Both individual
and corporate
Company's leadership styles,
innovation potential of staff
Evaluation of IT innovation adoption for a company's corporate culture
Potential implementation of technology
Model
Medium, large
Individual
Staff competences
and qualification
IT innovation use scenarios for company's employees
TOE model
Model
Small, medium, large
Corporate
+ / - 5
Organizational, technological and environmental context
Evaluation of IT innovation adoption for a certain company
1
2
3
4
5
5 Can be defined by evaluators but not a mandatory one BUSINESS INFORMATICS Vol. 13 No 2 - 2019
tunities to evaluate IT innovation alignment with corporate strategy.
The authors of this model distinguish three contexts (the processes and characteristics of which influence the success of the IT innovation "adoption"), these contexts are technological, organizational, and environmental [34].
The technological context determines how ready IT companies are to implement new technology, how this technology is spread and adopted on the market in general and how this technology is realizable with the current technical conditions and capabilities of the company. The organizational context describes how the size of the company, its organizational structure, communication processes and internal growth drivers will affect the adaptation of IT innovation. The environmental context evaluates the impact of IT innovation on the company's position in the industry and among competitors. It also takes into account the limitations that an IT innovation can face from the government and other regulatory institutions.
In articles [35—37], the authors illustrate each context in more details. However, the original model lacks the strategic alignment of IT innovation — it can be defined by an evalua-tor in one of three contexts, but not necessarily. Thus, a new context, the strategic one, for the TOE model modification is suggested. The strategic context allows us to assess the coherence between IT innovations and the company's strategy, as well as IT innovation's impact on the achievement of strategic goals (Figure 1).
Several works are dedicated to the use of the TOE model, where their authors describe contexts with well-known models, frameworks and tools of strategic, organizational and IT management (for example, Porter's five forces framework, the value chain, the Osterwalder — Pigneur outline, PEST analysis, SWOT analysis, models of enterprise architecture TOGAF, Zachman, etc.). The choice of models remains up to the company or organization that evaluates the IT innovation adoption. Based on the recommendations for research into innovation activities (the Oslo Manual [15]) three stages of the ranking process have been outlined (Figure 2).
Fig. 1. Scheme of modified TOE model of IT innovation adoption
Event planning Data collection Analysis
y for project y y for further / y of ranking
ranking projects ranking results
Fig. 2. Main stages for IT innovation project ranking
Stage 1. Initialization and event planning for the project ranking process. Within this stage preparatory activities such as selection of innovative IT projects, forming plans and schedules for project evaluation activities, appointing decision makers and members of the expert team using the responsibility matrix are carried out.
To determine the innovativeness of the project and its selection to the short-list of candidate projects for further evaluation and ranking a number of methods and recommendations are proposed in [38—40]. In our study the procedure of innovative projects selection is not considered: by default it is assumed that the evaluated projects are truly innovative.
Stage 2. Activities for data collection for further projects ranking. During this stage the main activities of the approach are carried out based on the modified TOE model of IT innovation adoption model. These activities include the preparation of a base for project evaluation (definition of evaluation criteria, forming of an evaluation tool and procedure), collecting data for evaluation and calculation of projects scores.
To highlight the criteria in four contexts the following models can be used:
♦ strategic context: Osterwalder—Pigneur business model canvas, value chain, Porter's five-force model, a balanced scorecard;
♦ environmental context: SWOT analysis, PEST analysis, competitive benchmarking, T.E.M.P.L.E.S. analysis;
♦ organizational context: the organizational structure model, organizational layer of
TOGAF and Zachman enterprise architecture models and frameworks;
♦ technological context: technology, applications, information systems, data layers of TOGAF and Zachman enterprise architecture models and frameworks.
Expert evaluations can be held both collectively while open discussions (brainstorming) or open rating followed by score calibration, and individual closed evaluation from each expert with further calculation of the arithmetic average of all scores.
Stage 3. Analysis of ranking results. The
final stage is dedicated to the processing of the data collected and project ranking based on this data. At this stage, the goal is achieved and we obtain a ranked list (rating) of innovative IT projects. The collected questionnaires with scores are processed, for each criterion the average score is set, and then each candidate project receives a final score that equally takes into account four contexts, similar to the original model. This means that the score of each context goes into the final score with the weight of 25%.
Realization of ranking stages described above depends on the specifics of a certain company such as its size, staff, experience and level of maturity in project management. At the same time, the internal knowledge base about innovative projects will determine the specifics for this approach —frequency of use, list and number of positions and roles in expert teams, the format of communication during the assessment and other organizational aspects.
4. Implementation of the approach developed in a Russian IT company
Testing the approach we developed was carried out in one of the leading Russian internet technologies companies. The company owns a large internet search engine and site and also provides services on this site. The main markets for the company are Russia, the CIS countries, Turkey and Israel. The company's services include various advertising opportunities, as well as consulting and analytical support for advertisers.
The approach we developed was applied in the company in 2017 for the first time. Within the first stage, specialists of the group of perspective products formed a list of six candidate projects. Each project was assigned a responsible manager who provided all the necessary information about it (i.e. project description, its contents, goals, a list of necessary resources, expected results, etc.) and participated in activities for project ranking. A team of experts was also formed at the first stage. It was important to involve specialists from different areas which correspond to the four contexts of the IT innovation adoption model. Thus, experts from several departments were involved.
In addition, a meeting schedule was formed and the deadlines for project managers to submit information were determined as well as document templates for submitting information.
During the second stage of the modified IT adoption model, a list of criteria for evaluating innovative IT projects was determined. To form criteria of the strategic context, where it is necessary to analyze the project's impact on the company's business logic and strategy, the Osterwalder-Pigneur business model canvas has been used. This canvas describes the company's activities in nine blocks: key resources, partner network, key activities, value proposi-
tion, customer segments, channels, customer relationships, cost structure, revenue streams [41]. The completed canvas was handed over to the experts who identified the problematic elements in the blocks (Figure 3). In particular, in the "partner network" block the problematic element is advertising agencies; in the "key activities" block - the sale of advertising services; in the "value proposition" block — advertising surfaces and customized analytics; in the "consumer segments" block - small and medium business; in the "customer relationships" block - a personal manager; in the "income structure" block - advertising. Thus, the projects evaluated should somehow assist and support identified problem entities.
For the other contexts, the following models have been used:
♦ organizational context: a motivational model of a unified approach to the enterprise architecture by TOGAF and Archimate [42];
♦ technological context: a multi-layered model of enterprise architecture [43] (in this case, for a commercial department);
♦ environmental context: SWOT analysis.
Similarly to the strategic context, experts
identified the problem areas for other contexts, and this became the basis for the questionnaire. For questions of the strategic context, it was proposed to use a qualitative scale; for technological, environmental and organizational questions — a quantitative scale. The choice of scale may vary depending on the models chosen for each context.
In order to evaluate the impact of candidate projects on the identified problem areas, a questionnaire was developed (Table 3), and a qualitative scale was introduced where values vary from one to five: 1 — a project will have a negative effect; 2 — a project will have no effect; 3 — a project will have indirect effect; 4 — a project will have a positive effect; 5 — the project is directly focused on improving a problem area.
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIWIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
Key
partners
Internet sites
Internet
publishers
Advertisement agencies
Key
activities
Client acquisition
Client service
Client support
R&D
Value
proposition
Vast search engine on internet portal Various supporting services on portal
Customer relationship
Personal manager (SMB clients)
i.
SIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIE
Key resources
Intellectual resources Human resources Physical resources
Advertisement placement with wide audience coverage
Customized analytics for advertisers
Personal manager (key clients) 27/7 support
Customer segments
Portal and search engine users Large businesses
Small and medium businesses
S2
%
SIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIE
Channels
| Internet site Mobile / tablet applications Physical offices
1
■iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii FFi iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii FFi iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii uj i ■■iaiaiaiaiiaaiiaaiiaaaiaaaiaa m iiaaiaiiaaiaiaaiiaiaaaiaaiiaiaaiiaaiaiiaaiaiaaiiaaiaiiaaiaia m ■iaaiiaaiaiiaaiaiaaiiaiaaiiaaiaiiaaiaiaaiiaiaaiiaaia 11 ■ i
Cost structure
Technical maintenance of data centers and other physical resources Marketing R&D
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIM
Revenue streams
Advertisements (search engine result page, partner sites, mobile applications)
Fig. 3. Graphic representation of the company's business model on the Osterwalder—Pigneur canvas6
As a result of the second stage, a survey has been conducted among respondents of various positions and roles. The questionnaire for each candidate project was collected.
On the third stage, the data received was processed, the final scores were given to each project and a ranked list of projects was drawn up. The score for each context was defined as a share of the maximum possible in this particular context. The final score of the project was calculated as an arithmetic average between four contexts similarly to the principle of equality of contexts in the original TOE model [34]. Further on, the project was assigned a rank depending on the score: the higher the score, the higher the project rank. Project evaluations in four contexts are presented in Figure 4 in the form of a radar chart,
6 Grey color of a cell defines problematic aspects in each block
where an "ideal" project would have a value of 100% in all contexts.
To verify the suitability of the results of the developed approach, the calculated project ranks were compared with the ranks of the same projects from internal corporate sources (Table 4). According to the results, four out of six projects received the same ranks. The ranks of the projects "Smart client base cauterization" and "Smart forecasting of advertisement", obtained from internal data and with developed approach, differed by one point.
Study of internal corporate information allowed us to justify the difference in project rank — due to the lower prioritization among department heads' projects, "Smart client base cauterization" was given a smaller rank. The product line did not take into account the fea-
Table 3.
Questionnaire for four context evaluation
Question groups according to modified TOE model of IT innovation adoption
1. Strategic context (based on Osterwalder-Pigneur)
1.1 How will project realization affect the "Small and medium businesses" customer segment?
1.2 How will project realization affect the "Advertisement agencies" key partners segment?
1.3 How will project realization affect the "Personal manager" customer relationship segment?
1.4 How will project realization affect the "Advertisement placement" value proposition segment?
1.5 How will project realization affect the "Customized analytics" value proposition segment?
1.6 How will project realization affect the "Client service" key activities segment?
1.7 How will project realization affect the "Client support" key activities segment?
1.8 How will project realization affect the "Advertisement" revenue stream segment?
2. Organizational context (based on SWOT analysis)
2.1 What share of identified opportunities is realized by a project?
2.2 What share of identified threats is minimized by a project?
2.3 What share of identified weaknesses is improved by a project?
2.4 What share of identified strengths is enlarged by a project?
3. Org by anizational context (based on the motivational model of a unified approach to the enterprise architecture rOGAF and Archimate)
3.1 What share of goals is reached with the help of a project?
3.2 What share of KPIs is increased with the help of a project?
3.3 What share of growth drivers is activated with the help of a project?
3.4 What share of external stakeholders is positively affected a project?
4. Tec by hnological context (based on a multi-layer model of a unified approach to the enterprise architecture rOGAF and Archimate)
4.1 What share of application components which will be integrated in a project requires significant change or development from scratch?
4.2 What share of services which will be integrated in a project requires significant change or development from scratch?
4.3 What share of supporting business processes is changed by a project?
4.4 What share of key business processes is changed by a project?
4.5 What share of business services is changed by a project?
4.6 What share of external contactors and communication with them is changed by a project?
tures of different types of advertisers, since its original goal was to provide universal analytical material.
Similarly, the approach we developed was reused in 2018. The results are presented in Table 5. Compared to the first implementation of the approach developed, this time the ranks of all
projects via the approach and via internal corporate data had no differences.
In the first implementation of the approach we developed, the deviation from the company's internal rating was around 6%; in the second implementation no deviations in the ratings have been revealed. The equality of project rat-
Strategic context
...... "Smart" advertisement proposal
---"Smart" media plan
= = = "Smart" client base cauterization
- "Smart" forecasting of advertisement
- "Smart" VC processing
"Smart" forecasting of revenue anomalies
Fig. 4. Graphic representation of the project evaluations on the axes of the four contexts
ings proves that the approach we developed is close to the way projects are ranked in a company. It is important to note that the approach use in 2017—2018 allowed us to create a base of reference models, developing documents templates, questionnaires and meeting scripts, all
of which will reduce the time needed to realize the approach next time. Moreover, the proposed approach allowed us to formalize the half intuitive and subjective factor of decision making. Previously, the project ranking process in the company was presented as a "black box": the input information was a set of candidate projects, a ranked list of projects was the output information. The general questions of ranking timing, communication formats, evaluation team and criteria were very relevant for employees who were working on a project or its idea.
Presentation and documents on the results of the approach use have been presented at the quarterly meetings of the commercial department.
After a series of discussions, it was decided that the special group of prospective products and services will continue using a new approach for IT innovative project ranking in order to accumulate data and confirm the appropriateness of the approach. At the same time, the approach will be used together with the current "intuitive" method of project prioritization; this will allow us to adjust the approach and correct minor flaws.
Regular use of the approach we developed will collect enough information to clarify the list of competencies and extend the roles and position of the expert team. Based on the first
Table 4.
Comparative evaluation of the rank lists of innovative IT projects (2017) from the developed approach and from internal corporate data
Project Based on the developed approach Based on internal data
Share of maximum (%) Rank Rank
"Smart" advertisement proposal 78% 1 1
"Smart" media plan 72% 2 2
"Smart" client base cauterization 67% 3 4
"Smart" forecasting of advertisement 64% 4 3
"Smart" CV processing 52% 5 5
"Smart" forecasting of revenue anomalies 48% 6 6
Table 5.
Comparative evaluation of the rank lists of innovative IT projects (2018) from the approach we developed and from internal corporate data
Project Based on the developed approach Based on internal data
Share of maximum (%) Rank Rank
"Smart" audit of client account 75% 1 1
"Smart" forecasting of geographical expansion of client account 71% 2 2
"Smart" client churn forecasting 67% 3 3
"Smart" forecasting of advertisement agency growth 65% 4 4
"Smart" forecasting of key performance indicators of commercial department 61% 5 5
"Smart" task allocation 50% 6 6
implementation of the approach, it is also possible to develop requirements for the forms, document template and content of supporting documents that will help experts conduct project evaluation activities in the future.
Conclusion
The approach we developed allows one to carry out a complex evaluation and ranking of innovative IT projects based on analysis of project results influence on four contexts of the company: strategic, environmental, organizational and strategic
This method can be applied in the following cases:
♦ a company chooses new innovative technologies from among several alternatives;
♦ a company selects innovative projects among candidate projects for further realization;
♦ a company prioritizes projects for forming a plan and order of their realization.
The approach we developed has been successfully tested and applied in a large Russian IT company, making it possible to form practical recommendations about its further usage.
The prospects of this study lie in the conceptual development of the approach, such as research into the significance of each of the four contexts and calibration of their coefficients in the final score. Another area of further research is dedicated to the development of a procedure for determining the innovative-ness of a project so as to allow preselection of innovative IT projects for further ranking with the approach developed. ■
References
1. McKinsey and Company CIS (2017) Digital Russia: new reality. Available at: https://www.mckinsey.com/ ru/~/media/mckinsey/locations/europe%20and%20middle%20east/russia/our%20insights/digital%20russia/ digital-russia-report.ashx (accessed 12 September 2018) (in Russian).
2. Bower J.L., Christensen C.M. (1995) Disruptive technologies: Catching the wave. Harvard Business Review, no 1, pp. 43—53.
3. Huawei (2018) Huawei drives convergence ofdisruptive technologies for digital transformation success.
The Wall Street Journal Partners. Available at: https://partners.wsj.com/huawei/news/digitally-advanced-traditional-enterprises-will-lead-digital-era/ (accessed 13 September 2018).
4. Blanck T. (2018) Disruptive innovation as a catalyst for digital transformation. Available at: https://www.luxoft. com/podcasts/disruptive-innovation-as-a-catalyst-for-digital-transformation/ (accessed 13 September 2018).
5. Qualman E. (2018) Digital transformation in the workplace. Available at: https://insights.samsung. com/2018/07/05/digital-transformation-in-the-workplace/ (accessed 13 September 2018).
6. Likholetov V.V., Lisienkova L.N., Baranova E.V. (2016) Virtual tour as a marketing instrument in tourism. Economics and Innovations Management, no 1, pp. 136—138.
7. Martin W.J. (2017) The global information society. Abingdon: Routledge.
8. PWC Consulting (2018) Global research ofdigital operations in 2018: "Digital champions". Available at: https://www.pwc.ru/ru/iot/digital-champions.pdf (accessed 23 November 2018).
9. Rot E. (2010) Innovations as the way to increasing efficiency. McKinsey Bulletin. Available at: http://vestnikmckinsey.ru/strategic-planning/innovacii-put-k-povyhsheniyu-ehffektivnosti (accessed 23 November 2018) (in Russian).
10. Fisher A. (2014) Why most innovations are great big failures. The Fortune. Available at: http://fortune. com/2014/10/07/innovation-failure/ (accessed 13 December 2018).
11. Nieto-Rodriguez N. (2016) How to prioritize your company's projects. Harvard Business Review. Available at: https://hbr.org/2016/12/how-to-prioritize-your-companys-projects (accessed 22 November 2018).
12. Fedosova R.N., Pimenov S.V. (2009) Modern instruments for investment projects appraisal. Vestnik of the Orenburg State University, no 5, pp. 78—81 (in Russian).
13. Barancheev V.P., Maslemmikova N.P., Mishin V.M. (2015) Innovations management. Moscow: Urait (in Russian).
14. Kirsner S. (2017) The stage where most innovation projects fail. Harvard Business Review. Available at: https://hbr.org/2017/04/the-stage-where-most-innovation-projects-fail (accessed 22 November 2018).
15. OECD (2005) Statistical office of the European Communities. Oslo Manual: Guidelines for collecting and interpreting innovation data. Third edition. Available at: http://www. 1-innocert.my/Manual/0SL0%20 Manual%20V3%202005-EN.pdf (accessed 23 November 2018).
16. Schumpeter J. (2008) Theory of economic development. Moscow: Eksmo (in Russian).
17. Anshina A.A. (2009) Innovative management. Moscow: Delo (in Russian).
18. Kozlovskaya E.A., Demidenko D.S., Yakovleva E.L. (2012) Economics and innovations management. Moscow: Economika (in Russian).
19. Santo B. (1990) Innovation as a mean ofeconomic development. Moscow: Progress (in Russian).
20. Medysky V.G. (2008) Innovative management. Moscow: Infra-M (in Russian).
21. Barysheva A.V., Baldin K.V., Galditskaya S.N., Ischenko I.I. (2013) Innovations. Moscow: Dashkov & Co. (in Russian).
22. Glukhov V.V. (2003) Economics of knowledge. Saint Petersburg: Piter (in Russian).
23. Twiss B. (1989) Management of scientific and technical innovations. Moscow: Ekonomika (in Russian).
24. OECD (2015) Frascati Manual 2015: Guidelines for collecting and reporting data on research and experimental development. The measurement ofscientific, technological and innovation activities. Available at: https://read.oecd-ilibrary.org/science-and-technology/frascati-manual-2015_9789264239012-en#page1 (accessed 21 November 2018).
25. Edison H., Bin Ali N., Torkar R. (2013) Towards innovation measurement in the software industry. Journal of Systems and Software, vol. 86, no 5, pp. 1390—1407.
26. Sotavov A.K. (2016) Improvement ofinnovations project management methods in the sphere of information technologies. Doctoral thesis. Saint Petersburg (in Russian).
27. Renkema T.J.W., Berghout E.W. (1997) Methodologies for innovation evaluation at the proposal stage: a comparative review. Innovation Technology, vol. 39, no 1, pp. 1—13.
28. Mumford M.D., Bedell-Avers K.E., Hunter S.T. (2008) Planning for innovation: A multi-level perspective. Research in Multi-Level Issues, vol. 7, pp. 107—154.
29. Stevens S.S. (1946) On the theory ofscales of measurement. Science. New Series. Available at: http:// psychology.okstate.edu/faculty/jgrice/psyc3214/Stevens_FourScales_1946.pdf (accessed 25 November 2018).
30. Davis F.D. (1989) Perceived usefulness, perceived ease ofuse, and user acceptance of information technology. MIS Quarterly, vol. 13, no 3, pp. 319-340.
31. Venkatesh V., Morris M.G., Davis G.B., Davis F.D. (2003) User acceptance of information technology: Toward a unified view. MIS Quarterly, vol. 27, no 3, pp. 425-478.
32. Rogers E.M. (2010) Diffusion of innovations. N.Y.: Simon and Schuster.
33. Ajzen I. (1991) The theory ofplanned behavior. Organizational Behavior and Human Decision Processes, vol. 50, no 2, pp. 179-211.
34. Tornatzky L.G., Fleischer M., Chakrabarti A.K. (1996) Processes of technological innovation. Lanham: Lexington Books.
35. Kuan K., Chau P. (2001) A perception-based model for EDI adoption in small businesses using a technology-organization-environment framework. Information & Management, vol. 38, no 8, pp. 507-521.
36. Pan M.J., Jang W.Y. (2008) Determinants of the adoption of ERP within the technology-organization-environment framework: Taiwan's communications industry. Journal ofComputerInformation Systems, vol. 48, no 3, pp. 94-102.
37. Lim K.H. (2009) Knowledge management systems diffusion in Chinese enterprises: A multistage approach using the technology-organization-environment framework. Journal ofGlobal Information Management, vol. 17, no 1, pp. 70-84.
38. Mutanov G.M., Esengalieva Zh.S. (2012) Method for assessing the innovativeness and competitiveness of innovative projects. Fundamental Research, vol. 3, no 3, pp. 712-717 (in Russian).
39. Sergeev V.A., Kipcharskaya E.V., Podymalo D.K. (2010) Fundamentals of innovation design. Ulyanovsk: UlSTU (in Russian).
40. Linton J.D., Walsh S.T., Morabito J. (2002) Analysis, ranking and selection of R&D projects in a portfolio.
R&D Management, vol. 32, no 2, pp. 139-148.
41. Osterwalder A. (2013) A better way to think about your business model. Harvard Business Review. Available at: https://hbr.org/2013/05/a-better-way-to-think-about-yo (accessed 26 November 2018).
42. Rubenchik A. (2014) Modeling of enterprise architecture. ArchiMate language overview. Information Management, no 6, pp. 55-59 (in Russian).
43. Zaramenskih E.P. (2018) Enterprise architecture. Moscow: Urait (in Russian).
About the authors
Vladimir I. Grekul
Cand. Sci. (Tech.);
Professor, Department of Information Systems and Digital Infrastructure Management, National Research University Higher School of Economics, 20, Myasnitskaya Street, Moscow 101000, Russia; E-mail: grekoul@hse.ru
Eugene A. Isaev
Cand. Sci. (Tech.);
Professor, Head of Department of Information Systems and Digital Infrastructure Management, National Research University Higher School of Economics, 20, Myasnitskaya Street, Moscow 101000, Russia; E-mail: eisaev@hse.ru
Nina L. Korovkina
Associate Professor, Department of Information Systems and Digital Infrastructure Management, National Research University Higher School of Economics, 20, Myasnitskaya Street, Moscow 101000, Russia;
E-mail: nkorovkina@hse.ru Tatiana S. Lisienkova
Lecturer, Department of Information Systems and Digital Infrastructure Management, National Research University Higher School of Economics, 20, Myasnitskaya Street, Moscow 101000, Russia; E-mail: tlisienkova@hse.ru
