Qualification Evaluation of Computer Information System Integration Based on Weighted Products

1 System integration qualification and assessment system

The so-called system integration is a new process and method which combines different systems into an integrated, more powerful new one, depending on the application needs. Computer Information System Integration is engaged in the overall planning, design, development, implementation, service and protection of computer application system engineering and network system engineering. The Qualification Certification of Computer Information System Integration refers to the process in which those companies of computer information system integration must go through the certification held by third-party certification organizations, which are authorized by the Ministry of Industry and Information Technology Ministry, in order to obtain the “Computer Information System Integration Qualification Certificate”. These third-party certification organizations will assess the comprehensive and integrated capabilities of these enterprises on computer information systems, including technical level, management level, service level, quality assurance capabilities, technical equipment, system construction quality, personnel ratios and quality[3], operation results, assets condition and other factors[4].

At present, system integration enterprises of China can be divided into four levels — first grade enterprises have the capacity to build computer information system of those national, provincial (ministry), industrial level, prefectural (city and below) level enterprises independently, including large, medium and small ones and other types; second grade enterprises have the capacity to build computer information system of those provincial (ministry), industrial level, prefectural (city and below) level enterprises independently, including large, medium and small ones and other types; third grade enterprises have the capacity to build computer information system of those medium and small enterprises independently, or cooperative with the large-scale enterprises (or equivalent size ones); forth grade enterprises have the capacity to build computer information system of those small enterprises independently, or cooperative with the midsize enterprises (or equivalent size ones)[5].

Assessments of all grades of enterprises have strict evaluation criteria, such as first grade qualification selection criteria are divided into seven categories, consisting of 27 rules. See the first and second column of Table 1.

From the first and second column of Table 1 it can be seen that the content of the table has too many connotations, so the index analysis is necessary. Then the results are shown in the third column of Table 1 clearly.

2 Evaluation methods of system integration qualification

The evaluation processes of Enterprise System Integration Qualification are basically similar in China, but slight differences can still be found, for example, first grade enterprise qualification assessment process consists of four links, as shown below.

The third-party accrediting organization holds the document review and on-site evaluation, and verifies the enterprise application materials. → Ministry of Industry and Information Technology holds expert review meeting and the enterprise defense for itself. → Experts will vote according to the defense situation and application materials provided by the enterprise. → Report to the qualification management committee based on the voting results.

This evaluation method is essentially a combination of repeated and complex on-site manual work by assessors and qualitative voting by experts. Primarily, the evaluation result depends on the assessment experience and work attitude of those assessors. Secondly, it depends on the on-site defense effect. And finally, it is decided by experts’ intuitive feelings toward the status of the enterprise. Therefore, this evaluation method has some disadvantages — lack of fair, impartial and scientific assessment principles, and thus there is always dispute or disagreement on the evaluation results. A conclusion can be drawn that to explore the scientific and rational evaluation method is an vital issue in the face of government policy makers of China’s software industry, system integration enterprises, as well as qualification assessors.

Lots of evaluation methods are currently used. For the qualitative ones, there are expert-group method, Delphi method[6], brainstorming, etc. For the quantitative ones, there are weighted sum method, lexicographic method, AHP, the weighted product method, TOPSIS method, the main component analysis[7], etc. Besides, there is a fuzzy comprehensive evaluation method, which is the combination of qualitative and quantitative method[8], as well as extension goodness method[9–10] and so on. Selection of methods generally depends on the nature of the problem. These methods all have their own advantages, but they also have some shortcomings and assumptions. For example, the weighted sum method, the most widely used method, includes the following assumptions[11]:

① Index system appears to be a tree structure, in which each subordinate only associates with its upper index;

② The marginal value of each index is linear (the pros and cons is proportional to index value), the value of each two index are mutually independent;

③ The indexes are fully compensatory, that is, no matter how poor a certain index of a program is, it can be compensated by another index.

In fact, this assumption is not true in some evaluations. First, the index system may be reticulated, i.e., at least one subordinate associates with two or more than two upper indexes simultaneously. That is, an indicator can reflect both extents of the two upper indexes. Secondly, the linear condition of the marginal value of each index is often local, and there may be even a best value for a given interval or the midpoint; value independence conditions between indexes are also extremely difficult to meet, or at least extremely difficult to verify whether they meet. While compensatory between the indexes is usually only partial and conditional.

Unlike weighted sum method, the weighted product method has the following obvious characteristics: as long as there is a group (or class) indexes of which the value are zero, the comprehensive evaluation index of the weighted product method is zero; as long as there is a group (or class) indexes of which the values are lower, it will greatly affect the final value of the program; thus non-compensatory of indexes have been fully reflected.

During the system integration qualification evaluation process, a certain amount of compensatory can be reached between the first-grade indexes since each enterprise has its own characteristics and development process. From the second-grade indexes, it can be known that each enterprise should be compared with each other horizontally. Each condition must be met by the enterprises indispensably, so there cannot be compensatory between the first-grade and second-grade indexes. Even if there is partial compensation, it can not be linear, which is precisely the thought and meaning of the weighted product method. Therefore, in order to achieve fair, scientific and rational qualification evaluation results, a comprehensive evaluation method should be adopted.

3 System integration qualification index assignment

Scientific and fair evaluation process depends on the data. The nature, range, and standard requirements of various indexes related to the review process for the System Integration Qualification can be expressed in Table 2.

4 Case application

Take four companies (N₁, N₂, N₃, N₄), which participated qualification evaluation in the first half of 2012, to illustrate the data extraction and calculation in the entire evaluation process.

The second-grade index can be calculated through the relationship between the second-grade index and index analysis. Yet one thing should be noted that since there are incommensurability between various indexes, and the dimension difference as well, each index should be normalized and converted to a number between 0 and 1, wherein the quantitative data calculated by benchmarking.

Namely, Benefit-oriented Index takes Zij=xij/ximax, while Cost-oriented Index Zij=ximax/xij; Qualitative logic index takes 0 and 1; third-grade qualitative index takes 1, 0.75, and 0.5; forth-grade qualitative index takes 1, 0.75, 0.5, and 0.25.

Normalized results are shown in Table 4.

Suppose further that seven categories of the first-grade index are the same on weights, the comprehensive evaluation value of each enterprise will be: N₁ = 0.562, N₂ = 0.513, N₃ = 0.578, N₄ = 0.552, that is, N₃ ≻ N₁ ≻ N₄ ≻ N₂.

5 Conclusion

The evaluation of system integration qualification enterprise is significant towards our domestic software and information services industry, and it is also driving force to promote the steady development of China’s software and information service industry. This paper have reached a reasonably satisfactory result through the analysis of indexers, and the integrated use of selection criteria at different levels, combining the weighted product and weighted sum method to evaluate enterprise qualification. The idea and method proposed in this paper can help the government departments guide the development of the industry effectively with administrative means, thus carry out fair and impartial evaluation; At the same time, it will help reduce the repeatability workload of assessors from evaluation agencies, streamline workflow, and improve work efficiency; It will also help system integration enterprises to identify the development direction and standardize their development paths.

It should be noted that the paper showed the advantage of N₃ obviously, while little difference can be found between N₁ and N₄ in this evaluation. Therefore, before the passing number of the qualified enterprises is decided, differences in the evaluation results should be taken into consideration. To select the best from those excellent, N₃, N₁ and N₄ three companies should pass the evaluation in this case. It is unreasonable if only the N₃ and N₁ are allowed to pass the evaluation, after all the difference between N₁ and N₄ is quite small. Therefore, the evaluation results of the system integration enterprise group should be further analyzed with the help of aggregative rank[12-14] idea, which is one of the directions for further paper research.