CC BY
65
Detecting Network Attacks in Computer Networks by Using Data Mining Methods
Platonov V. V., Semenov P. O. Peter the Great St. Petersburg Polytechnic University St. Petersburg, Russia [email protected], [email protected]
Abstract. The article describes an approach to the development of an intrusion detection system for computer networks. It is shown that the usage of several data mining methods and tools can improve the efficiency of protection computer networks against network attacks due to the combination of the benefits of signature detection and anomalies detection and the opportunity of adaptation the system for hardware and software structure of the computer network.
Keywords: computer network, intrusion detection system, data mining methods, support vector machine, principal component analysis.
Fig. 1. The possibilities of IDS adaptation
Introduction
Nowadays there is a tendency of integration information resources into computer networks, this changing the order of information interaction and the software logic that leads to necessity of improvement the protection of computer networks. With the development of information technologies the problem of large data becomes particularly relevant. The main task of Data Mining methods is to detect information in unstructured data and presenting it in a visual form [1]. The set of parameters that are analyzing by intrusion detection systems is a considerable amount of data that determines the possibility of their processing exactly by Data Mining methods [2].
The purpose of the research is to improve the efficiency of network attacks detection in computer networks through the application of various Data Mining methods, used in the adaptive detection system. The novelty of the work lies in the design of dynamic intrusion detection system (IDS) architecture that can work both with a precise description of the attack, and to analyze a various statistical criteria, thereby combining the advantages of the classical anomaly detection systems and signature detection system. An important difference from other studies is the ability of IDS to adapt its functionality to the structure of the protected software and hardware environment. It worked out the possibility of using different methods and tools of Data Mining in the problem of intrusion detection.
The model of adaptive intrusion detection system
Most of the existing IDS can be represented as four main blocks: event, database, analyzer and the reaction block [3]. For effectively protection of the computer network it is necessary to distribute the iDs components to network units and create the ability to adapt to every unit. Fig. 1 shows all the possibilities of adaptation iDs components within a separate network unit.
The main task of the research is to develop the analyzer block that handles a lot of parallel detection processes on every physical network units. The detection module is an indivisible part of
analyzer block that allows detecting some kind of network attack or anomalous characteristics.
The developed model of IDS suggests different variants for distribution functional blocks in the computer network that are shown on Fig. 2. Depending on the computing capability, the basic information flow between units and performing tasks it is possible to create unified centers of decision making for the collected data from multiple units or signals from detection modules.
The ability of adapting the IDS to software and hardware structure of the computer network was created. Each detection module is associated with a set of attack classes, on which he responds with a certain probability. In the training dataset the considered attacks supplemented with options such as the types of potential targets and attack category. Detection module dependences are shown in Fig. 3.
Events
Modules
Reaction
Fig. 2. Distribution of IDS functional blocks in the computer network
List of event
Set analyzing basic features
Detection module
Set of attack classes
Potential target kinds | Attack category
[Hardware [Deny of service
¡Operating system [Probe / port scaning
|Service [Remote-to-Local
¡Software [User-to-Root
TT
r
T
|_The necessity of using detection module_|
Fig. 3. Detection module dependences
For automatically changing the event block each detection module is associated with a set of sensors that provides extraction of all necessary data features. Adaptive model consists in possibility of automatically change the set of using sensors and the set of detection modules, depending on the structure of the protected software and hardware environment and a variety of potential attacks. Depending on the potential invasion targets changed the list of processed network packet types and composition derived from traffic data, as well as the structure of the database tables used to calculate the statistical parameters of traffic. Depending on the detected attack categories in analyzer block varies composition of detection modules, and according to it changes the set of records in database decision rules.
The architecture of intrusion detection system
The analysis of the applicability of various methods of Data Mining was carried out to solve the problem of network attacks detection. Data Mining methods are used for classification, clustering and optimization. Among the existing approaches of network attacks and anomalies detection the most effective are Decision and Regression Trees, Support Vector Machines, Cluster Analysis, Association Rule Learning, Genetic Algorithms, Hidden Markov Models, Artificial Neural Networks and Fuzzy Logic [4-10]. As a result, the research highlighted a set of promising directions of Data Mining methods and their corresponding subtasks associated with the detection of network attacks in computer network, which are shown in tab. 1.
Table 1
The order of application Data Mining methods
Direction of Data Mining methods Realizable functions in network attacks detection
Classification The assignment of the analyzed vectors to sets of normal or attack
Dimension reduction Increasing the speed of network analysis by forming optimized features space
Clustering Construction an optimized set of detection modules
Fuzzy logic Organization of interaction detection modules; creating redundant IDS modular architecture
The classification methods categorize a sequence of net-work packets as either a set of attacks or as normal traffic. The dimen-
sionality reduction methods construct an optimal feature space based on network traffic parameters to detect specific sets of attacks. The clustering methods divide various attacks into classes for a particular attack-detection module. using fuzzy logic allows reducing the number of errors first and second kinds in the process of interaction among attack-detection modules.
For each of the subtasks corresponding directions were analyzed and the specific methods and tools were chosen. For the purposes of this investigation a program prototype of the intrusion detection system was developed using support vector machine (SVM) [11] as a classifier, principal component analysis (PCA) [12] for feature space formation that are most suitable for each attack under consideration and k-means method [13] for generate detection module set. C language has been used as a programming language. LibsVM was used to work with the sVM, with DARPA training files being used for investigation [14].
The designed architecture is based on the parallelization of the detection process, i. e., a number of attack-detection modules simultaneously run in the system in which each module is responsible for detecting a particular group of attacks (from a single network packet to the set of attacks according to some category). Fig. 4 shows a simplified scheme of a single detection module, which includes the following:
• extraction of relevant training information from network dumps;
• defining a set of clusters corresponding to the detection module with k-means method;
• training the PcA on the extracted dataset;
• translating the initial data into a lower-dimensional space according to the obtained transformation rules;
• training the sVM on the transformed data;
• testing the obtained SVM model;
• automatic adjustment of PCA and SVM parameters.
in the block of basic parameters extraction, the following basic parameters are extracted from the network traffic:
• network layer header parameters (IP, ICMP packets);
• transport layer header parameters (TCP, UDP packets);
• application layer header parameters (HTTP, FTP, DNS and SMTP packets);
• statistical and time parameters of the TCP session.
Fig. 4. Detection module formation
Both multi-bit parameters and divided eight-bit parts are taken into account, e. g., the IP address can be represented by one, two, or four parameters. The output of this block is a set of vectors. Each vector contains a label, i. e., attack (-1) or normal traffic (+1), and extracted parameters.
Fig. 5 shows a diagram of a network packet analysis. After extraction by a plurality of sensors, a common list of basic parameters is transferred to the detection module. In operation, the database system accumulates detection signals of the detection
Fig. 5. Network packet analysis
modules potentially dangerous traffic. The decision to invade is adopted on the basis of signals of all modules.
Dimension reduction of analyzing
TRAFFIC PARAMETERS
in the dimensionality reduction block, a dimensionality reduction method is applied to the extracted vectors to construct a new feature space [15-17]. Principal component analysis was chosen for the problem solution. its application resulted in construction of a weight matrix Wkp for calculation of new parameters as:
Skxn = WkxpXpxn ,
where, the k - dimension of new space; p - dimension of base space; n - number of vectors in base set; X- base vector set; S - new vector set.
For this method, it is important what kind of matrix is analyzed. Here, the following matrices are used: correlation matrix, covariance matrix, and sums of squares and cross products matrix.
For the automatic selection of a sufficient number of new parameters and the filtration of less important basic parameters, the following two threshold values are used:
• ^ is the minimum importance of a new parameter at which the parameter is usable;
• 5 is the minimum value of the coefficient in the transformation matrix of basic parameters.
From k parameters considered are those for which the corresponding eigenvalue of the analyzed matrix exceeds a threshold:
S' = [s{li e [1;k], \X{\ >^},
where, the L - eigenvalue of the analyzed matrix. Discarded x basic parameters, for which is true this condition:
X ' = {xjlje [1; n], Vi e [1; p]:\wy |<8}.
Upon training the PCA on the initial data, a list of new parameters is created that represents the new parameters via initial parameters and specifies the importance of the former. Based on the importance, different methods of reducing the dimensionality imply different objects; in particular, for the PCA, the importance is eigenvalues of the analyzing matrix. New parameters are represented as a linear combination of basic parameters and, to calculate new parameters, a matrix of linear representation coefficients is constructed.
The PCA is characterized by the need to scale the input data. in the detection system, upon extracting basic parameters, the following three scaling techniques are used:
• The maximum and minimum values for each basic parameter are found and this parameter is translated into the interval [-1;
+1] by the formula y = 2(x - min) / (max - min) - 1, where x is the basic parameter and y is the scaled parameter.
• The largest possible value of the basic parameter is taken as the maximum value (for example, the maximum lifetime of the IP packet is 28 = 256) and all basic parameters are divided by their maximums.
• The maximum and minimum values for all basic parameters are found and all basic parameters are translated by the formula y = 2(x - min) / (max - min) - 1.
For different types of data (various protocols and specific attacks), the best results are obtained by different data-scaling techniques.
The output of this block is the vectors containing attack labels and lists of new parameters.
Classification of network packets
The classification block is responsible for training and testing the SVM, as well as selecting the best parameters of the SVM kernel. For the SVM, the selection of the kernel is essential [18]. All used types of the SVM kernel are shown in tab. 2.
Table 2
SVM kernel types
Kernel name Kernel formula
Linear u' - v
Radial basis function g-yO'-v)2
Polynomial (y-u' xv + coe/0)deg ree
Sigmoid tanh(y -u' xv + coe/o)
in addition to kernel selection and kernel parameter adjustment, the effectiveness of the SVM also depends on the C parameter, which controls the error tolerance and the position of the hyperplane. This parameter allows one to adjust the ratio between the maximum width of the separator and the minimum total error.
SVM adjustment selection is based on the construction of the search grid for radial basis kernel by C and y parameters. An algorithm of SVM parameter selection for radial basis kernel by C and y parameters is suggested by the authors of library LibSVM [11]. When attempting to apply this algorithm for the considered data it was found that the suggested intervals for log2C and log2Y are not appropriate. Moreover, with log2C exceeding a certain constant the result of SVM work depends only on y parameter.
In the course of the performed study an automatic algorithm of SVM parameter selection was developed: first, log2Y parameters with the least number of support vector in training are determined for C = 225 (selection is performed with an interval A(log2Y) = 2). The log2C parameter in the area of the best log2Y values is found. Next, search grid with an interval A(log2C) = 2-4 and A(log2Y) = 2-4 is constructed and the best values in the area of best points obtained at the previous algorithm stage are found.
The distribution of the best points based on the experimental results for various attacks and basic parameter sets is shown in Fig. 6.
Fig. 6. SVM with radial basis kernel adjustments
Hues of color indicate the areas with identical detection percentage and support vector number. The darkest area corresponds to the best values set. All points below the first curve have a great number (up to several thousands) of support, vectors, with the training process lasting from several minutes to several hours. A dramatic reduction of support vectors number and, accordingly, a dramatic increase of training and testing speed is in the area to the right of the second curve.
To the right of the third curve the number of support vectors and the correct classification percentage do not change with further increase of C parameter. The best points can lie on the horizontal ray starting from curve 3 or at the point of the distinct area between the second and third curves After obtaining the best result for SVM testing (the least number of errors of the first and second kinds (FN and FP) and the least number of support vectors in SVM model) the automatic adjustment block changes the parameters in dimension reduction block and repeats the same operation cycle in SVM module.
To increase the capabilities of the programming complex the feature of principal component training for an attack set only was added. With such an approach the dimension reduction block determines internal dependencies between the parameters within the framework of attack packages rather than those in the entire training set. In the attack, set training application of new parameters with the least eigenvalues of the weight matrix is similar to the attack signature.
We noted that such an approach depends considerably on the training set, which can result in an overtraining problem and considerable number of false responses. This approach proves highly efficient for certain attacks (not exceeding 5 now parameters and less than 20 support vectors for 100 % recognition), however, it entirely inapplicable for others, which makes this approach only an additional means in the designed programming complex. Automatic adjustment in the dimension reduction block boils down to the selection of the matrix to be analyzed, a set of vectors to be considered (all packages or only attack ones) and the selection of two threshold values of 5 and As a result of the automatic adjustment, Mock work the system best, parameters are found:
• The least FP and FN values determine the attack detection quality.
• The least number of basic parameters, new parameters and support vectors of SVM-model determine the system's performance speed.
The training procedure yields a SVM model that is used to classify the vectors.
To test the SVM model, the following characteristics are evaluated: the number of correctly detected attacks (TP), the number of false positives (FP), the number of correctly classified normal packets (TN), the number of missed attacks (FN), and the percentage of correctly classified vectors.
Cluster analysis and fuzzy logic for intrusion detection module manage
For subtasks forming composition of detection modules different hierarchical and iterative cluster analysis methods were analyzed. The main drawback of hierarchical techniques is the inability to handle large amounts of data [19, 20], and for iterative methods - the need for a priori knowledge of the number of clusters [21]. The technique of cluster analysis, which allows eliminating both of these shortcomings as part of the problem is solved.
To determine the number of clusters and their centers to the set of attack vectors applying agglomerative hierarchical method. Then, using the method of k-means clusters are formed for the entire training sample. Experiments have shown that for different training samples clustering the best results are achieved by using the following distance metrics: Euclidean distance, squared Euclidean distance and Manhattan distance.
The use of fuzzy logic has improved the performance of the intrusion detection system due to generalization of methods of support vector machine and k-means (C-means) for the application of probabilistic assessment when the classification and construction of overlapping clusters. Also, fuzzy logic allows correlating the signals detection modules with the known attacks: while cluster analysis is formed by a matrix belonging to a variety of attacks:
Amxt ~
where, the t - number of known attacks, m - number of detection modules, c.. - the number of vectors attack j in the training set for module i.
Probability that the detected signal belongs to a specific attack, taking into account the previous signals from the same source:
Pi = an + Px-1(1 - au ^
where, the l - number of the module, issuing a signal of an attack, a - element of matrix A • x - the serial number of the current signal of attack from a given source.
Approach results
Experimental research were carried out with the IDS prototype on attacks categories User-to-Root and Remote-to-Local, which are the most difficult to detect. Experimental research was divided into five stages: performing classification by support vector machine for multi-bit traffic parameters and divided into eight-bit of the traffic parameters, the formation of a new parameter space by principal component analysis, the formation of the detection modules using the method of k-means and synthesis methods of support vector machine and k-means using fuzzy logic. The results are shown in Tab. 3.
Tab. 4 shows the results of other researches of using Data Mining methods for intrusion detection [4-10].
Table 3
The results of experimental research stages
Stage description TP, % FP, %
SVM (multi-bit) 85 5
SVM 91 2
SVM + PCA 94 3
SVM + PCA + k-means 98 1
SVM + PCA + k-means + fuzzy logic 99 0,6
The results of other researches Table 4
Using method TP, % FP, %
Quarter-sphere SVM 65 1
SVM 95,5 1
SVM + Genetic algorithm 99 -
SVM + fuzzy logic 99,56 0,44
C4.5 95 1
C4.5 + PCA 92,16 -
C4.5 + Artificial neural network 93,28 0,2
k-means clustering 65 1
Single leakage clustering 69 1
Y-means clustering 89,89 1
k-nearest neighbor 92 1
Artificial neural network + PCA 92,22 -
Genetic algorithm 97,47 0,69
Conclusion
The experiments performed with the program prototype show robustness of intrusion detection system and applicability of the selected data mining techniques for the specified purpose. The support vector machine allows identifying a considerable part of attacks under investigation with 100 % confidence, with an error for the rest not exceeding several percent of the total packages number. The principal component analysis allows reducing the information volume required for network packages classification for 2,5-3 times and considerably increases the system efficiency.
The method of k-means allows identifying fragments of typical attacks in separate detection modules and breaking complex attacks on several modules, resulting in improved detection performance by 3-5 %. The use of fuzzy logic has increased detection performance by 1 %, allowing classifying the vector having different labels in the training set. in modular architecture of the intrusion detection system while designing several classifiers of the same packages type, the dimension reduction methods allow to construct pretty simple SVM models that identify intrusions much faster and more accurately than a single general classifier with a great number of parameters and support vectors in a SVM model.
The application of various methods, possibility to adjust internal settings, threshold values enables one to obtain the best possible correlation between the system efficiency and intrusion detection accuracy.
References
1. Tan P. N., Steinbach M., Kumar V. Introduction to Data Mining, Addison-Wesley, 2005, 769 p.
2. Sequeira K., Zaki M. ADMIT: Anomaly-based data mining for intrusions, Proc. of the Eighth ACMSIGKDD Int'nl Conf. on Knowledge Discovery and Data Mining, NY, USA, 2002, ACM, 2002, pp. 386-395.
3. The Common Intrusion Detection Framework (CIDF). Available at: http://gost.isi.edu/cidf (accessed 03.12.2016).
4. Bhattacharyya D. K., Kalita J. K. Network Anomaly Detection. A Machine Learning Perspective, CRC Press, 2014, 364 p.
5. Portnoy L., Eskin E., Stolfo S. J. Intrusion detection with unlabeled data using clustering. Proc. of ACM Workshop on Data Mining Applied to Security, 2001, pp. 1-14.
6. Mabu S., Chen C., Lu N., Shimada K., Hirasawa K. An intrusion-detection model based on fuzzy class association-rule mining using genetic network programming. IEEE Trans. Syst. Man Cybern. Part CAppl. Rev, 2011, vol. 41, no. 1, pp. 130-139.
7. Bankovic Z., Stepanovich D., Bojanic S., Nieto-Ta-ladris O. Improving network security using genetic algorithm approach. Comput. Electr. Eng., 2007, vol. 33, no. 5-6, pp. 438-451.
8. Ghahramani Z. An Introduction to hidden Markov models and Bayesian networks. Int. J. Pattern Recognit Artif Intell., 2001, vol. 15, pp. 9-42.
9. Lee S. C., Heinbuch D. V. Training a neural-network based intrusion detector to recognize novel attacks. IEEE Trans. Syst. Man Cybern. Part A Syst. Humans, 2001, vol. 31, no. 4, pp. 294-299.
10. Tajbakhsh A., Rahmati M., Mirzaei A. Intrusion detection using fuzzy association rules. Appl. Soft Comput., 2009, vol. 9, no. 2, pp. 462-469.
11. Hsu C.-W., Chang C.-C., Lin C.-J. A Practical Guide to Support Vector Classification. Dep. Comput. Sci., Nat. Taiwan Univ., Taipei 106, Taiwan, 2003, 16 p.
12. Aivazyan, S.A., Bukhshtaber, V. M., Enyukov, I. S., Me-shalkin, L. D., Prikladnaya statistika: Klassifikatsiya i snizhenie razmernosti: Spravochnoe izdanie [Applied Statistics: Classification and Dimension Reduction: Reference Guide], Moscow: Finansy i Statistika, 1989.
13. Guha S., Rastogi R., Shim K. Cure: An efficient clustering algorithm for large databases. SIGMOD, 1998, vol. 27, ACM, pp. 73-84.
14. DARPA Intrusion Detection Data Sets. Available at: https://www.ll.mit.edu/ideval/data/(accessed 03.12.2016).
15. Fodor I. K. A Survey of Dimension Reduction Techniques. U. S. Dep. of Energy by Univ. of California, Lawrence Livennore Nat. Lab., 2002, 26 p.
16. Kayacik H. G., Zincir-Heywood A. N., Heywood M. I. Selecting Features for Intrusion Detection: A Feature Relevance Analysis on KDD 99 Intrusion Detection Datasets. Proc. Third Annual Conf. on Privacy, Security and Trust (PST-2005), 2006, pp. 85-89.
17. Miguel A. Carreira-Perpinan A Review of Dimension Reduction Techniques. Technical Report CS-96-09 Dept. of Comput. Sci. Univ. of Sheffield, 1997, 69 p.
18. Vapnik V. N. The Nature of Statistical Learning Theory. Springer, 2000, 314 p.
19. Zhambyu M. Ierarkhicheskiy klaster-analiz i sootvetst-viya [Hierarchical Cluster analisys and compliance]. Moscow: Finansy i Statistika, 1988, 345 p.
20. Karypsis G., Han H., Kumar V. Chameleon: A hierarchical clustering algorithm using dynamic modeling. IEEE Comput., 1999, vol. 32, no. 8, pp. 68-75.
21. Mandel I. D. Klasterniy analiz [Cluster analysis]. Moscow: Finansy i Statistika, 1988, 176 p.
Обнаружение сетевых атак в компьютерных сетях с помощью методов интеллектуального анализа данных
Платонов В. В., Семенов П. О. ФГАОУ ВО «Санкт-Петербургский политехнический университет Петра Великого»
Санкт-Петербург, Россия [email protected], [email protected]
Аннотация. Статья посвящена описанию подхода к построению системы обнаружения сетевых атак в распределённых вычислительных сетях. Показано, что применение комплекса методов и средств интеллектуального анализа данных позволяет повысить эффективность защиты распределённых вычислительных сетей от сетевых атак за счёт сочетания преимуществ поиска сигнатур и аномалий и возможности адаптации системы под программно-аппаратную структуру сети.
Ключевые слова: распределённая вычислительная сеть, система обнаружения сетевых атак, методы интеллектуального анализа данных, метод опорных векторов, метод главных компонент.
Литература
1. Tan P. N. Introduction to Data Mining / P. N. Tan, M. Steinbach, V. Kumar. - Addison-Wesley, 2005. - 769 p.
2. Sequeira K. ADMIT: Anomaly-based data mining for intrusions / K. Sequeira, M. Zaki // Proc. of the Eighth ACM SIG-KDD Int'nl Conf. on Knowledge Discovery and Data Mining, NY, USA, 2002. NY: ACM, 2002. - P. 386-395.
3. The Common Intrusion Detection Framework (CIDF) -URL: http://gost.isi.edu/cidf/ (дата обращения 03.12.2016).
4. Bhattacharyya D. K. Network Anomaly Detection. A Machine Learning Perspective / D. K. Bhattacharyya, J. K. Kalita. -CRC Press, 2014. - 364 p.
5. Portnoy L. Intrusion detection with unlabeled data using clustering / L. Portnoy, E. Eskin, S. J. Stolfo // Proc. of ACM Workshop on Data Mining Applied to Security. - 2001. - P. 1-14.
6. Mabu S. An intrusion-detection model based on fuzzy class association-rule mining using genetic network programming / S. Mabu, C. Chen, N. Lu, K. Shimada, K. Hirasa-wa // IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. - 2011. Vol. 41, № 1. - P. 130-139.
7. Bankovic Z. Improving network security using genetic algorithm approach / Z. Bankovic, D. Stepanovich, S. Bojan-ic, O. Nieto-Taladris // Comput. Electr. Eng. - 2007. - Vol. 33, № 5-6. - P. 438-451.
8. Ghahramani Z. An Introduction to hidden Markov models and Bayesian networks / Z. Ghahramani // Int. J. Pattern Recog-nit Artif Intell. - 2001. - Vol. 15. - P. 9-42.
9. Lee S. C. Training a neural-network based intrusion detector to recognize novel attacks / S. C. Lee, D. V. Heinbuch // IEEE Trans. Syst. Man Cybern. Part A Syst. Humans. - 2001. - Vol. 31, № 4. - P. 294-299.
10. Tajbakhsh, A. Intrusion detection using fuzzy association rules / A. Tajbakhsh, M. Rahmati, A. Mirzaei // Appl. Soft Comput. - 2009. - Vol. 9, № 2. - P. 462-469.
11. Hsu C.-W. A Practical Guide to Support Vector Classification / C.-W. Hsu, C.-C. Chang, C.-J. Lin; Dep. Comput. Sci., Nat. Taiwan Univ., Taipei 106. - Taiwan, 2003. - 16 p.
12. Айвазян С. А. Прикладная статистика: Классификация и снижение размерности: справ. изд. / С. А. Айвазян,
B. М. Бухштабер, И. С. Енюков, Л. Д. Мешалкин; под ред.
C. А. Айвазяна. - М.: Финансы и статистика, 1989. - 607 с.
13. Guha S. Cure: An efficient clustering algorithm for large databases / S. Guha, R. Rastogi, K. Shim // SIGMOD. - 1998. -Vol. 27, ACM. - P. 73-84.
14. DARPA Intrusion Detection Data Sets - URL: https://www. ll.mit.edu/ideval/data/ (дата обращения: 20.05.2016).
15. Fodor I. K. A Survey of Dimension Reduction Techniques / I. K. Fodor; U. S. Dep. of Energy by Univ. of California, Lawrence Livennore Nat. Lab., 2002. - 26 p.
16. Kayacik H. G. Selecting Features for Intrusion Detection: A Feature Relevance Analysis on KDD 99 Intrusion Detection Datasets / H. G. Kayacik, A. N. Zincir-Heywood, M. I. Heywood // Proc. Third Annual Conf. on Privacy, Security and Trust (PST-2005). - 2006. - P. 85-89.
17. Miguel A. Carreira-Perpinan A Review of Dimension Reduction Techniques / A. Carreira-Perpinan Miguel. - Technical Report CS-96-09 Dept. of Comput. Sci. Univ. of Sheffield, 1997. - 69 p.
18. Вапник В. Н. The Nature of Statistical Learning Theory / В. Н. Вапник. - 2-е изд. - Springer, 2000. - 314 с.
19. Жамбю М. Иерархический кластер-анализ и соответствия. - М.: Финансы и статистика, 1988. - 345 с.
20. Karypsis G. Chameleon: A hierarchical clustering algorithm using dynamic modeling / G. Karypsis, H. Han, V. Kumar // IEEE Comput. - 1999. - Vol. 32, № 8. - P. 68-75.
21. Мандель И. Д. Кластерный анализ / И. Д. Мандель. -М.: Финансы и статистика, 1988. - 176 с.
