Научная статья на тему 'Evaluation methods of image segmentation quality'

Evaluation methods of image segmentation quality Текст научной статьи по специальности «Компьютерные и информационные науки»

CC BY
480
67
i Надоели баннеры? Вы всегда можете отключить рекламу.
Ключевые слова
segmentation / quantitative segmentation evaluation / Fr chet metric / Hausdorff metric / Gromov-Hausdorff metric / GromovFr chet metric / polygon / cytological images / сегментація / кількісна оцінка сегментації / метрика Фреше / метрика Хаусдорфа / метрика Громова-Хаусдорфа / метрика Громова-Фреше / полігон / цитологічні зображення

Аннотация научной статьи по компьютерным и информационным наукам, автор научной работы — Berezsky O. M., Pitsun O. Y.

Context. The basic methods of quantitative evaluation of image segmentation quality are explored. They are used to select segmentation algorithms for specific image classes. The object of the study is cytological and histological images that are used in diagnosing the pathological processes in oncology. The subject of the study is quantitative methods for segmentation algorithms’ quality evaluation. Objective. The purpose of the work is to introduce the Gromov-Fr chet metric and develop a metric-based method for quantitative evaluation of segmentation quality for image segmentation algorithms’ comparison. Method. The quantitative evaluation criteria, which are based on comparison with etalon image and without the comparison with etalon image, are analyzed. The algorithms for measuring the distances between images based on the Fr chet, Hausdorff, and Gromov-Hausdorff metrics are analyzed. To calculate the distance between the contours of images, the Gromov-Fr chet distance was introduced. The condition of identity, symmetry and triangle is proved, and it is shown that the Gromov-Fr chet distance is a metric. The metric-based method of quantitative evaluation of segmentation quality is developed. It is based on the use of the Gromov-Hausdorff and Gromov-Fr chet metrics. The method is based on the algorithms for non-convex-into-convex polygon transformation, weighted chord algorithm, and algorithms for calculating the Fr chet and Hausdorff distances. To calculate the Hausdorff distance between convex regions, the Atalah’s algorithm was used. The Thierry and Manillo algorithm was used to find the discrete Fr chet distance. These algorithms have the lowest computational complexity among their class of algorithms. Results. The Gromov-Fr chet metric was introduced and the metric-based method of quantitative evaluation of segmentation quality was developed. Conclusions. The conducted experiments on the basis of cytological images confirmed the performance of software for evaluation the distances between images. The developed method showed a high accuracy of estimation the distances between images. The developed software module was used in intelligence systems for diagnosing the breast precancerous and cancerous conditions. The software can be used in various software systems of computer vision. Promising areas for further research are search for new metrics to evaluate the distances between images.

i Надоели баннеры? Вы всегда можете отключить рекламу.
iНе можете найти то, что вам нужно? Попробуйте сервис подбора литературы.
i Надоели баннеры? Вы всегда можете отключить рекламу.

МЕТОДИ ОЦІНКИ ЯКОСТІ СЕГМЕНТАЦІЇ ЗОБРАЖЕНЬ

Актуальність. Розглянуто основні методи кількісної оцінки якості сегментації зображень. Вони використовуються для вибору алгоритмів сегментації для конкретних класів зображень. Об’єктом дослідження є цитологічні та гістологічні зображення, які використовуються при діагностуванні патологічних процесів в онкології. Предметом дослідження є кількісні методи оцінки якості алгоритмів сегментації. Мета роботи – введення метрики Громова-Фреше та розроблення метричного методу кількісної оцінки якості сегментації для порівняння алгоритмів сегментації зображень. Метод. Проаналізовані критерії кількісної оцінки, які базуються на порівнянні та без порівняння з еталонним зображенням. Проаналізовано алгоритми знаходження відстані між зображеннями на основі метрик Фреше, Хаусдорфа, Громова-Хаусдорфа. Для знаходження відстані між контурами зображень введено відстань Громова-Фреше. Доведено умову тотожності, симетричності та трикутника і показано, що відстань Громова-Фреше є метрикою. Розроблено метричний метод кількісної оцінки якості сегментації. Він базується на використанні метрик Громова-Хаусдорфа та Громова-Фреше. Метод побудований на основі алгоритмів перетворення неопуклих полігонів у опуклі, алгоритму зважених хорд, алгоритмів обчислення відстаней Фреше та Хаусдорфа. Для знаходження відстані Хаусдорфа між опуклими областями використано алгоритм M. Аталаха. Для знаходження дискретної відстані Фреше використано алгоритм Томаса Ейтера і Хейкі Манілли. Дані алгоритми мають найнижчу обчислювальну складність серед свого класу алгоритмів. Результати. Введено метрику Громова-Фреше та розроблено метричний метод кількісної оцінки якості сегментації. Висновки. Проведені експерименти на основі цитологічних зображень підтвердили працездатність програмного забезпечення оцінки відстані між зображеннями. Розроблений метод показав високу точність оцінки відстаней між зображеннями. Розроблений програмний модуль використаний у інтелектуальній системі діагностування передракових і ракових станів молочної залози. Програмне забезпечення може бути використане в різних програмних системах комп’ютерного зору. Перспективними напрямами досліджень є пошук нових метрик для оцінки відстані між зображеннями.

Текст научной работы на тему «Evaluation methods of image segmentation quality»

UDC 004.93

Berezsky O. M.1, Pitsun O. Y.2

Professor, Head of Computer engineering department, Ternopil National Economic University, Ternopil, Ukraine

2Post-graduate student, Ternopil National Economic University, Ternopil, Ukraine

EVALUATION METHODS OF IMAGE SEGMENTATION QUALITY

Context. The basic methods of quantitative evaluation of image segmentation quality are explored. They are used to select segmentation algorithms for specific image classes. The object of the study is cytological and histological images that are used in diagnosing the pathological processes in oncology. The subject of the study is quantitative methods for segmentation algorithms' quality evaluation.

Objective. The purpose of the work is to introduce the Gromov-Fr echet metric and develop a metric-based method for quantitative evaluation of segmentation quality for image segmentation algorithms' comparison.

Method. The quantitative evaluation criteria, which are based on comparison with etalon image and without the comparison with etalon image, are analyzed. The algorithms for measuring the distances between images based on the Fr; chet, Hausdorff, and Gromov-Hausdorff metrics are analyzed.

To calculate the distance between the contours of images, the Gromov-Frechet distance was introduced. The condition of identity, symmetry and triangle is proved, and it is shown that the Gromov-Fr echet distance is a metric.

The metric-based method of quantitative evaluation of segmentation quality is developed. It is based on the use of the Gromov-Hausdorff and Gromov-Fr echet metrics. The method is based on the algorithms for non-convex-into-convex polygon transformation, weighted chord algorithm, and algorithms for calculating the Frechet and Hausdorff distances. To calculate the Hausdorff distance between convex regions, the Atalah's algorithm was used. The Thierry and Manillo algorithm was used to find the discrete Frechet distance. These algorithms have the lowest computational complexity among their class of algorithms.

Results. The Gromov-Fr echet metric was introduced and the metric-based method of quantitative evaluation of segmentation quality was developed.

Conclusions. The conducted experiments on the basis of cytological images confirmed the performance of software for evaluation the distances between images. The developed method showed a high accuracy of estimation the distances between images. The developed software module was used in intelligence systems for diagnosing the breast precancerous and cancerous conditions. The software can be used in various software systems of computer vision. Promising areas for further research are search for new metrics to evaluate the distances between images.

Keywords: segmentation, quantitative segmentation evaluation, Frechet metric, Hausdorff metric, Gromov-Hausdorff metric, Gromov-Fr chet metric, polygon, cytological images.

NOMENCLATURE

AMS - automated microscopy system;

AR - area relation;

FOM - figure of merit;

NR FOM - number relation figure of merit;

AUMA - absolute ultimate measurement accuracy;

RUMA - relative ultimate measurement accuracy;

FOC - figure of certainty;

FRAG- fragmentation;

CSP - correctly selected pixels;

PDE - pixel distance error;

I - a predefined image;

Ie - an expert (etalon) image;

I A j - an image, segmented by algorithm Aj;

Aj - a segmentation algorithm;

d(Ie, IA ) = dj - a distance between etalon image Ie

and segmented image IAi by algorithm Aj;

d (x, y) - an Euclidean distance between points x andy;

dft - a Hausdorff distance;

d2V2W2) - a distance between contours;

dp - a Frnchet distance;

cIqh - a Gromov-Hausdorff distance;

dQp - a Gromov-Fr é chet distance;

A, B - compact sets;

© Berezsky O. M., Pitsun O. Y., 2018 DOI 10.15588/1607-3274-2018-1-14

f, g - continuous curves;

a, P - arbitrary continuous non-decreasing function; (X, d) - metric space X with metric d ; Yi - parameterized curves in metric spaces,

f, g - isometric embeddings into a space; Z - a set of isometric transformations;

P, Q, Pi, Qj - polygons;

Cp - a contour (external boundary) of convex polygon

Pi; i

Op. - an internal boundary of convex polygon Pi;

di(Fi,Wi) - a distance between regions; D - a combined distance; 9, X - weighted coefficients; V - initial set of polygon vertices; n - a number of vertices in polygon; m - a number of vertices corresponding to condition

a. < 180°;

B - a set of polygon vertices after convex regions formation;

a. - interior angles of the polygon; O - algorithm computational complexity. INTRODUCTION

Image processing and analysis have been widely used in computed tomography, magnetic resonance tomography,

X-ray imaging (digital radiography), histology and cytology, etc. [1]. For diagnostics in oncology, automated microscopic system (AMS) are used to process and analyze cytological and histological images.

The object of this research is cytological and histological image segmentation.

Cytological image is a microscopic image of preparations containing cells and their components (nucleus, cytoplasm) [2]. Histological image is a microscopic image of preparation thin sections of fixed tissues that reflect their structure [3].

The main disadvantages of most histological and cytological images are low quality, non-uniform illumination of areas, presence of noise, lack of clear contours between the microscopic objects and the background. It is known that image segmentation is crucial for the average computer vision. There are many algorithms for image segmentation, such as threshold, watershed distribution, A-means, and others. Therefore, the choice of segmentation algorithms and their parameters is of great significance. To select the optimal parameters for segmentation algorithms, it is necessary to evaluate the results of segmentation.

The subject of the research is methods of segmentation quality evaluation.

There are the following segmentation evaluation criteria: non-standard segmentation criteria that do not require standard (etalon) segmentation and standard (etalon-based) segmentation criteria.

One of the main criteria of the first group is homogeneity of segments. This approach is based on calculation of value dispersion of a certain image feature used for segmentation [4]. Another criterion for evaluating segmentation is contrast between segments. Besides, a shape of a segment is also a criterion that can help evaluate segmentation quality. This criterion can be effectively used for a specific image analysis. The advantage of using non-standard criteria is simplicity and speed.

The most famous algorithms belonging to the second group are AR, FOM, NR FOM, AUMA, RUMA, FOC, Baddeley, Average Distance, Variance distance, FRAG [5]. This group of algorithms is based on the use of metrics [6]. The results of etalon-based segmentation are compared with the results of specific segmentation algorithm. The main advantage of the second group of algorithms is high accuracy of evaluation.

There is no single unified theory of image segmentation. Most algorithms are heuristic in nature. Therefore, the actual problem is objective quantitative evaluation of segmentation quality.

The purpose of this article is to analyze and compare the current evaluation methods of image segmentation, develop a metric-based method and algorithms for quantitative evaluation of image segmentation using histological and cytological images.

1 PROBLEM STATEMENT

Let I be a predefined image (fig. 1). Expert (etalon) image Ie is obtained after expert's segmentation. Let image IA be an image segmented by algorithm Aj. Then the distance

Figure 1 - Expert image and image segmented by algorithms

between an etalon image and an image segmented by algorithms Aj will be equal to d(Ie,IAj), i = 1,..., l.

It necessary to find out the algorithm A j that meets the requirement of distance minimum dj = min(di,d2,...,di) .

It is necessary to find out the algorithm. 2 LITERATURE REVIEW

Segmentation quality evaluation has become a research focus of many researchers since the late 80s - early 90s. Zhang, Mattana, and Huo [5] stated that it is accuracy of individual objects' selection that can serve as image segmentation evaluation. The authors considered individual characteristics of objects. To evaluate the threshold segmentation methods, Lee, Chung, and Park introduced segmentation error probability criterion based on counting wrongly classified pixels. Yasnoff and Mui [7] introduced pixel distance error (PDE) to find out the distances between pixels in the target and segmented images. The longer the distance is, the higher the segmentation error will be. Gerbrands [8] introduced FOM criterion to determine the distance between the segmented pixel and "correct" pixel location. Criterion RUMA, offered by Zhang [9], uses geometric object parameters to evaluate segmentation quality.

The analyzed criteria mainly evaluate the distances between individual objects in etalon and segmented images. In practice, it is necessary to evaluate the quality of segmentation for a group of objects.

The algorithms for comparing segmentation results by means of metrics are based on known Fmchet and Hasdorf metrics.

Thus, Lopez and Reisner [10] developed an algorithm to reduce a number of vertices of the convex polygon for a given error E in the Hausdorff metric. The algorithm can be used only for convex polygons. Alt and Scharfz [11] calculated the Hausdorff distance between algebraic plane curves using Voronoi diagrams. The algorithm is used for partial cases with algebraic curves and has a high computational complexity. Chew and Kedem [12] developed an algorithm for finding the minimum Hausdorff distance in metrics Lj and Lm. The resulting computational complexity

is O(n2 • log2 n). Knauer and Scherfenberg [13] developed a search method by a given image pattern, which has the least distance in the metric of Hausdorff. In this case, translation of the specified pattern is used to the search image. The algorithm has a high computational complexity. Alvarez and Seidel [14] developed a method for finding the minimum weight spanning tree based on the Hausdorff metric for d -dimensional space. The problem of approximation of such a tree is solved in polynomial time. Atallah [15] developed an algorithm for finding the Hausdorff distance between convex polygons. The computational complexity

of this algorithm is O(m • n) , where m and n refer to a number of vertices of the first and second polygons, respectively.

A number of publications are devoted to the development of algorithms for finding the Frnchet distance between curves. Alt and Godau [16] developed algorithms for finding the distance between parametric curves. The computational complexity is O(mn • log mn) . Mosig and Clausen [17] developed an algorithm for calculating the Frnchet discrete distance for polygonal curves. They used groups of rigid motion. The computational complexity is

O(m2 • n2), where m and n refer to a number of fragments on the first and second curves. K. Buchin, M. Buchin, and Wenk [ 18] developed an algorithm for calculating the Frnchet distance for surfaces that are represented by simple polygons. The algorithm has a polynomial complexity. Rote [19] developed the algorithm for calculating the Frnchet distance between two curves given by the set of m and linearly approximated segments. The computational

complexity is O(m • n) . Schlesinger, Vodolazskiy, and Yakovenko [20] obtained the computational complexity O(m • n) for the Frnchet metric for closed polygonal curves. Ahn et al [21] developed an algorithm for calculating the Frnchet distance with inaccurately given vertices. For d -dimensional space they obtained the computational complexity of O(d • m • n). Cook, Driemel, Sherette, and Wenk [22] developed an algorithm for calculating the Frnchet distance between non-flat surfaces. The authors reached the polynomial time in the Lx metric. Gudmundsson and Smid [23] developed a fast algorithm for finding the similarity of polygonal trees in the Frnchet metric. The algorithm has a polynomial complexity.

Eiter and Mannila [24] developed an algorithm for finding the Frnchet discrete distance between polygons. The computational complexity of the algorithm is O(m • n). The best algorithms for finding the Frnchet distance for plane closed curves have the computational complexity of O(m • n) . The best algorithms for finding the Hausdorff distance for convex regions have computational complexity O(m • n).

The mentioned algorithms calculate the distances separately between the curves (contours) of the images in the Frechet metric and between the regions of the images in the Hausdorff metric.

Therefore, it is necessary to develop a metric, method, and software for evaluating segmentation quality for complex images with many objects.

3 MATERIALS AND METHOD

In order to compute distances between images, we use Hausdorff and Frnchet metrics. To compute the shortest distances between images, we use Gromov-Hausdorff and Gromov-Fr e chet metrics.

We present the basic metrics below.

Frnchet metric. Let X be a metric space with a metric d . For two curves f : [a, b] ^ X and g : [a', b'] ^ X, the Frnchet distance between them is equal to [10]:

dF = inf sup d(f (a(i)),g(P(t)))

a,p te[0,1] ,

where a and P are arbitrary non-decreasing continuous functions from the interval [0,1] on intervals [a, b] and [a', b' ], respectively. The value of function is a(0) = 0 and a (1) = 1, and it is similar for function p.

Hausdorff metric. For a metric space (X, d) , the Hausdorff metric dh will be called a metric on the set 5 of

all non-empty compact subsets X, which is defined by means of the formula [25]:

dH(A,B) := max<jmaxmind(x,y),maxmind(x,y

[ xeA yeB yeB xeA J

Gromov-Hausdorff metric. The distance between two compact sets A and B is equal to [26]:

dGH (A,B):= inf dH (f (A), g(B)),

X, f ,g

where f : A ^ X , g : B ^ X denote isometric embeddings into a metric space ( X , d) .

Gromov-Frechet metric. To measure the distance between two curves, we use Gromov-Fr e chet metric:

dGP (S, Q ):= inf dF (f (S), g (Q)),

X ,S,Q

where f : S ^ X, g : Q ^ X - isometric embeddings into a metric space (X, d) . Suppose there are two parameterized curves in metric spaces y j : [0,1] ^ Xj, i = 1,2. (Xj, dj) is a metric, t ^Jj (t) e Xj (fig. 2). We embed isometrically jj : X1 ^ Z , j = 1,2, (Z, d).

Then we take a Fre chet distance between jjy1 and j^y 2,

dF(jlYl, j2Y2) . (1)

Finally, infimum of such (1) along all isometric embeddings j , j will be a Gromov-Fr echet distance:

dGF (Y^ Y2) = inf dF CAY^ j2Y2 ) .

iНе можете найти то, что вам нужно? Попробуйте сервис подбора литературы.

j^ j2,Z

Figure 2 - Gromov-Fr échet metric between two parameterized curves

Theorem. The function Cqf is a metric. Proof. First we check if the definition is correct. Having two spaces X1 and X2 we consider their bouquet

Z = X1 v X2, sticking together two arbitrary points x° and X20.

Let us define a metric d in a bouquet by means of the formula:

d ( x, y ) =

dj (x, y), if x,y e Xj.

dj(x, x0 ) + d2(x0,y), if x e Xj,y e X2.

Then we check the definition of a metric for dQF .

Obviously, dQp(yi, Y2) - 0 .

Besides, dQp (y, y) = 0 for each Y.

Let us denote that the definition implies that

dQF(Y1, Y2) - dQF(Yi([0,1]),Y2([0,1])).

If dQp (Y1, Y 2) = 0, then it implies that dQF (Y1 ([0,1]), Y 2([0,1])) = 0, therefore, we can assume that

Y1 ([0,1]) = Y 2 ([0,1]).

The last equality in case of embedded curves gives equality Y1 = Y2 exactly to parameterization. The symmetry

of function dQF is obvious.

Let us verify the triangle inequality. Let yi : [0,1] ^ X.,

i = 1,2,3 be parameterized curves; dQp(Y1, Y2) = a ,

dQF (Y 2, Y3) = b; and g> 0. There are metric spaces Z12

and Z23, and such isometric embeddings

j) :Xt ^Z12, i = 1,2,

so that

dF(hYb j2Y2) < a + g ,

dF(k2Y2,k3Y3) < b + g .

Let Z = Z12 -1 Z23 . In other words, Z is derived

k2j2

from Z12 ^ Z23 by identification of each point j2( x) with point (x) , x e X2.

Metric d on Z is defined by the formula:

|d'(x, y), if x,y e Z12. d"( x, y), if x,y e Z23.

inf{d'(x, j2 (z)) + d'(k2 (z), y)| z e X2>,if x e Z12, y e Z23.

(Here d is a metric on Z12 , d is a metric on Z 23 ). Then from the triangle inequality for d F it derives that

dQF (Y1, Y 2) ^ dF ( j1 y 1, k 3 y 3) < d

F ( j1Y1 , j 2 Y 2 ) +

+ d f (k 2 Y 2, k 3 Y 3) < a + b + 2 g = = dQF (Yl, Y 2 ) + dQF (Y 2 , Y3 ) + 2g

Since g > 0 is arbitrary, we obtain the required inequality. To evaluate segmentation quality, the researchers developed quantitative evaluation method of segmentation quality (QEMSQ), which is built on metric-based measuring the distance between images.

After segmentation, we obtain a set of segments that we approximate linearly and get a set of polygons. In general, they are not convex. Thus, the task is to compare two non-convex polygons after a specific algorithm segmentation and expert segmentation.

Let P and Q be two non-convex polygons (fig. 3).

k, : Xt ^ Z23, i = 2,3

Figure 3 - Polygons P and Q Let us divide polygons P and Q into sets of convex polygons, that is, P = P1 u...u Pi u... u Pn , Q = Q1 u ..u Qj u .. u Qm . Each convex polygon Pi and Qj is represented as P = Cp u Op. , where Cp and Op are contour (external boundary) and internal region of convex polygon p, respectively. Similarly, polygon

Q: Qj = CQj u OQj.

Then we receive P = (CP1 uOP1)u(Cp uOp)u...u(CPn uOPn)}| V,

Q = (CQ1 u OQQ1) u (CQjt u OQjt ) u ... u (CQm u OQm )} | W.

Let us represent the latter expressions in the following form:

P = (Cp1 u...uCp u...uCp )u(Op1 u...uOp u...uOp ), Q = (CQ1 u...uCQj u...uCQm)u(OQ1 u...uOQj u...uOQm).

Denote:

Cp u...uCp u...uCp = V1, Op u...uOp u...uOp = V2,

C2i u ... u CQj u ... u CQm = W1,

Oq u...uOQ u...uOQ = W2.

JQ

Qm

Then the distance between polygons p and Q is equal to a sum of distances between contours and internal regions

of convex polygons pi and Qj . The distance between regions equals:

d1(V1,W1) = inf {g1 > 0| Vi = 1~n, 3j = 1m, dH (O,, O}) < g1 and vice versa Vj = 1, m, 3i = 1, n, and dH (Oi,Oj ) < g1), dH - a Hausdorff distance.

Similarly, we calculate the distances between contours:

d 2(V2,W2) = inf {g 2 > 0| Vi = , 3j = dF (C,, C} ) < g 2

and vice versa Vj = 1, m, 3i = 1,n, and dF (Ci, Cj) < g2

dF - a Fr chet distance.

Quantitative evaluation method of segmentation quality is based on a combination of algorithms that ensure finding the shortest distances between images. This combination includes a set of algorithms: algorithm for non-convex-into-convex polygon transformation, weighted chords algorithm, Hausdorff distance algorithm, and discrete Frechet distance algorithm.

Stepwise quantitative evaluation method of segmentation quality can be represented as follows:

1. Formation of a convex polygons' set.

2. Conducting isometric transformations for embedding convex polygons with a maximum cross section.

3. Computing a Fre chet distance for convex polygons.

4. Computing a Hausdorff distance for convex polygons.

5. Finding the shortest distance based on weighted metrics (Frechet and Hausdorff metrics) between polygons

P and Q according to D = <pdH + Xdp.

Let us describe the basic algorithms that underpin this method.

Algorithm for non-convex-into-convex polygon transformation:

Let p be a non-convex polygon with vertices

V = {v0,V1,...,vn—1}. Algorithm for convex polygon

formation consists of the following steps:

1. Beginning with an upper vertex, we select the vertices with internal angles aj larger than 180°. If such angles do not exist, then the polygon is convex and the algorithm is completed. Otherwise, we get an array of vertices

B = ^bb h^.^ bm}.

2. We connect consistently the received vertices b1 beginning with the top vertex and get a polygon p1 .

3. We repeat step 1 with a received polygon p1 .

4. We repeat steps 1 and 2 with polygons p1 until the

angle of each vertex meets the condition a < 180°. The algorithm is graphically demonstrated in fig. 4

a b

Figure 4 - Partitioning non-convex polygon into convex polygons: a - polygon p, with internal angles larger than 180°, b - polygon i^,

convex region formation

Algorithm computational complexity is O(n + m), where n is a number of vertices of initial non-convex polygon and m is a number of vertices of non-convex polygon.

Weighted chords algorithm is described in the article of Berezsky, Melnyk, Batko, and Pitsun [27] Computational complexity of the algorithm is O(n ■ m), where n is a number of weighted chords of the first polygon and m is a number of weighted chords of the second polygon. For convex regions' comparison, the Atallah's algorithm [15] was used. For contour comparison, we used the Frechet discrete distance algorithm developed by Eiter and Mannila [25]. So, the developed QEMSQ algorithms have the least computational complexity. 4 EXPERIMENTS

For computer experiments, we used cytological images [28].

To compare segmentation quality of individual micro objects, we use etalon image shown in fig. 5.

Other images for these experiments were generated randomly. In this case, we use the following methods to evaluate segmentation quality: CSP, WSP, PDE, FRAG, AUMA, and RUMA.

To evaluate segmentation quality of micro objects' groups, we use cytological images. Fig. 6 shows the etalon image and images segmented by means of thresholding, A-means, and watershed distribution algorithms [29]. We used Hausdorff, Fre chet, Gromov-Hausdorff, and Gromov-Fr echet metrics.

Figure 5 - Etalon image

«

- A i

P

Figure 6 - Image segmentation: a - output image, b - etalon image, c - segmentation results by k-means + threshold algorithms, d segmentation results by watershed + threshold algorithms, e - segmentation results by threshold algorithm

5 RESULTS

Comparative analysis of image segmentation is demonstrated in Table 1.

Thus, the described methods of segmentation quality evaluation rated the most similar to etalon images and the most dissimilar images. These methods should be used to evaluate the segmentation quality of individual microscopic objects rather than the entire image as a whole. Table 2 shows the results of segmentation quality evaluation of images illustrated in fig. 6.

Analysis of data in Table 2 demonstrates that Gromov-Fre chet and Gromov-Hausdorff metrics showed better results than others. The combination of segmentation algorithms A-means and threshold optimally suits the segmentation of cytological images.

6 DISCUSSION

The analyzed methods of quantitative evaluation of segmentation quality, such as CSP, WSP, PDE, FRAG, AUMA, RUMA provide evaluation for only individual microscopic objects.

To evaluate the quality of segmentation of micro objects' groups, it is necessary to apply metrics. The most common metrics are the classical metrics of Fr e chet and Hausdorff. At present, the best known algorithms that implement the Frechet metric for contours (flat curves) have the computational complexity of O(m ■ n). Known algorithms for calculating the Hausdorff distance between convex regions (polygons) have the computational complexity of O(m ■ n). The Hausdorff and Frechet metrics allow separately measuring the distances between regions and contours of images. To find the shortest distances between regions, the Gromov-Hausdorff metric is used.

The disadvantages of the known algorithms are the following:

1. Algorithms based on the Fr e chet metric calculate the distance only between the contours of images;

iНе можете найти то, что вам нужно? Попробуйте сервис подбора литературы.

2. Algorithms based on the Hausdorff metric calculate the distance only between convex regions of images;

3. There are no algorithms that calculate the shortest distance between contours.

The advantages of the developed algorithms are the following:

1. The proposed Gromov-Frechet metric allows estimating the shortest distance between the contours of images;

2. The use of a combined metric based on the metrics of Gromov-Hausdorff and Gromov-Freche provided the possibility to calculate the shortest distances between contours and non-convex regions of images.

3. The developed EMISQ, which is based on the best known algorithms for calculating the Frechet and Hausdorff distances, automatically estimates the shortest distances between groups of micro objects.

CONCLUSIONS

In the article, the authors introduced the Gromov-Frechet distance and proved that distance is a metric. The method of quantitative evaluation of image segmentation quality is developed, on the basis of which a program module is designed and implemented, which allows calculating the shortest distance between images in an automatic mode.

The scientific novelty of the results is the following:

- for the first time, a Gromov-Frechet metric was proposed for measuring the shortest distance between the contours of images;

- for the first time, quantitative evaluation method of segmentation quality based on the integrated use of Gromov-Fre chet and Gromov-Hausdorff metrics was applied allowing to evaluate the shortest distances between images.

The practical significance of the results is in the development of software to evaluate the shortest distances between the images. Computer experiments that were conducted on the example of cytological and histological images showed high efficiency of software that was used in image automatic segmentation algorithms.

Table 1 - Comparative analysis of image segmentation

Methods of segmentation quality evaluation M, ii Jf Jt

CSP, % 80.96 93.52 70.5 97.84

WSP, % 53.72 47.9 58.26 44.01

FRAG, % 98 90 70 97

AUMA, -pixels 4347.0 1165.0 5113.0 1105.0

RUMA, % 15.22 4.08 17.9 3.87

Table 2 - Comparative analysis of image segmentation based on metrics

Metrics b) ^ b) b) ^ c) b) ^ d) b) ^ e)

Hausdorff 0 19.20 21.37 46.57

Gromov-Hausdorff 0 19.20 21.37 46.57

Frechet 0 36.76 26.4 52.77

Gromov-Frechet 0 36.76 24.18 52.77

Weighted metrics 0 27.98 22.78 49.67

Further areas of research embrace the development of algorithms of de-paralleling the metric quantitative evaluation method of segmentation quality, which will help speed up the process of segmentation quality evaluation and segmentation algorithm optimization. Besides, a promising area for further investigations is development of a metric for evaluation the similarities of non-convex polygons.

ACKNOWLEDGEMENTS

The proposed research has been developed within the state budget project "Hybrid Intelligent Information Technology Diagnosing of Precancerous Breast Cancer Based on Image Analysis" (state registration number 1016U002500). REFERENCES

1. Yinpeng J. Contrast Enhancement by Multi-scale Adaptive Histogram Equalization / J. Yinpeng, L.Fayadb, A. Laine // Proceedings of SPIE. - 2001. - Vol. 4478. - P. 206-213. DOI: 10.7916/D8QZ2M29

2. Baron T. H. A Prospective Comparison of Digital Image Analysis and Routine Cytology for the Identification of Malignancy in Biliary Tract Strictures / T. H. Baron, G. Harewood, A. Rumalla // Clinical Gastroenterology and Hepatology. - 2004. - Vol. 2, Issue 3. - P. 214-219. DOI: 10.1016/S1542-3565(04)00006-0

3. Petushi S. Large-scale computations on histology images reveal grade-differentiating parameters for breast cancer / S. Petushi, F. U. Garcia, M. M. Haber // BMC Medical Imaging. - 2006. -Vol. 6, Issue 14. - P. 496-499. DOI: 10.1186/1471-2342-6-14

4. Levine M. D. Dynamic measurement of computer generated image segmentations / M. D. Levine, A. Nazif// IEEE Transactions on Pattern Analysis and Machine Intelligence. - 1985. - Vol. 7, Issue 2. - P. 155-164. DOI: 10.1109/TPAMI.1985.4767640

5. Zhang Y. J. A review of recent evaluation methods for image segmentation / Y. J. Zhang // Signal Processing and its Applications (ISSPA) : Sixth International Symposium, Kuala Lumpur Malaysia, Aug 13-16, 2001 : proceedings. - Kuala Lumpur, 2001. - P. 148-151.

6. Berezsky O. Methods of quantitative evaluation of image segmentation quality / O. Berezsky // Signal/Image Processing and Pattern Recognition (UkrOBRAZ'2014) : XIIIth All-Ukraine International Conference, Kyiv, 3-7 November 2014 : proceedings. - Kyiv, 2014. - P. 51-54.

7. Lee S. U. A comparative performance study of several global thresholding techniques for segmentation / S. U. Lee, S. Y. Chung, R. H. Park // Computer Vision, Graphics, and Image Processing. -1990. - Vol. 52, Issue. 2. - P. 171-190.

8. Zhang Y. J. Segmentation evaluation using ultimate measurement accuracy / Y. J. Zhang, J. J. Gerbrands // Image Processing Algorithms and Technique. - 1992. - Vol. 1657. - P. 449-460.

9. Zhang Y. J. Objective and quantitative segmentation evaluation and comparison / Y. J. Zhang, J. J. Gerbrands // Signal Processing. -1994. - Vol. 39, Issue.2. - P. 43-54. DOI:10.1016/0165-1684(94)90122-8

10. Lopez M. Hausdorff approximation of convex polygons / M. A. Lopez, S. Reisner // Computational Geometry. - 2005. -Vol. 32, Issue 2. - P. 139-158. DOI: 10.1016/ j.comgeo.2005.02.002

11. Alt H. Computing the Hausdorff distance between curved objects / H. Alt, L. Scharfz // International Journal of Computational Geometry. - 2008. - Vol. 18. - P. 307-320. DOI: 10.1142/ S0218195908002647

12. Chew L. P. Getting around a lower bound for the minimum Hausdorff distance / L. P. Chew, K. Kedem // Computational

Geometry. - 1998. - Vol. 10, Issue 3. - P. 197-202. DOI: S0925-7721(97)00032-1

13. Knauer C. Approximate nearest neighbor search under translation invariant hausdorff distance / C. Knauer, M. Scherfenberg // International Journal of Computational Geometry. - 2011. -Vol. 21, Issue 3. - P. 369-381. DOI: S0218195911003706

14. Alvarez V. Approximating the minimum weight spanning tree of a set of points in the Hausdorff metric / V. Alvarez, R. Seidel // Computational Geometry. - 2010. - Vol. 43. - P. 94-98.

15. Atallah M. J. Computing Some Distance Functions Between Polygons / M. J. Atallah, C. Celso // Computer Science Technical Reports. - 1990. - Vol. 9. - P. 1-10.

16. Alt H. Computing the Frechet distance between two polygonal curves / H. Alt, M. Godau // International Journal of Computational Geometry and Applications. - 1995. - Vol. 5. - P. 75-91.

17. Mosig A. Approximately matching polygonal curves with respect to the Fre chet distance / A. Mosig, M. Clausen // Computational Geometry. - 2005. - Vol. 30, Issue 2. - P. 113-127. DOI: 10.1016/ j.comgeo.2004.05.004

18. Buchin K. Computing the Frechet distance between simple polygons / K. Buchin, M. Buchin, C. Wenk // Computational Geometry. - 2008. - Vol. 441, Issue 1-2. - P. 2-20. DOI: 10.1145/ 1137856.1137870

19. Rote G. Computing the Fr e chet distance between piecewise smooth curves / G. Rote // Computational Geometry. - 2007. -Vol. 37. - P. 162-174. DOI: 10.1016/j.comgeo.2005.01.004

20. Schlesinger M. I. Frechet Similarity of Closed Polygonal Curves / M. I. Schlesinger, E. V. Vodolazskiy, V. M. Yakovenko // International Journal of Computational Geometry. - 2016. -Vol. 26. - P. 53-66. DOI: 10.1142/S0218195916500035

21. Computing the discrete Fr echet distance with imprecise impute / [H.-K. Ahn, C. Knauer, M. Scherfenberg et al.] // International Journal of Computational Geometry. - 2016. - Vol. 22. - P. 2744. DOI: 10.1142/S0218195912600023

22. Computing the Fr e chet distance between folded polygons / [A. F. Cook, Anne Driemel, Jessica Sherette et al.] // Computational Geometry. - 2015. - Vol. 50. - P. 1-16.

23. Gudmundsson J. Fast algorithms for approximate Frechet matching queries in geometric trees / J. Gudmundsson, M. Smid // Computational Geometry. - 2015. - Vol. 48. - P. 479-494. DOI:10.1016/j.comgeo.2015.02.003

24. Computing discrete Fr e chet distance: Technical Report: CD-TR 94/64 / T. Eiter, H. Mannila // Information Systems Department, Technical University of Vienna. - Vienna, 1994. - 7 p.

25. Deza M. M. Encyclopedia of Distances / M. M. Deza. - Berlin : Springer-Verlag, 2009. - 590 p.

26. Gromov M. Metric Structures for Riemannian and Non-Riemannian Spaces / M. Gromov. - Boston, MA, Progress in Mathematics, 1999. - 1041 p.

27. Regions Matching Algorithms Analysis to Quantify the Image Segmentation Results / [O. Berezsky, Y. Batko, O. Pitsun et al.] // Sensors & Transducers. - 2017. - Vol. 208, Issue 1. - P. 44-49. DOI: 10.1109/STC-CSIT.2016.7589862

28. Fuzzy system diagnosing of precancerous and cancerous conditions of the breast / [O. Berezsky, S. Verbovyy, L. Dubchak et al.] // Computer Sciences and Information Technologies (CSIT) : XI th International Scientific and Technical Conference, Lviv, 6-10 September 2016 : proceedings. - Lviv, 2016. - P. 200-203. DOI: 10.1109/STC-CSIT.2016.7589906

29.Berezsky O. Automated Processing of Cytological and Histological Images / Oleh Berezsky, Oleh Pitsun // Perspective Technologies and Methods in MEMS Design (MEMSTECH'2016) : XII th International Conference, Lviv-Polyana, 20-24 April 2016 : proceedings. - Lviv, 2016. - P. 51-53. DOI:10.1109/ MEMSTECH.2016.7507518

Article was submitted 14.06.2017.

After revision 20.08.2017.

Березький О. М.1, Пщун О. Й.2

'Д-р техн. наук, професор, завщувач кафедри комп'ютерно1 шженерн, Тернопiльський нацiональний економiчний ушверситет, Тернопiль, Укра1на

2Аспiрант, Терношльський нацiональний економiчний унiверситет, Тернопiль, Укра1на

МЕТОДИ ОЦ1НКИ ЯКОСТ1 СЕГМЕНТАЦИ ЗОБРАЖЕНЬ

Актуальтсть. Розглянуто основнi методи кшьгасно1 оцiнки якостi сегментаци зображень. Вони використовуються для вибору алrоритмiв сегментаци для конкретних класiв зображень. Об'ектом дослiдження е цитолоriчнi та пстолопчш зображення, якi використовуються при дiаrностуваннi патолоriчних процесiв в онкологи. Предметом дослщження е кiлькiснi методи ощнки якостi алrоритмiв сегментаци.

Мета роботи - введення метрики Громова-Фреше та розроблення метричного методу кшькюно! оцiнки якостi сегментаци для порiвняння алrоритмiв сегментаци зображень.

Метод. Проаналiзованi критери кшькюно1 оцiнки, якi базуються на порiвняннi та без порiвняння з еталонним зображенням. Проаналiзовано алгоритми знаходження вiдстанi мiж зображеннями на основi метрик Фреше, Хаусдорфа, Громова-Хаусдорфа.

Для знаходження вщсташ мiж контурами зображень введено вщстань Громова-Фреше. Доведено умову тотожносп, симетричностi та трикутника i показано, що вiдстань Громова-Фреше е метрикою.

Розроблено метричний метод кшькюно! ощнки якосп сегментаци. Вш базуеться на використанш метрик Громова-Хаусдорфа та Громова-Фреше. Метод побудований на основi алгорштшв перетворення неопуклих полшошв у опуклi, алгоритму зважених хорд, алгорштшв обчислення вiдстаней Фреше та Хаусдорфа. Для знаходження вщсташ Хаусдорфа мiж опуклими областями використано алгоритм М. Аталаха. Для знаходження дискретное' вщсташ Фреше використано алгоритм Томаса Ейтера i Хейга Маншли. Даш алгоритми мають найнижчу обчислювальну складшсть серед свого класу алrоритмiв.

Результати. Введено метрику Громова-Фреше та розроблено метричний метод кшькюно1 ощнки якосп сегментаци.

Висновки. Проведенi експерименти на основi цитолоriчних зображень пiдтвердили працездатшсть програмного забезпечення оцiнки вiдстанi мiж зображеннями. Розроблений метод показав високу точшсть оцiнки вiдстаней мiж зображеннями. Розроблений програмний модуль використаний у штелектуальнш системi дiаrностування передракових i ракових станiв молочно1 залози. Програмне забезпечення може бути використане в рiзних програмних системах комп'ютерного зору. Перспективними напрямами дослщжень е пошук нових метрик для оцiнки вщсташ мiж зображеннями.

Ключовi слова: сегментащя, кiлькiсна оцiнка сегментаци, метрика Фреше, метрика Хаусдорфа, метрика Громова-Хаусдорфа, метрика Громова-Фреше, полшон, цитолопчш зображення.

Березский О. Н.1, Пицун О. Й.2

'Д-р техн. наук, профессор, заведующий кафедрой компьютерной инженерии, Тернопольский национальный экономический университет, Тернополь, Украина

2Аспирант, Тернопольский национальный экономический университет, Тернополь, Украина

МЕТОДЫ ОЦЕНКИ КАЧЕСТВА СЕГМЕНТАЦИИ ИЗОБРАЖЕНИЙ

Актуальность. Рассмотрены основные методы количественной оценки качества сегментации изображений. Они используются для выбора алгоритмов сегментации для конкретных классов изображений. Объектом исследования являются цитологические и гистологические изображения, используемые при диагностировании патологических процессов в онкологии. Предметом исследования являются количественные методы оценки качества алгоритмов сегментации.

Цель работы - введение метрики Громова-Фреше и разработка метрического метода количественной оценки качества сегментации для сравнения алгоритмов сегментации изображений.

Метод. Проанализированы критерии количественной оценки, основанные на сравнении и без сравнения с эталонным изображением. Проанализированы алгоритмы нахождения расстояния между изображениями на основе метрик Фреше, Хаусдорфа, Громова-Хаусдорфа.

Для нахождения расстояния между контурами изображений введено расстояние Громова-Фреше. Доказано условие тождественности, симметричности и треугольника и показано, что расстояние Громова-Фреше является метрикой.

Разработан метрический метод количественной оценки качества сегментации. Он базируется на использовании метрик Громова-Хаусдорфа и Громова-Фреше. Метод построен на основе алгоритмов преобразования невыпуклых полигонов в выпуклые, алгоритма взвешенных хорд, алгоритмов вычисления расстояний Фреше и Хаусдорфа. Для нахождения расстояния Хаусдорфа между выпуклыми областями использован алгоритм М. Аталаха. Для нахождения дискретного расстояния Фреше использован алгоритм Томаса Рейтера и Хейкки Маниллы. Данные алгоритмы имеют самую низкую вычислительную сложность среди своего класса алгоритмов.

Результаты. Введено метрику Громова-Фреше и разработано метрический метод количественной оценки качества сегментации.

Выводы. Проведенные эксперименты на основе цитологических изображений подтвердили работоспособность программного обеспечения оценки расстояния между изображениями. Разработанный метод показал высокую точность оценки расстояний между изображениями. Разработанный программный модуль использован в интеллектуальные системе диагностирования предраковых и раковых состояний молочной железы. Программное обеспечение может быть использовано в различных программных системах компьютерного зрения. Перспективными направлениями исследований является поиск новых метрик для оценки расстояния между изображениями.

Ключевые слова: сегментация, количественная оценка сегментации, метрика Громова-Хаусдорфа, метрика Громова-Фреше, полигон, цитологические изображения.

REFERENCES

1. Yinpeng J., Fayadb L., Laine A. Contrast Enhancement by Multi-scale Adaptive Histogram Equalization, Proceedings of SPIE, 2001, Vol. 4478, pp. 206-213. DOI: 10.7916/D8QZ2M29

2. Baron T. H., Harewood G., Rumalla A. A Prospective Comparison of Digital Image Analysis and Routine Cytology for the

Identification of Malignancy in Biliary Tract Strictures, Clinical Gastroenterology and Hepatology, 2004, Vol. 2, Issue 3, pp. 214-219. DOI: 10.1016/S1542-3565(04)00006-0 3. Petushi S., Garcia F. U., Haber M. M. Large-scale computations on histology images reveal grade-differentiating parameters for breast cancer, BMC Medical Imaging, 2006, Vol. 6, Issue 14, pp. 496-499. DOI: 10.1186/1471-2342-6-14

4. Levine M. D., Nazif A. Dynamic measurement of computer generated image segmentations, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1985, Vol. 7, Issue 2, pp. 155-164. DOI: 10.1109/TPAMI.1985.4767640

5. Zhang Y. J. A review of recent evaluation methods for image segmentation, Signal Processing and its Applications (ISSPA), Sixth International Symposium, Kuala Lumpur Malaysia, Aug 13-16, 2001, proceedings. Kuala Lumpur, 2001, pp. 148-151.

6. Berezsky O. Methods of quantitative evaluation of image segmentation quality, Signal/Image Processing and Pattern Recognition (UkrOBRAZ'2014) : XIIIth All-Ukraine International Conference, Kyiv, 3-7 November 2014 : proceedings. Kyiv, 2014, pp. 51-54.

7. Lee S. U., Chung S. Y., Park R. H. A comparative performance study of several global thresholding techniques for segmentation, Computer Vision, Graphics, and Image Processing, 1990, Vol. 52, Issue. 2, pp. 171-190.

8. Zhang Y. J., Gerbrands J. J. Segmentation evaluation using ultimate measurement accuracy, Image Processing Algorithms and Technique, 1992, Vol. 1657, pp. 449-460.

9. Zhang Y J., Gerbrands J. J. Objective and quantitative segmentation evaluation and comparison Signal Processing, 1994, Vol. 39, Issue.2, pp. 43-54. D0I:10.1016/0165-1684(94)90122-8

10. Lopez M. A., Reisner S. Hausdorff approximation of convex polygons, Computational Geometry, 2005, Vol. 32, Issue 2, pp. 139-158. DOI: 10.1016/j.comgeo.2005.02.002

11. Alt H., Scharfz L. Computing the Hausdorff distance between curved objects, International Journal of Computational Geometry, 2008, Vol. 18, pp. 307-320. DOI: 10.1142/S0218195908002647

12. Chew L. P., Kedem K. Getting around a lower bound for the minimum Hausdorff distance, Computational Geometry, 1998, Vol. 10, Issue 3, pp. 197-202. DOI: S0925-7721(97)00032-1

13. Knauer C., Scherfenberg M. Approximate nearest neighbor search under translation invariant hausdorff distance, International Journal of Computational Geometry, 2011, Vol. 21, Issue 3, pp. 369-381. DOI: S0218195911003706

14. Alvarez V., Seidel R. Approximating the minimum weight spanning tree of a set of points in the Hausdorff metric, Computational Geometry, 2010, Vol. 43, pp. 94-98.

iНе можете найти то, что вам нужно? Попробуйте сервис подбора литературы.

15. Atallah M. J., Celso C. Computing Some Distance Functions Between Polygons, Computer Science Technical Reports, 1990, Vol. 9, pp. 1-10.

16. Alt H., Godau M. Computing the Fr échet distance between two polygonal curves, International Journal of Computational Geometry and Applications, 1995,Vol. 5, pp. 75-91.

17. Mosig A., Clausen M. Approximately matching polygonal curves with respect to the Fr é chet distance, Computational Geometry, 2005, Vol. 30, Issue 2, pp. 113-127. DOI: 10.1016/ j.comgeo.2004.05.004

18. Buchin K., Buchin M., Wenk C. Computing the Frachet distance between simple polygons, Computational Geometry, 2008, Vol. 441, Issue 1-2, pp. 2-20. DOI: 10.1145/1137856.1137870

19. Rote G. Computing the Fr é chet distance between piecewise smooth curves, Computational Geometry, 2007, Vol. 37, pp. 162-174. DOI: 10.1016/j.comgeo.2005.01.004

20. Schlesinger M. I., Vodolazskiy E. V., Yakovenko V. M. Frechet Similarity of Closed Polygonal Curves, International Journal of Computational Geometry, 2016, Vol. 26, pp. 53-66. DOI: 10.1142/S0218195916500035

21. Ahn H.-K., Knauer C., Scherfenberg M. et al. Computing the discrete Fr échet distance with imprecise impute, International Journal of Computational Geometry, 2016, Vol. 22, pp. 27-44. DOI: 10.1142/S0218195912600023

22. Cook A. F., Driemel Anne, Sherette Jessica et al. Computing the Fréchet distance between folded polygons, Computational Geometry, 2015, Vol. 50, pp. 1-16.

23. Gudmundsson J., Smid M. Fast algorithms for approximate Fréchet matching queries in geometric trees, Computational Geometry, 2015, Vol. 48, pp. 479-494. DOI:10.1016/ j .comgeo.2015.02.003

24. Eiter T., Mannila H. Computing discrete Fréchet distance: Technical Report: CD-TR 94/64 / Information Systems Department, Technical University of Vienna. Vienna, 1994, 7 p.

25. Deza M. M. Encyclopedia of Distances. Berlin, Springer-Verlag, 2009, 590 p.

26. Gromov M. Metric Structures for Riemannian and Non-Riemannian Spaces. Boston, MA, Progress in Mathematics, 1999, 1041 p.

27. Berezsky O., Batko Y., Pitsun O. et al. Regions Matching Algorithms Analysis to Quantify the Image Segmentation Results, Sensors & Transducers, 2017, Vol. 208, Issue 1, pp. 44-49. DOI: 10.1109/STC-CSIT.2016.7589862

28. Berezsky O., Verbovyy S., Dubchak L. et al. Fuzzy system diagnosing of precancerous and cancerous conditions of the breast, Computer Sciences and Information Technologies (CSIT) : XI th International Scientific and Technical Conference, Lviv, 6-10 September 2016 : proceedings. Lviv, 2016, pp. 200-203. DOI: 10.1109/STC-CSIT.2016.7589906

29. Berezsky O., Pitsun O. Automated Processing of Cytological and Histological Images, Perspective Technologies and Methods in MEMS Design (MEMSTECH'2016) : XII th International Conference, Lviv-Polyana, 20-24April 2016 : proceedings. Lviv, 2016, pp. 51-53. DOI: 10.1109/MEMSTECH.2016.7507518

i Надоели баннеры? Вы всегда можете отключить рекламу.