HariyaniYuli Sun1,2
ParkCheolsoo1
-
(Department of Computer Engineering, Kwangwoon University / Seoul, Korea {yulisun,
parkcheolsoo}@kw.ac.kr )
-
(School of Applied Science, Telkom University / Bandung, Indonesia yulisun@telkomuniversity.ac.id
)
Copyright © The Institute of Electronics and Information Engineers(IEIE)
Keywords
White blood cells, Nailfold capillary, Event-based data, Classification, Dynamic vision sensor
1. Introduction
White blood cells (WBCs) are a crucial part of the human immune system. A lack of
WBCs can lead to a number of illnesses, such as sepsis [1], infectious disorders [2,3], and cancer [4]. Typically, a blood sample must be drawn by experienced medical personnel for WBC
monitoring using advanced technology. Numerous noninvasive methods based on optical
technologies have been suggested to enhance WBC monitoring. One example is observing
nailfold capillaries, where WBCs are identified as visible gaps [5]. Nailfold capillaroscopy is a non-invasive imaging method to assess micro-circulation
in tiny blood vessels under the fingernail [6]. The location of nailfold capillaries in fingers and capillary visualization is shown
in Fig. 1. By illuminating the capillaries at a specific wavelength, a microscope can reveal
WBC flow in order to evaluate WBC status [7,8]. Moreover, absorption gaps or plasma gaps in a nailfold capillary can be used to
quantify the number of WBCs.
Fig. 1. Location of nailfold capillaries (left) and visualization of the capillaries (right).
Fig. 2. Differences in output of a standard camera versus an event-based camera (adapted from[16]).
Recently, measurement of the plasma gap has usually been done with a standard camera
[8,9]. However, difficulties with that method are low visibility of the gap, the small
size of the capillary, the high speed of WBC movement, and non-contrast between capillaries
and their environment/background. To address these problems, this study uses a dynamic
vision sensor (DVS) camera, a so-called event-based camera, and bio-inspired vision
sensors that work differently from conventional frame cameras to detect the change
in luminance on a pixel basis and to produce a stream of asynchronous event output
at a microsecond time resolution [10]. Event cameras such as the dynamic vision sensor are bio-inspired devices that work
differently from conventional frame cameras. While the conventional camera captures
images at a fixed rate, event cameras asynchronously measure per-pixel brightness
changes and provide a stream of events that encode the time, location (X, Y), and
sign (polarity) of the brightness changes, as seen in Fig. 2. DVS cameras have been utilized in various applications, such as object detection/tracking
[11-13] and gesture detection/ classification [14,15]. DVS sensors capture detailed movement phases and fast movements without blurring
[15]. With the advantages of DVS cameras, the existence and movement of WBCs in the nailfold
can be captured.
This paper proposes event-based white blood cell classification in the nailfold capillary
using various machine learning algorithms. To this end, we recorded a new dataset
using the DVS camera for nailfold capillary videos, and then conducted feature engineering
and classification. Extensive results are provided to evaluate the various machine
learning algorithms.
2. The Proposed Method
This section describes our proposed method. In this study, we propose a classification
approach using various machine learning algorithms given input from the WBC event-data
stream. Fig. 3 shows the overall procedure of the proposed model consisting of event accumulation,
feature engineering, and machine learning classification.
Fig. 3. Proposed WBC classification method.
2.1 Event-data Stream
An event-based camera (the DVS camera) is a type of vision sensor that can capture
changes in intensity levels at each pixel independently to within microseconds (the
so-called $\textit{events}$). Different from a standard camera’s output frame, a DVS
camera provides a stream of event data consisting of $\textit{(t, x, y, p)}$ information,
where $\textit{t}$ is the timestamp generated when an event occurs, $\textit{x}$ is
the x-axis of the spatial location of an event, $\textit{y}$ is the y-axis of the
spatial location of the event, and $\textit{p}$ is the event polarity.
A DVS sensor identifies an event in a corresponding pixel only when the intensity
change at the pixel surpasses threshold ${\theta}$, as shown in Eq. (1):
where $I_{now}^{\left(x,y\right)}$ and $I_{\textit{previous}}^{\left(x,y\right)}$
are the current and previous intensities at the corresponding pixel (x, y) [15]. Table 1 is an example of the event-data stream in our experiment.
The WBC flows through the nailfold capillary. Due to this WBC flow, the pixel intensity
around the WBC changes, and hence, the DVS camera captures WBC movement.
Table 1. Event-data streams.
|
t
|
X
|
Y
|
P
|
|
1631582467514400
|
340
|
241
|
0
|
|
1631582467514410
|
135
|
15
|
1
|
|
1631582467514410
|
8
|
182
|
1
|
|
1631582467514410
|
248
|
113
|
1
|
|
1631582467514420
|
136
|
52
|
1
|
|
1631582467514420
|
110
|
94
|
1
|
|
1631582467514440
|
55
|
48
|
1
|
|
1631582467514450
|
129
|
64
|
1
|
|
1631582467514450
|
24
|
119
|
0
|
|
1631582467514460
|
67
|
100
|
1
|
|
.
.
.
|
.
.
.
|
.
.
.
|
.
.
.
|
|
1631582518334360
|
326
|
106
|
1
|
|
1631582518334370
|
332
|
162
|
1
|
|
1631582518334380
|
77
|
93
|
0
|
2.2 Data Acquisition and Event Accumulation
The original data come from pre-recorded WBC video during previous research [17], and we selected two videos from two subjects. To obtain the event data, we re-recorded
the video using a DVS camera and display monitor.
The acquisition process of our event-data stream dataset employed a DVS camera by
iniVation, the DAVIS346, which captures event-data stream output at an area size of
346x260. All the event-data streams have x- and y-axes within this area. Alongside
the event stream, the DAVIS346 outputs a simultaneous intensity frame or an active
pixel sensor (APS) image, which is a grayscale image of a global shutter sensor. However,
in this study, the grayscale image is only used to assist with the data labeling procedure.
After the event-data streams are acquired, event-data segmentation and accumulation
are conducted. Segmented event data are 200ms long, and all event data within this
window are accumulated. Fig. 4 visualizes the event accumulation of a single window with or without the existence
of WBCs. This visualization image is only used for easy readability. For the experiments,
all data are in the form $\textit{t, x, y, p}$.
Fig. 4. Visualization of the accumulated events: (a) without WBC; (b) with WBC. The red circle highlights the WBCs.
2.3 Feature Reduction
One characteristic of the event-based camera is that it works asynchronously, resulting
in a non-uniform and arbitrary number of events for each segment. Owing to the nature
of the classifiers, there is a necessity to create the same $\textit{N}$ events for
each segment. In our experiments, we set $\textit{N}$ to 100,000 events and added
zero padding for the segments that had fewer than 100,000 events. Then, we selected
prominent features of the input data by employing principal component analysis (PCA).
PCA is commonly used to transform a large set of data into smaller sets without losing
information from projecting the data to a new set of features called principal components.
By selecting a subset of principal components with high variance, the feature space
dimension will be reduced. In this research, with 95% variance, the data size of each
sample was reduced by 99%. Originally each sample had a length of 100,000, which shrank
to 346 features after the PCA process.
2.4 Classifiers
2.4.1 k-Nearest Neighbors
k-Nearest neighbors (KNN) is a non-parametric machine learning algorithm usually applied
to classification tasks. The KNN algorithm works as follows: define the class number
and a distance metric; find the nearest neighbor for every sample; and then, assign
a class to the samples based on the distance between the sample and the center of
the class.
2.4.2 Decision Tree
A decision tree (DT) is a non-parametric machine learning algorithm for regression
and classification tasks. The model predicts the target after learning a simple rule-based
decision method based on data features.
2.4.3 Random Forest
Random forest (RF) is an algorithm for both regression and classification tasks considering
multiple decision trees that create a forest. Specifically, it combines several weak
models to create a better and more robust model. RF employs a bagging strategy to
prevent overfitting problems, and thus results in greater generalization.
3. Performance Evaluation
3.1 The Dataset
A dataset was generated from two videos with a total of 490 segments, among which
279 segments contained WBCs, and 211 did not. The event coordinates, $\textit{x}$
and $\textit{y}$, were used as input. The data were divided into training and testing
datasets at an 8:2 ratio where 392 segments were used for training and 98 for testing.
3.2 Performance Evaluation
The performance of the experiments was evaluated using accuracy and F1-score. Accuracy
is calculated as follows:
where \textit{TP} and \textit{TN} are true positive and true negative, respectively.
Precision, recall, and F1-score are calculated as follows:
The parameters of each algorithm used in the simulations are listed in Table 2. For KNN, the number of neighbors used in the simulation was five. For the decision
tree and random forest, the entropy function was used to measure the quality of a
split tree. While the maximum tree depth value was 100 for the decision tree, 20 was
chosen as the maximum tree depth and the number of estimators for the random forest
algorithm.
Table 2. Parameters for each algorithm.
|
Classifier
|
Parameter
|
|
KNN
|
Number of Neighbors = 5
|
|
Decision Tree
|
Criterion = Entropy
Max. Depth = 100
|
|
Random Forest
|
Criterion = Entropy
Max. Depth = 20
Number of Estimators = 20
|
Table 3 shows the performance comparison from KNN, the decision tree, and random forest in
terms of accuracy and F1-score, while Table 4 provides the precision and recall value for each classifier. The best accuracy and
F1-score were 75.51% and 75.08%, respectively, obtained using the random forest algorithm.
As well as the best accuracy, random forest yielded the best precision and recall
at 75.71% and 75.51%, respectively. Fig. 5 shows the confusion matrices for all algorithms. Among them, random forest produced
the most balanced performance.
Table 3. Performance Evaluation.
|
Classifier
|
Accuracy (%)
|
F1-score (%)
|
|
KNN
|
65.31
|
61.11
|
|
Decision Tree
|
67.35
|
67.25
|
|
Random Forest
|
75.51
|
75.08
|
Table 4. Precision and Recall.
|
Classifier
|
Precision (%)
|
Recall (%)
|
|
KNN
|
68.89
|
65.31
|
|
Decision Tree
|
67.20
|
67.34
|
|
Random Forest
|
75.71
|
75.51
|
Fig. 5. Confusion matrices.
4. Conclusion
This paper proposed a new way to detect WBC existence using event-based data. Classification
was conducted using three different machine learning algorithms: k-nearest neighbors,
the decision tree, and random forest. Based on our evaluation, event-based WBC classification
is a promising new method to detect WBC existence in nailfold capillaries. This proposed
state-of-the-art method can be used for developing a more efficient, non-invasive
WBC classification method. The limitation of this paper is that we focus on fixed
length numbers of events in every segment. For future directions, the process could
address arbitrary lengths of event-data streams, WBC counting, and implementation
of recent deep learning methods.
ACKNOWLEDGMENTS
This research was supported by the Basic Science Research Program through the
National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2020R1F1A1071712).
REFERENCES
Newman T. B., Draper D., Puopolo K. M., Wi S., Escobar G. J., Aug. 2014, Combining
Immature and Total Neutrophil Counts to Predict Early Onset Sepsis in Term and Late
Preterm Newborns, Pediatric Infectious Disease Journal, Vol. 33, No. 8, pp. 798-802
Honda T., Uehara T., Matsumoto G., Arai S., Sugano M., Jun. 2016, Neutrophil left
shift and white blood cell count as markers of bacterial infection, Clinica Chimica
Acta, Vol. 457, pp. 46-53
van Wolfswinkel M. E., et al. , Dec. 2013, Predictive value of lymphocytopenia and
the neutrophil-lymphocyte count ratio for severe imported malaria, Malaria Journal,
Vol. 12, No. 1, pp. 101
Crawford J., Dale D. C., Lyman G. H., Jan. 2004, Chemotherapy-induced neutropenia,
Cancer, Vol. 100, No. 2, pp. 228-237
Golan L., Yeheskely-Hayon D., Minai L., Dann E. J., Yelin D., Jun. 2012, Noninvasive
imaging of flowing blood cells using label-free spectrally encoded flow cytometry,
Biomedical Optics Express, Vol. 3, No. 6, pp. 1455
Shih T.-C., et al. , 2011, Hemodynamic analysis of capillary in finger nail-fold using
computational fluid dynamics and image estimation, Microvascular Research, Vol. 81,
No. 1, pp. 68-72
Bourquard A., et al. , 2015, Analysis of white blood cell dynamics in nailfold capillaries,
in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and
Biology Society (EMBC), pp. 7470-7473
Bourquard A., et al. , 2018, Non-invasive detection of severe neutropenia in chemotherapy
patients by optical imaging of nailfold microcirculation, Scientific Reports, Vol.
8, No. 1, pp. 1-12
McKay G. N., et al. , 2020, Visualization of blood cell contrast in nailfold capillaries
with high-speed reverse lens mobile phone microscopy, Biomed Opt Express, Vol. 11,
No. 4, pp. 2268-2276
Gallego G., et al. , Jan. 2022, Event-Based Vision: A Survey, IEEE Transactions on
Pattern Analysis and Machine Intelligence, Vol. 44, No. 1, pp. 154-180
Perot E., de Tournemire P., Nitti D., Masci J., Sironi A., 2020, Learning to Detect
Objects with a 1 Megapixel Event Camera, in Advances in Neural Information Processing
System, Vol. 33, pp. 16639-16652s
Wan J., et al. , Apr. 2021, Event-Based Pedestrian Detection Using Dynamic Vision
Sensors, Electronics (Basel), Vol. 10, No. 8, pp. 888
Cannici M., Ciccone M., Romanoni A., Matteucci M., Jun. 2019, Asynchronous Convolutional
Networks for Object Detection in Neuromorphic Cameras, in 2019 IEEE/CVF Conference
on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1656-1665
Belbachir A. N., Schraml S., Nowakowska A., Jun. 2011, Event-driven stereo vision
for fall detection, in CVPR 2011 WORKSHOPS, pp. 78-83
Wang Y., et al. , Jun. 2019, EV-Gait: Event-Based Robust Gait Recognition Using Dynamic
Vision Sensors, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 6351-6360
Mueggler E., Huber B., Scaramuzza D., Sep. 2014, Event-based, 6-DOF pose tracking
for high-speed maneuvers, in 2014 IEEE/RSJ International Conference on Intelligent
Robots and Systems, pp. 2761-2768
Hariyani Y. S., Eom H., Park C., 2020, DA-Capnet: Dual Attention Deep Learning Based
on U-Net for Nailfold Capillary Segmentation, IEEE Access, Vol. 8, pp. 10543-10553
Author
Yuli Sun Hariyani received a BS in telecommunication engineering and an MS in electrical
engineering from Telkom University, Bandung, Indonesia, in 2010 and 2013, respectively.
She is currently pursuing a PhD in the Computer Engineering Department at Kwangwoon
University, Seoul, South Korea. Since 2014, she has been a Lecturer with Telkom University,
Indonesia. Her research interests include pattern recognition, medical image processing,
and biomedical signal processing.
Cheolsoo Park is an associate professor in the Computer Engineering Department
at Kwangwoon University, Seoul, South Korea. He received a B.Eng. in Electrical Engineering
from Sogang University, Seoul, and an MSc from the Biomedical Engineering Department
at Seoul National University, South Korea. In 2012, he received his PhD in Adaptive
Nonlinear Signal Processing from the Imperial College London, U.K., and worked as
a postdoctoral researcher in the Bioengineering Department at the University of California,
San Diego, U.S.A. His research interests are in the areas of machine learning and
adaptive and statistical signal processing, with applications in brain computer interfaces,
computational neuroscience, and wearable technology.