FOREX Press I. J. of Electrical & Electronics Research
Support Open Access
  • Home/
  • IJEER-100346

Research Article |

Revaluating Pretraining in Small Size Training Sample Regime

Author(s): Vandana Khobragade1, Jagannath Nirmal2 and Shreyansh Chedda3

Published In : International Journal of Electrical and Electronics Research (IJEER)        Volume 10, Issue 3

Publisher : FOREX Publication

Published : 22 September 2022

e-ISSN : 2347-470X

Page(s) : 694-704

DOI: https://doi.org/10.37391/IJEER.100346

Abstract

Deep neural network (DNN) based models are highly acclaimed in medical image classification. The existing DNN architectures are claimed to be at the forefront of image classification. These models require very large datasets to classify the images with a high level of accuracy. However, fail to perform when trained on datasets of small size. Low accuracy and overfitting are the problems observed when medical datasets of small sizes are used to train a classifier using deep learning models such as Convolutional Neural Networks (CNN). These existing methods and models either always overfit when training on these small datasets or will result in classification accuracy which tends towards randomness. This issue stands even when using Transfer Learning (TL), the current standard for such a scenario. In this paper, we have tested several models including ResNet and VGGs along with more modern models like MobileNets on different medical datasets with transfer learning and without transfer learning. We have proposed solid theories as to why there exists a need for a more novel approach to this issue, and how the current methodologies fail when applied to the aforementioned datasets. Larger, more complex models are not able to converge for smaller datasets. Smaller models with less complexity perform better on the same dataset than their larger model counterparts.

Keywords: Acute Lymphoblastic Leukemia, Convolutional Neural Network, Deep Neural Network, Transfer Learning, White Blood Cells




Vandana Khobragade*, Department of Electronic and Telecommunication, LTCE, University of Mumbai, Mumbai, India; Email: vandanakhobragade@ltce.in

Jagannath Nirmal, Department of Electronics, KJSCE, Mumbai, India; Email: jhnirmal@somaiya.edu

Shreyansh Chedda, Department of Computer Science, VJTI, Mumbai, India; Email: shreyansh.chheda@gmail.com

    [1] B. J. Bain, “Diagnosis from the blood smear,” New England Journal of Medicine, vol. 353, no. 5, pp. 498-507, 2005.[Cross Ref]
    [2] P. G. Gallagher. Red cell membrane disorders. ASH Education Program Book, vol. 1, pp. 13-18.[Cross Ref]
    [3] T. J.Durant, E.M.Olson, W.L.Schulz, R.Torres, “Very deep convolutional neural networks for morphologic classification of erythrocytes,” Clinical Chemistry, vol. 63, no.12, 2017, pp. 1847-1855.[Cross Ref]
    [4] J. Ford, "Red blood cell morphology,” International journal of laboratory hematology, vol. 35, no. 3, pp. 351-357, 2013.[Cross Ref]
    [5] V.Sanjay and P.Swarnalatha, “A survey on various machine learning techniques for an efficient brain tumor detection from MRI images,” IJEER, vol. 10, no. 2, pp. 177-182,2022.[Cross Ref]
    [6] H.Singh and R.Singh Solanki, “Classification and feature extraction of brain tumor from MRI images using modified ANN approach,” IJEER, vol. 2, no. 2, pp. 10-15,2021.[Cross Ref]
    [7] J. Zhao, M. Zhang, Z. Zhou, J. Chu, “Automatic detection and classification of leukocytes using convolutional neural networks,” Medical & biological engineering & computing, vol. 55, no. 8, 2017, pp. 1287-1301.[Cross Ref]
    [8] G. Litjens et al.Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Scientific reports, vol. 6, no. 1, 2016 pp. 1-11.[Cross Ref]
    [9] C. McClanahan, “History and evolution of GPU architecture,” A Survey Paper, vol. 9, 2010.[Cross Ref]
    [10] Ciresan. D., Meier. U., Masci, J., Gambardella.L., and Schmidhuber, J., 2011. Flexible, high-performance convolutional neural networks for image classification. Twenty-second international joint conference on artificial intelligence.[Cross Ref]
    [11] Deng, J., Dong,W., Socher,R., Li,Li-Jia., Li, K., Fei-Fei, Li., 2009. Imagenet: A large-scale hierarchical image database.IEEE conference on computer vision and pattern recognition.[Cross Ref]
    [12] O. Russakovsky et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, no. 3, pp. 211-252, 2015.[Cross Ref]
    [13] A. Krizhevsky, I. Sutskever, G. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.[Cross Ref]
    [14] Zeiler, M. and Fergus, R. 2014.Visualizing and understanding convolutional networks. European conference on computer vision, Springer, Cham.[Cross Ref]
    [15] A. G. Howard, et al, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” 2017.[Cross Ref]
    [16] F. Iandola, M. Moskewicz, S. Karayev, R. Girshick, T. Darrell and K. Keutzer, “Densenet: Implementing efficient convent descriptor pyramids,” 2014. [Cross Ref]
    [17] M. Sajjad, et al., “Leukocytes classification and segmentation in microscopic blood smear: a resource-aware healthcare service in smart cities,” IEEE Access, vol. 5, pp. 3475-3489, 2016.[Cross Ref]
    [18] J. Prinyakupt and C. Pluempitiwiriyawej, “Segmentation of white blood cells and comparison of cell morphology by linear and naïve Bayes classifiers,” Biomedical engineering online, vol. 14, no. 1, pp. 1-19, 2015.[Cross Ref]
    [19] A. Abdeldaim, A. Sahlol, M. Elhosney, A.Hassanien, “Computer-aided acute lymphoblastic leukemia diagnosis system based on image analysis,” Advances in Soft Computing and Machine Learning in Image Processing, Springer, Cham, 2018, pp. 131-147.[Cross Ref]
    [20] Vogado, L., Veras, R., Andrade, A., Araujo, F., Silva, R. and Aires, K. 2017.Diagnosing leukemia in blood smear images using an ensemble of classifiers and pre-trained convolutional neural networks.30th SIBGRAPI Conference on Graphics, Patterns, and Images (SIBGRAPI). IEEE.[Cross Ref]
    [21] S. Pan, and Q. Yang, “A survey on transfer learning." IEEE Transactions on knowledge and data engineering,” vol. 22, no.10, pp.1345-1359, 2009.[Cross Ref]
    [22] Torrey, L. and Shavlik, J. 2010. Handbook of research on machine learning applications and trends: algorithms, methods, and techniques. IGI Global.[Cross Ref]
    [23] Oquab, M., Bottou, L., Laptev, I. and . Sivic, J. 2014. Learning and transferring mid-level image representations using convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724.[Cross Ref]
    [24] You, Q., Luo, J., Jin, H., and Yang, J. 2015. Robust image sentiment analysis using progressively trained and domain transferred deep networks. Twenty-ninth AAAI conference on artificial intelligence.[Cross Ref]
    [25] Yao, Y. and Doretto, G. 2010. Boosting for transfer learning with multiple sources. 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE.[Cross Ref]
    [26] N. Srivastava, G. Hinton, A. Krizhevsky, “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research, vol. 15,no. 1, pp. 1929-1958, 2014.[Cross Ref]
    [27] Xie,S., Yang, T., Wang, X. and Lin, Y. 2015 . Hyper-class augmented and regularized deep learning for fine-grained image classification. Proceedings of the IEEE conference on computer vision and pattern recognition.[Cross Ref]
    [28] Wong, S., Gatt, A., Stamatescu, V. and Mcdonnell, M. 2016. Understanding data augmentation for classification: when to warp? 2016 international conference on digital image computing: techniques and applications (DICTA). IEEE.[Cross Ref]
    [29] L. Perez, and J. Wang, “The effectiveness of data augmentation in image classification using deep learning,” 2017.[Cross Ref]
    [30] T. Chan, K. Jia, S. Gao, J. Lu, Z. Zeng, Y. Ma, “PCANet: A simple deep learning baseline for image classification?,” IEEE transactions on image processing, vol. 24, no. 12,pp. 5017-5032, 2015.[Cross Ref]
    [31] H. C. Shin, et al., “Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics, and transfer learning.” IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1285-1298, 2016.[Cross Ref]
    [32] B. Q. Huynh, H. Li, and M. L. Giger, “Digital mammographic tumor classification using transfer learning from deep convolutional neural networks,” Journal of Medical Imaging, vol. 3, 2016.[Cross Ref]
    [33] M.Geng, Y. Wang, T.Xiang, Y.Tian, “Deep transfer learning for person re-identification,” arXiv preprint arXiv:1611.05244, 2016.[Cross Ref]
    [34] A. Kumar, J. Kim, D. Lyndon, M. Fulham, and D. Feng, “An ensemble of fine-tuned convolutional neural networks for medical image classification,” IEEE journal of biomedical and health informatics, vol. 21, no.1, pp. 31-40, 2016.[Cross Ref]
    [35] Yanai, K. and Kawano, Y. 2015. Food image recognition using a deep convolutional network with pre-training and fine-tuning. 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE.[Cross Ref]
    [36] Mersha Nigus and H.L Shashirekha (2022), A Comparison of Machine Learning and Deep Learning Models for Predicting Household Food Security Status. IJEER 10(2), 308-311. DOI: 10.37391/IJEER.100241.[Cross Ref]
    [37] Carneiro, G. and Vasconcelos, N. 2005. Formulating semantic image annotation as a supervised learning problem. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). vol. 2. IEEE.[Cross Ref]
    [38] Fergus,R., Perona, P. and Zisserman, A. 2003.Object class recognition by unsupervised scale-invariant learning. 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Proceedings, vol. 2. IEEE.[Cross Ref]
    [39] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wo, A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011.[Cross Ref]
    [40] Liu, S. and Deng, W. 2015. Very deep convolutional neural network-based image classification using a small training sample size.2015 3rd IAPR Asian conference on pattern recognition (ACPR), IEEE.[Cross Ref]
    [41] S. Hira, Anita Bai, and S. Hira, “An automatic approach based on CNN architecture to detect Covid-19 disease from chest X-ray images,” Applied Intelligence, vol. 51, no.5, pp. 2864-2889, 2021.[Cross Ref]
    [42] M. Loey, M. Naman, and Z. Hala, “Deep transfer learning in diagnosing leukemia in blood cells,” Computers, vol. 9, no. 2, 2020.[Cross Ref]
    [43] Habibzadeh, M., Jannesari, M., Rezaei,Z., Baharvand, H. and Totonchi, M. 2017. Automatic white blood cell classification using pre-trained deep learning models: ResNet and Inception. Tenth international conference on machine vision (ICMV 2017), vol. 10696, International Society for Optics and Photonics.[Cross Ref]
    [44] V.Narasimha and Dr.M.Dhanalakshmi, “Detection and severity identification of Covid-19 in chest X-ray images using deep learning,” IJEER, vol. 10, No. 2, pp. 364-369.[Cross Ref]
    [45] Labati, R., Piuri, V. and Scotti, F. 2011.All-IDB: The acute lymphoblastic leukemia image database for image processing. 2011 18th IEEE international conference on image processing. IEEE.[Cross Ref]
    [46] Duggal, R., Gupta, A., Gupta, R., Wadhwa, M. and Ahuja, C. 2016. Overlapping cell nuclei segmentation in microscopic images using deep belief networks. Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing.[Cross Ref]
    [47] Das, P. K., and S. Meher. 2021. An efficient deep convolutional neural network-based detection and classification of acute lymphoblastic leukemia. Expert Systems with Applications 183:115311. doi:10.1016/j.eswa.2021.115311.[Cross Ref]

Vandana Khobragade, Jagannath Nirmal, and Shreyansh Chedda (2022), Revaluating Pretraining in Small Size Training Sample Regime. IJEER 10(3), 694-704. DOI: 10.37391/IJEER.100346.