I. J. of Electrical & Electronics Research
Research Article | 
A Cost-Effective and Scalable Processing of Heavy Workload with AWS Batch
Author(s) : Nagresh Kumar1 and Sanjay Kumar Sharma2
Published In : International Journal of Electrical and Electronics Research (IJEER) Volume 10, Issue 2 , Special Issue on RDCTML
Publisher : FOREX Publication
Published : 22 May 2022
e-ISSN : 2347-470X
Page(s) : 144-149
Abstract
Recent technological advancements in the IT field have pushed many products and technologies into the cloud. In the present scenario, the cloud service providers mainly focus on the delivery of IT services and technologies rather than throughput. In this research paper, we used a scalable cost-effective approach to configure AWS Batch with AWS Fargate and CloudFormation and implemented it in order to handle a heavy workload. The AWS service configuration procedure, GitHub repository, and Docker desktop applications have been clearly described in this work. A cost-effective configuration and architecture of AWS Batch processing are given to provide high throughput. The processing of heavy workload by AWS Batch is represented in terms of execution time and the result shows that the concurrent execution reduces the execution time. To enhance the throughput heavy workload using batch processing an "Amazone FSx for Lustre" can also be used.
Keywords: AWS, CloudFormation, Fargate, GitHub, Docker, Amazon Web service (AWS)
Nagresh Kumar, Department of Computer Science, Banasthali Vidyapith, Rajasthan, India; Email: nagresh@gmail.com
Sanjay Kumar Sharma, Department of Computer Science, Banasthali Vidyapith, Rajasthan, India; Email: skumar2.sharma@gmail.com
[1] AWS documentation on AWS Batch. Accessed on: Feb. 20, 2022 [Online]Available: https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html. [Cross Ref]
[2] AWS documentation on AWS Batch Features. Accessed on: Feb. 20, 2022 [Online] Available: https://aws.amazon.com/batch/features/.[Cross Ref]
[3] Chandrajeet Yadav, Vikash Yadav et al, “Authentication, Access Control, VM Allocation and Energy efficiency towards Securing Computing Environments in Cloud Computing”, Annals of the Romanian Society for Cell Biology, Association of Cell Biology Romania Publication, ISSN 1583-6258, Vol. 25, No. 6, pp. 17939-17954, June 2021.[Cross Ref]
[4] Kyle M. D. Sweeney and Douglas Thain, 2018. Early Experience Using Amazon Batch for Scientific Workflows. In Proceedings of the 9th Workshop on Scientific Cloud Computing (ScienceCloud'18). Association for Computing Machinery, New York, NY, USA, Article 5, 1–8. [Cross Ref]
[5] D. Cui et al., "Cloud Workflow Task and Virtualized Resource Collaborative Adaptive Scheduling Algorithm Based on Distributed Deep Learning," 2020 IEEE International Conference on Advances in Electrical Engineering and Computer Applications( AEECA), 2020, pp. 137-14.[Cross Ref]
[6] Vikash Yadav et al, “Healthcare, IoT, and Big Data Support”, “Empowering Artificial Intelligence through Machine Learning New Advances and Applications”, Print ISBN: 9781771889308, pp. 57-81 Published by “CRC Press & Apple Academic Press”, July 2021.[Cross Ref]
[7] Yadav, V., Kundra, P., Verma, D. (2021), “Role of IoT and Big Data Support in Healthcare”, In: Gao, XZ. Tiwari, S., Trivedi, M., Mishra, K. (eds) Advances in Computational Intelligence and Communication Technology. Advances in Intelligent Systems and Computing, Vol. 1086, pp. 445-455, ISBN 978-981-15-1275-9, Springer, Singapore, June 19, 2020.[Cross Ref]
[8] C. Byun et al., "Best of Both Worlds: High Performance Interactive and Batch Launching," 2020 IEEE High Performance Extreme Computing Conference (HPEC), 2020, pp. 1-7.[Cross Ref]
[9] C. Lin and S. Lu, "Scheduling Scientific Workflows Elastically for Cloud Computing," 2011 IEEE 4th International Conference on Cloud Computing, 2011, pp. 746-747.[Cross Ref]
[10] V. March, S. See, M. Garg, P. Gupta and T. Atrey, "Batch Scheduler for Personal Multi-Core Systems," in 2013 12th International Symposium on Distributed Computing and Applications to Business, Engineering & Science, Hong Kong, China, 2010 pp. 584-587.[Cross Ref]
[11] V. Vinothina, "Scheduling scientific workflow tasks in cloud using swarm intelligence," 2017 IEEE International Conference on Current Trends in Advanced Computing (ICCTAC), 2017, pp. 1-5.[Cross Ref]
[12] J. Agarkhed and R. Ashalatha, "Optimal workflow scheduling for scientific workflows in cloud computing," 2016 International Conference on Emerging Technological Trends (ICETT), 2016, pp. 1-6.[Cross Ref]
[13] J. Abawajy, "Job Scheduling Policy for High Throughput Computing Environments," in Proceedings of the Ninth International Conference on Parallel and Distributed Systems, Taiwan, China, 2002 pp. 605.[Cross Ref]
[14] D. Kumar, Z. Shae and H. Jamjoom, "Scheduling Batch and Heterogeneous Jobs with Runtime Elasticity in a Parallel Processing Environment," 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, 2012, pp. 65-78. 10.1109/IPDPSW.2012.10 . [Cross Ref]
[15] S. Rana, A. Choudhary and K. J. Mathai, "A critical analysis of workflow scheduling algorithms in infrastructure as a Serivce Cloud and its research issues," 2016 IEEE Students' Conference on Electrical, Electronics and Computer Science (SCEECS), 2016, pp. 1-6.[Cross Ref]
[16] Y. -T. Chou, S. -J. Liu, T. -C. Wu, C. -L. Wu, C. -W. Tsai and M. -C. Chiang, "An Effective Algorithm for Cloud Workflow Scheduling," 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2018, pp. 3603-3608.[Cross Ref]
[17] R. Balasubramonian and N. Madan, "Power Efficient Approaches to Redundant Multithreading" in IEEE Transactions on Parallel & Distributed Systems, vol. 16, no. 08, pp. 1066-1079, 2007.[Cross Ref]
[18] Docker Manuals on Docker Hub Quickstart. Accessed on: Feb. 20, 2022 [Online] Available: https://docs.docker.com/docker-hub/.[Cross Ref]
[19] AWS documentation Installation of AWS CLI. Accessed on: Feb. 20, 2022 [Online] Available:https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html retrieve from www.aws.amazon.com [Cross Ref]
[20] AWS documentation on docker installation on Linux AMI. Accessed on: Feb. 20, 2022 [Online] Available: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html [Cross Ref]
[21] AWS documentation on AWS CloudFormation. Accessed on: Feb. 20, 2022 [Online] Available: https://docs.aws.amazon.com/cfn-guard/latest/ug/cfn-guard.pdf [Cross Ref]
[22] GitHub documentation on Publishing Docker images. Accessed on: Feb. 20, 2022 [Online] Available: https://docs.github.com/en/actions/publishing-packages/publishing-docker-images [Cross Ref]
[23] AWS documentation on AWS Batch API Reference. Accessed on: Feb. 20, 2022 [Online] Available:https://docs.aws.amazon.com/batch/latest/APIReference/batch-api.pdf [Cross Ref]
[24] AWS documentation on AWS Batch -Array Jobs. Accessed on: Feb. 20, 2022 [Online] Available: https://docs.aws.amazon.com/batch/latest/userguide/array_jobs.html [Cross Ref]
[25] Amazon EC2, On-Demand Pricing. Accessed on: Feb. 20, 2022 [Online] Available: https://aws.amazon.com/ec2/pricing/on-demand/[Cross Ref]
[26] Cloud Storage on AWS, Amazon S3 pricing. Accessed on: Feb. 20, 2022 [Online] Available: https://aws.amazon.com/s3/pricing/ [Cross Ref]
Nagresh Kumar and Sanjay Kumar Sharma (2022), A Cost-Effective and Scalable Processing of Heavy Workload with AWS Batch . IJEER 10(2), 144-149. DOI: 10.37391/IJEER.100216.