Distributed Data Locality-Aware Job Allocation

TY - GEN

T1 - Distributed Data Locality-Aware Job Allocation

AU - Markovic, Ana

AU - Kolovos, Dimitris

AU - Soares Indrusiak, Leandro

N1 - Publisher Copyright: © 2023 ACM. This is an author-produced version of the published paper. Uploaded in accordance with the University’s Research Publications and Open Access policy.

PY - 2023/11/12

Y1 - 2023/11/12

N2 - Scheduling tasks close to their associated data is crucial in distributed systems to minimize network traffic and latency. Some Big Data frameworks like Apache Spark employ locality functions and job allocation algorithms to minimize network traffic and execution times. However, these frameworks rely on centralized mechanisms, where the master node determines data locality by allocating tasks to available workers with minimal data transfer time, ignoring variances in worker configurations and availability. To address these limitations, we propose a decentralized approach to locality-driven scheduling that grants workers autonomy in the job allocation process while factoring in workers' configurations, such as network and CPU speed differences. Our approach is developed and evaluated on Crossflow, a distributed stream processing platform with data-aware independent worker nodes. Preliminary evaluation experiments indicate that our approach can yield up to 3.57x faster execution times when compared to the baseline centralized approach where the master controls data locality.

AB - Scheduling tasks close to their associated data is crucial in distributed systems to minimize network traffic and latency. Some Big Data frameworks like Apache Spark employ locality functions and job allocation algorithms to minimize network traffic and execution times. However, these frameworks rely on centralized mechanisms, where the master node determines data locality by allocating tasks to available workers with minimal data transfer time, ignoring variances in worker configurations and availability. To address these limitations, we propose a decentralized approach to locality-driven scheduling that grants workers autonomy in the job allocation process while factoring in workers' configurations, such as network and CPU speed differences. Our approach is developed and evaluated on Crossflow, a distributed stream processing platform with data-aware independent worker nodes. Preliminary evaluation experiments indicate that our approach can yield up to 3.57x faster execution times when compared to the baseline centralized approach where the master controls data locality.

KW - Big Data processing

KW - Data-aware job scheduling

KW - Distributed processing

KW - Locality scheduler

UR - http://www.scopus.com/inward/record.url?scp=85178157515&partnerID=8YFLogxK

U2 - 10.1145/3624062.3624287

DO - 10.1145/3624062.3624287

M3 - Conference contribution

AN - SCOPUS:85178157515

T3 - ACM International Conference Proceeding Series

SP - 2089

EP - 2096

BT - Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

PB - Association for Computing Machinery, Inc

T2 - 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

Y2 - 12 November 2023 through 17 November 2023

ER -