Efficient Dynamic Cost Scheduling Algorithm for Data Batch Processing

Abstract

Batch scheduling and processing play a critical role in many manufacturing and service industries. They are widely used in service industries such as banking to process data which makes them of great importance since data communication, monitoring and execution are essential whether they are done online or offline. Batch processing is defined as the execution of a set of required tasks within a specific time frame without violating predecessors' requirements and constraints set by the client. The goal is to achieve the agreed service level contracted with clients using the minimum amount of resources. This research investigates the scheduling problem of processing a set of tasks of non-identical sizes and priority using a set of processors. The objective is to minimize the data batch processing cost while taking into consideration the available resources and the tasks predecessors and constraints. Different types of costs will be included which are: servers and software basic leasing cost, rental cost for additional resources needed in case of overload and extra work, penalty cost of failing to execute the batch process as per the Service Level Agreement (SLA), and the opportunity cost representing the cost of idling a resource for any period of time due to inefficient task allocation. An iterative algorithm with an optimization model at each iteration was developed to optimize the data batching process while minimizing the aforementioned costs. A sensitivity analysis is conducted by varying the main model parameters, one at a time to study their impact on the total cost and the problem under study. Also, different network sizes and complexities were tested to study the effectiveness of the developed algorithm. It was found that it is more effective to include all types of costs in one optimization model along with priority, weight, predecessor and time factors. The algorithm proved its effectiveness by allocating files with higher priority and weight prior to other files while taking into consideration time and different types of costs which led to lower batch process total cost. It is recommended that penalty cost and extra processors different costs should be negotiated thoroughly between stakeholders prior to signing the SLA since it was found that those costs affect the time and number of rented extra processors which consequently affects the whole batch process.