Optimization and Management of Scientific Workflows in Distributed Network Environments: From Theory to Practice
SESSION: Doctoral Research Showcase II (Workflows and Parallel and Distributed IO Optimization)
EVENT TYPE: Doctoral Research Showcase
TIME: 1:48PM - 2:06PM
SESSION CHAIR: Sadaf R. Alam
ABSTRACT: Mapping large-scale scientific workflows in distributed environments and optimizing their end-to-end performance in terms of latency and throughput are crucial to the success of collaborative applications that require fast system response and smooth data flow. We construct analytical cost models and formulate workflow mapping as optimization problems for minimum end-to-end delay (MED) and maximum frame rate (MFR). The difficulty of these problems arises from the topological matching nature in the spatial domain, which is further compounded by the resource sharing complicacy in the temporal dimension. For MED, we propose an efficient algorithm to compute the exact end-to-end delay of a mapped workflow with arbitrary node reuse and develop a distributed workflow mapping scheme based on a recursive critical path optimization procedure; while for MFR, we conduct a rigorous workflow stability analysis and develop a distributed layer-oriented dynamic programming solution based on topological sorting to identify and minimize the global bottleneck.
Sadaf R. Alam (Chair) - Swiss National Supercomputing Centre