Streamline: Dynamic Auto-Tuning in Apache Flink

, ,

I have spent a lot of time working with distributed systems, and one challenge consistently stands out: managing unbounded data streams. Whether you are dealing with smart cities, healthcare sensors, or V2X communications, the data workloads often fluctuate dynamically and unpredictably.

In practice, I noticed engineers were constantly forced into a frustrating, manual compromise. When relying on static configurations, you either over-provision your resources and burn through your infrastructure budget, or you under-provision and watch your pipeline choke under backpressure.

That is exactly why I built Streamline. I wanted to build a multi-layer auto-tuning framework that doesn’t just react to problems, but anticipates them.

Quick Start & Installation

Getting Streamline running locally or on your cluster is straightforward.

1
2
3
4
5
6
7
8
9
10
# Clone repository
git clone https://github.com/stefanpedratscher/streamline
cd streamline

# Create and activate virtual environment
python3 -m venv env
source env/bin/activate

# Install dependencies
pip install -r requirements.txt

You can test the implementation using the provided examples in the /examples directory:

1
2
3
4
5
# Run an example of a single data pipeline
python examples/single_pipeline.py

# Run an example of an ensemble of data pipelines
python examples/ensemble.py

Step-by-Step: How Streamline Works

Step 1: Model Training

Before touching any live application, Streamline trains its underlying machine learning models. It pre-trains a Time Series Transformer (for workload prediction), a Neural Network (as a surrogate to estimate Apache Flink performance), and a Polynomial Regression model (to map max processing limits).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
INFO: Starting Workload Predictor training using Transformers...
INFO: Epoch 0/200, Loss: 1.2165
INFO: Epoch 50/200, Loss: 0.0241
INFO: Epoch 100/200, Loss: 0.0127
INFO: Epoch 150/200, Loss: 0.0076
INFO: Epoch 200/200, Loss: 0.0052
INFO: Starting Neural Network training...
INFO: Epoch 0/200, Train Loss: 0.0946, Val Loss: 0.0582
INFO: Epoch 50/200, Train Loss: 0.0045, Val Loss: 0.0079
INFO: Epoch 100/200, Train Loss: 0.0035, Val Loss: 0.0065
INFO: Epoch 150/200, Train Loss: 0.0028, Val Loss: 0.0065
INFO: Epoch 200/200, Train Loss: 0.0027, Val Loss: 0.0075
INFO: R^2 Score: Train=0.96, Val=0.90
INFO: Starting Polynomial Regression training...
INFO: Finished training for 0: ./polynomial_regeression_0.png
INFO: Finished training for 1: ./polynomial_regeression_1.png
INFO: Finished training for 2: ./polynomial_regeression_2.png
INFO: Finished training for 3: ./polynomial_regeression_3.png

Step 2: Proactive Prediction & Auto-Tuning

When a new application is submitted, Streamline predicts the incoming workload for the source operators using the Transformer model, and calculates downstream loads using historical aspect ratios.

img

It then feeds these predictions into an NSGA-II evolutionary algorithm. Using the Neural Network as a surrogate evaluator, it finds optimized configuration parameters (parallelism and buffer-size) and estimates cpu, end-to-end-latency, and throughput for each operator.

1
2
3
4
5
INFO: Predicting workload of operators using Transformers and aspect ratio...
INFO: Best configuration for v_0: {'parallelism': 3.0, 'segment-size': 16384.0, 'est_tp': 1614.5908279224252, 'est_l': 170.6463427734375, 'est_cpu': 1.72125346660614}.
INFO: Best configuration for v_1: {'parallelism': 3.0, 'segment-size': 16384.0, 'est_tp': 834.103070678711, 'est_l': 96.64015594482422, 'est_cpu': 1.3095434522628784}.
INFO: Best configuration for v_2: {'parallelism': 2.0, 'segment-size': 16384.0, 'est_tp': 834.103070678711, 'est_l': 128.06282302856445, 'est_cpu': 1.0709905421733856}.
INFO: Best configuration for v_3: {'parallelism': 2.0, 'segment-size': 16384.0, 'est_tp': 1319.449833651006, 'est_l': 111.70156860351562, 'est_cpu': 1.601874589920044}.

Step 3: Resource-Aware Scheduling

With the configurations set, the scheduler dynamically maps the operator instances to your available compute resources (TaskManagers). 

1
2
INFO: Total required CPU: 5.703662050962448
{'R11': ['pipeline_1_v_3_0', 'pipeline_1_v_3_1', 'pipeline_1_v_0_0', 'pipeline_1_v_0_1', 'pipeline_1_v_0_2', 'pipeline_1_v_2_0', 'pipeline_1_v_2_1', 'pipeline_1_v_1_0', 'pipeline_1_v_1_1', 'pipeline_1_v_1_2']}

Step 4: Periodic Re-evaluation (Dynamic Adaptation)

Streamline runs at regular intervals to verify if the applications need updates based on shifting data streams. For instance, if the workload of the application drops, Streamline automatically scales down its parallelism (from 4.0 to 2.0) to save resources, while leaving other applications untouched.

Evaluation

Streamline was evaluated on the Grid 5000 testbed using real-world IoT and streaming benchmarks.

Metric Maximum Improvement
End-to-end Latency 10x
Monetary Costs 10x
CPU Utilization 9x
Throughput 4x

If you are dealing with large-scale, volatile data stream applications, or are interested in a more in depth description of Streamline or its evaluation results, I encourage you to read the full paper or explore the Streamline repository on GitHub.