流式计算[Flink Beam Spark]

Flink vs. Spark#

特性	Flink	Spark
本质	event	batch
State	Keyed State(Value, List, Map) Operator State(List)	RDD
算子[What]	Source Transform Sink	Transformation Action
API [What]	1. Table 2. DataStream 3.DataSet	1.SparkSQL(DataFrame， DataSet) 2. Spark Streaming
Time[When]	Event Time Ingestion Time Processing Time	无
Windows[Where]	Tumbling Sliding Session	Sliding batchDuration,windowDuration,slideDuration
可靠性	Savepoint External Checkpoint	linage master checkpoint

Beam vs. Flink vs. Spark [6]#

• What

	Beam model	Flink	Spark
ParDo	√	√	√
GroupByKey	√	√	~
Flatten	√	√	√
Combine	√	√	√
Composite Transfrom	√	~	~
Side Inputs	√	~	~
Source API	√	~	√
Aggregators	~	~	~
Keyed States	×	×	×

• Where

	Beam model	Flink	Spark
Global windows	√	√	√
Fixed windows	√	√	~
Sliding windows	√	√	×
Session windows	√	√	×
Custom windows	√	√	×
Custom merging windows	√	√	×
Timestamp control	√	√	×

• When

	Beam model	Flink	Spark
Configurable triggering	√	√	×
Event-time triggers	√	√	×
Proccessing-time triggers	√	√	√
Count triggers	√	√	×
[Meta]data driven triggers	×	×	×
Composite triggers	√	√	×
Allowed lateness	√	√	×
Timers	×	×	×

• How

	Beam model	Flink	Spark
Discarding	√	√	√
Accumulating	√	√	×
Accumulating @ Retracting	×	×	×

参考#

1. Apache Flink状态管理和容错机制介绍
2. Streaming System 第二章：The What- Where- When- and How of Data Processing 未
3. Streaming System 第三章：Watermarks 未
4. xxx
5. <<Spark大数据处理技术>> 10.2节
6. Apache Beam是什么？