Apache Flink

Apache Flink

Developer(s)	Apache Software Foundation
Stable release	1.3.2 / 5 August 2017
Development status	Active
Written in	Java and Scala
Operating system	Cross-platform
Type	Data analytics, machine learning algorithms
License	Apache License 2.0
Website	flink.apache.org

Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala.^[1][2] Flink executes arbitrary dataflow programs in a data-parallel and pipelined manner.^[3] Flink's pipelined runtime system enables the execution of bulk/batch and stream processing programs.^[4][5] Furthermore, Flink's runtime supports the execution of iterative algorithms natively.^[6]

Flink provides a high-throughput, low-latency streaming engine^[7] as well as support for event-time processing and state management. Flink applications are fault-tolerant in the event of machine failure and support exactly-once semantics.^[8] Programs can be written in Java, Scala,^[9] Python,^[10] and SQL^[11] and are automatically compiled and optimized^[12] into dataflow programs that are executed in a cluster or cloud environment.^[13]

Flink does not provide its own data storage system and provides data source and sink connectors to systems such as Amazon Kinesis, Apache Kafka, HDFS, Apache Cassandra, and ElasticSearch.^[14]

Development

Apache Flink is developed under the Apache License 2.0^[15] by the Apache Flink Community within the Apache Software Foundation. The project is driven by over 25 committers and over 275 contributors.

data Artisans is a company that was founded by the original creators of Apache Flink.^[16] 12 Apache Flink committers are currently employed by data Artisans.^[17]

Overview

Apache Flink’s dataflow programming model provides event-at-a-time processing on both finite and infinite datasets. At a basic level, Flink programs consist of streams and transformations. “Conceptually, a stream is a (potentially never-ending) flow of data records, and a transformation is an operation that takes one or more streams as input, and produces one or more output streams as a result.”^[18]

Apache Flink includes two core APIs: a DataStream API for bounded or unbounded streams of data and a DataSet API for bounded data sets. Flink also offers a Table API, which is a SQL-like expression language for relational stream and batch processing that can be easily embedded in Flink’s DataStream and DataSet APIs. The highest-level language supported by Flink is SQL, which is semantically similar to the Table API and represents programs as SQL query expressions.

Programming Model and Distributed Runtime

Upon execution, Flink programs are mapped to streaming dataflows.^[18] Every Flink dataflow starts with one or more sources (a data input, e.g. a message queue or a file system) and ends with one or more sinks (a data output, e.g. a message queue, file system, or database). An arbitrary number of transformations can be performed on the stream. These streams can be arranged as a directed, acyclic dataflow graph, allowing an application to branch and merge dataflows.

Flink offers ready-built source and sink connectors with Apache Kafka, Amazon Kinesis, HDFS, Apache Cassandra, and more.^[14]

Flink programs run as a distributed system within a cluster and can be deployed in a standalone mode as well as on YARN, Mesos, Docker-based setups along with other resource management frameworks.^[19]

State: Checkpoints, Savepoints, and Fault-tolerance

Apache Flink includes a lightweight fault tolerance mechanism based on distributed checkpoints.^[8] A checkpoint is an automatic, asynchronous snapshot of the state of an application and the position in a source stream. In the case of a failure, a Flink program with checkpointing enabled will, upon recovery, resume processing from the last completed checkpoint, ensuring that Flink maintains exactly-once state semantics within an application. The checkpointing mechanism exposes hooks for application code to include external systems into the checkpointing mechanism as well (like opening and committing transactions with a database system).

Flink also includes a mechanism called savepoints, which are manually-triggered checkpoints.^[20] A user can generate a savepoint, stop a running Flink program, then resume the program from the same application state and position in the stream. Savepoints enable updates to a Flink program or a Flink cluster without losing the application's state . As of Flink 1.2, savepoints also allow to restart an application with a different parallelism—allowing users to adapt to changing workloads.

DataStream API

Flink’s DataStream API enables transformations (e.g. filters, aggregations, window functions) on bounded or unbounded streams of data. The DataStream API includes more than 20 different types of transformations and is available in Java and Scala.^[21]

A simple example of a stateful stream processing program is an application that emits a word count from a continuous input stream and groups the data in 5-second windows:

import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.api.windowing.time.Time

case class WordCount(word: String, count: Int)

object WindowWordCount {
  def main(args: Array[String]) {

    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val text = env.socketTextStream("localhost", 9999)

    val counts = text.flatMap { _.toLowerCase.split("\\W+") filter { _.nonEmpty } }
      .map { WordCount(_, 1) }
      .keyBy("word")
      .timeWindow(Time.seconds(5))
      .sum("count")

    counts.print

    env.execute("Window Stream WordCount")
  }
}

Apache Beam - Flink Runner

Apache Beam “provides an advanced unified programming model, allowing (a developer) to implement batch and streaming data processing jobs that can run on any execution engine.”^[22] The Apache Flink-on-Beam runner is the most feature-rich according to a capability matrix maintained by the Beam community.^[23]

data Artisans, in conjunction with the Apache Flink community, worked closely with the Beam community to develop a robust Flink runner.^[24]

DataSet API

Flink’s DataSet API enables transformations (e.g. filters, mapping, joining, grouping) on bounded datasets. The DataSet API includes more than 20 different types of transformations.^[25] The API is available in Java, Scala and an experimental Python API. Flink’s DataSet API is conceptually similar to the DataStream API.

Table API and SQL

Flink's Table API is a SQL-like expression language for relational stream and batch processing that can be embedded in Flink’s Java and Scala DataSet and DataStream APIs. The Table API and SQL interface operate on a relational Table abstraction. Tables can be created from external data sources or from existing DataStreams and DataSets. The Table API supports relational operators such as selection, aggregation, and joins on Tables.

Tables can also be queried with regular SQL. The Table API and SQL offer equivalent functionality and can be mixed in the same program. When a Table is converted back into a DataSet or DataStream, the logical plan, which was defined by relational operators and SQL queries, is optimized using Apache Calcite and is transformed into a DataSet or DataStream program.

Flink Forward

Flink Forward is an annual conference about Apache Flink. The first edition of Flink Forward took place in 2015 in Berlin. The two-day conference had over 250 attendees from 16 countries. Sessions were organized in two tracks with over 30 technical presentations from Flink developers and one additional track with hands-on Flink training.

In 2016, 350 participants joined the conference and over 40 speakers presented technical talks in 3 parallel tracks. On the third day, attendees were invited to participate in hands-on training sessions.

In 2017, the event expands to San Francisco, as well. The conference day is dedicated to technical talks on how Flink is used in the enterprise, Flink system internals, ecosystem integrations with Flink, and the future of the platform. It features keynotes, talks from Flink users in industry and academia, and hands-on training sessions on Apache Flink.

Speakers from the following organizations have presented at Flink Forward conferences: Alibaba, Amadeus, Bouygues Telecom, Capital One, Cloudera, data Artisans, EMC, Ericsson, Hortonworks, Huawei, IBM, Google, MapR, MongoDB, Netflix, New Relic, Otto Group, Red Hat, ResearchGate, Uber, and Zalando.^[26][27]

History

In 2010, the research project "Stratosphere: Information Management on the Cloud"^[28] (funded by the German Research Foundation (DFG)^[29]) was started as a collaboration of Technical University Berlin, Humboldt-Universität zu Berlin, and Hasso-Plattner-Institut Potsdam. Flink started from a fork of Stratosphere’s distributed execution engine and it became an Apache Incubator project in March 2014.^[30] In December 2014, Flink was accepted as an Apache top-level project.^{[31][32][33][34]}

Release Dates

• 02/2017: Apache Flink 1.2
• 08/2016: Apache Flink 1.1 (08/2016: v1.1.1; 09/2016 v1.1.2; 10/2016 v1.1.3;; 12/2016 v1.1.4))
• 03/2016: Apache Flink 1.0 (04/2016: v1.0.1; 04/2016: v1.0.2; 05/2016 v1.0.3)
• 11/2015: Apache Flink 0.10 (11/2015: v0.10.1; 02/2016: v0.10.2)
• 06/2015: Apache Flink 0.9 (09/2015: v0.9.1)
  • 04/2015: Apache Flink 0.9-milestone-1

Apache Incubator Release Dates

• 01/2015: Apache Flink 0.8-incubating
• 11/2014: Apache Flink 0.7-incubating
• 08/2014: Apache Flink 0.6-incubating (09/2014: v0.6.1-incubating)
• 05/2014: Stratosphere 0.5 (06/2014: v0.5.1; 07/2014: v0.5.2)

Pre-Apache Stratosphere Release Dates

• 01/2014: Stratosphere 0.4 (version 0.3 was skipped)
• 08/2012: Stratosphere 0.2
• 05/2011: Stratosphere 0.1 (08/2011: v0.1.1)

See also

• List of Apache Software Foundation projects
• Other comparable data processing engines such as Storm and Spark.^[35]
• Apache Beam, a shared programming model for which Flink is a founding backend.

References

External links

• "Data Artisans Blog". 
• "Apache Flink Knowledge Base".  A directory of over 600 categorized Apache Flink web resources
• "Announcing Google Cloud Dataflow runner for Apache Flink". 
• "Apache Flink is now running on Google Cloud Dataflow". 
• "Google Cloud Dataflow bekommt Runner für Apache Flink".  (in German)
• "Flink@Twitter". 
• "Going with the stream: Unbounded data processing with Apache Flink"
• "Apache Flink 1.2 Released with Dynamic Rescaling, Security and Queryable State"
• "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing"

Apache Software Foundation
Top level projects	Accumulo ActiveMQ Ambari Ant Apex Aries Apache HTTP Server APR Avro Axis Axis2 Beam Bloodhound Brooklyn Buildr Calcite Camel Cassandra Cayenne Chemistry CloudStack Cocoon Cordova CouchDB cTAKES CXF Derby Directory Drill Empire-db Felix Flex Flink Flume Forrest Geronimo Gora Gump Hadoop Hama HBase Hive Jackrabbit James Jini JMeter Kafka Kudu Kylin Lucene Mahout Marmotta Maven MINA mod_perl MyFaces Nutch ODE OFBiz Oozie OpenEJB OpenJPA OpenNLP OpenOffice PDFBox Parquet Phoenix POI Pig Pivot Qpid Roller Samza ServiceMix Shiro Sling Solr Spark Stanbol Storm SpamAssassin Sqoop Struts 1 Struts 2 Subversion SystemML Tapestry Thrift Tika Tomcat Traffic Server UIMA Velocity Wave Wicket Xalan Xerces ZooKeeper
Commons	BCEL BSF Daemon Jelly Logging
Incubator	SINGA Trafodion XAP
Other projects	Batik Chainsaw FOP Ivy Log4j
Attic	Abdera AxKit Beehive Bluesky iBATIS Cactus Click Continuum Deltacloud Excalibur Harmony HiveMind Jakarta Lenya Shale Shindig Slide stdcxx Tuscany Wink XMLBeans
Licenses	Apache License
Category