Apache Spark

Apache Spark

Original author(s)	Matei Zaharia
Developer(s)	Apache Software Foundation, UC Berkeley AMPLab, Databricks
Initial release	May 30, 2014 (2014-05-30)
Stable release	v1.6.0 / January 4, 2016 (2016-01-04)
Development status	Active
Written in	Scala, Java, Python, R
Operating system	Linux, Mac OS, Windows
Type	data analytics, machine learning algorithms
License	Apache License 2.0
Website	spark.apache.org

Apache Spark is an open source cluster computing framework originally developed in the AMPLab at University of California, Berkeley but was later donated to the Apache Software Foundation where it remains today. In contrast to Hadoop's two-stage disk-based MapReduce paradigm, Spark's multi-stage in-memory primitives provides performance up to 100 times faster for certain applications.^[1] By allowing user programs to load data into a cluster's memory and query it repeatedly, Spark is well-suited to machine learning algorithms.^[2]

Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone (native Spark cluster), Hadoop YARN, or Apache Mesos.^[3] For distributed storage, Spark can interface with a wide variety, including Hadoop Distributed File System (HDFS),^[4] Cassandra,^[5] OpenStack Swift, Amazon S3, Kudu, or a custom solution can be implemented. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in such a scenario, Spark is run on a single machine with one executor per CPU core.

Spark had in excess of 1000 contributors in 2015,^[6] making it not only the most active project in the Apache Software Foundation but one of the most active open source big data projects.^[7]

History

Spark was initially started by Matei Zaharia at UC Berkeley AMPLab in 2009, and open sourced in 2010 under a BSD license.

In 2013, the project was donated to the Apache Software Foundation and switched its license to Apache 2.0. In February 2014, Spark became a Top-Level Apache Project.^[8]

In November 2014, the engineering team at Databricks used Spark and set a new world record in large scale sorting.^[9]

Project components

The Spark project consists of multiple components.

Spark Core and Resilient Distributed Datasets

Spark Core is the foundation of the overall project. It provides distributed task dispatching, scheduling, and basic I/O functionalities. The fundamental programming abstraction is called Resilient Distributed Datasets (RDDs), a logical collection of data partitioned across machines. RDDs can be created by referencing datasets in external storage systems, or by applying coarse-grained transformations (e.g. map, filter, reduce, join) on existing RDDs.

An RDD is a collection of items distributed across many nodes that can be manipulated in parallel. Spark Core provides many APIs for building and manipulating these collections. An RDD in Spark is simply an immutable distributed collection of objects. Each RDD is split into multiple partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes. Users can create RDDs in two ways: by loading an external dataset, or by distributing a collection of objects in their driver program. RDDs offer two types of operations after creation: transformations and actions. Transformations construct a new RDD. Actions, on the other hand, compute a result based on an RDD, and either return it to the driver program or save it to an external storage system.

The RDD abstraction is exposed through a language-integrated API in Java, Python, Scala, and R similar to local, in-process collections. This simplifies programming complexity because the way applications manipulate RDDs is similar to manipulating local collections of data.

A Spark cluster is composed of one Driver JVM and one or many Executor JVMs.

Spark SQL

Spark SQL is a component on top of Spark Core that introduces a new data abstraction called DataFrames, which provides support for structured and semi-structured data. Spark SQL provides a domain-specific language to manipulate DataFrames in Scala, Java, or Python. It also provides SQL language support, with command-line interfaces and ODBC/JDBC server. Prior to version 1.3 of Spark, DataFrames were referred to as SchemaRDDs.

Spark Streaming

Spark Streaming leverages Spark Core's fast scheduling capability to perform streaming analytics. It ingests data in mini-batches and performs RDD transformations on those mini-batches of data. This design enables the same set of application code written for batch analytics to be used in streaming analytics, on a single engine.

MLlib Machine Learning Library

Spark MLlib is a distributed machine learning framework on top of Spark Core that, due in large part of the distributed memory-based Spark architecture, is as much as nine times as fast as the disk-based implementation used by Apache Mahout (according to benchmarks done by the MLlib developers against the Alternating Least Squares (ALS) implementations, and before Mahout itself gained a Spark interface), and scales better than Vowpal Wabbit.^[10] Many common machine learning and statistical algorithms have been implemented and are shipped with MLlib which simplifies large scale machine learning pipelines, including:

• summary statistics, correlations, stratified sampling, hypothesis testing, random data generation^[11]
• classification and regression: support vector machines, logistic regression, linear regression, decision trees, naive Bayes classification
• collaborative filtering techniques including alternating least squares (ALS)
• cluster analysis methods including k-means, and Latent Dirichlet Allocation (LDA)
• dimensionality reduction techniques such as singular value decomposition (SVD), and principal component analysis (PCA)
• feature extraction and transformation functions
• optimization primitives such as stochastic gradient descent, limited-memory BFGS (L-BFGS)

GraphX

GraphX is a distributed graph processing framework on top of Spark. It provides an API for expressing graph computation that can model the Pregel abstraction. It also provides an optimized runtime for this abstraction.

Like Spark, GraphX initially started as a research project at UC Berkeley's AMPLab and Databricks, and was later donated to the Apache Software Foundation and the Spark project.^[12]

Features

• APIs and libraries for Java, Scala, Python, R, and other languages.^[13]
• Scalability to over 8000 nodes in production.^[14]
• Ability to cache datasets in memory for interactive data analysis: extract a working set, cache it, query it repeatedly.
• Interactive command line interface (in Scala or Python) for low-latency, horizontally scalable, data exploration.
• Higher level library for stream processing, using Spark Streaming.
• Support for structured and relational query processing (SQL), through Spark SQL.
• Higher level libraries for machine learning and graph processing.

References

^[7]

External links

• Official website
• Spark SQL
• Spark Streaming
• MLlib machine learning library
• GraphX graph processing library

Apache Software Foundation Top level projects Abdera Accumulo ActiveMQ Ambari Ant Aries Apache HTTP Server APR Avro Axis Axis2 Bloodhound Buildr Calcite Camel Cassandra Cayenne Chemistry Click CloudStack Cocoon Continuum Cordova CouchDB cTAKES CXF Deltacloud Derby Directory Drill Empire-db Felix Flex Flink Flume Forrest Geronimo Gora Gump Hadoop Hama HBase Hive Jackrabbit James JMeter Kafka Karaf Kylin Lucene Mahout Marmotta Maven MINA mod_perl MyFaces Nutch ODE OFBiz Oozie OpenEJB OpenJPA OpenNLP OpenOffice PDFBox Phoenix POI Pig Pivot Qpid River Roller Samza ServiceMix Shindig Shiro Sling Spark Stanbol Storm SpamAssassin Sqoop Struts Subversion Tapestry Thrift Tiles Tomcat Traffic Server Turbine Tuscany UIMA Velocity Wave Wicket Wink Xalan Xerces XMLBeans ZooKeeper Commons projects Apache Commons Logging BCEL BSF Daemon Jelly Lucene projects Lucene Java Solr Hadoop projects Hadoop HDFS Other projects Batik Chainsaw FOP Ivy Log4j Log4Net XAP Incubator projects XAP Apache Attic AxKit Beehive Bluesky Cactus Jakarta Excalibur Harmony HiveMind Lenya Slide Shale stdcxx iBATIS Licenses Apache License Category Commons