SUMMARY.adoc

Summary

1. Introduction

2. Overview of Apache Spark

Spark SQL

1. Spark SQL — Queries Over Structured Data on Massive Scale

2. SparkSession — The Entry Point to Spark SQL

   1. Builder — Building SparkSession using Fluent API

   2. SharedState — Shared State Across SparkSessions

3. Dataset — Strongly-Typed Structured Query with Encoder

   1. Encoders — Internal Row Converters

   2. ExpressionEncoder — Expression-Based Encoder

   3. LocalDateTimeEncoder — Custom ExpressionEncoder for java.time.LocalDateTime

   4. DataFrame — Dataset of Rows

      1. Row

      2. RowEncoder — Encoder for DataFrames

4. Schema — Structure of Data

   1. StructType

   2. StructField

   3. Data Types

5. Dataset Operators

   1. Column Expressions and Operators

   2. TypedColumn Expressions

   3. Standard Functions — functions Object

      1. Standard Functions for Date and Time

      2. Window Aggregate Functions

   4. User-Defined Functions (UDFs)

   5. Basic Aggregation — Typed and Untyped Grouping Operators

      1. RelationalGroupedDataset — Untyped Row-based Grouping

      2. KeyValueGroupedDataset — Typed Grouping

   6. Joins

      1. Broadcast Joins (aka Map-Side Joins)

   7. Multi-Dimensional Aggregation

   8. UserDefinedAggregateFunction — Contract for User-Defined Aggregate Functions (UDAFs)

   9. Dataset Caching and Persistence

      1. User-Friendly Names Of Cached Queries in web UI’s Storage Tab

6. DataSource API — Loading and Saving Datasets

   1. DataFrameReader — Reading Datasets from External Data Sources

   2. DataFrameWriter

   3. DataSource — Pluggable Data Provider Framework

      1. CreatableRelationProvider — Data Sources That Save Rows Per Save Mode

      2. RelationProvider — Data Sources With Schema Inference

      3. SchemaRelationProvider — Data Sources With Mandatory User-Defined Schema

   4. DataSourceRegister

      1. CSVFileFormat

      2. JdbcRelationProvider

      3. JsonFileFormat

      4. JsonDataSource

      5. ParquetFileFormat

   5. Custom Formats

7. CacheManager — In-Memory Cache for Tables and Views

8. BaseRelation — Collection of Tuples with Schema

   1. HadoopFsRelation

   2. JDBCRelation

9. QueryExecution — Query Execution of Dataset

10. Spark SQL’s Performance Tuning Tips and Tricks (aka Case Studies)

    1. Number of Partitions for groupBy Aggregation

11. Expression — Executable Node in Catalyst Tree

    1. AggregateExpression — Expression Container for AggregateFunction

    2. AggregateFunction

       1. DeclarativeAggregate

       2. ImperativeAggregate — Contract for Aggregate Function Expressions with Imperative Methods

       3. TypedImperativeAggregate — Contract for Imperative Aggregate Functions with Custom Aggregation Buffer

    3. Attribute Leaf Expression

    4. BoundReference Leaf Expression — Reference to Value in InternalRow

    5. CallMethodViaReflection Expression

    6. Generator — Catalyst Expressions that Generate Zero Or More Rows

    7. JsonToStructs Unary Expression

    8. Literal Leaf Expression

    9. ScalaUDAF — Catalyst Expression Adapter for UserDefinedAggregateFunction

    10. StaticInvoke Non-SQL Expression

    11. TimeWindow Unevaluable Unary Expression

    12. UnixTimestamp TimeZoneAware Binary Expression

    13. WindowExpression Unevaluable Expression

        1. WindowSpecDefinition Unevaluable Expression

    14. WindowFunction

        1. AggregateWindowFunction

        2. OffsetWindowFunction

        3. SizeBasedWindowFunction

12. LogicalPlan — Logical Query Plan / Logical Operator

    1. Aggregate Unary Logical Operator

    2. BroadcastHint Unary Logical Operator

    3. DeserializeToObject Logical Operator

    4. Expand Unary Logical Operator

    5. GroupingSets Unary Logical Operator

    6. Hint Logical Operator

    7. InMemoryRelation Leaf Logical Operator For Cached Query Plans

    8. Join Logical Operator

    9. LocalRelation Logical Operator

    10. LogicalRelation Logical Operator — Adapter for BaseRelation

    11. Pivot Unary Logical Operator

    12. Repartition Logical Operators — Repartition and RepartitionByExpression

    13. RunnableCommand — Generic Logical Command with Side Effects

        1. AlterViewAsCommand Logical Command

        2. ClearCacheCommand Logical Command

        3. CreateDataSourceTableCommand Logical Command

        4. CreateViewCommand Logical Command

        5. ExplainCommand Logical Command

    14. SubqueryAlias Logical Operator

    15. UnresolvedFunction Logical Operator

    16. UnresolvedRelation Logical Operator

    17. Window Unary Logical Operator

    18. WithWindowDefinition Unary Logical Operator

13. Analyzer — Logical Query Plan Analyzer

    1. CheckAnalysis — Analysis Validation

    2. ResolveWindowFrame Logical Evaluation Rule

    3. WindowsSubstitution Logical Evaluation Rule

14. SparkOptimizer — Logical Query Optimizer

    1. Optimizer — Base for Logical Query Plan Optimizers

    2. ColumnPruning

    3. CombineTypedFilters

    4. ConstantFolding

    5. CostBasedJoinReorder

    6. DecimalAggregates

    7. EliminateSerialization

    8. GetCurrentDatabase / ComputeCurrentTime

    9. LimitPushDown

    10. NullPropagation — Nullability (NULL Value) Propagation

    11. PropagateEmptyRelation

    12. PushDownPredicate — Predicate Pushdown / Filter Pushdown Logical Plan Optimization

    13. ReorderJoin

    14. SimplifyCasts

15. SparkPlan — Physical Query Plan / Physical Operator

    1. BroadcastExchangeExec Unary Operator for Broadcasting Joins

    2. BroadcastHashJoinExec Binary Physical Operator

    3. BroadcastNestedLoopJoinExec Binary Physical Operator

    4. CoalesceExec Unary Physical Operator

    5. DataSourceScanExec — Contract for Leaf Physical Operators with Code Generation

       1. FileSourceScanExec Physical Operator

       2. RowDataSourceScanExec Physical Operator

    6. ExecutedCommandExec Physical Operator

    7. HashAggregateExec Aggregate Physical Operator for Hash-Based Aggregation

    8. InMemoryTableScanExec Physical Operator

    9. LocalTableScanExec Physical Operator

    10. ObjectHashAggregateExec Aggregate Physical Operator

    11. ShuffleExchange Unary Physical Operator

    12. ShuffledHashJoinExec Binary Physical Operator

    13. SortAggregateExec Aggregate Physical Operator for Sort-Based Aggregation

    14. SortMergeJoinExec Binary Physical Operator

    15. InputAdapter Unary Physical Operator

    16. WindowExec Unary Physical Operator

        1. AggregateProcessor

        2. WindowFunctionFrame

    17. WholeStageCodegenExec Unary Operator with Java Code Generation

16. Partitioning — Specification of Physical Operator’s Output Partitions

17. SparkPlanner — Query Planner with no Hive Support

    1. SparkStrategy — Base for Execution Planning Strategies

    2. SparkStrategies — Container of Execution Planning Strategies

    3. Aggregation Execution Planning Strategy for Aggregate Physical Operators

    4. BasicOperators Execution Planning Strategy

    5. DataSourceStrategy Execution Planning Strategy

    6. FileSourceStrategy Execution Planning Strategy

    7. InMemoryScans Execution Planning Strategy

    8. JoinSelection Execution Planning Strategy

18. Physical Plan Preparations Rules

    1. CollapseCodegenStages Physical Preparation Rule — Collapsing Physical Operators for Whole-Stage CodeGen

    2. EnsureRequirements Physical Preparation Rule

19. SQL Parsing Framework

    1. SparkSqlParser — Default SQL Parser

       1. SparkSqlAstBuilder

    2. CatalystSqlParser — DataTypes and StructTypes Parser

    3. AstBuilder — ANTLR-based SQL Parser

    4. AbstractSqlParser — Base SQL Parsing Infrastructure

    5. ParserInterface — SQL Parser Contract

20. SQLMetric — Physical Operator Metric

21. Catalyst — Tree Manipulation Framework

    1. TreeNode — Node in Catalyst Tree

    2. QueryPlan — Structured Query Plan

    3. RuleExecutor — Tree Transformation Rule Executor

    4. GenericStrategy

    5. QueryPlanner — Converting Logical Plan to Physical Trees

    6. Catalyst DSL — Implicit Conversions for Catalyst Data Structures

22. ExchangeCoordinator and Adaptive Query Execution

23. ShuffledRowRDD

24. Debugging Query Execution

25. Datasets vs DataFrames vs RDDs

26. SQLConf

    1. CatalystConf

27. Catalog

    1. CatalogImpl

28. ExternalCatalog — System Catalog of Permanent Entities

29. SessionState

    1. BaseSessionStateBuilder — Base for Builders of SessionState

30. SessionCatalog — Metastore of Session-Specific Relational Entities

31. UDFRegistration

32. FunctionRegistry

33. ExperimentalMethods

34. SQLExecution Helper Object

35. CatalystSerde

36. Tungsten Execution Backend (aka Project Tungsten)

    1. Whole-Stage Code Generation (CodeGen)

    2. CodegenSupport — Physical Operators with Optional Java Code Generation

    3. InternalRow — Abstract Binary Row Format

       1. UnsafeRow — Mutable Raw-Memory Unsafe Binary Row Format

    4. CodeGenerator

    5. UnsafeProjection — Generic Function to Project InternalRows to UnsafeRows

       1. GenerateUnsafeProjection

    6. BytesToBytesMap Append-Only Hash Map

37. UnsafeHashedRelation

38. ExternalAppendOnlyUnsafeRowArray — Append-Only Array for UnsafeRows (with Disk Spill Threshold)

39. AggregationIterator — Generic Iterator of UnsafeRows for Aggregate Physical Operators

    1. TungstenAggregationIterator — Iterator of UnsafeRows for HashAggregateExec Physical Operator

40. JdbcDialect

41. HadoopFileLinesReader

42. KafkaWriter — Writing Dataset to Kafka

    1. KafkaSourceProvider

    2. KafkaWriteTask

43. Hive Integration

    1. Spark SQL CLI — spark-sql

    2. DataSinks Strategy

44. Thrift JDBC/ODBC Server — Spark Thrift Server (STS)

    1. SparkSQLEnv

45. (obsolete) SQLContext

46. Settings

Spark MLlib

1. Spark MLlib — Machine Learning in Spark

2. ML Pipelines and PipelineStages (spark.ml)

   1. ML Pipeline Components — Transformers

      1. Tokenizer

   2. ML Pipeline Components — Estimators

   3. ML Pipeline Models

   4. Evaluators

   5. CrossValidator

   6. Params and ParamMaps

   7. ML Persistence — Saving and Loading Models and Pipelines

   8. Example — Text Classification

   9. Example — Linear Regression

3. Latent Dirichlet Allocation (LDA)

4. Vector

5. LabeledPoint

6. Streaming MLlib

7. GeneralizedLinearRegression

Structured Streaming

1. Spark Structured Streaming — Streaming Datasets

Spark Core / Tools

1. Spark Shell — spark-shell shell script

2. Web UI — Spark Application’s Web Console

   1. Jobs Tab

   2. Stages Tab — Stages for All Jobs

      1. Stages for All Jobs

      2. Stage Details

      3. Pool Details

   3. Storage Tab

      1. BlockStatusListener Spark Listener

   4. Environment Tab

      1. EnvironmentListener Spark Listener

   5. Executors Tab

      1. ExecutorsListener Spark Listener

   6. SQL Tab

      1. SQLListener Spark Listener

   7. JobProgressListener Spark Listener

   8. StorageStatusListener Spark Listener

   9. StorageListener — Spark Listener for Tracking Persistence Status of RDD Blocks

   10. RDDOperationGraphListener Spark Listener

   11. SparkUI

3. Spark Submit — spark-submit shell script

   1. SparkSubmitArguments

   2. SparkSubmitOptionParser — spark-submit’s Command-Line Parser

   3. SparkSubmitCommandBuilder Command Builder

4. spark-class shell script

   1. AbstractCommandBuilder

5. SparkLauncher — Launching Spark Applications Programmatically

Spark Core / Architecture

1. Spark Architecture

2. Driver

3. Executor

   1. TaskRunner

   2. ExecutorSource

4. Master

5. Workers

Spark Core / RDD

1. Anatomy of Spark Application

2. SparkConf — Programmable Configuration for Spark Applications

   1. Spark Properties and spark-defaults.conf Properties File

   2. Deploy Mode

3. SparkContext

   1. HeartbeatReceiver RPC Endpoint

   2. Inside Creating SparkContext

   3. ConsoleProgressBar

   4. SparkStatusTracker

   5. Local Properties — Creating Logical Job Groups

4. RDD — Resilient Distributed Dataset

   1. RDD Lineage — Logical Execution Plan

   2. TaskLocation

   3. ParallelCollectionRDD

   4. MapPartitionsRDD

   5. OrderedRDDFunctions

   6. CoGroupedRDD

   7. SubtractedRDD

   8. HadoopRDD

   9. NewHadoopRDD

   10. ShuffledRDD

   11. BlockRDD

5. Operators

   1. Transformations

      1. PairRDDFunctions

   2. Actions

6. Caching and Persistence

   1. StorageLevel

7. Partitions and Partitioning

   1. Partition

   2. Partitioner

      1. HashPartitioner

8. Shuffling

9. Checkpointing

   1. CheckpointRDD

10. RDD Dependencies

    1. NarrowDependency — Narrow Dependencies

    2. ShuffleDependency — Shuffle Dependencies

11. Map/Reduce-side Aggregator

Spark Core / Optimizations

1. Broadcast variables

2. Accumulators

   1. AccumulatorContext

Spark Core / Services

1. SerializerManager

2. MemoryManager — Memory Management

   1. UnifiedMemoryManager

3. SparkEnv — Spark Runtime Environment

4. DAGScheduler — Stage-Oriented Scheduler

   1. Jobs

   2. Stage — Physical Unit Of Execution

      1. ShuffleMapStage — Intermediate Stage in Execution DAG

      2. ResultStage — Final Stage in Job

      3. StageInfo

   3. DAGScheduler Event Bus

   4. JobListener

      1. JobWaiter

5. TaskScheduler — Spark Scheduler

   1. Tasks

      1. ShuffleMapTask — Task for ShuffleMapStage

      2. ResultTask

   2. TaskDescription

   3. FetchFailedException

   4. MapStatus — Shuffle Map Output Status

   5. TaskSet — Set of Tasks for Stage

   6. TaskSetManager

      1. Schedulable

      2. Schedulable Pool

      3. Schedulable Builders

         1. FIFOSchedulableBuilder

         2. FairSchedulableBuilder

      4. Scheduling Mode — spark.scheduler.mode Spark Property

      5. TaskInfo

   7. TaskSchedulerImpl — Default TaskScheduler

      1. Speculative Execution of Tasks

      2. TaskResultGetter

   8. TaskContext

      1. TaskContextImpl

   9. TaskResults — DirectTaskResult and IndirectTaskResult

   10. TaskMemoryManager

       1. MemoryConsumer

   11. TaskMetrics

       1. ShuffleWriteMetrics

   12. TaskSetBlacklist — Blacklisting Executors and Nodes For TaskSet

6. SchedulerBackend — Pluggable Scheduler Backends

   1. CoarseGrainedSchedulerBackend

      1. DriverEndpoint — CoarseGrainedSchedulerBackend RPC Endpoint

7. ExecutorBackend — Pluggable Executor Backends

   1. CoarseGrainedExecutorBackend

   2. MesosExecutorBackend

8. BlockManager — Key-Value Store for Blocks

   1. MemoryStore

   2. DiskStore

   3. BlockDataManager

   4. ShuffleClient

   5. BlockTransferService — Pluggable Block Transfers

      1. NettyBlockTransferService — Netty-Based BlockTransferService

      2. NettyBlockRpcServer

   6. BlockManagerMaster — BlockManager for Driver

      1. BlockManagerMasterEndpoint — BlockManagerMaster RPC Endpoint

   7. DiskBlockManager

   8. BlockInfoManager

      1. BlockInfo

   9. BlockManagerSlaveEndpoint

   10. DiskBlockObjectWriter

   11. BlockManagerSource — Metrics Source for BlockManager

   12. StorageStatus

9. MapOutputTracker — Shuffle Map Output Registry

   1. MapOutputTrackerMaster — MapOutputTracker For Driver

      1. MapOutputTrackerMasterEndpoint

   2. MapOutputTrackerWorker — MapOutputTracker for Executors

10. ShuffleManager — Pluggable Shuffle Systems

    1. SortShuffleManager — The Default Shuffle System

    2. ExternalShuffleService

    3. OneForOneStreamManager

    4. ShuffleBlockResolver

       1. IndexShuffleBlockResolver

    5. ShuffleWriter

       1. BypassMergeSortShuffleWriter

       2. SortShuffleWriter

       3. UnsafeShuffleWriter — ShuffleWriter for SerializedShuffleHandle

    6. BaseShuffleHandle — Fallback Shuffle Handle

    7. BypassMergeSortShuffleHandle — Marker Interface for Bypass Merge Sort Shuffle Handles

    8. SerializedShuffleHandle — Marker Interface for Serialized Shuffle Handles

    9. ShuffleReader

       1. BlockStoreShuffleReader

    10. ShuffleBlockFetcherIterator

    11. ShuffleExternalSorter — Cache-Efficient Sorter

    12. ExternalSorter

11. Serialization

    1. Serializer — Task SerDe

    2. SerializerInstance

    3. SerializationStream

    4. DeserializationStream

12. ExternalClusterManager — Pluggable Cluster Managers

13. BroadcastManager

    1. BroadcastFactory — Pluggable Broadcast Variable Factories

       1. TorrentBroadcastFactory

       2. TorrentBroadcast

    2. CompressionCodec

14. ContextCleaner — Spark Application Garbage Collector

    1. CleanerListener

15. Dynamic Allocation (of Executors)

    1. ExecutorAllocationManager — Allocation Manager for Spark Core

    2. ExecutorAllocationClient

    3. ExecutorAllocationListener

    4. ExecutorAllocationManagerSource

16. HTTP File Server

17. Data Locality

18. Cache Manager

19. OutputCommitCoordinator

20. RpcEnv — RPC Environment

    1. RpcEndpoint

    2. RpcEndpointRef

    3. RpcEnvFactory

    4. Netty-based RpcEnv

21. TransportConf — Transport Configuration

(obsolete) Spark Streaming

1. Spark Streaming — Streaming RDDs

Spark Deployment Environments

1. Deployment Environments — Run Modes

2. Spark local (pseudo-cluster)

   1. LocalSchedulerBackend

   2. LocalEndpoint

3. Spark on cluster

Spark on YARN

1. Spark on YARN

2. YarnShuffleService — ExternalShuffleService on YARN

3. ExecutorRunnable

4. Client

5. YarnRMClient

6. ApplicationMaster

   1. AMEndpoint — ApplicationMaster RPC Endpoint

7. YarnClusterManager — ExternalClusterManager for YARN

8. TaskSchedulers for YARN

   1. YarnScheduler

   2. YarnClusterScheduler

9. SchedulerBackends for YARN

   1. YarnSchedulerBackend

   2. YarnClientSchedulerBackend

   3. YarnClusterSchedulerBackend

   4. YarnSchedulerEndpoint RPC Endpoint

10. YarnAllocator

11. Introduction to Hadoop YARN

12. Setting up YARN Cluster

13. Kerberos

    1. ConfigurableCredentialManager

14. ClientDistributedCacheManager

15. YarnSparkHadoopUtil

16. Settings

Spark Standalone

1. Spark Standalone

2. Standalone Master

3. Standalone Worker

4. web UI

5. Submission Gateways

6. Management Scripts for Standalone Master

7. Management Scripts for Standalone Workers

8. Checking Status

9. Example 2-workers-on-1-node Standalone Cluster (one executor per worker)

10. StandaloneSchedulerBackend

Spark on Mesos

1. Spark on Mesos

2. MesosCoarseGrainedSchedulerBackend

3. About Mesos

Execution Model

1. Execution Model

Security

1. Spark Security

2. Securing Web UI

Spark Core / Data Sources

1. Data Sources in Spark

2. Using Input and Output (I/O)

   1. Parquet

3. Spark and Cassandra

4. Spark and Kafka

5. Couchbase Spark Connector

(obsolete) Spark GraphX

1. Spark GraphX — Distributed Graph Computations

2. Graph Algorithms

Monitoring, Tuning and Debugging

1. Unified Memory Management

2. Spark History Server

   1. HistoryServer

   2. SQLHistoryListener

   3. FsHistoryProvider

   4. HistoryServerArguments

3. Logging

4. Performance Tuning

5. MetricsSystem

   1. MetricsConfig — Metrics System Configuration

   2. Metrics Source

   3. Metrics Sink

6. SparkListener — Intercepting Events from Spark Scheduler

   1. LiveListenerBus

   2. ReplayListenerBus

   3. SparkListenerBus — Internal Contract for Spark Event Buses

   4. EventLoggingListener — Spark Listener for Persisting Events

   5. StatsReportListener — Logging Summary Statistics

7. JsonProtocol

8. Debugging Spark using sbt

Varia

1. Building Apache Spark from Sources

2. Spark and Hadoop

   1. SparkHadoopUtil

3. Spark and software in-memory file systems

4. Spark and The Others

5. Distributed Deep Learning on Spark

6. Spark Packages

Interactive Notebooks

1. Interactive Notebooks

   1. Apache Zeppelin

   2. Spark Notebook

Spark Tips and Tricks

1. Spark Tips and Tricks

2. Access private members in Scala in Spark shell

3. SparkException: Task not serializable

4. Running Spark Applications on Windows

Exercises

1. One-liners using PairRDDFunctions

2. Learning Jobs and Partitions Using take Action

3. Spark Standalone - Using ZooKeeper for High-Availability of Master

4. Spark’s Hello World using Spark shell and Scala

5. WordCount using Spark shell

6. Your first complete Spark application (using Scala and sbt)

7. Spark (notable) use cases

8. Using Spark SQL to update data in Hive using ORC files

9. Developing Custom SparkListener to monitor DAGScheduler in Scala

10. Developing RPC Environment

11. Developing Custom RDD

12. Working with Datasets from JDBC Data Sources (and PostgreSQL)

13. Causing Stage to Fail

Further Learning

1. Courses

2. Books

Spark Distributions

1. DataStax Enterprise

2. MapR Sandbox for Hadoop (Spark 1.5.2 only)

Spark Workshop

1. Spark Advanced Workshop

   1. Requirements

   2. Day 1

   3. Day 2

Spark Talk Ideas

1. Spark Talks Ideas (STI)

2. 10 Lesser-Known Tidbits about Spark Standalone

3. Learning Spark internals using groupBy (to cause shuffle)