builder

Interface for building the session configuration.

Syntax

builder

Methods

Method Description
appName(name) Sets a name for the application, which is shown in the Spark web UI.
create() Creates a new SparkSession.
config(key, value, conf, map) Sets a config option. Options are automatically propagated to both SparkConf and SparkSession's own configuration.
enableHiveSupport() Enables Hive support, including connectivity to a persistent Hive metastore.
getOrCreate() Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.
master(master) Sets the Spark master URL to connect to.
remote(url) Sets the Spark remote URL to connect via Spark Connect.

Examples

spark = (
    SparkSession.builder
        .master("local")
        .appName("Word Count")
        .config("spark.some.config.option", "some-value")
        .getOrCreate()
)

spark.sql("SELECT * FROM range(10) where id > 7").show()

+---+
| id|
+---+
|  8|
|  9|
+---+

spark.createDataFrame([('Alice', 1)], ['name', 'age']).show()

+-----+---+
| name|age|
+-----+---+
|Alice|  1|
+-----+---+

spark.range(1, 7, 2).show()

+---+
| id|
+---+
|  1|
|  3|
|  5|
+---+
  • Last updated on