Great questions!

When you start the Spark shell, a SparkSession is created and is accessible via the spark variable.

When you run sbt console a SparkSession is not created, so you need to create your own with val spark = SparkSession.builder().master(“local”).appName(“my-app”).getOrCreate()

The WARN SparkSession$Builder: Using an existing SparkSession; some configuration may not take effect. warning message is a bit confusing because it makes it seem like something is wrong, when everything is working fine. The getOrCreate() method let’s us reuse an existing SparkSession. It gives this warning when you use an existing SparkSession to let you know that an existing SparkSession is being used, even if you tried to change the configuration (we aren’t changing the configuration and we want to reuse the existing SparkSession, so this warning message can be ignored).