Matthew PowersSpeed up a pandas query 10x with these 6 Dask DataFrame tricksThis post demonstrates how to speed up a pandas query to run 10 times faster with Dask using six performance optimizations. You’ll often…Feb 22, 2022Feb 22, 2022
Matthew PowersCreating a Java Spark project with Maven and junitThis blog post shows how to organize a Spark Java project, write some application code and run a simple test.Jul 4, 20192Jul 4, 20192
Matthew PowersDependency Injection with SparkDependency injection is a great design pattern to write code that’s more flexible and easier to test.Jun 25, 2019Jun 25, 2019
Matthew PowersSpeaking Slack Notifications from SparkThe spark-slack library can be used to speak notifications to Slack from your Spark programs and handle Slack Slash command responses.Apr 3, 20184Apr 3, 20184
Matthew PowersDocumenting Spark Code with ScaladocYou can use Scaladoc to generate nicely formatted documentation for your Spark projects, just like the official Spark documentation.Feb 18, 20182Feb 18, 20182
Matthew PowersThe different type of Spark functions (custom transformations, column functions, UDFs)Spark code can be organized in custom transformations, column functions, or user defined functions (UDFs).Jan 21, 20187Jan 21, 20187
Matthew PowersAdding StructType columns to Spark DataFramesStructType objects define the schema of Spark DataFrames. StructType objects contain a list of StructField objects that define the name…Jan 16, 20186Jan 16, 20186
Matthew PowersAdding ArrayType columns to Spark DataFrames with concat_ws and splitThe concat_ws and split Spark SQL functions can be used to add ArrayType columns to DataFrames.Jan 15, 2018Jan 15, 2018
Matthew PowersHow to write Spark ETL ProcessesSpark is a powerful tool for extracting data, running transformations, and loading the results in a data store.Jan 5, 201811Jan 5, 201811
Matthew PowersSpark User Defined Functions (UDFs)Spark let’s you define custom SQL functions called user defined functions (UDFs). UDFs are great when built-in SQL functions aren’t…Dec 27, 20178Dec 27, 20178