Tuesday, January 31, 2023

PySpark's reduceByKey Method, Word count and Line count Programs

pyspark.RDD.reduceByKey()

Merge the values for each key using an associative and commutative reduce function.

What is the associative property? 
The associative property is a math rule that says that the way in which factors are grouped in a multiplication problem does not change the product.

What is the commutative property? 
The commutative property is a math rule that says that the order in which we multiply numbers does not change the product.

spark.apache.org/docs

Note: This program can also be used to compute the number of duplicate occurrences of records in a dataset Tags: Spark,Technology,

No comments:

Post a Comment