Requirement: Compute revenue generated by all postpaid and prepaid service customers. Schema details: CustomerID, Mobile Number, Gender, SeniorCitizen Flag, Mode, Calls, SMS, Internet Service Status, MonthlyCharges (USD), CustomerChurn Flag Before solving the requirement, let us understand the concept of Paired RDD. In most analytical programs, data in the <Key, Value> format provides a feasible way to perform computations. RDD with this data format is called Paired RDD. Every record contains only two fields, key and value. Solution: Step 1: Create a paired RDD with the <key, value> structure. In this requirement, it is <Mode, MonthlyCharges>. Step 2: Group all the fields and apply the sum arithmetic function Step 3: Use the same paired RDD we created and apply the reduceByKey() transformation.
In [1]:
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
23/02/02 15:05:38 WARN Utils: Your hostname, ashish-Lenovo-ideapad-130-15IKB resolves to a loopback address: 127.0.1.1; using 192.168.1.187 instead (on interface wlp2s0) 23/02/02 15:05:38 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/02 15:05:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
No comments:
Post a Comment