Printing contents of RDD, SQL DataFrame, and PySpark.Pandas DataFrame

Download Code and Data

In [1]:

from pyspark import SparkContext
sc = SparkContext.getOrCreate()

23/02/08 08:21:23 WARN Utils: Your hostname, ashish-Lenovo-ideapad-130-15IKB resolves to a loopback address: 127.0.1.1; using 192.168.1.108 instead (on interface wlp2s0)
23/02/08 08:21:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

23/02/08 08:21:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/02/08 08:21:26 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

1: For RDD¶

In [2]:

rdd = sc.textFile("../in/TelecomData.csv")

Using collect()¶

In [5]:

rdd.collect()

Out[5]:

['TXCUST00001,982120000,Male,N,PrePaid,Active,Active,InActive,20,N',
 'TXCUST00002,982120001,Male,N,PostPaid,Active,Active,InActive,25,N',
 'TXCUST00003,982120002,Male,N,PrePaid,Active,Active,InActive,20,Y',
 'TXCUST00004,982120003,Male,Y,PrePaid,Active,Active,InActive,25,N',
 'TXCUST00005,982120004,Male,N,PrePaid,Active,Active,InActive,15,N',
 ...
 'TXCUST00497,982120496,Male,N,PostPaid,Active,Active,Active,15,N',
 'TXCUST00498,982120497,Male,N,PostPaid,Active,Active,Active,20,N',
 'TXCUST00499,982120498,Male,N,PostPaid,Active,Active,Active,25,N',
 'TXCUST00500,982120499,Male,N,PostPaid,Active,Active,Active,20,N']

Using take()¶

In [6]:

rdd.take(10)

Out[6]:

['TXCUST00001,982120000,Male,N,PrePaid,Active,Active,InActive,20,N',
 'TXCUST00002,982120001,Male,N,PostPaid,Active,Active,InActive,25,N',
 'TXCUST00003,982120002,Male,N,PrePaid,Active,Active,InActive,20,Y',
 'TXCUST00004,982120003,Male,Y,PrePaid,Active,Active,InActive,25,N',
 'TXCUST00005,982120004,Male,N,PrePaid,Active,Active,InActive,15,N',
 'TXCUST00006,982120005,Male,N,PrePaid,Active,Active,InActive,15,N',
 'TXCUST00007,982120006,Male,N,PrePaid,Active,Active,InActive,12,N',
 'TXCUST00008,982120007,Male,N,PrePaid,Active,Active,InActive,10,N',
 'TXCUST00009,982120008,Male,Y,PrePaid,Active,Active,InActive,15,Y',
 'TXCUST00010,982120009,Female,N,PrePaid,Active,Active,InActive,15,N']

For RDD: show(), head() and tail() would give error "AttributeError: 'RDD' object has no such attribute"¶

show(), head() and tail() work for SQL DataFrame along with collect() and take().¶

2: For SQL DataFrame¶

In [9]:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

In [10]:

df = spark.read.option("header", False).csv('../in/TelecomData.csv')

In [13]:

df.collect()

Out[13]:

[Row(_c0='TXCUST00001', _c1='982120000', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='20', _c9='N'),
 Row(_c0='TXCUST00002', _c1='982120001', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='InActive', _c8='25', _c9='N'),
 Row(_c0='TXCUST00003', _c1='982120002', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='20', _c9='Y'),
 Row(_c0='TXCUST00004', _c1='982120003', _c2='Male', _c3='Y', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='25', _c9='N'),
 Row(_c0='TXCUST00005', _c1='982120004', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='15', _c9='N'),
 ...
 Row(_c0='TXCUST00497', _c1='982120496', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='15', _c9='N'),
 Row(_c0='TXCUST00498', _c1='982120497', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='20', _c9='N'),
 Row(_c0='TXCUST00499', _c1='982120498', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='25', _c9='N'),
 Row(_c0='TXCUST00500', _c1='982120499', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='20', _c9='N')]

In [15]:

df.take(5)

Out[15]:

[Row(_c0='TXCUST00001', _c1='982120000', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='20', _c9='N'),
 Row(_c0='TXCUST00002', _c1='982120001', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='InActive', _c8='25', _c9='N'),
 Row(_c0='TXCUST00003', _c1='982120002', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='20', _c9='Y'),
 Row(_c0='TXCUST00004', _c1='982120003', _c2='Male', _c3='Y', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='25', _c9='N'),
 Row(_c0='TXCUST00005', _c1='982120004', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='15', _c9='N')]

In [16]:

df.show()

+-----------+---------+------+---+--------+------+------+--------+---+---+
|        _c0|      _c1|   _c2|_c3|     _c4|   _c5|   _c6|     _c7|_c8|_c9|
+-----------+---------+------+---+--------+------+------+--------+---+---+
|TXCUST00001|982120000|  Male|  N| PrePaid|Active|Active|InActive| 20|  N|
|TXCUST00002|982120001|  Male|  N|PostPaid|Active|Active|InActive| 25|  N|
|TXCUST00003|982120002|  Male|  N| PrePaid|Active|Active|InActive| 20|  Y|
|TXCUST00004|982120003|  Male|  Y| PrePaid|Active|Active|InActive| 25|  N|
|TXCUST00005|982120004|  Male|  N| PrePaid|Active|Active|InActive| 15|  N|
|TXCUST00006|982120005|  Male|  N| PrePaid|Active|Active|InActive| 15|  N|
|TXCUST00007|982120006|  Male|  N| PrePaid|Active|Active|InActive| 12|  N|
|TXCUST00008|982120007|  Male|  N| PrePaid|Active|Active|InActive| 10|  N|
|TXCUST00009|982120008|  Male|  Y| PrePaid|Active|Active|InActive| 15|  Y|
|TXCUST00010|982120009|Female|  N| PrePaid|Active|Active|InActive| 15|  N|
|TXCUST00011|982120010|Female|  N| PrePaid|Active|Active|InActive| 15|  N|
|TXCUST00012|982120011|Female|  N|PostPaid|Active|Active|InActive| 20|  N|
|TXCUST00013|982120012|Female|  N|PostPaid|Active|Active|InActive| 25|  Y|
|TXCUST00014|982120013|Female|  N|PostPaid|Active|Active|InActive| 25|  N|
|TXCUST00015|982120014|Female|  Y|PostPaid|Active|Active|InActive| 25|  N|
|TXCUST00016|982120015|Female|  N|PostPaid|Active|Active|InActive| 30|  N|
|TXCUST00017|982120016|Female|  N|PostPaid|Active|Active|InActive| 15|  N|
|TXCUST00018|982120017|Female|  N|PostPaid|Active|Active|InActive| 15|  N|
|TXCUST00019|982120018|Female|  N|PostPaid|Active|Active|InActive| 15|  N|
|TXCUST00020|982120019|Female|  N|PostPaid|Active|Active|InActive| 20|  N|
+-----------+---------+------+---+--------+------+------+--------+---+---+
only showing top 20 rows

In [18]:

df.head(5)

Out[18]:

[Row(_c0='TXCUST00001', _c1='982120000', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='20', _c9='N'),
 Row(_c0='TXCUST00002', _c1='982120001', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='InActive', _c8='25', _c9='N'),
 Row(_c0='TXCUST00003', _c1='982120002', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='20', _c9='Y'),
 Row(_c0='TXCUST00004', _c1='982120003', _c2='Male', _c3='Y', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='25', _c9='N'),
 Row(_c0='TXCUST00005', _c1='982120004', _c2='Male', _c3='N', _c4='PrePaid', _c5='Active', _c6='Active', _c7='InActive', _c8='15', _c9='N')]

In [19]:

df.tail(5)

Out[19]:

[Row(_c0='TXCUST00496', _c1='982120495', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='15', _c9='N'),
 Row(_c0='TXCUST00497', _c1='982120496', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='15', _c9='N'),
 Row(_c0='TXCUST00498', _c1='982120497', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='20', _c9='N'),
 Row(_c0='TXCUST00499', _c1='982120498', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='25', _c9='N'),
 Row(_c0='TXCUST00500', _c1='982120499', _c2='Male', _c3='N', _c4='PostPaid', _c5='Active', _c6='Active', _c7='Active', _c8='20', _c9='N')]

3: For PySpark.Pandas¶

In [20]:

from pyspark import pandas as ppd
df_pandas = ppd.read_csv('../in/TelecomData.csv', 
                         names = ['CustomerID', 'MobileNumber', 'Gender', 'SeniorCitizen', 'Mode', 'Calls', 'SMS', 'InternetServiceStatus', 'MonthlyCharges', 'CustomerChurn'])

WARNING:root:'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to set this environment variable to '1' in both driver and executor sides if you use pyarrow>=2.0.0. pandas-on-Spark will set it for you but it does not work if there is a Spark context already launched.
/home/ashish/anaconda3/envs/pyspark/lib/python3.9/site-packages/pyspark/pandas/utils.py:975: PandasAPIOnSparkAdviceWarning: If `index_col` is not specified for `read_csv`, the default index is attached which can cause additional overhead.
  warnings.warn(message, PandasAPIOnSparkAdviceWarning)

In [28]:

# AttributeError: 'DataFrame' object has no attribute 'collect'
# AttributeError: 'DataFrame' object has no attribute 'show'
# df_pandas.take(2) # TypeError: `indices` must be a list-like except dict or set
df_pandas.take([0, 1, 2])

Out[28]:

	CustomerID	MobileNumber	Gender	SeniorCitizen	Mode	Calls	SMS	InternetServiceStatus	MonthlyCharges	CustomerChurn
0	TXCUST00001	982120000	Male	N	PrePaid	Active	Active	InActive	20	N
1	TXCUST00002	982120001	Male	N	PostPaid	Active	Active	InActive	25	N
2	TXCUST00003	982120002	Male	N	PrePaid	Active	Active	InActive	20	Y

In [29]:

df_pandas.head()

Out[29]:

	CustomerID	MobileNumber	Gender	SeniorCitizen	Mode	Calls	SMS	InternetServiceStatus	MonthlyCharges	CustomerChurn
0	TXCUST00001	982120000	Male	N	PrePaid	Active	Active	InActive	20	N
1	TXCUST00002	982120001	Male	N	PostPaid	Active	Active	InActive	25	N
2	TXCUST00003	982120002	Male	N	PrePaid	Active	Active	InActive	20	Y
3	TXCUST00004	982120003	Male	Y	PrePaid	Active	Active	InActive	25	N
4	TXCUST00005	982120004	Male	N	PrePaid	Active	Active	InActive	15	N

In [30]:

df_pandas.tail()

Out[30]:

	CustomerID	MobileNumber	Gender	SeniorCitizen	Mode	Calls	SMS	InternetServiceStatus	MonthlyCharges	CustomerChurn
495	TXCUST00496	982120495	Male	N	PostPaid	Active	Active	Active	15	N
496	TXCUST00497	982120496	Male	N	PostPaid	Active	Active	Active	15	N
497	TXCUST00498	982120497	Male	N	PostPaid	Active	Active	Active	20	N
498	TXCUST00499	982120498	Male	N	PostPaid	Active	Active	Active	25	N
499	TXCUST00500	982120499	Male	N	PostPaid	Active	Active	Active	20	N

In [ ]:

Moni Singh (YOB: 1990, Cook)

Go to Index of Guest Interviews

Moni Singh

1990-Nov-30, Cook

1. What is the one factor that contributed most in what you have achieved in life?

   Hard work.

   Mehnat
2. What is your single biggest achievement?

   Without education, without knowledge I came to Haryana for work.

   Bina padhai, bina jankari ke Haryana mei aa gayi.
3. What is the one lesson that you learnt the hard way?

   Be strong, don't be emotional.

   Strong bano, emotional mat bano.
4. What is the one thing that you would like to change about India to make it a better country?

   Women are not safe in this country. Something should be done about it. So that they go come and go anywhere anytime.

   Desh mei ladkiyan surakshit nahi. Unke liye kuch hona chahiye. Voh din mei, raat mei kahi bhi aa-jaa sake.
5. What would be your message to anyone aspiring success?

   Work hard and don't involve yourself too much in societal matters.

   Mehnat karo. Kisi se koi matlab nhi rakho.
6. What is the biggest mistake that you have ever made?

   I trusted Sandeep Pandey and gave him Rs 25000 for scooter but he neither gave me the scooter nor gave me my money back.

7. Please tell us about the toughest times you have ever had, the most testing times that you have been through.

   When I was coming to Haryana by train, all my belongings and money was stolen. So I had to work as a labor at 600 Rs per day.

   Jab mai yaha Haryana mei aai job karne toh mere documents kho gaye the aur mere paise bhi kho gaye the. Train mei samaan chori ho gaya tha.
8. What is the one thing that motivates you the most?

   God. Goddess Durga. Lord Krishna.

   Bhagwan. Durga maiya. Krishna Kanaiya.
9. What is the one thing that worries you the most?

   Education of my daughters.

10. Please tell us about your five year plan. Personally and professionally.

    I would save for two years. Then I would send my daughters to college. Then I would open my open PG accomodation.

    Do saal bachat karna hai. Uske baad beti ko college karana hai. Uske baad PG kholungi.
11. Define 'hard work'.

    Hard work is a good thing. It give you mental energy.

    Mehnat achi cheez hai. Isse shakti milti hai.
12. Define 'family'.

    Zero divided by zero.

    Zero bataa zero.
13. Daughters are also family then why did you say that family means 'zero by zero' to you?

    Whether the relation is of daughter or son or sister or husband, nobody would be there if you do not have money.
    There was a boy who used to ask his mother "where is my mama? Do I have a mama? Everyone's mama comes to visit him, but not mine."
    Then the mother said "go to that farm, your mama is there." Boy went there and scouted but he could not find his mama. So he came back. Then his mother gave a handful of mustard seeds and said "spill it across farmland and your mama will come". When the boy went back to the farm and spilled the mustard seeds, this produced a very good crop of mustard. Next day his mama came and asked "I have heard that you have got a very good crop of mustard?"

Dated: 07 Feb, 2023 Tags: Communication Skills,Management,Success Story