Using Python to Read and Write Data | Read and Write Data For Data Programming | Realcode4you

Getting Data

To recap, this is how we put data/information into a variable in Python:

data = ['John', 'Tom', 'Mary', 'Jane'] 
class_list = data

If the information comes in a file, say classdata.txt, in Python we open the file and then put each line into our class_list variable:

with open('classdata.txt') as file:  
	class_list = file.readlines() 

class_list here will be an array of line of texts from the file classdata.txt. Assuming that each line contains just the name of the student (like the array data above), we can use a loop to print out the name of each student:

for name in class_list:  
	print name 

Python dictionary and JSON format

A dictionary, in Python-speak, is essentially sets of key values. For example, say a student will have a name, an email and student id. Hence, a dictionary representing this person can be defined as

student = {'name': 'John', 'email': 'john@gmail.com', 'id': 1}

A class list of students, as an array, can therefore be represented as:

class_list = [{'name': 'John', 'email': 'john@gmail.com', 'id':  1},  {'name': 'Mary', 'email': 'mary@gmail.com', 'id': 2},  {'name': 'Peter', 'email': 'peter@gmail.com', 'id': 3}  ] 

Such a list can then be easily parsed by Python into JSON (JavaScript Object Notation) format, which we will see later, is a popular data exchange format in internet web services. The class_list, represented in JSON format, is as follows:

class_list = [  {"id": 1, "name": "John", "email": "john@gmail.com"},   {"id": 2, "name": "Mary", "email": "mary@gmail.com"},   {"id": 3, "name": "Peter", "email": "peter@gmail.com"}  ]

Working with CSV, TSV and other data formats

In the real world, data likely comes in the form of comma separated values (CSV) or tab separated values (TSV) file formats. For example, Excel exports its workbooks in CSV format. In other cases, which we will see later, information could come via the internet in data packets in various data exchange formats (eg JSON).

Let us look at CSV format first. Python has a csv module that enable you to read and write these files.

A typical CSV file comes in the format below:

Figure 1.5 CSV Format

To read the CSV file, import the csv module first and use it to read in the file. The output will be an array of rows each being a listing of the individual row items as a string:

import csv 
with open('data.csv') as f:   
	reader = csv.reader(f)  
	for row in reader:  
		print row

Output:

['car_park_no', 'address', 'x_coord', 'y_coord', 'car_park_type', 'type_of_parking_system', 'short_term_parking', 'free_parking', 'night_parking']

['ACB', 'BLK 270/271 ALBERT CENTRE BASEMENT CAR PARK', '30314.7936', '31490.4942', 'BASEMENT CAR PARK', 'ELECTRONIC PARKING', 'WHOLE DAY', 'NO', 'YES']

['ACM', 'BLK 98A ALJUNIED CRESCENT', '33758.4143', '33695.5198', 'MULTI-STOREY CAR PARK', 'ELECTRONIC PARKING', 'WHOLE DAY', 'SUN & PH FR 7AM-10.30PM', 'YES']

Let’s say we want to save the data in the CSV file into a data variable. We can simply append each row into an array

carpark_list = [] 
for row in reader:  
	car_list.append(row)

or use the shorthand method (Python calls this List Comprehension), as follows

carpark_list = [row for row in reader]

The first line of the CSV file contains the header. To skip the header row, we call a next() function to get the first item (ie first row) in the iterator reader, so that the first row does not appear when the rest of the data is put inside the variable carpark_list. Here is the full code:

Reading TSV File Reading a tab-separated value (TSV) file essentially follows the same format. The only addition here is to pass in delimiter='\t' as keyword argument.

If the separator is not a tab but some other values, the same pattern (ie delimiter = something) here applies.

reader = csv.reader(f, delimiter='\t') 

Output CSV file as Dictionary

Python CSV module also comes with a convenient DictReader to convert each row of the CSV file into Python dictionaries. In the example below, we use that function to convert parse each row into dictionary, and finally insert all the dictionary items into the carpark_list:

with open('data.csv') as f:   
	reader = csv.DictReader(f)  
carpark_list = list(reader) 

Output Python Dictionary as CSV file If you have a carparklist containing data in the form of a list of Python dictionary, e.g

carparklist = [  
{'carpark_no': 'ACB', 'address': 'BLK 270/271 ALBERT  CENTRE BASEMENT CAR PARK', 'x_coord': '30314.7936', 'y_coord':  '31490.4942'},  
{'carpark_no': 'ACM', 'address': 'BLK 98A ALJUNIED  CRESCENT', 'x_coord': '33758.4143', 'y_coord': '33695.5198'},  
{'carpark_no': 'AH1', 'address': 'BLK 101 JALAN DUSUN',  'x_coord': '29257.7203', 'y_coord': '34500.3599'} ] 

and you want to save the data as a CSV file, you can use the example script here:

# extract the names of each data column using the dict keys 
cols = carparklist[0].keys() 

# and write them in the first line of the CSV file 
with open('newdata.csv', 'w') as f:  
	f.write(','.join(cols)+'\n') 

# then write the rest of the CSV file  
for o in carparklist:  
		row = [str(o[col])  for col in cols] 
		f.write(','.join(row)+'\n') 

You have now generated a newdata.csv file which you can open using Excel for further processing.