Python - Get Data through API with Bearer Token (II) - Data Processing

After printing out sample returned json data, I know the object is supposed to be converted to list of dictionary and then data frame.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#Process data
#Type of object is 'str', so we need further processing
type(pro_res.text)

#Split json text into a list
dat = pro_res.text.split('\n')
#Remove last element: ""
ind = list(range(len(dat)-1))
dat = [dat[x] for x in ind]
#Convert list into data frame
df = pd.DataFrame([json.loads(x) for x in dat])

#Two approaches to write output
##Approach 1: directly write csv
#df.to_csv('ProductCategory.csv', index = False, encoding = "utf-8", sep = "\t")

##Approach 2: write to compressed file to save storage resources
compression_opts = dict(method='zip',archive_name='ProductCategory.csv')
df.to_csv('./API Data/ProductCategory/ProductCategory.zip', index = False, encoding = "utf-8", compression = compression_opts)

Python - Get Data through API with Bearer Token (II) - Data Processing
http://arwenzhou.github.io/2020/11/12/Python-requests-API-2/
Author
Arwen
Posted on
November 12, 2020
Licensed under