- There is a server that contains a json dataset that I need
- I can manually use chrome to login to the url and use chrome developer tool to read the request header for said json data
- I determined that the minimum required headers that should be sent to the json endpoint are ['cookie', 'x-xsrf-token', 'user-agent']
- I don't know how I can get these values so that I can automate fetching this data. I would like to use request module to get the data
- I tried using selenium, to navigate to the webpage that exposes these header values, but cannot get said headers values (not sure if selenium supports this)
- Is there a way for me to use request module to inch towards getting these header values...by following the request header "bread crumbs" so to speak?
- Is there an alternative module that excels at this?
- To note, I have used selenium to get the required datapoints successfully, but selenium is resource heavy and prone to crash;
- By using the request module with header values greatly simplifies the workflow and makes my script reliable
↧
get json data from host that requires headers={'user-agent', 'cookie', x-xsrf-token'}
↧