Quantcast
Channel: Active questions tagged selenium - Stack Overflow
Viewing all articles
Browse latest Browse all 98807

get json data from host that requires headers={'user-agent', 'cookie', x-xsrf-token'}

$
0
0
  • There is a server that contains a json dataset that I need
  • I can manually use chrome to login to the url and use chrome developer tool to read the request header for said json data
  • I determined that the minimum required headers that should be sent to the json endpoint are ['cookie', 'x-xsrf-token', 'user-agent']
  • I don't know how I can get these values so that I can automate fetching this data. I would like to use request module to get the data
  • I tried using selenium, to navigate to the webpage that exposes these header values, but cannot get said headers values (not sure if selenium supports this)
  • Is there a way for me to use request module to inch towards getting these header values...by following the request header "bread crumbs" so to speak?
  • Is there an alternative module that excels at this?
  • To note, I have used selenium to get the required datapoints successfully, but selenium is resource heavy and prone to crash;
  • By using the request module with header values greatly simplifies the workflow and makes my script reliable

Viewing all articles
Browse latest Browse all 98807


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>