I wrote a program that sends a GET request to the API and writes a response to a file every 1 million lines. Please let me know if there is any solution.
StreamConsumedError Traceback (most recent call last)
3 decoded_lines = ''
4 with open (path.format (i), 'w') as f:
---->5 for line in r.iter_lines ():
6 count + = 1
7 if count == i * 1000000:
~/opt/anaconda3/lib/python3.7/site-packages/requests/models.py in iter_lines (self, chunk_size, decode_unicode, delimiter)
792 pending = None
793
->794 for chunk in self.iter_content (chunk_size = chunk_size, decode_unicode = decode_unicode):
795
796 if pending is not None:
~/opt/anaconda3/lib/python3.7/site-packages/requests/models.py in iter_content (self, chunk_size, decode_unicode)
767
768 if self._content_consumed and isinstance (self._content, bool):
->769 raise StreamConsumedError ()
770 elif chunk_size is not None and not isinstance (chunk_size, int):
771 raise TypeError ("chunk_size must be an int, it is instead a% s."% Type (chunk_size))
StreamConsumedError:
base = 'https://api.nazuki-oto.com/historical/tweets?type=stream&estid= {}'
endpoint = 'kargrNyc'
header = {'Accept-Encoding': 'deflate, gzip'}
url = base.format (endpoint)
s = requests.Session ()
r = s.get (url, auth = ('id', 'pass'), headers = header, stream = True)
for i in range (1, 7):
count = 0
decoded_lines = ''
with open (path.format (i), 'w') as f:
for line in r.iter_lines ():
count + = 1
if count == i * 1000000:
decoded_lines + = line.decode ('utf-8')
break
elif count>(i-1) * 1000000:
decoded_lines + = line.decode ('utf-8') + '\ n'
else:
pass
f.write (decoded_lines)
-
Answer # 1
Related articles
- api - classes using python websocket will be closed immediately
- i don't understand the exercises using python trigonometric functions
- python 3x - i want to get the nth array with an argument using python3 argparse
- please explain the function using the python dictionary
- parameter estimation using python's weighted least squares method (wls)
- parallel processing using python multiprocessingpool and multiprocessingqueue does not work well
- about external libraries when using multiple versions of python
- python - i want to separate by a specific word using the split function
- python - image recognition using cnn keras multiple inputs
- about batch change of file name using python
- python 3x - how to rename a folder created using jupyternotebook
- python - handling of 1d array data when using the predict function in keras
- python 3x - processing to jump to the link destination using chrome driver in python
- python - error in image binarization using cv2adaptivethreshold function
- i want to adjust the execution result using the while statement in python as expected
- python - i want to put the image file path in a variable and open it using that variable
- python - aggregation processing using pandas
- specifying the range of the graph using python date
- python - about the fibonacci sequence using recursion
- python - i'm using selenium the text sent by send_keys to the input tag disappears when the text is sent by send_key to the next
- api - classes using python websocket will be closed immediately
- i want to extract the coordinate values from the recognition result obtained by cloudvision api
- [fast api/bearer authentication] i don't know how to fill the value from js (axios) to oauth2passwordrequestform
- reading python text module assignment error occurred
- about api authentication using python requests module
- python - missing 1 required positional argument error
- api - please tell me how to specify the url parameter (/: xxx) of backlog
Isn't there any more data because Stream is a Consumed Error?