Yahoo! Auctions using Python! I want to get the shipping cost.
Since I have just started learning programming recently, please forgive me if there is something wrong with the words.
"¥ 2,980"
(tax included)
This is a part of HTML including the shipping cost of the target page.
Page URL:
I wanted to scrape the shipping cost using the CSS selector, but it's missing.
Error message[Execution result]
IndexError: list index out of range
Applicable source code
import requests
import lxml.html
response = requests.get (URL)
HTML = lxml.html.fromstring (response.text)
SOURYOU = HTML.cssselect ('# method0>div') [0] .text.strip ()
print (SOURYOU)
Since the prompt decision price "4,350 yen" in the same page was stored in almost the same structure, the following code was executed as a test, and "4,350 yen" was obtained.
[HTML including prompt decision price]
- Prompt decision price
<\ dt>
-
"
4,350 yen "
(0 yen tax)<\ span>
<\ dd>import requests import lxml.html response = requests.get (URL) HTML = lxml.html.fromstring (response.text) KAKAKU = HTML.cssselect ('# l-sub>div.ProductInformation>ul>li.ProductInformation__item.js-stickyNavigation-start>div>dl>dd.Price__value') [0] .text.strip () print (KAKAKU)
Since the structure is almost the same for the shipping cost and the prompt decision price, it was impossible to understand why the prompt decision price of "4,350 yen" could be obtained and only the shipping cost of "2,980 yen" could not be obtained.
Thanks for your support.
Supplemental information (FW/tool version etc.)Python 3.8.0
Microsoft Windows 10
Version 1903Please provide more information here.
-
Answer # 1
-
Answer # 2
Click on the "Details" section on the page → The shipping cost will be displayed
It seems that it cannot be obtained because of the Javascript mechanism.It seems to be possible to get the HTML that appeared after running Googlescript by running Google Chrome in Python using a library called Selenium.
Reference blog:
Python Web Scraping Techniques Collection "No value can not be obtained" JavaScript support
[https://qiita.com/Azunyan1111/items/b161b998790b1db2ff7a]
Related articles
- python - elements cannot be acquired after screen transition in selenium
- javascript - cannot call python program by post communication using aws amplify
- python - regular expression: sub () cannot replace the matched part
- [rails] date cannot be displayed with javascript [html] [css]
- i can't get child elements by scraping python
- javascript - line bot error created with gas: 400 cannot be resolved
- javascript - i want to make an array of the elements obtained by clicking
- python: i want to check the match of the elements of the data frame
- python - regression analysis with statsmodels cannot resolve the error that it contains infinity or nan
- python - ngrok cannot be executed
- python - cannot convert csv file to excel file (contents are not copied )
- python - can the characteristics of pivot elements be `7n/10`?
- python - icrawler cannot save images
- python - cuda cannot be downgraded by rebuilding anaconda's gpu environment
- python - i want to get multiple elements with find_elements and iterate with a for statement
- python 3x - pillow cannot be installed on macos
- javascript - react conditional branch cannot be implemented well
- python - cannot read csv file with google colaboratory
- javascript - i want to use the processing result of python without opening the port in the web application
- python - image cannot be read by opencv
- javascript : Function not working for onclick event on link
- javascript : Alignment. Adding icons to the menu
- javascript : Why, when printing, the canvas is cleared and nothing else appears on it?
- python : Django How to display data from db into multiple pages?
- javascript : Displaying data from a google table on the site
- javascript : How to make sure that if a person has an incorrect path in the address bar, then the site will automatically redire
- javascript : Correct imitation of a long page load
- javascript : Display messages from the list in random order
- javascript : localStorage js removeBlock function
- javascript : Move text by button
Because it is the text of the div element
OK.