In order to collect the desired data by scraping,
Normally, log in to the website → display the top page after login → move to the page with data and then collect
I think I'll do it, but I've been researching various things, but I just can't figure out how to move pages.
You can log in with the code below and get the data of the page displayed after login,
How can I move around the site and get the desired data to the acquisition page?
Please kindly advise.
from selenium import webdriver from time import sleep import pandas as pd USER = "yamada" PASS = "taro" browser = webdriver.Chrome ('chromedriver.exe') #Access the login page url_login = "https://xxxx.com/login_page" browser.get (url_login) sleep (3) #Set the ID/PW required for login elem_username = browser.find_element_by_id ('username') elem_password = browser.find_element_by_id ('password') elem_username.send_keys (USER) elem_password.send_keys (PASS) #Click the login button elem_login_btn = browser.find_element_by_id ('login-btn') elem_login_btn.click () # It works fine so far and i am logged in. # From here, for example, I want to move to any link destination.
Answer # 1
It depends on your requirements.
Case 1: The URL is known in advance.
browser.get ("URL of target page")
(If page transitions are managed, you may not be able to go directly to that page)
Case 2: The URL of the target page is written somewhere on the page after login
browser.find_element_ ～～Identify the target element with and get the URL
Case 3: Click the button on the page after logging in to go to the target page
browser.find_element_ ～～Identify the purpose button with
- python - about pythinista3 selenium browser controls
- python - about scraping using selenium
- python - read and write values from another file
- python, about the fire spread step of forest fire simulation
- about image output of python
- python 3x - please teach about the exchange of information with qthead
- python - about x-axis adjustment in matplotlib
- python - i want to assign an element of one list to an element of another list
- python - error when scraping with selenium and firefox
- python - about "" "of" "" select === = "" "
- python - about hamiltonian neural networks
- python - what i don't understand about yolo9000 (v2)
- python - about write loop to csv
- python - selenium: element specification an error occurs in the code that can be executed once
- about python argument and data definition
- python 3x - about downloading anaconda
- python - about the optimum angle of rotation matrix
- python - about downloading youtube videos by youtube-dl
- about processing to exclude the character string group specified from list in python
- python - you may need to restart the kernel to use updated packages error
- php - coincheck api authentication doesn't work
- php - i would like to introduce the coincheck api so that i can make payments with bitcoin on my ec site
- [php] i want to get account information using coincheck api
- the emulator process for avd pixel_2_api_29 was killed occurred when the android studio emulator was started, so i would like to
- python 3x - typeerror: 'method' object is not subscriptable
- i want to call a child component method from a parent in vuejs
- dart - flutter: the instance member'stars' can't be accessed in an initializer error
- xcode - pod install [!] no `podfile 'found in the project directory