Home>

In order to collect the desired data by scraping,
Normally, log in to the website → display the top page after login → move to the page with data and then collect
I think I'll do it, but I've been researching various things, but I just can't figure out how to move pages.

You can log in with the code below and get the data of the page displayed after login,
How can I move around the site and get the desired data to the acquisition page?

Please kindly advise.

from selenium import webdriver
from time import sleep
import pandas as pd
USER = "yamada"
PASS = "taro"
browser = webdriver.Chrome ('chromedriver.exe')
#Access the login page
url_login = "https://xxxx.com/login_page"
browser.get (url_login)
sleep (3)
#Set the ID/PW required for login
elem_username = browser.find_element_by_id ('username')
elem_password = browser.find_element_by_id ('password')
elem_username.send_keys (USER)
elem_password.send_keys (PASS)
#Click the login button
elem_login_btn = browser.find_element_by_id ('login-btn')
elem_login_btn.click ()
# It works fine so far and i am logged in.
# From here, for example, I want to move to any link destination.
  • Answer # 1

    It depends on your requirements.

    Case 1: The URL is known in advance.
    browser.get ("URL of target page")
    (If page transitions are managed, you may not be able to go directly to that page)

    Case 2: The URL of the target page is written somewhere on the page after login
    browser.find_element_ ~~Identify the target element with and get the URLgetTo do

    Case 3: Click the button on the page after logging in to go to the target page
    browser.find_element_ ~~Identify the purpose button withclickTo do

    Such.