Home>
I want to achieve in the end

I am currently programming with the ultimate goal of developing an information gathering application in my research.

Programming flow

(1) Enter a specific word ➤ (2) Set the period ➤ (3) Display the number of searches within the period

Example: ① Enter the word "apple" ➤ ② Select the period (3 items today, yesterday, one week ago) ➤ ③ Number of searches within the period [Apple: ○○○]

I would like to develop this flow of program operation using web scraping (Google) and Tkinter (GUI).

problem

Currently, I am creating programming by dividing it into Tkinter and web scraping.

This time, as a problem on the web scraping side, I would like to show "the number of searches within the period when a specific word is entered".

Whole source code
import argparse
from time import sleep
from string import ascii_lowercase
from string import digits
import requests
import urllib.parse

class GoogleAutoComplete:
    def __init __ (self, test_mode = False, recurse_mode = False):
                        'hl = ja&output = toolbar&ie = utf-8&oe = utf-8&'\
                        'client = firefox&q ='
        self.test_mode = test_mode
        self.recurse_mode = recurse_mode
        a = open ('table_hiragana.dat','r')
        tango_list = ['a','i','u','e','o','ka','ki','ku','ke','ko','sa','shi' ,'Su','se','so','ta','chi','tsu','te',' and',' na',' ni','nu','ne',' No',' is','hi','fu','he','ho','ma','mi','mu','me','mo','ya','yu' ,'Yo',' et al','ri',' ru','re','ro','wa','wo','n','kya','kyu','kyo' ,'Sha',' Shu',' Sho',' Cha',' Chu',' Cho',' Nya',' Nyu',' Nyo','Hya','Hyu','Hyo ',' Mya',' Myu',' Myo',' Rya',' Ryu',' Ryo',' Gya',' Gyu',' Gyo',' Ja',' Ju' ,'Jo','Bya','Byu','Byo','Pya','Pyu','Pyo']
        for tango in a:
            tango = tango.rstrip ()
            tango_list.append (tango)
        if test_mode:
            self.chrs = ['A','g', '1']
        else: else:
            self.chrs = [str (i) for i in tango_list] # Exactly the same as the Perl Japanese syllabary file

    def get_suggest (self, query):
        buf = requests.get (self.base_url +
                           urllib.parse.quote_plus (query)). json ()
        suggests = [ph for ph in buf [1]]
        print ("query: [{0}] ({1})". format (query, len (suggests)))
        for ph in suggests:
            print ("", ph)
        sleep (1)
        return suggests
    def get_suggest_with_one_char (self, query):
        #Suggest word for the keyword itself
        ret = self.get_suggest (query)
        #Keyword + Suggestion word when blank
        ret.extend (self.get_suggest (query +''))
        #Keyword + space + suggestion word for 1 character
        for ch in self.chrs:
            ret.extend (self.get_suggest (query +'' + ch))
        return self.get_uniq (ret)
    # Exclude duplicates
    def get_uniq (self, arr):
        uniq_ret = []
        for x in arr:
            if x not in uniq_ret:
                uniq_ret.append (x)
        return uniq_ret
if __name__ == "__main__":
    phrase = input ('Enter keyword.')
    #Google Suggest keyword acquisition
    gs = GoogleAutoComplete (recurse_mode = "--recure")
    ret = gs.get_suggest_with_one_char (phrase)

    #Save to file
    fname = "suggest.csv"
    with open (fname,'w', encoding ='UTF-8') as fs:
        for key in ret:
            fs.write (key + "\ n")
Execution result (input word is "apple")

When the command prompt is started

Result ↓
query: [apple] (10)
Apple
Ringo-chan
Apple music festival
Apple recipe
Apple jam
Apple calories
Candy apple
Apple english
Apple cider vinegar
Apple loop

query: [apple] (10)
Ringo-chan
Apple recipe
Apple jam
Apple calories
Apple daughter
Apple english
Apple disease
Ringoro
Apple type
Apple cake

query: [Apple] (10)
ringo apple pie
ringo apple pie calories
ringo apple pie price
ringo apple pie best-by date
Apple Akiei
Apple grip strength
Apple allergy
Apple arrangement
Apple apple pie
Apple allergy baby

query: [Apple] (10)
Apple illustration
Apple dog
No need for an apple doctor
Apple meaning
Weight of one apple
Apple illustration handwritten
One apple calorie
Apple illustration cute
ringo Ikebukuro
Apple illustration

query: [Apple] (10)
Apple rabbit
Apple whiskey
Apple rabbit
Sweetened apples
Sliced ​​apples
Apple Rabbit Real
Apple whiskey
Apple wiki
Other than apple rabbit
Apple rabbit illustration

query: [Apple] (10)
Apple english
Apple nutrition
Apple salt water
Apple picture
How to choose an apple
Apple picture book
Apple ethylene gas
Apple emoji
Apple nutrition composition table
Apple salt water why

query: [Apple] (10)
Apple lunch
Apple weight
Apple sweets
Apple origami
Apple snack
Apple Orin
Apple size
Apple liquor
Apple sweets easy
Apple side dish

query: [Apple] (10)
Apple calories
Apple decoration
With apple skin
Apple kanji
Apple Korean
Apple potassium
Apple decoration rose
Apple cocktail
Apple calorie photo
Apple kanji

query: [Apple] (10)
Apple season
How to cut apples
Apple Kiou
Apple caramelized
Rare varieties of apples
Apple tree
Apple crate
Apple character
Cut and save apples
Apple season word

query: [Apple] (10)
Apple cream cheese
Apple rot
Apple cookies
Apple crumble
Apple comb
Apple cook pad
Apple black vinegar
Apple black
Apple cream cheese snacks
ringo coupon

query: [Apple Ke] (10)
Apple cake
Apple health
Apple blood sugar
Apple blood pressure
Apple cake rice cooker
Apple cake pancake mix
Apple cake frying pan
How to eat apples healthy
Apple chicken
ringo Keio Mall

query: [Apple] (10)
Apple compote
Apple effect
Apple Jonathan
Apple efficacy
Apple compote range
Apple perfume
Apple love sky
Apple compote honey
Apple compote longevity
Apple Kotoku

query: [Apple] (10)
Apple salad
Apple sweet potato
Apple oxidation
Apple sansa
Apple sugar water
Apple sweet potato cake
Apple producing area
Apple Sun Tsugaru
Apple cultivation
Apple San Fuji

query: [Apple] (10)
Apple season
Apple type
Best-by date for apples
Apple harvest time
Apple dietary fiber
Apple cinnamon
Apple Shinano Sweet
Apple consumption
ringo shinjuku
Apple season

query: [Apple] (10)
Apple smoothie
Apple sweets
Grated apples
Apple rice cooker
Apple Spanish
Apple sparkling wine
Apple water content
Soak apples in water
Apple straight juice
Ringo starr

query: [Apple] (10)
The best apple in the world
Apple production
Apple production ranking
Apple making
Apple ingredients
Apple Chiaki
Apple production three-dimensional
From when apples are raw
Apple production world
ringo subconscious

query: [Apple So] (10)
How to grow apples
Apple market
Sauteed apples
Apple sauce
Apple as it is
Apple material
Apple Sonia
From when the apple is as it is
How to grow apples From species
Free shipping on apples

query: [Apple] (10)
Mass consumption of apples
Apple tart
Apple tarte tatin
How to eat an apple
Apple high 2020
Apple carbs
Eat too much apple
Apples expensive why 2020
Apple protein
ringo Tachikawa

query: [Apple Chi] (10)
Ringo-chan
Ringo-chan marriage
Apple Chinese
Apple cheesecake
Apple brown
Apple chocolate
Apple cheese
Long-term storage of apples
Apples don't turn brown
Apple brown

query: [Apple] (10)
Apple mail order
Apple Tsugaru
Grip strength to crush apples
Apple mail order Nagano
Apple mail order Amazon
Apple Tsugaru Features
Apple Tsukurepo
Apple mail order cheap
Apple mail order for home use
Apple ripening

query: [Apple] (10)
ringo store
Apple iron
Apple tempura
Apple regular flight
Apple hand play
Apple natural yeast
Apple fruit picking
Gripping apples
Apple top plate cake
Low temperature cooking of apples

query: [with apples] (10)
Apples and butter
Apple sugar
Apple crested ibis
Apple features
Apple toast
Apple sugar content
Apple Touhou
Order apples
ringo Toyosu
ringo tokyo

query: [Apple] (10)
Apple family
Apple Nakameguro
Apple summer
How many grams of apples
Apple Nagano
Apple pear
Apple name
Long-lasting apple
Apple summer price
Apple whipped cream

query: [to apples] (10)
Boil apples
Crush the apples
Apple carrot smoothie
Apple newton
Apple garlic
Apple pregnant
Apple meat
Apples new Zealand
Apple popular recipe
ringo Nishiazabu

query: [Apple Nu] (10)
Pickled apples
Apple coloring book
How to apply apples
Apple plush toy
Warm apples
Apple cloth
Apple tree
How to apply apples Digital
Apple coloring book illustration
Apple slime

query: [Apple] (10)
Apple price
Apple price is high
Apple cat
Apple necklace
Apple nails
Apple price increase
Apple net
Apple price transition
Apple clay
ringo price

query: [Apple] (10)
Apple tree
Apple compote
Apple time
Types of apples
Apple pound cake
How to store apples
Apple pesticide
Apple forest
Directly from an apple farmer
Apple pesticide

query: [Apple] (10)
Apple flower language
Apple flower
Apple honey
Save half an apple
Apple leaves
Pickled apples in honey
Apple disposal rate
Half an apple calorie
Apple flower
Apple high heel

query: [Apple Hi] (10)
Apple varieties
Apple shelf life
Apple varieties Nagano
Apple varieties time
Apple varieties Tsugaru
One apple weighs
Cool apples
ringo hibiya
ringo Higashiazabu
Apple varieties

query: [Apple Fu] (10)
Apple french
Apple Fuji
Apple free material
Apple fat
Apple filling
Apple hometown tax payment
Apple Fuji season
Apple frying pan
Apple fluffy
Apple felt

query: [to apples] (10)
Apple discoloration
Apple wall
Reason for discoloration of apples
Apple
Apple discoloration time
Apple wall decoration
Apple discoloration sugar
Apple discoloration back
Apple discoloration free study
Apple wall making

query: [Apple Ho] (10)
Apple storage
Apple pancake mix
Apple storage period
Apple pancake mix easy
Apple pancake mix rice cooker
Apple pancake mix pound cake
Apple pancakes
Apple storage room temperature
Apple Hokuto
Apple pancake mix yogurt

query: [Apple] (10)
Apple muffin
Apple round bite
Apples every day
Apple bean knowledge
Whole apple
Apples daily skin
Apples are bad
Apple cartoon
ringo Marunouchi
ringo lineo

query: [Apple] (10)
How to identify apples
Apple honey
Apple mixer
Apple Miki Life
Apple flavor
Apple taste expression
Apple oranges
Apple green
ringo midtown
Miki apple

query: [Apple] (10)
How to peel apples
Apple pesticide-free
Steamed apple bread
Apple mutsu
How to peel apples
Apple swelling
Apple cavities
Steamed apples baby food
Apple mousse
How to peel an apple Rabbit

query: [Apple] (10)
Apple moon
Apple specialty
Apple brand
Apple metaphor
Apple main dish
Apple connoisseur
Apple benefits
Apple menu
ringo menu
Apple mercy

query: [Apples] (10)
Apples
Apple motif
Apple character
Brightness of apples
Apple peach crossing
Apple mold
Apples in the first place
Apple peach
Apple peach allergy
ringo peach

query: [Apples] (10)
Bake apples
Apple baked goods
Apple vegetables
Apple in translation
Apple soft
Roasted apple
Apples lose weight
Apples cheap
Apple yataka
Apple Yamagata

query: [Apple Yu] (10)
Apple dream fortune
Apple bow and arrow
Apples famous prefecture
Apple Yuru Chara
Derived from apples
Apple famous
Apple import
Boil apples
Apple organic acid
Apple Yume Hikari

query: [Apple] (10)
Apple yogurt
Apple sunlight
Apple yogurt cake
Apple night
Apples eat at night
Apple infant food
Apple yogurt recipe
Apple yogurt smoothie
Apple folic acid
How to read apples

query: [Apples] (10)
Apple ranking
Apple Rakuten
Apple Latin
Apple rum
Apple lamp
Apple light
Apple ranking prefectures
Apple wrap
Apple rank
ringo LaLaport Toyosu

query: [Apple] (10)
Apple baby food
Apple dishes
Apple baby food mid-term
Apple baby food late
Apple liqueur
Apple baby food range
Apple baby food recipe
Apple baby food from when
Apple baby food blender
Apple dishes ... omitted

Improvement points

As a result, "apple" and "related words" related to it are given.

In the end, I would like to do the following.

① Select the period (today, yesterday, one week ago)
(2) Extract the number of searches for the word "apple" from Google. (Excluding related words, ex apple + "related words")

I've been searching for hours, but I haven't been able to find the content that gives the number of searches ...
I tried combining it with reference to various blogs, but it became difficult due to malfunctions ... m (__) m

I would like to refer to your answers as much as possible.

  • Answer # 1

    Excuse me, not the answer, but just in case, scraping to Google is a violation of the rules.

    Automated query
    Google's Terms of Service prohibits automated queries, regardless of type, from being sent to Google's system without prior explicit permission from Google. To submit automated queries that waste resources, use software such as WebPosition Gold to send automated queries to Google to find out the ranking of websites and web pages in Google search results with various queries. Includes the act of Not only checking rankings, but also automatic access to other types of Google without permission is considered a violation of our webmaster guidelines and terms of use.
    Automated query

    There seems to be an API, so I'll post it for reference (although I haven't used it).
    Get Google search results using the Custom Search API