Webscrapping using BeautifulSoup

 

  • Assign the URL of interest to the variable url.
  • Package the request to the URL, send the request and catch the response with a single function requests.get(), assigning the response to the variable r.
  • Use the text attribute of the object r to return the HTML of the webpage as a string; store the result in a variable html_doc.
  • Create a BeautifulSoup object soup from the resulting HTML using the function BeautifulSoup().
  • Use the method prettify() on soup and assign the result to pretty_soup.
  • # Import packages
    import requests
    from bs4 import BeautifulSoup

    # Specify url: url
    url = 'https://www.python.org/~guido/'


    # Package the request, send the request and catch the response: r
    r = requests.get(url)

    # Extracts the response as html: html_doc
    html_doc = r.text

    # Create a BeautifulSoup object from the HTML: soup
    soup = BeautifulSoup(html_doc)


    # Prettify the BeautifulSoup object: pretty_soup
    pretty_soup = soup.prettify()

    # Print the response
    print(pretty_soup)
    • In the sample code, the HTML response object html_doc has already been created: your first task is to Soupify it using the function BeautifulSoup() and to assign the resulting soup to the variable soup.
    • Extract the title from the HTML soup soup using the attribute title and assign the result to guido_title.
    • Print the title of Guido's webpage to the shell using the print() function.
    • Extract the text from the HTML soup soup using the method get_text() and assign to guido_text.
    • Hit submit to print the text from Guido's webpage to the shell.
# Import packages
import requests
from bs4 import BeautifulSoup

# Specify url: url
url = 'https://www.python.org/~guido/'

# Package the request, send the request and catch the response: r
r = requests.get(url)

# Extract the response as html: html_doc
html_doc = r.text

# Create a BeautifulSoup object from the HTML: soup
soup = BeautifulSoup(html_doc)


# Get the title of Guido's webpage: guido_title
guido_title = soup.title


# Print the title of Guido's webpage to the shell
print(guido_title)

# Get Guido's text: guido_text
guido_text = soup.get_text()


# Print Guido's text to the shell
print(guido_text)
  • Use the method find_all() to find all hyperlinks in soup, remembering that hyperlinks are defined by the HTML tag <a> but passed to find_all() without angle brackets; store the result in the variable a_tags.
  • The variable a_tags is a results set: your job now is to enumerate over it, using a for loop and to print the actual URLs of the hyperlinks; to do this, for every element link in a_tags, you want to print() link.get('href').
  • # Import packages
    import requests
    from bs4 import BeautifulSoup

    # Specify url
    url = 'https://www.python.org/~guido/'

    # Package the request, send the request and catch the response: r
    r = requests.get(url)

    # Extracts the response as html: html_doc
    html_doc = r.text

    # create a BeautifulSoup object from the HTML: soup
    soup = BeautifulSoup(html_doc)

    # Print the title of Guido's webpage
    print(soup.title)

    # Find all 'a' tags (which define hyperlinks): a_tags
    a_tags = soup.find_all('a')


    # Print the URLs to the shell
    for link in a_tags:
        print(link.get('href'))

Comments

Post a Comment

Popular posts from this blog

Binomial Test in Python

  Binomial Test Let’s imagine that we are analyzing the percentage of customers who make a purchase after visiting a website. 1000 customers visited the site this month, and 58 of them made a purchase. The marketing department reports that historical data suggests about 72 of every 1000 visitors make a purchase. Thus, they estimate that the probability of any particular customer making a purchase is 7.2%. We would like to know if this month’s number, 58 purchases, is significantly different from normal or a reasonable fluctuation due to random chance. In previous exercises, we collected samples of numerical information (eg. order price) and then used the mean and standard deviation of those samples to make comparisons. In contrast, we now have a sample where each unit (a visitor) falls into one of two discrete categories: “made a purchase” “did not make a purchase” Instead of comparing sample means, we want to compare the percent in the “made a purchase” category to some expectatio...
Read more

Two - Sample T test

  2 Sample T-Test Suppose that a company has recently updated their website to make it more colorful and inviting. The company wants to know whether the new design is resulting in visitors staying on the site for a longer period of time. A sample of 100 visitors who saw the old design spent an average of 25 minutes on the site. A second sample of 100 visitors who saw the new version spent an average of 28 minutes on the site. Did the average time spent per visitor vary across groups? Or is this difference attributable to random chance? One way of testing whether this difference is significant is by using a  2 Sample T-Test . A 2 Sample T-Test compares two sets of numerical data. The null hypothesis of a 2 Sample T-Test is that the two observed samples come from populations with the same mean. In the example above, this means: if we could observe all site visitors in two alternate universes (one where they see each version of the site), the average visiting times in these unive...
Read more

Web Scraping using JSON and API

JSON–from the web to Python Wow, congrats! You've just queried your first API programmatically in Python and printed the text of the response to the shell. However, as you know, your response is actually a JSON, so you can do one step better and decode the JSON. You can then print the key-value pairs of the resulting dictionary. That's what you're going to do now! Pass the variable  url  to the  requests.get()  function in order to send the relevant request and catch the response, assigning the resultant response message to the variable  r . Apply the  json()  method to the response object  r  and store the resulting dictionary in the variable  json_data . # Import package import   requests # Assign URL to variable: url url  =  'http://www.omdbapi.com/?apikey=72bc447a&t=social+network' # Package the request, send the request and catch the...
Read more