Posts

Web Scraping Navigation in Tree Data Structures

  To navigate through a tree, we can call the tag names themselves. Imagine we have an HTML page that looks like this: < h1 > World's Best Chocolate Chip Cookies </ h1 > < div class = "banner" > < h1 > Ingredients </ h1 > </ div > < ul > < li > 1 cup flour </ li > < li > 1/2 cup sugar </ li > < li > 2 tbsp oil </ li > < li > 1/2 tsp baking soda </ li > < li > ½ cup chocolate chips </ li > < li > 1/2 tsp vanilla < li > < li > 2 tbsp milk </ li > </ ul > If we made a  soup  object out of this HTML page, we have seen that we can get the first  h1  element by calling: print ( soup . h1 ) <h1>World's Best Chocolate Chip Cookies</h1> We can get the children of a tag by accessing the  .children  attribute: for child in soup . ul . children : print ( child ) <li> 1 cup ...

Web Scraping in Python (CodeCademy

BeautifulSoup is a Python library that makes it easy for us to traverse an HTML page and pull out the parts we’re interested in. We can import it by using the line: "html.parser"  is one option for parsers we could use. There are other options, like  "lxml"  and  "html5lib"  that have different advantages and disadvantages, but for our purposes we will be using  "html.parser"  throughout. With the requests skills we just learned, we can use a website hosted online as that HTML: import   requests from   bs4   import   BeautifulSoup webpage_response  =  requests . get ( 'https://s3.amazonaws.com/codecademy-content/courses/beautifulsoup/shellter.html' ) webpage  =  webpage_response . content soup  =  BeautifulSoup ( webpage ,  "html.parser" ) print( soup ) BeautifulSoup breaks the HTML page into several types of objects. Tags A Tag corresponds to an HTML Tag in the original document. These lines of co...