Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Hint: You can adjust the default video playback speed in your account settings.
Hint: You can set your subtitle preferences in your account settings.
Sorry! Looks like there’s an issue with video playback 🙁 This might be due to a temporary outage or because of a configuration issue with your browser. Please refer to our video player troubleshooting guide for assistance.

Inspect Your Data Source (Recap)

Martin Breuss
Web Scraping With Beautiful Soup and Python Martin Breuss 02:11
Recommended Tutorial Course Slides (.pdf) Sample Code (.zip) Ask a Question

00:00 Let’s do a quick recap on Part 1, inspecting your website. This is actually a very important part of the web scraping process because you really need to understand how is the website structured and how do you interact with it before you know how to scrape it.

00:15 What you learned in here is that you first want to go and explore the website as a normal user, which means just heading to the URL with your browser and clicking around the website, essentially seeing how would you interact with it if you weren’t going to be using it with the script to scrape the information, but just using your hands and your eyes, essentially.

00:37 Next, you learned how to decipher the information that is hidden inside of URLs—the parts that a browser uses to communicate with the site, but that you, as a normal user, usually don’t have to worry about.

00:51 But there’s information in there, and you learned about query parameters and how these are structured and how you can add information in here. Change it even up in the URL and the website is going to change.

01:03 And the other way around—if you change something on the website and submit the information, then the URL is going to change. This connection is important, so keep it in mind.

01:12 Your way of interacting with websites here are the query parameters.

01:18 Then, finally, you looked at using developer tools to even deeper inspect the website that you’re working with. You learned how to open them up through a menu item or right-clicking, and then you can hover over different pieces of the website and see the corresponding code in HTML. Or the other way around—click on the HTML, and then see the page structure that relates to it light up. And like this, you can expand, collapse, figure out what are the actual pieces of information in the HTML structure that contain the information that you want, and that you’re going to want to target in your scraping and parsing later on.

01:58 Okay! So, that’s enough working with the browser for now. It’s time to write some Python code. We’re going to move over to Part 2 where you will learn about scraping the website.

Become a Member to join the conversation.