topic Re: Python script to screen scrape a web page? in Installation

Python script to screen scrape a web page?

jambajuice — Wed, 15 Dec 2010 04:34:59 GMT

I've been experimenting with lookup tables and I'd like to try using an external lookup command. The goal is to extract data from a web page. I've never done any programming in python and the little bit of research I've done is pretty daunting.

All the script needs to do is read the HTML of a web page where the link includes a field value from the event, such as www.externalsite.com/$event_code

Can anyone point me to some python examples that will accomplish this?

Thanks.

Re: Python script to screen scrape a web page?

gkanapathy — Wed, 15 Dec 2010 05:42:05 GMT

I don't have one, but I would recommend if you're using Python, you use the Beautiful Soup HTML parsing library, which is specifically intended for this. The standard library's HtmlParser and htmllib are rather less robust:

http://www.crummy.com/software/BeautifulSoup/

The other side of this is that you need to fetch the HTML page using an HTTP library. For this, the standard Python httplib is fine.

Re: Python script to screen scrape a web page?

dwaddle — Wed, 15 Dec 2010 06:43:16 GMT

Beautiful Soup is most awesome