Installation

Python script to screen scrape a web page?

jambajuice
Communicator

I've been experimenting with lookup tables and I'd like to try using an external lookup command. The goal is to extract data from a web page. I've never done any programming in python and the little bit of research I've done is pretty daunting.

All the script needs to do is read the HTML of a web page where the link includes a field value from the event, such as www.externalsite.com/$event_code

Can anyone point me to some python examples that will accomplish this?

Thanks.

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I don't have one, but I would recommend if you're using Python, you use the Beautiful Soup HTML parsing library, which is specifically intended for this. The standard library's HtmlParser and htmllib are rather less robust:

http://www.crummy.com/software/BeautifulSoup/

The other side of this is that you need to fetch the HTML page using an HTTP library. For this, the standard Python httplib is fine.

dwaddle
SplunkTrust
SplunkTrust

Beautiful Soup is most awesome

0 Karma
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Unmerging HTML Tables

[Puzzles] Solve, Learn, Repeat: Unmerging HTML TablesFor a previous puzzle, I needed some sample data, and ...

Enterprise Security (ES) Essentials 8.3 is Now GA — Smarter Detections, Faster ...

As of today, Enterprise Security (ES) Essentials 8.3 is now generally available, helping SOC teams simplify ...

AI for AppInspect

We’re excited to announce two new updates to AppInspect designed to save you time and make the app approval ...