Installation

Python script to screen scrape a web page?

jambajuice
Communicator

I've been experimenting with lookup tables and I'd like to try using an external lookup command. The goal is to extract data from a web page. I've never done any programming in python and the little bit of research I've done is pretty daunting.

All the script needs to do is read the HTML of a web page where the link includes a field value from the event, such as www.externalsite.com/$event_code

Can anyone point me to some python examples that will accomplish this?

Thanks.

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I don't have one, but I would recommend if you're using Python, you use the Beautiful Soup HTML parsing library, which is specifically intended for this. The standard library's HtmlParser and htmllib are rather less robust:

http://www.crummy.com/software/BeautifulSoup/

The other side of this is that you need to fetch the HTML page using an HTTP library. For this, the standard Python httplib is fine.

dwaddle
SplunkTrust
SplunkTrust

Beautiful Soup is most awesome

0 Karma
Get Updates on the Splunk Community!

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureOn Demand Now Step boldly into the AI revolution with enhanced security ...