Installation

Python script to screen scrape a web page?

jambajuice
Communicator

I've been experimenting with lookup tables and I'd like to try using an external lookup command. The goal is to extract data from a web page. I've never done any programming in python and the little bit of research I've done is pretty daunting.

All the script needs to do is read the HTML of a web page where the link includes a field value from the event, such as www.externalsite.com/$event_code

Can anyone point me to some python examples that will accomplish this?

Thanks.

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I don't have one, but I would recommend if you're using Python, you use the Beautiful Soup HTML parsing library, which is specifically intended for this. The standard library's HtmlParser and htmllib are rather less robust:

http://www.crummy.com/software/BeautifulSoup/

The other side of this is that you need to fetch the HTML page using an HTTP library. For this, the standard Python httplib is fine.

dwaddle
SplunkTrust
SplunkTrust

Beautiful Soup is most awesome

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...

Introduction to Splunk AI

How are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. Lucky for ...