Splunk Search

Hit URL grab data and index it


The concept seems simply yet there doesn't seem to be a straightforward way of doing it. I have URL which I want splunk to hit and index all the data off the page ever X seconds. Thats it. I don't want to only have it grab certain field just all the data on the page.

I initially tried the add-on feedparser but I'm not having any luck with it.

Tags (2)
0 Karma

Splunk Employee
Splunk Employee


Seems like this is two problems in one.

The first thing you need to solve is grabbing the contents of the web page. Splunk doesn't have anything built in to do this, but it should be easily scriptable via a wget or curl script.

Secondly, how to index that data. Once your script is outputting an html file, just input it into Splunk as you would any other file. It won't look pretty because it's not really a log message, but Splunk will index it fine.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...