Hit URL grab data and index it

[tb5821]

The concept seems simply yet there doesn't seem to be a straightforward way of doing it. I have URL which I want splunk to hit and index all the data off the page ever X seconds. Thats it. I don't want to only have it grab certain field just all the data on the page.

I initially tried the add-on feedparser but I'm not having any luck with it.

[mloven_splunk]

tb5821,

Seems like this is two problems in one.

The first thing you need to solve is grabbing the contents of the web page. Splunk doesn't have anything built in to do this, but it should be easily scriptable via a wget or curl script.

Secondly, how to index that data. Once your script is outputting an html file, just input it into Splunk as you would any other file. It won't look pretty because it's not really a log message, but Splunk will index it fine.