Splunk Search

Hit URL grab data and index it

tb5821
Communicator

The concept seems simply yet there doesn't seem to be a straightforward way of doing it. I have URL which I want splunk to hit and index all the data off the page ever X seconds. Thats it. I don't want to only have it grab certain field just all the data on the page.

I initially tried the add-on feedparser but I'm not having any luck with it.

Tags (2)
0 Karma

mloven_splunk
Splunk Employee
Splunk Employee

tb5821,

Seems like this is two problems in one.

The first thing you need to solve is grabbing the contents of the web page. Splunk doesn't have anything built in to do this, but it should be easily scriptable via a wget or curl script.

Secondly, how to index that data. Once your script is outputting an html file, just input it into Splunk as you would any other file. It won't look pretty because it's not really a log message, but Splunk will index it fine.

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...