Getting Data In

What is the best way to monitor a web page containing links to xml to index RSS feeds in Splunk?

Motivator

I want to monitor a web page containing links to xml for rss feeds to be indexed into Splunk in real-time. Let's use the following website as an example: https://spotcrime.com/rss.php

What is (are) the best way(s) to monitor that site for indexing? I want to test the method on my Windows machine first. I am interested in learning more about Powershell or Python scripted inputs if that is the best approach.

0 Karma

Champion

If you want to get all of those XML feeds into Splunk without manually entering them making a custom script might be the best approach. I have been considering writing a search command that will allow you to scrape web-pages recursively. I'll take a look at doing that soon (perhaps in the next couple of days); you can monitor progress on that here.

For reference, there are a couple of apps that may be useful to you:

0 Karma

Motivator

I found a couple of links on the subject, but I don't understand specifically what I should do to make it work: https://splunkbase.splunk.com/app/278/ and http://blogs.splunk.com/2012/03/14/indexing-feeds/ Also, some of these articles are a bit old (2012, etc.) so I am unsure about how much has changed since then and if they are not better ways to do it.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!