All Apps and Add-ons

How to capture the RSS Timestamp for the Syndication Input app?

ahartge
Path Finder

I'm hoping this may help someone, or if I have made a mistake, someone could help with the syntax etc.

When bringing in RSS feeds using the Syndication app, they were being recorded at the time they were imported, not the publish time - so looking for articles published in the past 24 hours was a pain.

I have updated the local props.conf & it appears to be working properly, I would be interested if anyone else has an alternate or better idea on how to do it ?

The key for me was the timestamp lookahead, because the published="<TIME>" is not until way down in the data, instead of in the first 150 characters (default) that Splunk likes.

I realize this may have an impact on indexing speed - but it's RSS feeds, so not expecting high volume.

root@splunkbox:/opt/splunk/etc/apps/syndication/local# cat props.conf 
[syndication]
TIME_PREFIX = published="
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 4000

I would also like to say thank you to the app creator @LukeMurphey for an excellent job delivering this for us.

Cheers,
Ash

vasuparvatham
New Member

Can you please send me instructions to configure this?

Thanks in advance.

0 Karma

LukeMurphey
Champion

I think that is the right way to handle this. You might have to use a different time prefix for other types of feeds (ATOM, RDF) because they may not have a published field. That shouldn't be a problem in your case though.

This also makes me wonder that I should handle them differently in the input itself. I'm considering adding the option to use the published date as the event date (see the ticket here).

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...