All Apps and Add-ons

How to capture the RSS Timestamp for the Syndication Input app?

ahartge
Path Finder

I'm hoping this may help someone, or if I have made a mistake, someone could help with the syntax etc.

When bringing in RSS feeds using the Syndication app, they were being recorded at the time they were imported, not the publish time - so looking for articles published in the past 24 hours was a pain.

I have updated the local props.conf & it appears to be working properly, I would be interested if anyone else has an alternate or better idea on how to do it ?

The key for me was the timestamp lookahead, because the published="<TIME>" is not until way down in the data, instead of in the first 150 characters (default) that Splunk likes.

I realize this may have an impact on indexing speed - but it's RSS feeds, so not expecting high volume.

root@splunkbox:/opt/splunk/etc/apps/syndication/local# cat props.conf 
[syndication]
TIME_PREFIX = published="
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 4000

I would also like to say thank you to the app creator @LukeMurphey for an excellent job delivering this for us.

Cheers,
Ash

vasuparvatham
New Member

Can you please send me instructions to configure this?

Thanks in advance.

0 Karma

LukeMurphey
Champion

I think that is the right way to handle this. You might have to use a different time prefix for other types of feeds (ATOM, RDF) because they may not have a published field. That shouldn't be a problem in your case though.

This also makes me wonder that I should handle them differently in the input itself. I'm considering adding the option to use the published date as the event date (see the ticket here).

Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...