All Apps and Add-ons

How to capture the RSS Timestamp for the Syndication Input app?

ahartge
Path Finder

I'm hoping this may help someone, or if I have made a mistake, someone could help with the syntax etc.

When bringing in RSS feeds using the Syndication app, they were being recorded at the time they were imported, not the publish time - so looking for articles published in the past 24 hours was a pain.

I have updated the local props.conf & it appears to be working properly, I would be interested if anyone else has an alternate or better idea on how to do it ?

The key for me was the timestamp lookahead, because the published="<TIME>" is not until way down in the data, instead of in the first 150 characters (default) that Splunk likes.

I realize this may have an impact on indexing speed - but it's RSS feeds, so not expecting high volume.

root@splunkbox:/opt/splunk/etc/apps/syndication/local# cat props.conf 
[syndication]
TIME_PREFIX = published="
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 4000

I would also like to say thank you to the app creator @LukeMurphey for an excellent job delivering this for us.

Cheers,
Ash

vasuparvatham
New Member

Can you please send me instructions to configure this?

Thanks in advance.

0 Karma

LukeMurphey
Champion

I think that is the right way to handle this. You might have to use a different time prefix for other types of feeds (ATOM, RDF) because they may not have a published field. That shouldn't be a problem in your case though.

This also makes me wonder that I should handle them differently in the input itself. I'm considering adding the option to use the published date as the event date (see the ticket here).

Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...