topic Re: How to capture the RSS Timestamp for the Syndication Input app? in All Apps and Add-ons

How to capture the RSS Timestamp for the Syndication Input app?

ahartge — Wed, 08 Jul 2015 04:23:22 GMT

I'm hoping this may help someone, or if I have made a mistake, someone could help with the syntax etc.

When bringing in RSS feeds using the Syndication app, they were being recorded at the time they were imported, not the publish time - so looking for articles published in the past 24 hours was a pain.

I have updated the local props.conf & it appears to be working properly, I would be interested if anyone else has an alternate or better idea on how to do it ?

The key for me was the timestamp lookahead, because the published="<TIME>" is not until way down in the data, instead of in the first 150 characters (default) that Splunk likes.

I realize this may have an impact on indexing speed - but it's RSS feeds, so not expecting high volume.

root@splunkbox:/opt/splunk/etc/apps/syndication/local# cat props.conf 
[syndication]
TIME_PREFIX = published="
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 4000

I would also like to say thank you to the app creator @LukeMurphey for an excellent job delivering this for us.

Cheers,
Ash

Re: How to capture the RSS Timestamp for the Syndication Input app?

LukeMurphey — Wed, 08 Jul 2015 05:50:22 GMT

I think that is the right way to handle this. You might have to use a different time prefix for other types of feeds (ATOM, RDF) because they may not have a published field. That shouldn't be a problem in your case though.

This also makes me wonder that I should handle them differently in the input itself. I'm considering adding the option to use the published date as the event date (see the ticket here).

Re: How to capture the RSS Timestamp for the Syndication Input app?

vasuparvatham — Mon, 04 May 2020 11:38:25 GMT

Can you please send me instructions to configure this?

Thanks in advance.