All Apps and Add-ons

How to capture the RSS Timestamp for the Syndication Input app?

ahartge
Path Finder

I'm hoping this may help someone, or if I have made a mistake, someone could help with the syntax etc.

When bringing in RSS feeds using the Syndication app, they were being recorded at the time they were imported, not the publish time - so looking for articles published in the past 24 hours was a pain.

I have updated the local props.conf & it appears to be working properly, I would be interested if anyone else has an alternate or better idea on how to do it ?

The key for me was the timestamp lookahead, because the published="<TIME>" is not until way down in the data, instead of in the first 150 characters (default) that Splunk likes.

I realize this may have an impact on indexing speed - but it's RSS feeds, so not expecting high volume.

root@splunkbox:/opt/splunk/etc/apps/syndication/local# cat props.conf 
[syndication]
TIME_PREFIX = published="
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 4000

I would also like to say thank you to the app creator @LukeMurphey for an excellent job delivering this for us.

Cheers,
Ash

vasuparvatham
New Member

Can you please send me instructions to configure this?

Thanks in advance.

0 Karma

LukeMurphey
Champion

I think that is the right way to handle this. You might have to use a different time prefix for other types of feeds (ATOM, RDF) because they may not have a published field. That shouldn't be a problem in your case though.

This also makes me wonder that I should handle them differently in the input itself. I'm considering adding the option to use the published date as the event date (see the ticket here).

Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...