I'm hoping this may help someone, or if I have made a mistake, someone could help with the syntax etc.
When bringing in RSS feeds using the Syndication app, they were being recorded at the time they were imported, not the publish time - so looking for articles published in the past 24 hours was a pain.
I have updated the local props.conf & it appears to be working properly, I would be interested if anyone else has an alternate or better idea on how to do it ?
The key for me was the timestamp lookahead, because the published="<TIME>" is not until way down in the data, instead of in the first 150 characters (default) that Splunk likes.
I realize this may have an impact on indexing speed - but it's RSS feeds, so not expecting high volume.
root@splunkbox:/opt/splunk/etc/apps/syndication/local# cat props.conf
TIME_PREFIX = published="
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD = 4000
I would also like to say thank you to the app creator @LukeMurphey for an excellent job delivering this for us.
I think that is the right way to handle this. You might have to use a different time prefix for other types of feeds (ATOM, RDF) because they may not have a published field. That shouldn't be a problem in your case though.
This also makes me wonder that I should handle them differently in the input itself. I'm considering adding the option to use the published date as the event date (see the ticket here).