Splunk 6 Can it index old rotated log files?

[aliimami]

Hi.

I am testing out splunk and splunk storm for our cluster deployment. In our pilot, I have set up a single host quite well and am receiving data on splunk storm for now.

However, we have old log files that go back almost 4 years. Is it possible to add those files to splunk, by using the same sourcetypes as the ones I have designated for the live data? We have a lot of custom applications logging to /var/log and we have given them all custom types.

Thanks in advance.

[martin_mueller · Accepted solution]

Sure.

Point a Splunk forwarder at the old files like you do with new files, and it'll eat them all up. Two things to keep in mind: Try to do it all in one day, so you only cause one license violation... and, if you have data older than five years, you need to set your sourcetypes to allow such old data, by default Splunk drops data older than 2000 days.

As for Splunk Storm, I'm not quite sure if you need to do additional changes or if there are other restrictions. The above approach is based on Splunk Enterprise.

[aliimami]

Yeah. Storm is space based. Thanks for the Licensing tip off.

[martin_mueller]

Yeah, see http://docs.splunk.com/Documentation/Splunk/6.0.1/Admin/HowSplunklicensingworks for reference.

When bulk-backfilling old data it's best to do it all in one go. That's Splunk Enterprise licensing of course, not sure how Splunk Storm works. Isn't that space-based? Maybe also with restrictions on the retention time?

[aliimami]

License violation???

And yes, Enterprise helps because I will eventually be deploying splunk enterprise at $WORK. Thanks.

[aliimami]

To clarify, I can manually upload old files. But can Splunk automatically load old logrotated files?