topic Re: How to send indexed logs from Splunk to third party in Getting Data In

How to send indexed logs from Splunk to third party

ayushchoudhary — Wed, 29 May 2019 10:10:51 GMT

Is it possible to have old archived (indexed) logs from Splunk to send to
any other third party solution ensuring that the log keeps their original raw format and timestamps?

Re: How to send indexed logs from Splunk to third party

wyfwa4 — Wed, 29 May 2019 15:05:03 GMT

Assuming you mean that you want to send the data before it is deleted from Splunk - then the answer is yes. See this link. Please note that the sending to the third party is not managed by Splunk you will need to manage that part yourself.

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Automatearchiving

if you set-up a frozen path for an index, then Splunk will write the data to that path before removing/deleting from the index. You can then use another tool to send the data. For example you could set-up syslog to write the contents of the frozen folder to a third party system. Another option is to define a frozen script which will run when data is being deleted - which would allow a more complex data processing function to be implemented.

Re: How to send indexed logs from Splunk to third party

ayushchoudhary — Fri, 31 May 2019 01:53:09 GMT

Thanks @wyfwa4 , but can you please let me know the data will be in raw format, and how to read the data as it was already compressed by Splunk.

Re: How to send indexed logs from Splunk to third party

wyfwa4 — Fri, 31 May 2019 13:08:31 GMT

There are two options for the frozen data - either writing the Splunk formatted buckets to disk (coldToFrozenDir) or run a script (coldToFrozenScript), both are configured on the index, in the indexes.conf file see - https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf.

The coldToFrozenScript will run any defined script before the data is deleted - so it depends on what your script does to determine whether the raw data is in its original form or not. There is a sample script provided ($SPLUNK_HOME/bin/coldToFrozenExample.py) which can be modified for your requirements. The script contains a sample function called "handleOldFlatfileExport". This is not run unless you modify the code to use this function. There are also some articles about using this script - https://answers.splunk.com/answers/338594/does-anyone-have-a-working-example-of-coldtofrozen.html

You would need to modify the script and test to see if you can get it working with your data. I would set-up a test index and set the "frozenTimePeriodInSecs" to a low number and then use a test file to import data into that index. You can then test the script in a short timeframe