topic Re: Data location to send to a remote server in Splunk Enterprise

Data location to send to a remote server

davidbien — Thu, 26 Apr 2018 14:27:09 GMT

Hi,
I have setup Splunk to receive syslogs from our network. These work fine and I can view them in Splunk.
Next step I want to take is to send those logs to logz.io to keep them in cloud. The problem I have is that I don't know where the data collected by Splunk is stored. Where can i find the logs collected by Splunk? What format is it in?

Thanks.

Re: Data location to send to a remote server

marian_coman — Tue, 29 Sep 2020 19:15:55 GMT

Hi,
Splunk logs are collected in buckets (hot, warm, cold, frozen, thawed) and depending on the environment settings they follow the process of roll out from hot -> warm -> cold. It has an entire process for this and you could better have a look here for more details:

http://docs.splunk.com/Documentation/Splunk/7.0.3/Indexer/HowSplunkstoresindexes

One note here is that hot buckets (most recent data) is in active writing stage, so you can't backup/move that data until in reaches warm stage.

If you want to sent those logs to cloud, for archiving purposes, you could copy the relevant archives to the cloud and you will be able later to bring them back to thawed and search them. To find out the buckets you need to backup you could use in search | dbinspect
to identify them. Location is usually in $SPLUNK_HOME/var/lib/splunk/your_index/db/ but cold data can be moved to an external storage (other Volume).

| dbinspect index=your_index
| table bucketId, startEpoch, endEpoch, id, index, modTime, path, sizeOnDiskMB, splunk_server, state

Regards

Re: Data location to send to a remote server

davidbien — Fri, 27 Apr 2018 09:11:39 GMT

Thank you for your answer, Marian.
Is it possible to send the logs in a human readable format to cloud? Or would I need to use Splunk every time I want to read them again?

Re: Data location to send to a remote server

marian_coman — Fri, 27 Apr 2018 10:40:35 GMT

The logs (Splunk db - tsidx) are not human readable, they're Splunk proprietary format. You'll have to re-index them to Splunk to be able to query them.

Re: Data location to send to a remote server

davidbien — Fri, 27 Apr 2018 10:44:25 GMT

Thank you Marian, you've been very helpful.