Splunk Enterprise

Data location to send to a remote server

davidbien
Engager

Hi,
I have setup Splunk to receive syslogs from our network. These work fine and I can view them in Splunk.
Next step I want to take is to send those logs to logz.io to keep them in cloud. The problem I have is that I don't know where the data collected by Splunk is stored. Where can i find the logs collected by Splunk? What format is it in?

Thanks.

0 Karma
1 Solution

marian_coman
Explorer

Hi,
Splunk logs are collected in buckets (hot, warm, cold, frozen, thawed) and depending on the environment settings they follow the process of roll out from hot -> warm -> cold. It has an entire process for this and you could better have a look here for more details:

http://docs.splunk.com/Documentation/Splunk/7.0.3/Indexer/HowSplunkstoresindexes

One note here is that hot buckets (most recent data) is in active writing stage, so you can't backup/move that data until in reaches warm stage.

If you want to sent those logs to cloud, for archiving purposes, you could copy the relevant archives to the cloud and you will be able later to bring them back to thawed and search them. To find out the buckets you need to backup you could use in search | dbinspect
to identify them. Location is usually in $SPLUNK_HOME/var/lib/splunk/your_index/db/ but cold data can be moved to an external storage (other Volume).

| dbinspect index=your_index
| table bucketId, startEpoch, endEpoch, id, index, modTime, path, sizeOnDiskMB, splunk_server, state

Regards

View solution in original post

marian_coman
Explorer

Hi,
Splunk logs are collected in buckets (hot, warm, cold, frozen, thawed) and depending on the environment settings they follow the process of roll out from hot -> warm -> cold. It has an entire process for this and you could better have a look here for more details:

http://docs.splunk.com/Documentation/Splunk/7.0.3/Indexer/HowSplunkstoresindexes

One note here is that hot buckets (most recent data) is in active writing stage, so you can't backup/move that data until in reaches warm stage.

If you want to sent those logs to cloud, for archiving purposes, you could copy the relevant archives to the cloud and you will be able later to bring them back to thawed and search them. To find out the buckets you need to backup you could use in search | dbinspect
to identify them. Location is usually in $SPLUNK_HOME/var/lib/splunk/your_index/db/ but cold data can be moved to an external storage (other Volume).

| dbinspect index=your_index
| table bucketId, startEpoch, endEpoch, id, index, modTime, path, sizeOnDiskMB, splunk_server, state

Regards

davidbien
Engager

Thank you for your answer, Marian.
Is it possible to send the logs in a human readable format to cloud? Or would I need to use Splunk every time I want to read them again?

0 Karma

marian_coman
Explorer

The logs (Splunk db - tsidx) are not human readable, they're Splunk proprietary format. You'll have to re-index them to Splunk to be able to query them.

0 Karma

davidbien
Engager

Thank you Marian, you've been very helpful.

0 Karma
Get Updates on the Splunk Community!

New Case Study Shows the Value of Partnering with Splunk Academic Alliance

The University of Nevada, Las Vegas (UNLV) is another premier research institution helping to shape the next ...

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...