Splunk Search

How do I export large amounts of data?

XOJ
Path Finder

I am having issues with finding a way to export two reports.

I have two reports, which I'll call search1 and search2. Both searches were run, then ran in the background. According to the jobs tab, both searches completed. The customer wanted this search run for "all-time" and thus is quite large. Search1 is 9.22GB and Search2 is 4.97GB.

The issue is getting access to the logs.

I've tried using | loadjob sid, and it just hangs and fails.

I've tried exporting from the jobs tab, and it fails.

I can't use the api, because from what I can tell, you must put the password into the search, when then makes the password searchable for anyone with access to that log.

I went to the $SPLUNK_HOME/var/run/splunk/dispatch folder and found both jobs where this link, https://docs.splunk.com/Documentation/Splunk/8.2.1/Troubleshooting/CommandlinetoolsforusewithSupport... says to run "splunk cmd splunkd toCsv ./results.srs.gz". the .gz file appears to now be .zst, but I ran the command.

Search1 after a while simply said "killed".

Search2 as I'm writing this appears to be working, as it appears comma delimited text is scrolling on the console. I assume that once changed, I will be able to export this one.

So how do I export Search1 and other large files in the future? The toCsv command was the last thing I found to try. Perhaps there is a setting in a .conf file I can modify and then run something else? Any assistance is appreciated.

Labels (1)
Tags (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Consider running multiple searches over smaller time ranges and then combining the results.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @XOJ,

dump command may help you to export a large amount of data.

https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Dump 

Below will create daily dump files

index=yourindex | eval _dstpath=strftime(_time, "%Y%m%d") | dump basefilename=search1
If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Consider running multiple searches over smaller time ranges and then combining the results.

---
If this reply helps you, Karma would be appreciated.
0 Karma

XOJ
Path Finder

I hate that this is the answer. People have businesses much bigger than ours, and even they have to make tiny searches?

That being said, you are the only one that gave an answer, so I will mark it as such.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...