topic issue pulling large data sets from Splunk using splunk cli - truncated output in Monitoring Splunk

issue pulling large data sets from Splunk using splunk cli - truncated output

ebailey — Tue, 29 Sep 2020 06:57:55 GMT

I need to pull a large set data from Splunk as a scheduled job and then redirect the output to shared storage. I have used the splunk cli for this sort of work before and tried it again.

/opt/splunk/bin/splunk search "index=os sourcetype=iostat bandwUtilPct > 0 earliest=-d@d latest=@d | ta
ble Device,_time,avgSvcMillis,avgWaitMillis,bandwUtilPct,host,rKB_PS,rReq_PS,wKB_PS,wReq_PS" -auth 'test:test' -output csv -maxout 0 > /shared/test/SPLUNK/IO_CSV/test_io.csv

If I run this query from the UI i get arond 13-14 million events, but if I run this query from the cli i get a little over 6 million events. I had thought using "-maxout 0 " preventing truncation or could this be something else? I don't see any error messages for the search so I do not know what else could cause the issue.

Any thoughts?

Thanks

Re: issue pulling large data sets from Splunk using splunk cli - truncated output

somesoni2 — Wed, 12 Aug 2015 15:45:48 GMT

Executing regular search from CLI may hit a memory limit. The best way to do it is by using Splunk RESTFUL API . See more details here http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

Re: issue pulling large data sets from Splunk using splunk cli - truncated output

ebailey — Wed, 12 Aug 2015 17:06:09 GMT

Kinda of what I thought - i am lazy so was hoping the built-in tools would work though the rest-api is easy enough.

Thanks!