issue pulling large data sets from Splunk using splunk cli - truncated output

[ebailey]

I need to pull a large set data from Splunk as a scheduled job and then redirect the output to shared storage. I have used the splunk cli for this sort of work before and tried it again.

/opt/splunk/bin/splunk search "index=os sourcetype=iostat bandwUtilPct > 0 earliest=-d@d latest=@d | ta
ble Device,time,avgSvcMillis,avgWaitMillis,bandwUtilPct,host,rKBPS,rReqPS,wKBPS,wReqPS" -auth 'test:test' -output csv -maxout 0 > /shared/test/SPLUNK/IOCSV/test_io.csv

If I run this query from the UI i get arond 13-14 million events, but if I run this query from the cli i get a little over 6 million events. I had thought using "-maxout 0 " preventing truncation or could this be something else? I don't see any error messages for the search so I do not know what else could cause the issue.

Any thoughts?

Thanks

[somesoni2 · Accepted solution]

Executing regular search from CLI may hit a memory limit. The best way to do it is by using Splunk RESTFUL API . See more details here http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

[ebailey]

Kinda of what I thought - i am lazy so was hoping the built-in tools would work though the rest-api is easy enough.

Thanks!