Monitoring Splunk

issue pulling large data sets from Splunk using splunk cli - truncated output

ebailey
Communicator

I need to pull a large set data from Splunk as a scheduled job and then redirect the output to shared storage. I have used the splunk cli for this sort of work before and tried it again.

/opt/splunk/bin/splunk search "index=os sourcetype=iostat bandwUtilPct > 0 earliest=-d@d latest=@d | ta
ble Device,_time,avgSvcMillis,avgWaitMillis,bandwUtilPct,host,rKB_PS,rReq_PS,wKB_PS,wReq_PS" -auth 'test:test' -output csv -maxout 0 > /shared/test/SPLUNK/IO_CSV/test_io.csv

If I run this query from the UI i get arond 13-14 million events, but if I run this query from the cli i get a little over 6 million events. I had thought using "-maxout 0 " preventing truncation or could this be something else? I don't see any error messages for the search so I do not know what else could cause the issue.

Any thoughts?

Thanks

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

Executing regular search from CLI may hit a memory limit. The best way to do it is by using Splunk RESTFUL API . See more details here http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

View solution in original post

somesoni2
Revered Legend

Executing regular search from CLI may hit a memory limit. The best way to do it is by using Splunk RESTFUL API . See more details here http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

View solution in original post

ebailey
Communicator

Kinda of what I thought - i am lazy so was hoping the built-in tools would work though the rest-api is easy enough.

Thanks!

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.