Monitoring Splunk

issue pulling large data sets from Splunk using splunk cli - truncated output

ebailey
Communicator

I need to pull a large set data from Splunk as a scheduled job and then redirect the output to shared storage. I have used the splunk cli for this sort of work before and tried it again.

/opt/splunk/bin/splunk search "index=os sourcetype=iostat bandwUtilPct > 0 earliest=-d@d latest=@d | ta
ble Device,_time,avgSvcMillis,avgWaitMillis,bandwUtilPct,host,rKB_PS,rReq_PS,wKB_PS,wReq_PS" -auth 'test:test' -output csv -maxout 0 > /shared/test/SPLUNK/IO_CSV/test_io.csv

If I run this query from the UI i get arond 13-14 million events, but if I run this query from the cli i get a little over 6 million events. I had thought using "-maxout 0 " preventing truncation or could this be something else? I don't see any error messages for the search so I do not know what else could cause the issue.

Any thoughts?

Thanks

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

Executing regular search from CLI may hit a memory limit. The best way to do it is by using Splunk RESTFUL API . See more details here http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

View solution in original post

somesoni2
Revered Legend

Executing regular search from CLI may hit a memory limit. The best way to do it is by using Splunk RESTFUL API . See more details here http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

ebailey
Communicator

Kinda of what I thought - i am lazy so was hoping the built-in tools would work though the rest-api is easy enough.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Leveraging Detections from the Splunk Threat Research Team & Cisco Talos

  Now On Demand  Stay ahead of today’s evolving threats with the combined power of the Splunk Threat Research ...

New in Splunk Observability Cloud: Automated Archiving for Unused Metrics

Automated Archival is a new capability within Metrics Management; which is a robust usage & cost optimization ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...