CLI search with FATAL: The search job terminated unexpectedly

[vnguyen46]

Hello,

I try to export a large log with CLI search below. It works well with a smaller log return, but giving error on large logs, FATAL: The search job terminated unexpectedly.

For instance, this search on Pan_logs terminated:

/opt/splunk/bin/splunk search "index=pan_logs earliest=-7d" -preview 0 -maxout 0 -output rawdata | gzip > pan_logs_7days.gz

Anyone knows how to resolve this issue?

Thanks,

[twesty]

I would suggest running the search in smaller batches. there are many reasons why you could have a failure however the likelihood is that the returning message is just too large to handle.

If you're looking for 7 days worth of data, run 7 separate queries over a day each and then stitch the output together outside of Slpu

[vnguyen46 · Accepted solution]

@twesty That's another idea.

For small log, I run CLI search directly on the SH: 

/opt/splunk/bin/splunk search "index=small_log earliest=-14d" -preview 0 -maxout 0 -output rawdata | gzip > small_log_14days.gz

I used dump for large logs by running this query on the SH homepage:

index=wineventlog | dump basefilename=WinEventLog rollsize=20000 compress=9 format=raw
the output file saved at this dir: /opt/splunk/var/run/splunk/dispatch/(sid)/dump/

Best,