Splunk Search

CLI search with FATAL: The search job terminated unexpectedly

vnguyen46
Contributor

Hello,

I try to export a large log with CLI search below. It works well with a smaller log return, but giving error on large logs, FATAL: The search job terminated unexpectedly.

For instance, this search on Pan_logs terminated:

/opt/splunk/bin/splunk search "index=pan_logs earliest=-7d" -preview 0 -maxout 0 -output rawdata | gzip > pan_logs_7days.gz

Anyone knows how to resolve this issue?

Thanks,

Labels (1)
Tags (2)
0 Karma
1 Solution

vnguyen46
Contributor

@twesty That's another idea.

For small log, I run CLI search directly on the SH: 

/opt/splunk/bin/splunk search "index=small_log earliest=-14d" -preview 0 -maxout 0 -output rawdata | gzip > small_log_14days.gz

I used dump for large logs by running this query on the SH homepage:

index=wineventlog | dump basefilename=WinEventLog rollsize=20000 compress=9 format=raw
the output file saved at this dir: /opt/splunk/var/run/splunk/dispatch/(sid)/dump/

Best,

View solution in original post

0 Karma

twesty
Path Finder

I would suggest running the search in smaller batches. there are many reasons why you could have a failure however the likelihood is that the returning message is just too large to handle.

If you're looking for 7 days worth of data, run 7 separate queries over a day each and then stitch the output together outside of Slpu

0 Karma

vnguyen46
Contributor

@twesty That's another idea.

For small log, I run CLI search directly on the SH: 

/opt/splunk/bin/splunk search "index=small_log earliest=-14d" -preview 0 -maxout 0 -output rawdata | gzip > small_log_14days.gz

I used dump for large logs by running this query on the SH homepage:

index=wineventlog | dump basefilename=WinEventLog rollsize=20000 compress=9 format=raw
the output file saved at this dir: /opt/splunk/var/run/splunk/dispatch/(sid)/dump/

Best,

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...