Splunk Search

CLI search with FATAL: The search job terminated unexpectedly

vnguyen46
Contributor

Hello,

I try to export a large log with CLI search below. It works well with a smaller log return, but giving error on large logs, FATAL: The search job terminated unexpectedly.

For instance, this search on Pan_logs terminated:

/opt/splunk/bin/splunk search "index=pan_logs earliest=-7d" -preview 0 -maxout 0 -output rawdata | gzip > pan_logs_7days.gz

Anyone knows how to resolve this issue?

Thanks,

Labels (1)
Tags (2)
0 Karma
1 Solution

vnguyen46
Contributor

@twesty That's another idea.

For small log, I run CLI search directly on the SH: 

/opt/splunk/bin/splunk search "index=small_log earliest=-14d" -preview 0 -maxout 0 -output rawdata | gzip > small_log_14days.gz

I used dump for large logs by running this query on the SH homepage:

index=wineventlog | dump basefilename=WinEventLog rollsize=20000 compress=9 format=raw
the output file saved at this dir: /opt/splunk/var/run/splunk/dispatch/(sid)/dump/

Best,

View solution in original post

0 Karma

twesty
Path Finder

I would suggest running the search in smaller batches. there are many reasons why you could have a failure however the likelihood is that the returning message is just too large to handle.

If you're looking for 7 days worth of data, run 7 separate queries over a day each and then stitch the output together outside of Slpu

0 Karma

vnguyen46
Contributor

@twesty That's another idea.

For small log, I run CLI search directly on the SH: 

/opt/splunk/bin/splunk search "index=small_log earliest=-14d" -preview 0 -maxout 0 -output rawdata | gzip > small_log_14days.gz

I used dump for large logs by running this query on the SH homepage:

index=wineventlog | dump basefilename=WinEventLog rollsize=20000 compress=9 format=raw
the output file saved at this dir: /opt/splunk/var/run/splunk/dispatch/(sid)/dump/

Best,

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...