Monitoring Splunk

How to catch ERROR events in search processes

lukasz92
Communicator

Hi,

Is it possible to create a search, that finds all "ERROR" messages in search.log for all search jobs?
I tried to search it in _internal - but not found.

Tags (2)
0 Karma
1 Solution

hunters_splunk
Splunk Employee
Splunk Employee

Hi lukasz92,

The short answer is No. search.log files are not stored under $SPLUNK_HOME/var/log/splunk/ but are written to SPLUNK_HOME/var/run/splunk/dispatch// .
Scheduled jobs (scheduled saved searches) include the saved search name as part of the directory name.

Search jobs manifest as a process in the OS. There are two processes in Linux for each search job: search-launcher and process-runner. You can isolate all the Splunk search processes with: ps -ef | grep search. The main job is the one using system resources and contains search --id in its name.

Hope this helps. Thanks!
Hunter

View solution in original post

0 Karma

hunters_splunk
Splunk Employee
Splunk Employee

Hi lukasz92,

The short answer is No. search.log files are not stored under $SPLUNK_HOME/var/log/splunk/ but are written to SPLUNK_HOME/var/run/splunk/dispatch// .
Scheduled jobs (scheduled saved searches) include the saved search name as part of the directory name.

Search jobs manifest as a process in the OS. There are two processes in Linux for each search job: search-launcher and process-runner. You can isolate all the Splunk search processes with: ps -ef | grep search. The main job is the one using system resources and contains search --id in its name.

Hope this helps. Thanks!
Hunter

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...