topic What is the SPL for cancelled / timed out searches? in Splunk Search

What is the SPL for cancelled / timed out searches?

McMac84 — Tue, 06 Jun 2023 13:07:38 GMT

Hi everyone, I've a scenario where Splunk is timing out in querying customer SIEM environments and reporting as potential dropped logs.

If I check in the customer's SIEM, I see that there are no dropped logs, so I know the issue is to do with Splunk querying their environment.

Knowing that, I am trying to craft an SPL that looks for searches that were canceled or timed out. Can someone help me with this?

What I've tried using is below but it's not accurately correlating to the time stamps I am seeing in the ticket for my customer.

Can someone help me build this out more.

index=<customer name> cancelled

danspav — Tue, 06 Jun 2023 04:32:16 GMT

Running searches are all logged to the _audit index - you can get details on cancelled searches there:

index=_audit action=search info IN("cancel","cancelled") | timechart span=15m count by user limit=15

I believe "cancel" is system-cancelled, and "cancelled" is user-cancelled, but that's just my guess based on this docs page.

Cheers,
Daniel

isoutamo — Tue, 06 Jun 2023 07:32:21 GMT

You could/should use MC on splunk. There is dashboards where you could look skipped and deferred searches.

Settings -> MC

Search -> Scheduler Activity -> Instance or Deployment based on your system