How to run selected correlation searches in a time-frame?

[ishanmeena]

Is there a way we can run selected correlation searches in a certain time-frame at once or in queue?

Use Case: In case something happens to the SIEM / Pipeline and the searches are not performed and SOC doesn't get alerts. We can use the query to run the selected correlation searches in the selected time-frame via ES Search app.

Is that possible? 

Also, if it is. Can we create a dashboard out of it for ease of use.

[PickleRick]

If I decipher your use case correctly, it looks like this:

1. Your Splunk goes down at - let's say - noon

2. You restore it to functionality at 9PM

3. Hopefully your data from noon to 9PM was buffered in your forwarders or is ingested on a "pull" basis and can be "backfilled"

And now you're at around 9:30PM but your scheduled searches didn't fire between noon and 9PM so you'd want to re-run them against corresponding time ranges as if they had been run at 1PM, 1:30 and so on. Do I get it right?

There is no built-in mechanics for something like that. You'd have to run each search manually, unfortunatelly. You could try to script it and dispatch the searches via REST API but I don't know of any ready-made app for this.

[ishanmeena]

Yee, exactly. 

So, we've a SOC with multiple use-cases. Sometimes the pipeline between the ticketing tool and Splunk goes down or sometimes Splunk itself goes down. So, we need to run the correlation searches in that time-frame manually to check for any alerts. What we're trying to do is create a query / dashboard to do that for you.

As of now, I could only find the below query but it goes irregular results or sometimes doesn't work at all.

| makeresults 
| eval search=[| rest splunk_server=local /servicesNS/-/-/saved/searches | where title="SomeAlert" | fields qualifiedSearch | rename qualifiedSearch as query | format "" "" "" "" "" ""]
| map search="| makeresults | map search="$search$

[PickleRick]

Well, you can try to do some duct-tape and zip-ties walkarounds but your main problem apparently is that your splunk instance goes down. Apart from non-fired saved searches it can result in data loss when - for example - your forwarders are not able to buffer the data.

As to your search (pasted with errors 😉 but don't worry, I get the idea) - of course it would need some deeper diagnostics in your particular case but most typicaly you'd have problems with time ranges of searches spawned this way.

Let's say the original search was supposed to run at 1:05 am. The search definition contans "earliest=-1h@h latest=@h". If you run it at 7:15am it won't look for events from midnight till 1am but for the ones betweem 6am and 7am.

Another thing which could affect your results is quality of your data. If you have events which are timestamped at "current", not at timestamp extracted from the event itself it will be the opposite to previous paragraph - if your event got buffered for several hours it would get indexed at, let's say 7:12am (even if it was generated at 0:43am) so if you were searching for it back in the "original" time range you wouldn't find it.

It's quite complicated and many thing can (and will ;-)) go wrong if you require HA but don't have it.

[gcusello]

Hi @ishanmeena,

as I said, you can have two situations:

•  indexers down: you cannot access data!
• Search Heads down, you can have a rescue Splunk installation, but the first thing to do is to have an HA infrastructure with a Search Head Cluster or a VM-Ware solution that restore the SHs very quickly.

If you cannot have none of the above solutions.

The only solution is anothr SH ready to intervene, bt you muste be sure that it's aligned with the main installation.

If you have a SOC, you MUST have one of the above solution to be sure to have a continous monitoring with fail over capabilities.

Ciao.

Giuseppe

[gcusello]

Hi @ishanmeena,

there are only two situations in which one or more Correlation Searches aren't executed:

• The indexers are down
• the Seach Heads are down.

In  the first case, you don't have any data to search, so it isn't possible to run any search on data abd the only solution is restart the indexers.

In the second case, your architecture should provide a back-up Search Head to use to run all the searches of the primary.

If you have a Search Head Cluster this is automatic.

If you don't have A SHC, you could have up and running another Search Head, containing all the Correlation Searches but disabled, and you could enable them if the primary is down.

But in this case, the question is why you don't use both of them in the nornal conditions and only one in case of fault of one?

The problem is that you should manually enable the Correlation Searches, because (without SHC) there isn't any orchestration between the two Search Heads.

For this reason, the best solution I hint is a Search Head Cluster.

The only alternative solution, that gives you only a very little lack of service, is having a cold copy of the search Head, continously updated by the VM-Ware infrastruture , that automatically starts when the first is down.

Ciao.

Giuseppe