topic Re: Splunk query for bayesian check in Splunk Search

Splunk query for bayesian check- Why am I getting error?

mahesh21894 — Tue, 30 May 2023 03:31:50 GMT

I am trying to refine search based on a sub query, where sub query is not a filter of outer query. I need to check if certain event happend in the past time(which is different from outer query).

Say current logline is : "Timestamp 9am Log:Info found x=2$ on day1"

I want to search something like this:
app=my-app "found x=2$ on day1" | eval isThereAEventBefore=(subQuery greater than 0, 1, 0)

replace subQuery with: (app=my-app "found x=*$ on day1 earliest=-1h" | stats count)

When i tried to write this query, i s:

Error in 'eval' command: The expression is malformed. Expected ).

Re: Splunk query for bayesian check

yuanliu — Sat, 27 May 2023 10:57:54 GMT

When your code/pseudo code is already giving error or undesired output, it is best to describe the use case/intention/desire in words. If I understand correctly, you are trying to inject a field isThereAEventBefore based on what you call a subquery; when that query returns a count greater than 0, set isThereAEventBefore to 1, otherwise set to 0.

The illustrated pseudo code is not how SPL works. (Additionally, the second search which you enclose in parentheses also contains a misplaced quotation mark. It is more probable that your desired second search contains ("found x=*$ on day1" earliest=-1h) instead of ("found x=*$ on day1 earliest=-1h"). Because you didn't say what was the search period of the first search, I also suspect that if the field you wanted is isThereAEventBefore (as oppsosed to isThereAEventAfter), earliest should be latest. But that I'll leave it to you.

Here is a literal way to implement my speculation of your intention.

app=my-app "found x=2$ on day1" | append [search app=my-app "found x=*$ on day1" earliest=-1h | stats count | eval isThereAEventBefore = if(count > 0, 1, 0)] | eventstats values(isThereAEventBefore) as isThereAEventBefore

In this, both append and eventstats are expensive, especially considering the subsearch is so close to the main search.

You can get away from a single index search with no append in order to improve performance. The logic requires some getting used to. You do a broader search to include both index queries, then mark events to separate, perform stats on one set, then filter out that set that you only need for stats. Assuming "earliest=-1h" is still the correct logic for isThereAEventBefore,

app=my-app "found x=*$ on day1" earliest=-2h ``` 2h is just an example; any value greater than 1h will do ``` | eval is_before = if((now() - relative_time(now, "-1h")) < 0, 1, 0) ``` mark events within 1h for stats ``` | eventstats sum(is_before) as are_before ``` perform stats on x=*$ ``` | search "found x=2$ on day1" ``` only keep events where x=2$ ``` | eval isThereAEventBefore = if(are_before > 0, 1, 0)

Hope this helps.

Re: Splunk query for bayesian check

mahesh21894 — Wed, 31 May 2023 21:57:51 GMT

Apologies for not posting query clearly, i was little confused with my usecase & new to splunk as well.

Here is the quick description of use case:
I would like to flag a splunk record(X) when there exists certain splunk record(Y) in +15mins(future) w.r.t event time of X. This is to filter out some false positives in logs.

Looks like
-> append accumulates results
-> and doesn't run a subsearch for every event faced in outer query.

So it felt like i need a map instead of append, below is a sample query:

<common_search> "Inconsistency with item=*, on Date=*"
| eval pid=itemId, dt=ItemDate, a_latest=_time+900, a_earliest=_time
| map maxsearches=20000 search='search earliest=\"$a_earliest$\" latest=\"$a_latest$\"<common_search> \"consistent with item=$pid$, on Date=$Dt$\" | stats count | eval isThereAEventBefore=if(count>0, 1, 0)'
| eventstats values(isThereAEventAfter) as isThereAEventAfter

common_search:-
index=.. splunk_server_group=.. sourcetype=.. host="*beta*" source="<path>"

Notes:
-> As i don't future logs handy yet, i was looking back on a random log by hardcoding it in subsearch.
-> My query seems to be going into queue & after sometime, i get 0 matches. Though i need in future, currently i am checking back in time _time-900. Can i assume it works similarly for future time as well _time+900

could you please help me if i framed query in right fashion w.r.t my use case ? & why its still gives no matches?