How to search two different strings from the same source, but different timestamps?

[Abilan1]

Hi ,

I would like to know how to search two different search strings (Error and issue) from the same source file, but the error and issue both have different timestamps, so am unable to search in direct way. Please help me here. Thank You!

[diogofgm · Accepted solution]

it took me some time to figure this out but i believe this is what you are looking for. (math logic)
Not the most performant search query but works.
replace my_index with your index and try this:

index=my_index "Handle State structures to abandoned" | stats count by source | eval status = "keep" | append [search index=my_index "KERNEL RECYCLING" | stats count by source | eval status = "remove" ] | stats values(status) AS status by source | search status!=remove

Small explanation on this:
1- list the sources that have "Handle State structures to abandoned" and build a field called status with "keep" as value

index=my_index "Handle State structures to abandoned" | stats count by source | eval status = "keep"

2 - list the sources that have "KERNEL RECYCLING", build a field called status with "remove" as value and append the result to 1:

...| append [search index=my_index "KERNEL RECYCLING" | stats count by source | eval status = "remove" ] 

3 - list both results by status (sources that you want keep or remove) and filter the ones you do not want to show:

...| stats values(status) AS status by source | search status!=remove

[Abilan1]

Hi ,

Thank you so much! this is perfect. I wanted it in the dashboard..X axis (date_month) and count of event (Y axis). Can you please help me?

[diogofgm]

splitted by source?

[Abilan1]

Hi,
Total number of source is event count by date (X axis)

[diogofgm]

Try this:
index=my_index "Handle State structures to abandoned" | eval month=strftime(_time,"%m") | stats values(source) AS source by month | mvexpand source | eval status = "keep"
| append [search index=my_index "KERNEL RECYCLING" | eval month=strftime(_time,"%m") | stats values(source) AS source by month | mvexpand source | eval status = "remove" ]
| stats values(month) AS month values(status) AS status by source | search status!=remove | mvexpand month | stats count AS sources by month

[Abilan1]

Hi ,

Thanks again, this is perfect..is it possible to add colours in this dashboard. I am unable to open this in Pivot?

[diogofgm]

to have different colours for each month add the following in the end of the last search:
| xyseries month month sources
and then in the chart format options > general > stack mode choose stacked

what do you mean open this in pivot?

[Abilan1]

Hi ,

Thank you so much again!!!! never mind about that Pivot related question..

[diogofgm]

if you use OR you can search for both.

source=my_file Error OR issue

[Abilan1]

Hi ,

I am searching in real time , I cannot give specific source file name and I think the command which you have mentioned will work if Error OR issue will be in same indexed data (time stamp). I am looking for different time stamp.

[diogofgm]

I wrote source=my_file only as an example. You can use whatever fits your search query (e.g. index=my_index, host=my_host, etc ). It doesn't matter if you're doing a real-time search.
About the Error OR issue, this will show you all the events (different timestamps) that contains one of the strings or both in the same event.

[aljohnson_splun]

In Splunk you can use the boolean operators AND, OR, and NOT to select your results.

For example, say you have some data that looks like this

10:00:01 PM Error: Broken Panda
10:00:02 PM Error: Broken Red Panda
10:00:03 PM Issue: Red Panda Warning
10:00:04 PM Error: Red Panda Endangered
10:00:05 PM Issue: Panda Babies (IVF)

You could search for the "issue" and the "error" by searching:

error OR issue

But what if you only wanted to search for issues or errors with red pandas? They're much cuter anyways.
You could search for the phrase red panda with double quotes:

"red panda" issue OR error

Note that I didn't need to use a boolean between "red panda" and issue: that is because AND is implied between search terms.

[wrangler2x]

I don't see either error or issue in the sample logs you are showing here. Are you showing actual log entries?

Also, you say you are searching on error and issue... did you mean error OR issue? As for the timestamp, Splunk does not care... it will return anything that matches the search within the time-frame specified.

If you want to see log entries that have error or issue but not certain types of them, you can use NOT.

What is the search you are using now? Would you share that?

[Abilan1]

Hi ,

I told error or issue as an example. My logs look likes below

10:00:01 PM KERNEL RECYCLING: Terminated for recycling
10:00:02 PM API ipcSawUnregisterProcV1
10:00:03 PM Handle State structures to abandoned
10:00:04 PM Error: Failed to validate User handle

I want only the source which has this event "Failed to validate User handle". I don't want it which has both (Failed to validate User handle and KERNEL RECYCLING).

If you see my log above these event happened in two different time stamp.

[wrangler2x]

In the example logs you gave us "Error: Failed to validate User" only appears once, not twice. To search for this is quite simple. After you specify the basic search and before you use a pipe or just at the end of the basic search, just put "Error: Failed to validate User" and any log entry having that string will be shown to you. If you have two strings your are interested in, then do the same for each (put them in quotes) and separate them with OR. Have you tried that?

[Abilan1]

Hi ,

I have tried this "Handle State structures to abandoned" NOT "KERNEL RECYCLING", it is still showing the files which has KERNEL RECYCLING also. I think it is showing because this event happened in different time stamp and indexed in another line.

[Abilan1]

Hi ,

I would like to know boolean operators will work for different time stamp indexed data. What I understood is it look into only the indexed data on the same time stamp. I am not sure that it will look into the entire source file? Can you please confirm will it look for entire source file or only that specific time stamp indexed line. And In my case I have logs like

10:00:01 PM KERNEL RECYCLING: Terminated for recycling
10:00:02 PM API ipcSawUnregisterProcV1
10:00:03 PM Handle State structures to abandoned
10:00:04 PM Error: Failed to validate User handle

I want to get the alert only if that source has "Handle State structures to abandoned" this message and not this one in the same source "KERNEL RECYCLING" in the same source. I have tried the below in search
Handle State structures to abandoned NOT (KERNEL RECYCLING)
but it is not working for me. in this search am getting the results which has Kernel Recycling message. so I though as both the strings are in different index data, splunk is unable to pick it up. Please help me on this one,

Thank You!

[diogofgm]

boolean operators work for events with different timestamps.

in your search use quotes around the string instead of ( )
"Handle State structures to abandoned" NOT "KERNEL RECYCLING"

[Abilan1]

Hi ,

Thanks! I have tried that suggestion, still am getting results which has "KERNEL RECYCLING" in it's source file.

[diogofgm]

What do you mean in it's source file? You want to check files that don't contain KERNEL RECYCLING rather than checking individual events?