topic Re: How to search two different strings from the same source, but different timestamps? in Splunk Search

How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 12:08:09 GMT

Hi ,

I would like to know how to search two different search strings (Error and issue) from the same source file, but the error and issue both have different timestamps, so am unable to search in direct way. Please help me here. Thank You!

Re: How to search two different strings from the same source, but different timestamps?

aljohnson_splun — Tue, 25 Aug 2015 12:30:12 GMT

In Splunk you can use the boolean operators AND, OR, and NOT to select your results.

For example, say you have some data that looks like this

10:00:01 PM Error: Broken Panda
10:00:02 PM Error: Broken Red Panda
10:00:03 PM Issue: Red Panda Warning
10:00:04 PM Error: Red Panda Endangered
10:00:05 PM Issue: Panda Babies (IVF)

You could search for the "issue" and the "error" by searching:

error OR issue

But what if you only wanted to search for issues or errors with red pandas? They're much cuter anyways.
You could search for the phrase red panda with double quotes:

"red panda" issue OR error

Note that I didn't need to use a boolean between "red panda" and issue: that is because AND is implied between search terms.

Re: How to search two different strings from the same source, but different timestamps?

diogofgm — Tue, 25 Aug 2015 12:56:50 GMT

if you use OR you can search for both.

source=my_file Error OR issue

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 13:10:02 GMT

Hi ,

I am searching in real time , I cannot give specific source file name and I think the command which you have mentioned will work if Error OR issue will be in same indexed data (time stamp). I am looking for different time stamp.

Re: How to search two different strings from the same source, but different timestamps?

diogofgm — Tue, 29 Sep 2020 07:07:18 GMT

I wrote source=my_file only as an example. You can use whatever fits your search query (e.g. index=my_index, host=my_host, etc ). It doesn't matter if you're doing a real-time search.
About the Error OR issue, this will show you all the events (different timestamps) that contains one of the strings or both in the same event.

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 16:09:11 GMT

Hi ,

I would like to know boolean operators will work for different time stamp indexed data. What I understood is it look into only the indexed data on the same time stamp. I am not sure that it will look into the entire source file? Can you please confirm will it look for entire source file or only that specific time stamp indexed line. And In my case I have logs like

10:00:01 PM KERNEL RECYCLING: Terminated for recycling
10:00:02 PM API ipcSawUnregisterProcV1
10:00:03 PM Handle State structures to abandoned
10:00:04 PM Error: Failed to validate User handle

I want to get the alert only if that source has "Handle State structures to abandoned" this message and not this one in the same source "KERNEL RECYCLING" in the same source. I have tried the below in search
Handle State structures to abandoned NOT (KERNEL RECYCLING)
but it is not working for me. in this search am getting the results which has Kernel Recycling message. so I though as both the strings are in different index data, splunk is unable to pick it up. Please help me on this one,

Thank You!

Re: How to search two different strings from the same source, but different timestamps?

diogofgm — Tue, 25 Aug 2015 16:45:39 GMT

boolean operators work for events with different timestamps.

in your search use quotes around the string instead of ( )
"Handle State structures to abandoned" NOT "KERNEL RECYCLING"

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 16:49:28 GMT

Hi ,

Thanks! I have tried that suggestion, still am getting results which has "KERNEL RECYCLING" in it's source file.

Re: How to search two different strings from the same source, but different timestamps?

diogofgm — Tue, 25 Aug 2015 20:29:51 GMT

What do you mean in it's source file? You want to check files that don't contain KERNEL RECYCLING rather than checking individual events?

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 20:38:44 GMT

Hi ,

I want to create the dashboard, for the "Handle State structures to abandoned" this event, but If this "KERNEL RECYCLING" string is also in the same source file. I want to leave this source file from my Count. Like this I want to search all my source file and update the count in dashboard. Please let me know if you need more information. Thank You!!

Re: How to search two different strings from the same source, but different timestamps?

wrangler2x — Tue, 25 Aug 2015 20:55:12 GMT

I don't see either error or issue in the sample logs you are showing here. Are you showing actual log entries?

Also, you say you are searching on error and issue... did you mean error OR issue? As for the timestamp, Splunk does not care... it will return anything that matches the search within the time-frame specified.

If you want to see log entries that have error or issue but not certain types of them, you can use NOT.

What is the search you are using now? Would you share that?

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 22:07:22 GMT

Hi ,

I told error or issue as an example. My logs look likes below

10:00:01 PM KERNEL RECYCLING: Terminated for recycling
10:00:02 PM API ipcSawUnregisterProcV1
10:00:03 PM Handle State structures to abandoned
10:00:04 PM Error: Failed to validate User handle

I want only the source which has this event "Failed to validate User handle". I don't want it which has both (Failed to validate User handle and KERNEL RECYCLING).

If you see my log above these event happened in two different time stamp.

Re: How to search two different strings from the same source, but different timestamps?

wrangler2x — Tue, 25 Aug 2015 22:17:14 GMT

In the example logs you gave us "Error: Failed to validate User" only appears once, not twice. To search for this is quite simple. After you specify the basic search and before you use a pipe or just at the end of the basic search, just put "Error: Failed to validate User" and any log entry having that string will be shown to you. If you have two strings your are interested in, then do the same for each (put them in quotes) and separate them with OR. Have you tried that?

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 22:23:42 GMT

Hi ,

I have tried this "Handle State structures to abandoned" NOT "KERNEL RECYCLING", it is still showing the files which has KERNEL RECYCLING also. I think it is showing because this event happened in different time stamp and indexed in another line.

Re: How to search two different strings from the same source, but different timestamps?

diogofgm — Tue, 25 Aug 2015 23:08:36 GMT

it took me some time to figure this out but i believe this is what you are looking for. (math logic)
Not the most performant search query but works.
replace my_index with your index and try this:

index=my_index "Handle State structures to abandoned" | stats count by source | eval status = "keep" | append [search index=my_index "KERNEL RECYCLING" | stats count by source | eval status = "remove" ] | stats values(status) AS status by source | search status!=remove

Small explanation on this:
1- list the sources that have "Handle State structures to abandoned" and build a field called status with "keep" as value

index=my_index "Handle State structures to abandoned" | stats count by source | eval status = "keep"

2 - list the sources that have "KERNEL RECYCLING", build a field called status with "remove" as value and append the result to 1:

...| append [search index=my_index "KERNEL RECYCLING" | stats count by source | eval status = "remove" ]

3 - list both results by status (sources that you want keep or remove) and filter the ones you do not want to show:

...| stats values(status) AS status by source | search status!=remove

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 23:26:03 GMT

Hi ,

Thank you so much! this is perfect. I wanted it in the dashboard..X axis (date_month) and count of event (Y axis). Can you please help me?

Re: How to search two different strings from the same source, but different timestamps?

diogofgm — Tue, 25 Aug 2015 23:35:46 GMT

splitted by source?

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Tue, 25 Aug 2015 23:39:30 GMT

Hi,
Total number of source is event count by date (X axis)

Re: How to search two different strings from the same source, but different timestamps?

diogofgm — Tue, 29 Sep 2020 07:04:25 GMT

Re: How to search two different strings from the same source, but different timestamps?

Abilan1 — Wed, 26 Aug 2015 00:55:26 GMT

Hi ,

Thanks again, this is perfect..is it possible to add colours in this dashboard. I am unable to open this in Pivot?