topic Re: Search for Error only in the latest log file coming from different hosts in Splunk Search

Search for Error only in the latest log file coming from different hosts

nmohammed — Sun, 11 Mar 2018 21:26:16 GMT

My original search Query returns results-

index="ver_logs"  "ERORR detected" | rex field=source "VerLogs\\\(?.*?)\_"  | stats count by cid, host

but we only want to get results from the latest log file for each cid.

Appreciate your help.

Re: Search for Error only in the latest log file coming from different hosts

nmohammed — Sun, 11 Mar 2018 21:28:12 GMT

Corrected the rex extraction for field -

index="ver_logs" "ERORR detected" | rex field=source "VerLogs\\\(?.*?)\_" | stats count by cid, host

Re: Search for Error only in the latest log file coming from different hosts

esix_splunk — Mon, 12 Mar 2018 04:37:40 GMT

By the latest log file, do you mean the most recent log file?

Splunk data is indexed based on time series data. This means that as you search data, and as long as you have it indexed based on the proper event time stamps, then the results you get will be displayed in the time from latest to oldest. (In your results view.)

Drilling down a bit more without understanding your use case fully, you can also you the latest Event Order command in your stats pipeline in order to get the latest events based on the fields of your choosing : http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Eventorderfunctions

Those are a few options there...

Re: Search for Error only in the latest log file coming from different hosts

deepashri_123 — Mon, 12 Mar 2018 08:36:03 GMT

Hey nmohammed,

Are you trying something like this:

index="ver_logs" "ERROR detected" | rex field=source "VerLogs\(?.*?)_" | stats latest(cid) AS cid by host | stats count(cid) AS count values(cid) AS cid by host

Let me know if this helps!!

Re: Search for Error only in the latest log file coming from different hosts

nmohammed — Tue, 29 Sep 2020 18:25:10 GMT

Thanks Deepashri_123

I will try to explain my use-case :

1.We've multiple servers and on each server there are logs produced by our app during upgrades.
2. Each Log is identified by unique cid_date_time.log

D:\Logs\VerLogs\fe1234_3122018_233050.log
D:\Logs\VerLogs\fe1234_3122018_231020.log
D:\Logs\VerLogs\fe5678_3122018_212030.log
D:\Logs\VerLogs\fe5678_3122018_231020.log

Sometimes during our app upgrades , we run into errors. So, a redeployment is neede which is usually in a very shortly after the first failure scan. After re-doing the upgrade, we again want to validate from the newest log file (source) for each cid. The "cid" is unique and I am extracting it during search-time using the "rex" command.

It was easy to run a query and look for errors, but it scans all the logs available. Unfortunately, we only to validate if there are any errors from the newest log file produced by a cid.

In the above 4 log files; we saw error in the first log (D:\Logs\VerLogs\fe1234_3122018_191020.log), then after re-deployment there were no errors in the second log file (D:\Logs\VerLogs\fe1234_3122018_231020.log). These times may change and for some "cid" there may not be any errors as first attempt to upgrade can be successful.

Re: Search for Error only in the latest log file coming from different hosts

nmohammed — Tue, 13 Mar 2018 06:05:18 GMT

The data is indexed properly without any issues.
Example data -

[3/10/2018 11:32:34 PM] ERROR detected SQL migration failed, deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

Re: Search for Error only in the latest log file coming from different hosts

nmohammed — Tue, 13 Mar 2018 06:07:54 GMT

Thanks Esix,

Tried using latest; but not able to get the errors from the latest log file produced by unique cid extracted using the rex at search time. I have put my use-case in more detail in the comments to deepashri_123.

Re: Search for Error only in the latest log file coming from different hosts

niketn — Tue, 13 Mar 2018 09:03:27 GMT

@nmohammed have you tried the following:

 index="ver_logs"  "ERORR detected"
| rex field=source "VerLogs\\\(?<cid>[^\_]+)\_" 
| dedup cid host
| table cid host _time _raw

[Updated]

Added the rex to extract cid

Re: Search for Error only in the latest log file coming from different hosts

nmohammed — Wed, 14 Mar 2018 03:29:59 GMT

@niketniley

it did not help. So, what I am trying to do is.. search for any errors in the latest log file produced by each "cid". Some "cid" don't have errors and some do. When we run a regular search it does return only those cids which have error.
"index="ver_logs" "ERORR detected" | rex field=source "VerLogs\(?.*?)_" | stats count by cid, host"

but we look at the list of all "cid" obtained from above search and fix them running an upgrade which produces another log file for that "cid" which does not have error. So, in order to validate we should only run searches against the newest log file produced by each cid for any error occurrence.

Thanks for your input.

Re: Search for Error only in the latest log file coming from different hosts

deepashri_123 — Tue, 29 Sep 2020 18:28:00 GMT

Hey nmohammed,

I am not sure if this will work:
Can you try something like this:
index=ver_logs [search index=ver_logs | dedup _time | head 1 | return _time]

Let me know if this gives you result of only latest source.

Re: Search for Error only in the latest log file coming from different hosts

nmohammed — Wed, 14 Mar 2018 18:22:52 GMT

hi @deepashri_123

Doesn't give the results I want to see. Thanks for helping though.

Re: Search for Error only in the latest log file coming from different hosts

niketn — Wed, 14 Mar 2018 20:20:48 GMT

@nmohammed did you try with your rex for finiding cid as well?

The dedup command actually retains only the latest row based on the dedup criteria. This is based on the fact that latest event will be displayed first in Splunk (reverse chronological order of time). In this case I assumed cid + host should give unique record and that you would be interested only in the latest.

When you say it did not help, what is the output of query vs what is the expected output?

Also can you add sample data the scenario across multiple hosts where cid has error first and then it is fixed?