Splunk Search

How to compare search result of logs from production server to QA search result logs?

raghavendrasred
New Member

I want compare the 2 search result error logs and show the result.
I want to compare based on "Error" log of 2 search result.

One search result from prod:

*NullPointerException*  index=prod | search log_level="ERROR" OR log_severity="Error" OR status>399 | stats earliest(_time) as FirstOccurence latest(_time) as LastOccurence count by static_msg | fieldformat FirstOccurence=strftime(FirstOccurence,"%m/%d/%y %H:%M:%S") | fieldformat LastOccurence =strftime(LastOccurence,"%m/%d/%y %H:%M:%S") | eval Error = static_msg | Table Error  count FirstOccurence LastOccurence | sort –count

Search result from QA:

*NullPointerException*  host=$host$ | search log_level="ERROR" OR log_severity="Error" OR status>399 | stats earliest(_time) as FirstOccurence latest(_time) as LastOccurence count by static_msg | fieldformat FirstOccurence=strftime(FirstOccurence,"%m/%d/%y %H:%M:%S") | fieldformat LastOccurence =strftime(LastOccurence,"%m/%d/%y %H:%M:%S") | eval Error = static_msg | Table Error  count FirstOccurence LastOccurence | sort –count

I tried diff like eval diff = prod - qa it's showing difference of count or number of entries displayed, but I am looking for actual log differences.

0 Karma
1 Solution

somesoni2
Revered Legend

Try something like this

| set diff [search *NullPointerException*  index=prod log_level="ERROR" OR log_severity="Error" OR status>399 | stats earliest(_time) as FirstOccurence latest(_time) as LastOccurence count by static_msg | convert ctime(*Occurrence) timeformat="%m/%d/%y %H:%M:%S" | rename  static_msg as Error| Table Error  count FirstOccurence LastOccurence] [search *NullPointerException* host=$host$ log_level="ERROR" OR log_severity="Error" OR status>399 | stats earliest(_time) as FirstOccurence latest(_time) as LastOccurence count by static_msg | convert ctime(*Occurrence) timeformat="%m/%d/%y %H:%M:%S" | rename  static_msg as Error| Table Error  count FirstOccurence LastOccurence ] | sort –count

View solution in original post

0 Karma

somesoni2
Revered Legend

Try something like this

| set diff [search *NullPointerException*  index=prod log_level="ERROR" OR log_severity="Error" OR status>399 | stats earliest(_time) as FirstOccurence latest(_time) as LastOccurence count by static_msg | convert ctime(*Occurrence) timeformat="%m/%d/%y %H:%M:%S" | rename  static_msg as Error| Table Error  count FirstOccurence LastOccurence] [search *NullPointerException* host=$host$ log_level="ERROR" OR log_severity="Error" OR status>399 | stats earliest(_time) as FirstOccurence latest(_time) as LastOccurence count by static_msg | convert ctime(*Occurrence) timeformat="%m/%d/%y %H:%M:%S" | rename  static_msg as Error| Table Error  count FirstOccurence LastOccurence ] | sort –count
0 Karma

raghavendrasred
New Member

it not worked.

I tried another way..

NullPointerException index=prod | search log_level="ERROR" OR log_severity="Error" OR status>399 | stats count by static_msg | eval prod = static_msg | appendcols [search NullPointerException host=$host$ | search log_level="ERROR" OR log_severity="Error" OR status>399 | stats count by static_msg | eval qa = static_msg ] | eval err=if(prod == qa, "OK", "Error")

it should work like vlookup, but it is not working..

0 Karma

raghavendrasred
New Member

Thanks. now i got the result in single Error field.
in result we have both the records of production and QA in single field as Error. The Error contains the 2 records with same message(like 2 entry with message "Failed to process ospp message."). Is it possible to highlight common records present in result.

0 Karma

raghavendrasred
New Member

Please some one answer the above request.

0 Karma
Get Updates on the Splunk Community!

Celebrating Fast Lane: 2025 Authorized Learning Partner of the Year

At .conf25, Splunk proudly recognized Fast Lane as the 2025 Authorized Learning Partner of the Year. This ...

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...