I need to display the error message (DC1-4 connect fail host:port Connection refused (Connection refused)) in a column until logs have the info message (DC1-4 connected host:port)

[a508184]

Hi -

Need to create a Splunk dashboard for an application.

Am very new to Splunk and doesn't have any Splunk experts within my organization, so need your help.

Sample log format:
12-15 20:22:55,671 ERROR - logs/app/applog 1-426 DC1-4 connect fail host:port Connection refused (Connection refused)
12-15 20:25:13,860 INFO - logs/app/applog 1-423 DC1-4 connected host:port

Time = 12-15 20:25:13
Status = INFO
Log Path = logs/app/applog
Stats Status = 1-426
Process = DC1-4 and DC1-4

Whenever apps is having issues connecting to other sources, it throws an error message with the stats status 1-426 for the particular process DC1-4. Once its fixed then INFO (Success) message will be printed in the logs with the stats status - 1-423 for the particular process DC1-4.

Requirement:
- I need to display the error message (DC1-4 connect fail host:port Connection refused (Connection refused)) in a column until logs have the info message (DC1-4 connected host:port)

[renjith_nair · Accepted solution]

@a508184,

We need to extract the Status field and Process field from the log and show only if the latest status is down(1-426)

Based on your sample event, below regex can be used. Please let us know if there is a change in the format of Status or Process string

index="your index" "other search terms"
| rex field=_raw "applog (?<Status>\d-\d{3}) (?<Process>\w+-\w+)"
| stats latest(Status) as Status , latest(_raw) as _raw by Process|where Status="1-426"

You may adjust the regex according to your actual events.

[a508184]

Hi Renjith -

Thanks for your reply. Yes i can't exact the process, because log format differs for different process ID's.

Also, i need to display the error in the Dashboard until its fixed or success STATS status code (1-423).

Thanks,
Nithin Setty

[renjith_nair]

Hi Nithin,

Is it possible to provide sample events (anonymize sensitive data) for those different events? We might be able to provide a general extraction.
The error will be displayed until the STATUS code is 1-423 because we are filtering the events only for FAILURES by using |where Status="1-426".

[a508184]

Here is the sample logs

12/15/18
8:25:13.860 PM

"12-15 20:25:13,860 INFO - logs/app/fmapp1 12-15,20:24:52.019 ios11 1-423 0 4 DDC3-2 connected XXX:XXX".
Field2 = INFO Field4 = 1-423 0 4 Log_Time = 12-15 20:25:13,860 detailed_logs = 4 DDC3-2 connected XXX:XXX . host = servername source = /applog/log.txt sourcetype = prd_asg_app_logs

12/15/18
8:22:55.671 PM

"12-15 20:22:55,671 ERROR - logs/app/fmapp1 12-15,20:22:32.734 ios55 1-426 0 2 DDC1-4 connect fail XXX:XXX Connection refused (Connection refused)"
Field2 =ERROR Field4 = 1-426 0 2 Log_Time = 12-15 20:22:55,671 detailed_logs = DDC1-4 connect fail XXX:XXX Connection refused (Connection refused) host = servername source =/applogs/log.txt sourcetype = prd_asg_app_logs

[a508184]

Below is different senario,

Here for error its 6-908 stats status code will be returned for the f1/DC1/PRD03 counters.
12-15 17:25:02,318 ERROR - logs/stats/stats1 12-15,17:25:01.397 ios13 6-908 0 2 Stats create alert DDlag2 f1/DC1/PRD03 value 1544916240000
Counters = f1/DC1/PRD03 Field2 = ERROR Field4 = 6-908 0 2 Log_Time = 12-15 17:25:02,318 detailed_logs = Stats create alert DDlag2 applog/PRD03 value 1544916240000 host = source = /applogs/log.txt sourcetype = prd_asg_app_logs

Success stats status code is 6-904 for the f1/DC1/PRD03 counters.
12-15 21:29:05,289 INFO - logs/stats/stats1 12-15,21:29:00.214 ios13 6-904 0 4 Stats clear alert DDlag2 f1/DC1/PRD03 value 8270
Counters = f1/DC1/PRD03 Field2 = INFO Field4 = 6-904 0 4 Log_Time = 12-15 21:29:05,289 detailed_logs = 4 Stats clear alert DDlag2 f1/DC1/PRD03 value 8270 host = source = /applogs/log.txt sourcetype = prd_asg_app_logs

Above and below senario needs to be incudle in the same dashboard.

[renjith_nair]

@a508184, What are these Counters ,Field2, Field4 etc? Are these extracted fields ? If thats the case, your status is already in Field4 . And counters has the process information. Then we just need to combine these two scenario.

[a508184]

Yeah - they are extracted fields, however the below query is working for me for two different scenarios.
index=asg sourcetype=prd_asg_app_logs source=*
|rex field=_raw " (?\d-\d{3})" | rex field=_raw " (?DDC\w+-\w+)"
| stats last(Status) as Status , last(_raw) as _raw by Process, source
|where Status="6-908"

index=asg sourcetype=prd_asg_app_logs source=* NOT ("fmq" "dbWriter*")
|rex field=_raw " (?\d-\d{3})" |rex field=_raw "fmapp1/(?D\w+/\w+)"
|table Process Status source _raw
|where Status="6-908" |where isnotnull(Process)

Could you please help in combining both the queries?

Thanks,
Nithin Setty

[renjith_nair]

Try this. You may adjust the rex and conditions based on your actual events,

index=asg sourcetype=prd_asg_app_logs source=* 
|rex field=_raw "(?<Status>\d-\d{3})"
|rex field=_raw "(?<Process1>DDC\w+-\w+)"
|rex field=_raw "fmapp1/(?<Process2>D\w+/\w+)"
|eval Process=if(source!="fmq" OR source!="dbWriter",Process2,Process1)
|table Process Status source _raw

[a508184]

Thanks a lot for your help Renjith.

It works, but i cant achieve the below scenario

I need to display the error in the Dashboard until its fixed or success STATS status code 1-423 for first query and 6-904 for the second query.

First Query - Success Code 1-423 and Error Code 1-426
index=asg sourcetype=prd_asg_app_logs source=*
|rex field=_raw "(?\d-\d{3})"
|rex field=_raw "(?DDC\w+-\w+)"
|stats latest(Status) as Status , latest(_raw) as _raw by Process
|where Status="1-426"

First Query - Success Code 6-904 and Error Code 6-908
index=asg sourcetype=prd_asg_app_logs source=*
|rex field=_raw " (?\d-\d{3})" |rex field=_raw "fmapp1/(?\w+/\w+)"
| stats latest(Status) as Status , latest(_raw) as _raw by Process
|where Status="6-908" |where isnotnull(Process)

Need your help in combining both the queries which matches scenario.

[renjith_nair]

@a508184, since the above combined search is working for you, we add these conditions too the same.

 index=asg sourcetype=prd_asg_app_logs source=* 
 |rex field=_raw "(?<Status>\d-\d{3})"
 |rex field=_raw "(?<Process1>DDC\w+-\w+)"
 |rex field=_raw "fmapp1/(?<Process2>D\w+/\w+)"
 |eval Process=if(source!="fmq" OR source!="dbWriter",Process2,Process1)
 |stats latest(Status) as Status,latest(_raw) as Message by Process
 |where (Status!="1-426" AND Status!="6-904")

[a508184]

Thanks Renjith for your help.