topic Re: How to check application running status hosted in Linux server using application log in Splunk Search

How to check application running status hosted in Linux server using application log

mnarmada — Mon, 15 Jun 2020 20:00:57 GMT

Hello Team,

Here is my requirement:

I have to check the application running status, which is installed in Linux server. For this, I have a log generated by the application, which might not contain the continuous-time intervals. The log will get updated when the user is using the app. In the log, I have 3 high priority exceptions: TransactionRolledbackException, WIMSystemException, ConnectionWaitTimeoutException. When any of these exceptions occurred in the log, the status should be "DOWN". If any other exceptions occur, then the status should be "WARNING, and if no exception, it should show "OK". Also once the high priority exception occurs, we will notify the users by email alert. After the email alert, it would be cleared then the next events will generate. once the next event generates and does not contain any high priority exceptions, then the status should be shown in the dashboard as "OK" and low priority exceptions, warning. And if the latest event contains exception again, then "DOWN".

Noe: when the application is down in real time, the log will not generate.

Here are my sample codes but not satisfied with the results:

index=myIndex sourcetype=mySourcetpe | stats count as Total earliest(_time) as start_time latest(_time) as latest_time earliest(_raw) as Earliest_Event latest(_raw) as Latest_Event by _time | eval stop=strptime(stop, "%m/%d/%Y") | eval Earliest_Count= Total - 1 | eval Latest_Count= Total + 1 | eval status=case(((Latest_count > Total) AND match(_raw, "TransactionRolledbackException")), "Down",((Latest_count > Total) AND match(_raw, "WIMSystemException")), "Down",((Latest_count > Total) AND match(_raw, "ConnectionWaitTimeoutException")), "Down",((Latest_count > Total) AND match(_raw, "\w+Exception")), "Warning", 1!=2, "OK") | stats count by status

index=myIndex sourcetype=myscourcetype | eval status=case( match(_raw, "TransactionRolledbackException"), "Down", match(_raw, "WIMSystemException"), "Down", match(_raw, "ConnectionWaitTimeoutException"), "Down", match(_raw, "\w+Exception"), "WARNING" , 1!=2, "OK") | timechart count by status

Any Help or suggestion would be really appreciated!! Thanks!

Re: How to check application running status hosted in Linux server using application log

mnarmada — Wed, 17 Jun 2020 07:17:25 GMT

@richgalloway @dmaislin_splunk

Hello All,

Can someone please help.

Re: How to check application running status hosted in Linux server using application log

dmaislin_splunk — Wed, 17 Jun 2020 17:36:15 GMT

Can you upload a data sample? Please remove or substitute any sensitive data before you provide this information.

Re: How to check application running status hosted in Linux server using application log

mnarmada — Thu, 18 Jun 2020 07:44:07 GMT

Hello,

Below is the piece of data. Let me know if anything else required.

[3/14/18 0:12:41:610 EDT] 00000039 SystemErr     R  at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
46      3/14/18
12:12:41.610 AM 
[3/14/18 0:12:41:610 EDT] 00000039 SystemErr     R  at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
47      3/14/18
12:12:41.610 AM 
[3/14/18 0:12:41:610 EDT] 00000039 SystemErr     R  at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
48      3/14/18
1:00:02.465 AM  
[3/14/18 1:00:02:465 EDT] 0000006b SystemErr     R com.ibm.wcc.service.intf.ProcessingException

Re: How to check application running status hosted in Linux server using application log

mnarmada — Fri, 19 Jun 2020 10:02:29 GMT

Hello,

I have tried like below and is working for me. Thanks!!

Still I accept any better solution than this.. haha..

index=myindex sourcetype=mysourcetype
| stats latest(_raw) as latest_event
| eval status=case(match(latest_event, "TransactionRolledbackException"), "Down", match(latest_event, "WIMSystemException"), "latest_event", match(_raw, "ConnectionWaitTimeoutException"), "Down", match(latest_event, "\w+Exception"), "Warning", 1!=2, "OK")
| stats count by status
| eval status=case(status="OK", 0, status="Warning", 5, status="Down", 10)
| rangemap field=status low=0-4 elevated=4-6 default=severe
| eval status = replace (status,"0","OK")
| eval status = replace (status,"5","WARNING")
| eval status = replace (status,"10","DOWN")