How to check application running status hosted in Linux server using application log

[mnarmada]

Hello Team,

Here is my requirement:

I have to check the application running status, which is installed in Linux server.  For this, I have a log generated by the application, which might not contain the continuous-time intervals. The log will get updated when the user is using the app. In the log, I have 3 high priority exceptions: TransactionRolledbackException, WIMSystemException, ConnectionWaitTimeoutException. When any of these exceptions occurred in the log, the status should be "DOWN". If any other exceptions occur, then the status should be "WARNING, and if no exception, it should show "OK". Also once the high priority exception occurs, we will notify the users by email alert. After the email alert, it would be cleared then the next events will generate. once the next event generates and does not contain any high priority exceptions, then the status should be shown in the dashboard as "OK" and low priority exceptions, warning. And if the latest event contains exception again, then "DOWN". 

Noe: when the application is down in real time, the log will not generate.

Here are my sample codes but not satisfied with the results:

1.

index=myIndex sourcetype=mySourcetpe 
| stats count as Total earliest(_time) as start_time latest(_time) as latest_time earliest(_raw) as Earliest_Event latest(_raw) as Latest_Event by _time
| eval stop=strptime(stop, "%m/%d/%Y")
| eval Earliest_Count= Total - 1
| eval Latest_Count= Total + 1
| eval status=case(((Latest_count > Total) AND match(_raw, "TransactionRolledbackException")), "Down",((Latest_count > Total) AND match(_raw, "WIMSystemException")), "Down",((Latest_count > Total) AND match(_raw, "ConnectionWaitTimeoutException")), "Down",((Latest_count > Total) AND match(_raw, "\w+Exception")), "Warning", 1!=2, "OK")
| stats count by status

2. 

index=myIndex sourcetype=myscourcetype
| eval status=case( match(_raw, "TransactionRolledbackException"), "Down", match(_raw, "WIMSystemException"), "Down", match(_raw, "ConnectionWaitTimeoutException"), "Down", match(_raw, "\w+Exception"), "WARNING" , 1!=2, "OK")
| timechart count by status

Any Help or suggestion would be really appreciated!! Thanks!

[dmaislin_splunk]

Can you upload a data sample?  Please remove or substitute any sensitive data before you provide this information.

[mnarmada]

Hello,

Below is the piece of data. Let me know if anything else required.

[3/14/18 0:12:41:610 EDT] 00000039 SystemErr     R  at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
46      3/14/18
12:12:41.610 AM 
[3/14/18 0:12:41:610 EDT] 00000039 SystemErr     R  at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
47      3/14/18
12:12:41.610 AM 
[3/14/18 0:12:41:610 EDT] 00000039 SystemErr     R  at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
48      3/14/18
1:00:02.465 AM  
[3/14/18 1:00:02:465 EDT] 0000006b SystemErr     R com.ibm.wcc.service.intf.ProcessingException

[mnarmada · Accepted solution]

Hello,

I have tried like below and is working for me. Thanks!!

Still I  accept any better solution than this.. haha..

index=myindex  sourcetype=mysourcetype
| stats latest(_raw) as latest_event
| eval status=case(match(latest_event, "TransactionRolledbackException"), "Down", match(latest_event, "WIMSystemException"), "latest_event", match(_raw, "ConnectionWaitTimeoutException"), "Down", match(latest_event, "\w+Exception"), "Warning", 1!=2, "OK")
| stats count by status
| eval status=case(status="OK", 0, status="Warning", 5, status="Down", 10)
| rangemap field=status low=0-4 elevated=4-6 default=severe
| eval status = replace (status,"0","OK")
| eval status = replace (status,"5","WARNING")
| eval status = replace (status,"10","DOWN")

[mnarmada]

@richgalloway @dmaislin_splunk 

Hello All,

Can someone please help.