topic Re: Setting up Alert if jboss service went down in Alerting

Setting up Alert if jboss service went down

shakeel253 — Thu, 07 Dec 2017 18:04:47 GMT

I have 3 servers App-1, App-2 and App-3. The three application are running on Jboss. I need a query that Alerts me, if on any servers Jboss Service goes down.

Any help with this will be appreciated.

Thank You

Re: Setting up Alert if jboss service went down

MousumiChowdhur — Wed, 20 Dec 2017 11:05:08 GMT

Hi @shakeel253,

You can run a script which will check the jboss status on the machines and index the output of the command in Splunk. Then you can set up an alert on when you get the status as "jboss is not running".

Thanks!

Re: Setting up Alert if jboss service went down

shakeel253 — Wed, 20 Dec 2017 13:21:08 GMT

Can you give an example of how this would be possible?

Re: Setting up Alert if jboss service went down

MousumiChowdhur — Tue, 26 Dec 2017 09:36:37 GMT

Sure! You first create a script which is going to check the jboss status of your machines. You can refer to sample scripts in the below link which will check the jboss status.

https://stackoverflow.com/questions/8761374/how-to-check-if-jboss-is-running-on-unix-server

Save the script in the bin directory of your app and then create an index like you create index normally which will store the result of your script. Then create the inputs.conf in the local directory of your app and give the path to your script. This complete thing is going to index the output of your script. Post the indexing you can set up alerts which will be triggered once it finds "jboss is not running" in your events. I hope this sounds clear. Do let me know if you find any trouble in doing that.

Thanks!!

Re: Setting up Alert if jboss service went down

shakeel253 — Tue, 26 Dec 2017 14:44:49 GMT

@Mousumichowdhury i created a shell script and placed it inside /opt/splunkforwarder/bin with executable permission. I also made an entry inside inputs.conf, below is the how the entry looks like inside inputs.conf

[default]
host = svm
/opt/splunkforwarder/bin/tomcatscript.sh

how do you create an index?

Re: Setting up Alert if jboss service went down

niketn — Tue, 26 Dec 2017 17:31:56 GMT

@shakeel253, I am afraid you have duplicate questions opened for your JBOSS alerting issues with different description: https://answers.splunk.com/answers/597751/splunk-query-that-alert-if-services-on-a-jboss-ser.html

Please clarify if this is any different from the other post. Can you try the following? Where hosts will have list of servers you want to monitor and source is the JBOSS log path.

  | tstats latest(_time) as _time WHERE (host="ABC1" OR host="ABC2" OR host="ABC3") AND source="/opt/jboss-eap/standalone/log/server.log" by host
  | eval "downTime (in Min)"=round((now()-_time)/60,0)
  | append [
      | makeresults
      | eval host="ABC1", "downTime (in Min)"="999"]
  | append [
      | makeresults
      | eval host="ABC2", "downTime (in Min)"="999"]
  | append [
      | makeresults
      | eval host="ABC3", "downTime (in Min)"="999"]
  | dedup host
  | where 'downTime (in Min)'>5

Re: Setting up Alert if jboss service went down

MousumiChowdhur — Wed, 27 Dec 2017 05:14:42 GMT

Hi,

you can create the index with the below stanza:

[yourindexname]
coldPath = $SPLUNK_DB/yourindexname/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/yourindexname/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/yourindexname/thaweddb

If this is a single instance setup then you can place your indexes.conf stanza in /opt/splunk/etc/apps/search/local and in case of clustered environment you can place the indexes.conf file inside /opt/splunk/etc/master-apps//local

Re: Setting up Alert if jboss service went down

shakeel253 — Tue, 29 Sep 2020 17:26:37 GMT

@MousumiChowdhury i still dont see the new custom index in splunk search. I will describe how my splunk is setup, maybe, i am missing something.

1) tomcatscript.sh is inside /opt/splunkforwarder/bin

!/bin/bash

if [ -z "$(ps -ef | grep java | grep tomcat)" ]
then
echo "Tomcat is NOT running"
else
echo "Tomcat is running"
fi

2) inputs.conf inside /opt/splunkforwarder/etc/apps/search/local

[monitor:///opt/tomcat/logs/catalina.out]
disabled = false
index = tomcat

[monitor:///opt/splunkforwarder/bin/tomcatscript.sh]
disabled = false
index = tomcatindex

3)indexes.conf inside /opt/splunkforwarder/etc/apps/search/local

[tomcatindex]
coldPath = $SPLUNK_DB/tomcatindex/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/tomcatindex/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/tomcatindex/thaweddb

when i do a search on splunk host="abcvm" but the only index i see is "os". I also did a restart as well for the splunkforwarder but didnt see the newly created index. Maybe i am doing something wrong, would appreciate if you could direct me, thank you

Re: Setting up Alert if jboss service went down

MousumiChowdhur — Thu, 28 Dec 2017 05:02:55 GMT

Hi!

Follow the below steps for the single instance:

Create your script in /opt/splunk/etc/apps/search/bin/
Create your inputs.conf in /opt/splunk/etc/apps/search/local/
Create your indexes.conf in /opt/splunk/etc/apps/search/local/

Follow the below steps for clustered set up:

Create your script in /opt/splunk/etc/deployment-apps/<yourappname>/bin/
Create your inputs.conf in /opt/splunk/etc/deployment-apps/<yourappname>/local/
Create your indexes.conf in /opt/splunk/etc/master-apps/<yourappname>/local/

Do let me know if that's working for you.

Thanks!

Re: Setting up Alert if jboss service went down

shakeel253 — Tue, 29 Sep 2020 17:27:04 GMT

@MousumiChowdhury

firstoff thank you for assisting me with this, having said that bin directory is not inside /opt/splunk/etc/apps/search, so i had to manually create the bin folder and gave it splunk ownership and copy the tomcat script inside bin.

2) This is how the indexes.conf

[root@ip-xx.xxx.xx.xxx local]# cat indexes.conf
[tomcatindex]
coldPath = $SPLUNK_DB/tomcatindex/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/tomcatindex/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/tomcatindex/thaweddb

[tomcatindex]
coldPath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh
maxTotalDataSizeMB = 512000
thawedPath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh

3)[root@ip-xx-xxx-xx-xx local]# cat inputs.conf
[monitor:///opt/tomcat/logs/catalina.out]
disabled = false
index = tomcat

[monitor:///opt/splunkforwarder/bin/tomcatscript.sh]
disabled = false
index = tomcatindex

[monitor:///opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh]
disabled = false
index = tomcatindex

After making these changes, i have restarted splunkforwarder and search for the index, but i do not see the index still, what am i missing?

Re: Setting up Alert if jboss service went down

MousumiChowdhur — Fri, 05 Jan 2018 06:22:32 GMT

Hi, may I know why are you creating the files in your splunkforwarder?