Archive
Highlighted

Setting up Alert if jboss service went down

Explorer

I have 3 servers App-1, App-2 and App-3. The three application are running on Jboss. I need a query that Alerts me, if on any servers Jboss Service goes down.

Any help with this will be appreciated.

Thank You

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Contributor

Hi @shakeel253,

You can run a script which will check the jboss status on the machines and index the output of the command in Splunk. Then you can set up an alert on when you get the status as "jboss is not running".

Thanks!

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Explorer

Can you give an example of how this would be possible?

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Contributor

Sure! You first create a script which is going to check the jboss status of your machines. You can refer to sample scripts in the below link which will check the jboss status.

https://stackoverflow.com/questions/8761374/how-to-check-if-jboss-is-running-on-unix-server

Save the script in the bin directory of your app and then create an index like you create index normally which will store the result of your script. Then create the inputs.conf in the local directory of your app and give the path to your script. This complete thing is going to index the output of your script. Post the indexing you can set up alerts which will be triggered once it finds "jboss is not running" in your events. I hope this sounds clear. Do let me know if you find any trouble in doing that.

Thanks!!

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Explorer

@Mousumichowdhury i created a shell script and placed it inside /opt/splunkforwarder/bin with executable permission. I also made an entry inside inputs.conf, below is the how the entry looks like inside inputs.conf

[default]
host = svm
/opt/splunkforwarder/bin/tomcatscript.sh

how do you create an index?

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Contributor

Hi,

you can create the index with the below stanza:

[yourindexname]
coldPath = $SPLUNK_DB/yourindexname/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/yourindexname/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/yourindexname/thaweddb

If this is a single instance setup then you can place your indexes.conf stanza in /opt/splunk/etc/apps/search/local and in case of clustered environment you can place the indexes.conf file inside /opt/splunk/etc/master-apps//local

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Explorer

@Mousumichowdhury i still dont see the new custom index in splunk search. I will describe how my splunk is setup, maybe, i am missing something.

1) tomcatscript.sh is inside /opt/splunkforwarder/bin

!/bin/bash

if [ -z "$(ps -ef | grep java | grep tomcat)" ]
then
echo "Tomcat is NOT running"
else
echo "Tomcat is running"
fi

2) inputs.conf inside /opt/splunkforwarder/etc/apps/search/local

[monitor:///opt/tomcat/logs/catalina.out]
disabled = false
index = tomcat

[monitor:///opt/splunkforwarder/bin/tomcatscript.sh]
disabled = false
index = tomcatindex

3)indexes.conf inside /opt/splunkforwarder/etc/apps/search/local

[tomcatindex]
coldPath = $SPLUNKDB/tomcatindex/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK
DB/tomcatindex/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/tomcatindex/thaweddb

when i do a search on splunk host="abcvm" but the only index i see is "os". I also did a restart as well for the splunkforwarder but didnt see the newly created index. Maybe i am doing something wrong, would appreciate if you could direct me, thank you

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Contributor

Hi!

Follow the below steps for the single instance:

Create your script in /opt/splunk/etc/apps/search/bin/
Create your inputs.conf in /opt/splunk/etc/apps/search/local/
Create your indexes.conf in /opt/splunk/etc/apps/search/local/

Follow the below steps for clustered set up:

Create your script in /opt/splunk/etc/deployment-apps/<yourappname>/bin/
Create your inputs.conf in /opt/splunk/etc/deployment-apps/<yourappname>/local/
Create your indexes.conf in /opt/splunk/etc/master-apps/<yourappname>/local/

Do let me know if that's working for you.

Thanks!

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Explorer

@MousumiChowdhury

firstoff thank you for assisting me with this, having said that bin directory is not inside /opt/splunk/etc/apps/search, so i had to manually create the bin folder and gave it splunk ownership and copy the tomcat script inside bin.

2) This is how the indexes.conf

[root@ip-xx.xxx.xx.xxx local]# cat indexes.conf
[tomcatindex]
coldPath = $SPLUNKDB/tomcatindex/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK
DB/tomcatindex/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/tomcatindex/thaweddb

[tomcatindex]
coldPath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh
maxTotalDataSizeMB = 512000
thawedPath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh

3)[root@ip-xx-xxx-xx-xx local]# cat inputs.conf
[monitor:///opt/tomcat/logs/catalina.out]
disabled = false
index = tomcat

[monitor:///opt/splunkforwarder/bin/tomcatscript.sh]
disabled = false
index = tomcatindex

[monitor:///opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh]
disabled = false
index = tomcatindex

After making these changes, i have restarted splunkforwarder and search for the index, but i do not see the index still, what am i missing?

0 Karma
Highlighted

Re: Setting up Alert if jboss service went down

Contributor

Hi, may I know why are you creating the files in your splunkforwarder?

0 Karma