topic Re: Splunk4JMX only works when I manually run poll_jmx.bat in Deployment Architecture

Splunk4JMX only works when I manually run poll_jmx.bat

RecoMark0 — Wed, 23 Dec 2015 23:14:30 GMT

Hello,
I have a few servers with UniversalForwarder installed, and Splunk4JMX app on them. I have two servers that appear to have the forwarders setup the same, however;
One server Splunk4JMX works great and jmx data gets sent to the indexers
The other server nothing is sent over(no errors in the Splunk4JMX\log folder either). The only way I can get data to splunk indexers is by manually running:

C:\Program Files\SplunkUniversalForwarder\bin>splunk cmd "C:\Program Files\SplunkUniversalForwarder\etc\apps\SPLUNK4JMX\bin\poll_jmx.bat" config.xml

Both servers Spunk4JMX inputs.conf file is the same:

[script://./bin/poll_jmx.sh config.xml]
interval = 60
sourcetype = jmx
index = jmx
disabled = 1

[script://$SPLUNK_HOME\etc\apps\SPLUNK4JMX\bin\poll_jmx.bat config.xml ]
interval = 60
index = jmx 
disabled = 0
sourcetype = jmx

[monitor://$SPLUNK_HOME/etc/apps/SPLUNK4JMX/logs]
disabled = false
followTail = 0
index = jmx
sourcetype = jmx_errors

Both servers config.xml is basically the same, except different server values.

According to the README file, Splunk4JMX is version 1.6

Question
Is there another setting I am missing to check between these two servers? I assume that since I can manually run it on the broken server, it is not a port/network issue between the server and the indexer. A permissions issue with the user that splunk was installed on maybe?

Update
From the splunkd log:
+ 4 12-06-2015 14:18:29.139 +0000 ERROR ExecProcessor - message from ""C:Program FilesSplunkUniversalForwarderetcappsSPLUNK4JMXbinpoll_jmx.bat" config.xml" Access is denied.

Re: Splunk4JMX only works when I manually run poll_jmx.bat

renjith_nair — Mon, 28 Dec 2015 14:07:24 GMT

Check your splunkd log for any errors. Try running the script manually as "splunkm user" and see if it works. splunkd log should complain in case of permission issues

Re: Splunk4JMX only works when I manually run poll_jmx.bat

jkat54 — Mon, 28 Dec 2015 15:00:06 GMT

Are the permissions good on both servers? Nope, you've since responded saying you found the access denied error.

It appears you're having the issue with windows machines , and by default splunkuniversalforwarder installs as the system account. Which means it should have full access to everything on the server. Perhaps you've used a best practice and installed as a service account in your windows domain instead of the system account. It appears this is true and that the service account user doesn't have execute permission on the batch file.

Perhaps you can change it to run as system account or administrator, restart splunk and see if that fixes the issue.

To find out what account it is running as, start -> run -> services.msc (click ok) look for splunkd service, right click go to properties, go to logo on tab, see/change what account it uses.

Re: Splunk4JMX only works when I manually run poll_jmx.bat

RecoMark0 — Mon, 28 Dec 2015 18:07:25 GMT

looks like a permission issue may be right. From the splunkd log:
+ 4 12-06-2015 14:18:29.139 +0000 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\etc\apps\SPLUNK4JMX\bin\poll_jmx.bat" config.xml" Access is denied.

Re: Splunk4JMX only works when I manually run poll_jmx.bat

RecoMark0 — Mon, 28 Dec 2015 18:52:08 GMT

So the permissions do not look good on the servers not working. What permissions does the splunk_service user(which runs splunkd) need in order to send the jmx data to the indexers?
All other splunk data is reaching the indexers from these servers without issue.

Re: Splunk4JMX only works when I manually run poll_jmx.bat

jkat54 — Mon, 28 Dec 2015 20:03:50 GMT

Oh sorry, I missed your new question. It will need execute permission on the batch file. might as well give it full access to the splunk directory and apply to sub directories & files (child objects) via inheritance

Re: Splunk4JMX only works when I manually run poll_jmx.bat

jkat54 — Mon, 28 Dec 2015 20:05:10 GMT

You voted on my answer, if it solved the problem then please check it as the solution too.

Re: Splunk4JMX only works when I manually run poll_jmx.bat

RecoMark0 — Mon, 28 Dec 2015 20:36:03 GMT

Sorry, I thought I did. Thank you very much, this helped a lot to figure out that one server had a different user running splunk for some reason. I have updated which service user runs splunk on the broken ones, and they are now sending jmx data.

Re: Splunk4JMX only works when I manually run poll_jmx.bat

jkat54 — Mon, 28 Dec 2015 21:04:16 GMT

Thanks and no need to say sorry!