I am trying to setup Splunk to monitor a remote tomcat instance ( catalina.out ) for messages like permGen Running out of Memory
Specifically:
Exception in thread "http-bio-8080-exec-36" java.lang.OutOfMemoryError: PermGen space
I was able to install Splunk on host A, and on B i have Tomcat running, plys Universal forwarder running with:
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf
[monitor:///usr/share/apache-tomcat-7.0.47/logs]
sourcetype = access_common
/opt/splunkforwarder/etc/system/local/outputs.conf
forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=<server where splunk server is installed>:9997
So how do i :
1. Make sure the forwarder HAS Connectivity and is able send logs, some command command line utilities perhaps
2. How do i setup the receiver / splunk server ?
Make sure your whitelist settings actually are .*
and _.*
... there should be no need to set them explicitly though, the defaults will work just fine.
As for the receiver, run this on the indexer CLI:
$SPLUNK_HOME/bin/splunk enable listen 9997
See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:
$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997
As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.
Make sure your whitelist settings actually are .*
and _.*
... there should be no need to set them explicitly though, the defaults will work just fine.
As for the receiver, run this on the indexer CLI:
$SPLUNK_HOME/bin/splunk enable listen 9997
See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:
$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997
As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.
and this Works!, thanks basic config is SO simple in Splunk, quite amazing. I wish the documentation was more use case driven.
next .....:
so the username and password for the command :
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
are local ? meaning, i can choose the password for user splunk, which would be local the forwarder ?
ok i used admin:changeme
/opt/splunkforwarder/bin/splunk add forward-server vm-jenkins-staging.3mhis.vm:9997
Splunk username: admin
Password:
Added forwarding to: vm-staging.vm:9997.
The forwarder has no clue about your indexer's credentials, use admin:changeme on the forwarder.
@martin_mueller
more /opt/splunkforwarder/etc/system/local/outputs.conf
forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=vm-staging.vm:9997
I tried to run add forward-server on the forwarder , with the same admin credentials as i use to login to the indexer, but getting error:
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
Splunk username: admin
Password:
Login failed