Archive

Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

Explorer

I am trying to setup Splunk to monitor a remote tomcat instance ( catalina.out ) for messages like permGen Running out of Memory
Specifically:

Exception in thread "http-bio-8080-exec-36" java.lang.OutOfMemoryError: PermGen space

I was able to install Splunk on host A, and on B i have Tomcat running, plys Universal forwarder running with:

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf

[monitor:///usr/share/apache-tomcat-7.0.47/logs]
sourcetype = access_common

/opt/splunkforwarder/etc/system/local/outputs.conf

forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=<server where splunk server is installed>:9997

So how do i :
1. Make sure the forwarder HAS Connectivity and is able send logs, some command command line utilities perhaps
2. How do i setup the receiver / splunk server ?

Tags (1)
0 Karma
1 Solution

SplunkTrust
SplunkTrust

Make sure your whitelist settings actually are .* and _.*... there should be no need to set them explicitly though, the defaults will work just fine.

As for the receiver, run this on the indexer CLI:

$SPLUNK_HOME/bin/splunk enable listen 9997

See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:

$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997

As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.

View solution in original post

SplunkTrust
SplunkTrust

Make sure your whitelist settings actually are .* and _.*... there should be no need to set them explicitly though, the defaults will work just fine.

As for the receiver, run this on the indexer CLI:

$SPLUNK_HOME/bin/splunk enable listen 9997

See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:

$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997

As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.

View solution in original post

Explorer

and this Works!, thanks basic config is SO simple in Splunk, quite amazing. I wish the documentation was more use case driven.
next .....:

  1. extract , or plot only the PermGen log
  2. Setup Notifications ( Email ) upon occurrence of Error
  3. Setup another log parser to get application errors
  4. Correlate the two errors temporally
0 Karma

Explorer

so the username and password for the command :
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
are local ? meaning, i can choose the password for user splunk, which would be local the forwarder ?
ok i used admin:changeme

/opt/splunkforwarder/bin/splunk add forward-server vm-jenkins-staging.3mhis.vm:9997
Splunk username: admin
Password:
Added forwarding to: vm-staging.vm:9997.

0 Karma

SplunkTrust
SplunkTrust

The forwarder has no clue about your indexer's credentials, use admin:changeme on the forwarder.

0 Karma

Explorer

@martinmueller
more /opt/splunkforwarder/etc/system/local/outputs.conf
forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default
index] server=vm-staging.vm:9997

0 Karma

Explorer

I tried to run add forward-server on the forwarder , with the same admin credentials as i use to login to the indexer, but getting error:
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
Splunk username: admin
Password:
Login failed

0 Karma