topic Re: Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM in Getting Data In

Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

kamal2222ahmed — Thu, 10 Apr 2014 13:55:40 GMT

I am trying to setup Splunk to monitor a remote tomcat instance ( catalina.out ) for messages like permGen Running out of Memory
Specifically:

Exception in thread "http-bio-8080-exec-36" java.lang.OutOfMemoryError: PermGen space

I was able to install Splunk on host A, and on B i have Tomcat running, plys Universal forwarder running with:

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf

[monitor:///usr/share/apache-tomcat-7.0.47/logs]
sourcetype = access_common

/opt/splunkforwarder/etc/system/local/outputs.conf

forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=<server where splunk server is installed>:9997

So how do i :
1. Make sure the forwarder HAS Connectivity and is able send logs, some command command line utilities perhaps
2. How do i setup the receiver / splunk server ?

Re: Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

martin_mueller — Thu, 10 Apr 2014 14:44:13 GMT

Make sure your whitelist settings actually are .* and _.*... there should be no need to set them explicitly though, the defaults will work just fine.

As for the receiver, run this on the indexer CLI:

$SPLUNK_HOME/bin/splunk enable listen 9997

See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_Splunk_CLI for more info on receiving. On the forwarder, run this to tell it where to forward its data:

$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997

As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.

Re: Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

kamal2222ahmed — Thu, 10 Apr 2014 17:20:00 GMT

I tried to run add forward-server on the forwarder , with the same admin credentials as i use to login to the indexer, but getting error:
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
Splunk username: admin
Password:
Login failed

Re: Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

kamal2222ahmed — Mon, 28 Sep 2020 16:21:40 GMT

@martin_mueller
more /opt/splunkforwarder/etc/system/local/outputs.conf
forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=vm-staging.vm:9997

Re: Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

martin_mueller — Thu, 10 Apr 2014 18:04:26 GMT

The forwarder has no clue about your indexer's credentials, use admin:changeme on the forwarder.

Re: Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

kamal2222ahmed — Thu, 10 Apr 2014 23:00:13 GMT

so the username and password for the command :
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
are local ? meaning, i can choose the password for user splunk, which would be local the forwarder ?
ok i used admin:changeme

/opt/splunkforwarder/bin/splunk add forward-server vm-jenkins-staging.3mhis.vm:9997
Splunk username: admin
Password:
Added forwarding to: vm-staging.vm:9997.

Re: Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

kamal2222ahmed — Fri, 11 Apr 2014 02:01:38 GMT

and this Works!, thanks basic config is SO simple in Splunk, quite amazing. I wish the documentation was more use case driven.
next .....:

extract , or plot only the PermGen log
Setup Notifications ( Email ) upon occurrence of Error
Setup another log parser to get application errors
Correlate the two errors temporally