Getting Data In

Splunk catalina.out for java.lang.OutOfMemoryError: PermGen space on remote VM

kamal2222ahmed
Explorer

I am trying to setup Splunk to monitor a remote tomcat instance ( catalina.out ) for messages like permGen Running out of Memory
Specifically:

Exception in thread "http-bio-8080-exec-36" java.lang.OutOfMemoryError: PermGen space

I was able to install Splunk on host A, and on B i have Tomcat running, plys Universal forwarder running with:

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf

[monitor:///usr/share/apache-tomcat-7.0.47/logs]
sourcetype = access_common

/opt/splunkforwarder/etc/system/local/outputs.conf

forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=<server where splunk server is installed>:9997

So how do i :
1. Make sure the forwarder HAS Connectivity and is able send logs, some command command line utilities perhaps
2. How do i setup the receiver / splunk server ?

Tags (1)
0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Make sure your whitelist settings actually are .* and _.*... there should be no need to set them explicitly though, the defaults will work just fine.

As for the receiver, run this on the indexer CLI:

$SPLUNK_HOME/bin/splunk enable listen 9997

See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:

$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997

As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Make sure your whitelist settings actually are .* and _.*... there should be no need to set them explicitly though, the defaults will work just fine.

As for the receiver, run this on the indexer CLI:

$SPLUNK_HOME/bin/splunk enable listen 9997

See http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Enableareceiver#Set_up_receiving_with_S... for more info on receiving. On the forwarder, run this to tell it where to forward its data:

$SPLUNK_HOME/bin/splunk add forward-server indexerhost:9997

As for connectivity, talk to your network administrators about possibly existing firewalls or other network hurdles.

kamal2222ahmed
Explorer

and this Works!, thanks basic config is SO simple in Splunk, quite amazing. I wish the documentation was more use case driven.
next .....:

  1. extract , or plot only the PermGen log
  2. Setup Notifications ( Email ) upon occurrence of Error
  3. Setup another log parser to get application errors
  4. Correlate the two errors temporally
0 Karma

kamal2222ahmed
Explorer

so the username and password for the command :
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
are local ? meaning, i can choose the password for user splunk, which would be local the forwarder ?
ok i used admin:changeme

/opt/splunkforwarder/bin/splunk add forward-server vm-jenkins-staging.3mhis.vm:9997
Splunk username: admin
Password:
Added forwarding to: vm-staging.vm:9997.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The forwarder has no clue about your indexer's credentials, use admin:changeme on the forwarder.

0 Karma

kamal2222ahmed
Explorer

@martin_mueller
more /opt/splunkforwarder/etc/system/local/outputs.conf
forwardedindex.0.whitelist = .
forwardedindex.1.whitelist = _.
[tcpout:default_index] server=vm-staging.vm:9997

0 Karma

kamal2222ahmed
Explorer

I tried to run add forward-server on the forwarder , with the same admin credentials as i use to login to the indexer, but getting error:
/opt/splunkforwarder/bin/splunk add forward-server vm-staging.vm:9997
Splunk username: admin
Password:
Login failed

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...