All Apps and Add-ons

splunkd crashing after installation of nmon 1.2.7

phiral
Explorer

Hello,

Installing this app crashes splunk-6.1.1-207789 for mac (during the installation), and after thad splunkd fails to start. Also tried to install with splunkd stopped by uncompressing it to $SPLUNK_HOME/etc/apps, and after that splunkd also fails to start with the following in splunkd_stderr.log:

~/bin/splunk/bin # cat ../var/log/splunk/splunkd_stderr.log 
2014-05-31 07:57:13.653 +0100 splunkd started (build 207789)
Conf mutator lockfile has disappeared; error condition possible.
2014-05-31 07:58:54.836 +0100 splunkd started (build 207789)
Conf mutator lockfile has disappeared; error condition possible.
2014-05-31 08:00:35.167 +0100 splunkd started (build 207789)
2014-05-31 08:03:37.358 +0100 Interrupt signal received
2014-05-31 08:13:15.794 +0100 splunkd started (build 207789)
Dying on signal #15 (si_code=0), sent by PID 38287 (UID 501)
2014-05-31 08:15:28.973 +0100 splunkd started (build 207789)
Cannot open manifest file inside "/Users/phiral/bin/splunk/var/lib/splunk/audit/db/db_1401520395_1401520395_3/rawdata": No such file or directory
Cannot open manifest file inside "/Users/phiral/bin/splunk/var/lib/splunk/_internaldb/db/db_1401520396_1401519817_4/rawdata": No such file or directory
Dying on signal #15 (si_code=0), sent by PID 38378 (UID 501)

Have tried repairing with splunk fsck repair --all-buckets-all-indexes --include-hots --metadata with no luck.
I am quite new to splunk and still don't know how to do a better debugging/investigation of the problem or gather more information.

Thanks!

0 Karma
1 Solution

guilmxm
SplunkTrust
SplunkTrust

Ok, after some tests i've done, i think the third party script "nmon_helper.sh" makes splunkd to crash under Mac OS X.

However, please note that nmon is ONLY compatible with AIX, Solaris and Linux distribution.

On Mac os X (or either Windows) you can ONLY use the App as a frontend or an indexer to receive and analyse data from your hosts.

Beyond this, could you please try:

  • have a functional splunk installation (so re-install if you can't get back splunk to work, in my tests deleting /Application/Splunk/etc/apps/nmon, /Application/Splunk/var/libs/* was enough to restart successfully Splunk)

  • stop splunk

  • extract manually the version 1.2.7 in etc/apps

  • before starting, cp defaults/inputs.conf to local/inputs.conf

  • edit your local/inputs.conf and deactivate the nmon_helper.sh and purge_nmon_repository with following code:

[script://./bin/nmon_helper.sh]
disabled = true

[script://./bin/purge_nmon_repository.sh]
disabled = true

  • Start Splunk

I do believe the problem comes from the nmon_helper.sh execution being activated by default (i'm checking it) under Mac, this unwanted to execute anyway as it will never generate any nmon data on Mac, but this behavior was unexpected

View solution in original post

guilmxm
SplunkTrust
SplunkTrust

For information, a new version V1.2.8 has been release to correct this unexpected crash when installation NMON App under Mac OS X

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Ok, after some tests i've done, i think the third party script "nmon_helper.sh" makes splunkd to crash under Mac OS X.

However, please note that nmon is ONLY compatible with AIX, Solaris and Linux distribution.

On Mac os X (or either Windows) you can ONLY use the App as a frontend or an indexer to receive and analyse data from your hosts.

Beyond this, could you please try:

  • have a functional splunk installation (so re-install if you can't get back splunk to work, in my tests deleting /Application/Splunk/etc/apps/nmon, /Application/Splunk/var/libs/* was enough to restart successfully Splunk)

  • stop splunk

  • extract manually the version 1.2.7 in etc/apps

  • before starting, cp defaults/inputs.conf to local/inputs.conf

  • edit your local/inputs.conf and deactivate the nmon_helper.sh and purge_nmon_repository with following code:

[script://./bin/nmon_helper.sh]
disabled = true

[script://./bin/purge_nmon_repository.sh]
disabled = true

  • Start Splunk

I do believe the problem comes from the nmon_helper.sh execution being activated by default (i'm checking it) under Mac, this unwanted to execute anyway as it will never generate any nmon data on Mac, but this behavior was unexpected

guilmxm
SplunkTrust
SplunkTrust

Great, can you set the question as answered.

You could join the google group:
https://groups.google.com/d/forum/nmon-splunk-app

I would be interested in having analysis of CPU load of Forwarders with the nmon app running (TA-nmon, check the scenario 2 within the help page for a full distributed scenario)

If you have the occasion, and have forwarders with the TA-nmon collecting data, could you send me some CPU utilization analysis (of splunkd process)

The App itself provides views to get CPU load of processes (TOP section)

I'm working currently on improving this CPU load on UF.

0 Karma

phiral
Explorer

Hi,
That was exactly the issue 🙂

The intention is to analyze data from aix hosts, we are running a large IBM/AIX infrastructure and just started to test-drive splunk to correlate events and performance statistics from power systems, aix lpars, database and storage arrays and service incidents.

As of now with my limited knowledge of splunk i am scripting lots of collectors/parsers to feed it with data. - And i am very happy to have found your app.

I would be glad to contribute when i have more experience with the overall splunk architecture.

Thanks!

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Hi,

This is quite strange, and as it does seem to be directly associated with a typical App issue...

You could try:

  • remove the App
  • start splunk
  • If everything is ok, re-install the App
  • restart Splunk

If your installation is not "critical", i mean a production environment for example, you can also simply remove the audit and _internal indexes if splunk still does not start. (they will be re-created upon start-up)

I will try myself upgrading from 1.2.6 to 1.2.7 on a Mac to check i have the same issue

0 Karma

guilmxm
SplunkTrust
SplunkTrust

also note that installing the Web Framework Toolkit is not required anymore as components have been included now in the App

phiral
Explorer

Hi,

I uncompressed a new splunk installation, started it and switched to a free license.

Installed the web framework toolkit and restarted splunk - everything ok.

Stopped splunk, uncompressed nmon app 1.2.7 in etc/apps and splunkd fails to start with:

Dying on signal #15 (si_code=0), sent by PID 70330 (UID 501)

Trying to start it again fails with the audit and _internal indexes errors. Removing the indexes did not help, same errors.

Removing the app did not help either, and starting splunk with --debug did not produce any more information.

Thanks for looking into this!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...