Where does Analytics for Nagios get sourcetype="nagios:hosts" data from?

[MartinBrown]

I have Analytics for Nagios (4.0.0), Splunk Add-on for Nagios Core (1.0.0), and Nagios (4.0.8) with mk_livestatus installed on some UNIX systems.

Many of the Analytics for Nagios dashboards work fine but none of the dashboards using the nagios_hosts_sourcetype macro show any hosts. This seems to be because none of the data received by the indexer has sourcetype="nagios:hosts" or sourcetype="nagios:core" for that matter. The only data the indexer sees has sourcetypes: nagiosserviceperf, nagioshostperf, and nagios.

What am I missing?

Thanks.

[jkat54]

If you look at macros.conf under the default directory you will see the following:

[nagios_index]
definition = index="nagios"
iseval = 0

[nagios_core_sourcetype]
definition = sourcetype="nagios:core"
iseval = 0

[nagios_gearman_sourcetype]
definition = sourcetype="nagios:gearman"
iseval = 0

[nagios_hosts_sourcetype]
definition = sourcetype="nagios:hosts"
iseval = 0

[HostAlert]
definition = eventname="INITIAL HOST STATE" OR eventname="CURRENT HOST STATE" OR eventname="HOST ALERT"

[ServiceAlert]
definition = eventname="INITIAL SERVICE STATE" OR eventname="CURRENT SERVICE STATE" OR eventname="SERVICE ALERT"

[nagios_hosts]
definition = `nagios_index` `nagios_core_sourcetype` `nagios_core_host_alert` | stats count by host_name
iseval = 0

[nagios_core_host_alert]
definition = eventname="INITIAL HOST STATE" OR eventname="CURRENT HOST STATE" OR eventname="HOST ALERT"
iseval = 0

[nagios_core_service_alert]
definition = eventname="INITIAL SERVICE STATE" OR eventname="CURRENT SERVICE STATE" OR eventname="SERVICE ALERT"
iseval = 0

nagios:hosts sourcetype is created with this search:
index="nagios" sourcetype="nagios:core" eventname="INITIAL SERVICE STATE" OR eventname="CURRENT SERVICE STATE" OR eventname="SERVICE ALERT" | stats count by host_name

nagios:core sourcetype is created in another application as per the app description:
it leverages inputs from the Splunk Supported "Splunk Add-on for Nagios Core".

[pcutter]

I also noticed this issue as the instructions for configuring tell you to add a sourcetype of nagios:core but most of the field extractions and macros are looking for a sourcetype of nagios.

I updated my sourcetype to be nagios rather than nagios:core and some things started working, however some parts are still not working.

[MartinBrown]

Thanks for that. So it seems my underlying problem is that the Splunk_TA_nagios-core app (1.0.0) on the universal forwarders is not assigning the sourcetypes correctly perhaps. The forwarder inputs.conf has:

[monitor:///apps/nagios/var/nagios.log]
sourcetype = nagios:core
disabled = 0
index=nagios

but the nagios index for /apps/nagios/var/nagios.log has sourcetype=nagios.

[jkat54]

That makes me believe you have a broken configuration stanza somewhere.

Something like this:

monitor:///.../.../*.log] <- missing [

Or this
[monitor://.../... <- missing ]

on the forwarder in question. Please check to see if the other .conf files on the forwarder are correct and if they have any typos, etc.

In the meantime you can solve the problem as follows (work around):

Change the nagios_core_sourcetype macro:

 [nagios_core_sourcetype]
 definition = sourcetype="nagios"     #<--- removed : core from nagios:core
 iseval = 0