How do I configure Splunk for Nagios to ingest nagios events from syslog

[lukeh]

Duplicate events are being captured by syslog (and therefore Splunk too). For example, nagios events are being written to the following two files on the central syslog server :-

user.log
syslog

Splunk ingests these log files into the default index called "main" with a sourcetype of "syslog".

How do I configure Splunk to discard duplicate events and ingest the nagios events into the "nagios" index with a sourcetype of "nagios"?

[avdheuvel]

You could stop nagios writing to syslog in nagios.conf on your nagios server.
Reduces disk I/O on your monitoing server to 🙂

[aandrieu]

Hi Luke 🙂

Thank you for your quick answer!
It didn't work because I receive these events from a forwarder: I just had to setup this configuration at the forwarder level.
For the record, I used this thread also: http://answers.splunk.com/answers/1958/cant-route-forwarded-data-to-different-index

Now it works well with the "syslog" as a stanza in props.conf and the modified regexp into transforms.conf

Thanks again 🙂

Arthur

[lukeh]

Hi Arthur 🙂

you can use a modified regex in a transform that applies the nagios sourcetype to events that come from your nagios host.

Please refer to the example here:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides#Example:_Assign_...

All the best,

Luke 🙂

[aandrieu]

Hi all,

I have a slightly different configuration: I try to capture nagios events from syslog events (sent by a forwarder), but I don't have anything in my index or sourcetype "nagios".
The host is determined correctly into syslog, so I should be able to filter on it.

In my props.conf:
[host::]
TRANSFORMS-set_nagios_index = set_nagios_index
TRANSFORMS-set_nagios_sourcetype = set_nagios_sourcetype

In my transforms.conf:
[set_nagios_index]
REGEX = \snagios(\d+|):\s
DEST_KEY = _MetaData:Index
FORMAT = nagios

[set_nagios_sourcetype]
REGEX = \snagios(\d+|):\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::nagios

Any thoughts ?
Thanks you in advance,
Arthur

[lukeh · Accepted solution]

You can configure Splunk to drop any nagios events that are read from the syslog file and only index the nagios events from user.log. You can also route the nagios events to a Splunk index called "nagios" with a sourcetype of "nagios".

Simply update props.conf and transforms.conf on your Splunk indexer as below :-

/opt/splunk/etc/system/local/props.conf

[source::/.../user.log]
TRANSFORMS-set_nagios_index = set_nagios_index
TRANSFORMS-set_nagios_sourcetype = set_nagios_sourcetype

[source::/.../syslog]
TRANSFORMS-set_nagios_null = set_nagios_null


/opt/splunk/etc/system/local/transforms.conf

[set_nagios_index]
REGEX = \snagios(\d+|):\s
DEST_KEY = _MetaData:Index
FORMAT = nagios

[set_nagios_sourcetype]
REGEX = \snagios(\d+|):\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::nagios

[set_nagios_null]
REGEX = \snagios(\d+|):\s
DEST_KEY = queue
FORMAT = nullQueue

Change the nagiosevent field extraction to work with your event data, just update the following configuration file:

$SPLUNK_HOME/etc/apps/SplunkForNagios/default/props.conf

Replace the following field extraction:

EXTRACT-nagiosevent = \[\d+] (?P<nagiosevent>[^:]*)(?=)

with this field extraction:

EXTRACT-nagiosevent = \snagios(\d+|):\s(?P<nagiosevent>[^:]*)(?=)

Restart Splunk 🙂

You should now be able to search for your nagios events in Splunk:

index=nagios

Luke 🙂