Getting Data In

host override from a forwarder

mloven
Path Finder

Hi all. I've got a 4.3 universal forwarder pointing to a 4.3 indexer, both on CentOS. The forwarder is monitoring a file where snmp traps are being dumped by snmpd. The events are being forwarded fine, but they are all showing as coming from the same host (the forwarder), so I'd like to override the host value with a value pulled from the trap data. Here's what I did:

This is all done on the forwarder, not the indexer.
All files were created in /opt/splunkforwarder/etc/system/local unless specified otherwise.

I created a transforms.conf that has this stanza (and only this stanza):

[h_o_transform]
DEST_KEY = MetaData:Host
REGEX = ZENOSS-MIB::evtDevice.?0? = STRING: "(\S+)"
FORMAT = host::$1

And then created a props.conf that has only this stanza:

[source::///var/log/snmptraps.log]
Transforms-hostoverride=h_o_transform

And just for completeness, I'll explain that what I want to do is, for all events in one particular file (/var/log/snmptraps.log), I want to pull out a string and use that string as the host name for that event.

Eventually there would be hundreds of forwarders pointing back at the indexer, so I felt that it would be better to put a small load on each forwarder rather than compound that load on the indexer and do the host override there.

The events are still coming in as before, all from one host. So I guess I have a few questions:

  1. Can the host override be applied on the forwarder, or does it have to be applied on the indexer?
  2. Is my regex correct? It tested out fine on several online regex testers I verified it with. The part of the event that I'm trying to strip is something like this:

ZENOSS-MIB::evtDevice.0 = STRING: "devicehostname.com",

With "devicehostname.com" being the hostname I'm trying to extract, obviously...

  1. Are the config files in the proper place? I want to apply this globally on each forwarder, and, as I understand it, /opt/splunkforwarder/etc/system/local is the right place for that. But the docs that I got that info from weren't specifically referring to forwarders, so I thought I'd check...

  2. Are my props.conf and transforms.conf written correctly? Again, I pulled this almost directly out of the docs, but they were talking about applying these on an indexer, so I'm not sure if different rules apply when dealing with a forwarder...

Thanks in advance for taking a look at this!

1 Solution

lguinn2
Legend

Your configuration files are almost correct, I think, but they aren't in the proper place!

The Universal Forwarder deals with the input as a data stream, not as individual events. Therefore, the transformation cannot be performed on the forwarder. You have several choices:

  • Change the Universal Forwarder to a regular forwarder (sometimes called a "heavy" forwarder). This will allow you to parse the data as individual events. But I don't recommend this, as it will increase the load on your production server - usually not a good thing for Splunk to do.
  • Move the props.conf and transforms.conf configuration to the indexer. There are a few advantages: first, you only have to do it on the indexer, not on a bunch of forwarders. Second, the work is done on the indexer instead of the production systems. This is what I suggest as your best solution.
  • Set up an intermediate "heavy" forwarder as a collection point, and parse the events there. I would not choose this solution. Yes, it is doable. But if your indexer is busy and you need to stand up another server, stand up an additional indexer. Adding an indexer will speed up your searches, increase resiliency, etc - in addition to increasing the speed of event parsing.

A few other things: the configuration files are case-sensitive. So change the line in props.conf to:

TRANSFORMS-hostoverride=h_o_transform

I am not entirely comfortable with the quotation marks in the regex. I think it will work, but if it doesn't, try this instead:

REGEX = ZENOSS-MIB::evtDevice.?0? = STRING:\s+.(\S+).,

This assumes that the comma actually follows the string as you showed it in your question. Put props.conf and transforms.conf in /opt/splunk/etc/system/local on the indexer. Restart Splunk and all future data will be properly indexed. These changes will not apply to data that has already been indexed.

View solution in original post

Lowell
Super Champion

Note: You also have a minor typo in your source stanza:

[source::///var/log/snmptraps.log]

Should be:

[source::/var/log/snmptraps.log]

The triple slash extra two slashes are only used in [monitor://...] segments.

lguinn2
Legend

Good catch!

0 Karma

lguinn2
Legend

Your configuration files are almost correct, I think, but they aren't in the proper place!

The Universal Forwarder deals with the input as a data stream, not as individual events. Therefore, the transformation cannot be performed on the forwarder. You have several choices:

  • Change the Universal Forwarder to a regular forwarder (sometimes called a "heavy" forwarder). This will allow you to parse the data as individual events. But I don't recommend this, as it will increase the load on your production server - usually not a good thing for Splunk to do.
  • Move the props.conf and transforms.conf configuration to the indexer. There are a few advantages: first, you only have to do it on the indexer, not on a bunch of forwarders. Second, the work is done on the indexer instead of the production systems. This is what I suggest as your best solution.
  • Set up an intermediate "heavy" forwarder as a collection point, and parse the events there. I would not choose this solution. Yes, it is doable. But if your indexer is busy and you need to stand up another server, stand up an additional indexer. Adding an indexer will speed up your searches, increase resiliency, etc - in addition to increasing the speed of event parsing.

A few other things: the configuration files are case-sensitive. So change the line in props.conf to:

TRANSFORMS-hostoverride=h_o_transform

I am not entirely comfortable with the quotation marks in the regex. I think it will work, but if it doesn't, try this instead:

REGEX = ZENOSS-MIB::evtDevice.?0? = STRING:\s+.(\S+).,

This assumes that the comma actually follows the string as you showed it in your question. Put props.conf and transforms.conf in /opt/splunk/etc/system/local on the indexer. Restart Splunk and all future data will be properly indexed. These changes will not apply to data that has already been indexed.

View solution in original post

lguinn2
Legend

Thanks for the correction - I have corrected my text above!

0 Karma

mloven
Path Finder

hmmm... ok. I kind of wanted to avoid doing the override on the indexer just so it didn't have any extra work to do, but if that's the way to go, then I can roll with it.

One thing I noticed, and I'm just pointing it out for any who come after me with the same issue... you said to change "Transforms-blah" to "transforms-blah"... Not sure if you meant to change that to "TRANSFORMS-blah" or not, but that seems to be what you have to do. I tried it with all lowercase and upon restarting it spit out an error about typos in props.conf.

Otherwise though, perfect answer. Thank you!

0 Karma