Sending certain logs from UDP port 514 to specific indexes

[aferone]

We have some Cisco devices that are sending syslog via port 514 natively (no splunk forwarder installed, obviously). I want to be able to send this log to a certain index, based on hostname. I tried using props and transforms as the following:

Since the original log from the Cisco device isn't coming from a Splunk forwarder, do I have to regex for the hostname?

props.conf


[host::hostnameommitted.com]
TRANSFORMS-force_index_for_your_host = force_index_perimeter

transforms.conf


[force_index_perimeter]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = perimeter

Thanks!

[hexx · Accepted solution]

Unfortunately, as of Splunk 4.3 you cannot have more than one functional stanza/input per UDP port. The devil is in the details of inputs.conf.spec where this limitation is made implicit by the following statement :

udp://<remote server>:<port>
* If <remote server> is specified, the specified port will only accept data from that server.

It's important to understand that this refers to the specified port, not to the specified data input.

This will be further clarified in our next maintenance release by the addition of the following statement to inputs.conf.spec :

* Only one stanza per port number is currently supported.

If you have configured more than one UDP input stanza using the same port but specifying different hosts of origin, pushing the UDPInputProcessor log channel to DEBUG level will typically yield messages like this one :

02-03-2011 11:19:07.167 DEBUG UDPInputProcessor - The UDP packet data from '10.1.5.139' was refused only accepting data from '10.1.8.120'

This means that in your case, you have to use transformations to route your UDP data to different indexes, as per Lisa's 2nd suggestion.

A few considerations I would add :

A) If you are using a [host::] stanza in props.conf to conditionally apply your routing, I would suggest to use something like this to add resilience to RDNS failures:

[host::(my.hostname.com|10.1.8.120)]

B) I personally am not too crazy about host-based filtering in the stanza name of props.conf. I would rather do something like this:

B.1) In props.conf, use a sourcetype-based stanza to apply a generic routing transformation to the data coming from the UDP input:

[syslog]
TRANSFORMS-idx_routing = generic_idx_routing

B.2) In transforms.conf, use a regular expression on SOURCE_KEY = MetaData:Host to route the events to a different index, whose name will be dynamically based on the value of "host":

[generic_idx_routing]
SOURCE_KEY = MetaData:Host
REGEX = (device1|device2|device3)\.domain\.com
DEST_KEY = _MetaData:Index
FORMAT = $1


In this example, events with a value of "device1.domain.com" for the "host" field will be sent to the "device1" index, while "host=device2.domain.com" and "host=device3.domain.com" will respectively go to indexes "device2" and "device3".

Note : This implies that you will have created the indexes necessary for the routing before-hand, based on the hostnames of the devices sending data to UDP:514. If you configure a new device to send data to that port, make sure that you create the corresponding index on the indexers first.

C) Finally, an alternative to dynamic index-routing would be to use several transformations with hard-coded REGEX and FORMAT values, like so:

C.1) In props.conf, use a sourcetype-based stanza to apply several routing transformation to the data coming from the UDP input:

[syslog]
TRANSFORMS-idx_routing = device1_idx_routing, device2_idx_routing

C.2) In transforms.conf, use a regular expression on SOURCE_KEY = MetaData:Host to route the events to a different index, whose name will be dynamically based on the value of "host":

[device1_idx_routing]
SOURCE_KEY = MetaData:Host
REGEX = device1\.domain\.com
DEST_KEY = _MetaData:Index
FORMAT = device1

[device2_idx_routing]
SOURCE_KEY = MetaData:Host
REGEX = device2\.domain\.com
DEST_KEY = _MetaData:Index
FORMAT = device2


This would be particularly useful if you want to use index names that are not included in the sending device's host name or in the event's raw data.

Note: In both scenarios, you might prefer to use the event raw data as the field on which to apply the REGEX rather than "host", in which case you would set SOURCE_KEY = _raw and modify your REGEX accordingly to dynamically extract the name of the target index from the event data.

[hexx]

Thank you, Alex!

[aferone]

Certainly:

*** inputs.conf ***
[udp://514]
connection_host = dns
disabled = 0
sourcetype = syslog

*** props.conf ***
[syslog]
TRANSFORMS-idx_routing = generic_idx_routing

*** transforms.conf ***
[generic_idx_routing]
SOURCE_KEY = MetaData:Host
REGEX = (host1|host2|host3).domain.com
DEST_KEY = _MetaData:Index
FORMAT = indexname

[hexx]

@aferone : I'm very glad to hear so. Could you please share the specifics of the configuration that worked for you?

[aferone]

OK, I made one more change, and I have this working now! Thank you!!

[aferone]

I appreciate the responses, but I cannot get any of these suggestions working! All of the 514 data is now being routed to the "main" index, instead of the one I'm specifying, which does exist on my Indexer.

[Rob]

Nice tight regex and great additional details!

[lguinn2]

Nice - I vote for this answer 🙂

[lguinn2]

Rob is right about setting the index as part of the input. You are right about using the regular (aka "heavy") forwarder as a collection point for UDP inputs - it adds resilience to your environment.

The configuration must go in the inputs.conf file on your forwarder, because you are not using the UF.

[udp://hostnameomitted.com:514]
index = MyForcedIndex
connection_host = dns
sourcetype = syslog

For this to work, you will need to specify a stanza for every possible device that is sending data via UDP. (More on this later.) If this is not working, then I have some questions:

• Does the Splunk forwarder have the ability to read port 514? On many systems, this is a privileged port. Splunk will be unable to read inputs from this port unless Splunk is running as root.
• Is the data UDP and not TCP? (I know, it would be weird if it was TCP, but I gotta ask.)
• Have you tried using the IP address in the udp stanza, instead of the host name ("hostnameomitted.com")?

Okay, let's assume that you have debugged your input stanza, and it is working. Hopefully you knew about Rob's advice and did this on a testing server, just in case. You may be done, but I have more comments.

While it is very efficient to set the index in inputs.conf, it means a lot of work for you. You have to set up a stanza for every device - there is no default. What if you want to change the index for some devices, but let most of them default to a common index? Here is my solution, and it is all done on the forwarder.

inputs.conf - sends all UDP:514 data to a single index

[udp://514]
index = defaultIndexForDevices
connection_host = dns
sourcetype=syslog

props.conf - enter a host stanza for any data to be re-routed to a different index. Or use source instead of host to define the stanza(s).

[host::hostnameA]
TRANSFORMS-routeA = routeHostnameA

transforms.conf - enter a stanza for each routing

[routeHostnameA]
REGEX =.*
DEST_KEY = _MetaData:Index
FORMAT = HostnameAIndex

In this solution, inputs.conf sends all inputs to a single index - one that is the "default" for all UDP inputs. Then, with props.conf and transforms.conf, you override this to route data to a different index. The stanza(s) in props.conf identify the inputs to be routed. The corresponding stanza(s) in transforms.conf do two things:

1. Select the data to be routed, REGEX=.* will match ALL the data that is sent to this transform.
2. Select the index. FORMAT = HostnameAIndex defines the index to be used.

Of course, if you have to do this for every input, you might as well use the first solution - specify things in inputs.conf and be done with it.

[Rob]

Very nicely put. +1

[lguinn2]

PS - and you could regex for the host name if you want, but it isn't necessary.

[Rob]

Actually I think you can do this a bit easier with editing the inputs.conf file for each source/index on the indexer.

[udp://hostnameommitted.com:514]
index = MyForcedIndex

Additionally, this can set the host field with different values as well using the connection_host = [ip|dns|none] config under the desired stanza.

You might wish to take a look at the inputs.conf.spec file for some additional info for setting this up. Here is a link to an example on Splunk docs..

http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

[aferone]

But, we are utilizing the main forwarder in case we have to take the indexer down, the logs will queue on the main indexer. So we have to use it.

[Rob]

Yep, you can still do that on the indexer using your inputs.conf to redirect to different indexes for all your sources. The Cisco devices dont need to have a forwarder in between the devices for the info to get parsed. You simply need to create a stanza for each source that you receive on the indexer using the inputs.conf file and specify the index.

[aferone]

No, I am not using a universal forwarder. That's the issue. The original log is coming from a Cisco device. Straight syslog, port 514. We can't change port 514 on the Cisco devices because of an older iOS. We do use universal forwarders and manipulate the index that way for other systems. We also use custom ports for indexes. But since 514 is the default for so many devices, we want to be able to send to different indexes on devices that cannot change port 514.

[Rob]

Ok, I take it you are using a Universal Forwarder. Is it possible for you to add the stanza's you want to the indexer? You would need to put your props and transforms there as well if you wanted to use those to redirect events to another index. That will definitely work.

[aferone]

Hmmm, doesn't seem to be working...

[Rob]

In that case put that on your main forwarder, but you may want to test everything with a test index first to be sure you are getting the data you are looking for 🙂

[aferone]

We send everything through a main forwarder. The log in question is coming over port 514, but the main forwarder to indexer communication is over 9997. Will it still work?

[Rob]

Sorry, this would be on the indexer. As long as your syslog source is pointed at the indexer then the indexer will listen on that port for event data and then put it in the index defined above.

If you need a full directory path as to where this goes, it should be in $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/customApp/local/.