Novice question: transforming index and sourcetype for UDP/syslog

[dsmith]

I'm trying to use our Splunk environment as a replacement for an older syslog server. We have multiple indexers, and we've set up a load-balancer in front of them to handle packets coming on on UDP port 514 and spread the packets out across the indexers. That part works well, but I'm having trouble with the appropriate props and transforms configurations to get those incoming events into the correct indexes. I assume I'm just overlooking something silly, but I need another set of eyes.

We're using a small app that's being deployed from a cluster master, to the indexers, with these three configuration files:

inputs.conf (yes, port 5140 is intentional, the load balancer handles the port translation)

[udp:5140]
disabled = 0
connection_host = ip
source = syslog
sourcetype = syslog

props.conf:

[source::udp:5140]
TRANSFORMS = override_index_f5, override_sourcetype_f5

transforms.conf:

[override_index_f5]
SOURCE_KEY = _raw
REGEX = (.*)f5-svc-ip=(.*)
DEST_KEY = _MetaData:Index
FORMAT = f5_connlog

[override_sourcetype_f5]
SOURCE_KEY = _raw
REGEX = (.*)f5-svc-ip=(.*)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::f5:connection-log

The intent of the above is to take events that look like this:

Jul 14 09:22:33 10.24.43.13 LOCAL1.INFO: Jul 14 2021 09:22:33 f5-svc-ip=1.2.3.201, f5-svc-port=636, externalip=2.3.4.91, externalport=13703, internalip=5.6.7.9, internalport=13703

and route them to the "f5_connlog" index with the "f5:connection-log" sourcetype. Instead, these events are landing in the "main" index (since no other index is specified), with the "syslog" sourcetype. I assume that's happening because the events aren't matching, but the regex I'm using is about as simple as can be.

(Obviously, once I figure out what I'm doing wrong, there will be more transforms, but this is a small simple test case.)

So, wise folks, what am I overlooking?

As a related question, is it possible to perform multiple actions on a single match? (In the above, I'm using the same source_key and same regex, so is it possible to combine the sourcetype and index transforms into a single stanza? I know they're two separate things, but it just feels slightly redundant to have to run the same regex twice.)

[dsmith · Accepted solution]

Following up to myself here: The problem was with my trying to be a bit too clever in inputs.conf. By specifying the source and sourcetype there, the transforms listed in props.conf were never triggered. So all my events had the source and sourcetype I explicitly specified, and landed in the default index (main).

This will be turning into a blog post in the near future, but the short version is "don't tell Splunk to make assumptions because Splunk will do exactly what you tell it to do". Here's our corrected inputs.conf:

[udp:5140]
disabled = 0
connection_host = ip

[richgalloway]

Friends don't let friends send syslog directly to Splunk.  Avoid it.  Consider replacing the old syslog server with the Splunk Connect for Syslog (SC4S) app.  It uses syslog-ng in a container to receive syslog and forward it to Splunk.  SC4S knows about many syslog sources and can set the sourcetype appropriately.

[dsmith]

Yes, but then I'd have to build a whole container infrastructure. 🙂 If I can use my existing large, highly-redundant Splunk environment, and my existing load balancers, that feels like a much wiser choice of limited IT budget.

(Aside from that, and this is diverging wildly off-topic for my original question, how does SC4S work in terms of redundancy and availability? It looks like it's ultimately a single point of failure, whereas using load balancers and sending traffic to my large cluster of indexers is likely to be more highly available and to lose fewer messages over time.)

[richgalloway]

Yes, you have to stand up a container or two, depending on the load, but SC4S does the rest for you.  No more having to worry about mapping sources, sourcetypes, etc.  Check the docs at https://splunk-connect-for-syslog.readthedocs.io/en/latest/

[dsmith]

Let's just assume I don't want to build multiple new servers, or learn a new load-balancer technology, or Docker or Kubernetes. 🙂 

(The containers wouldn't know my sourcetype anyway, these logs are something I literally created myself and are specific to my environment. So I'd have to do the same kind of transforms anyway, just somewhere else.)