Getting Data In

Novice question: transforming index and sourcetype for UDP/syslog

dsmith
Path Finder

I'm trying to use our Splunk environment as a replacement for an older syslog server. We have multiple indexers, and we've set up a load-balancer in front of them to handle packets coming on on UDP port 514 and spread the packets out across the indexers. That part works well, but I'm having trouble with the appropriate props and transforms configurations to get those incoming events into the correct indexes. I assume I'm just overlooking something silly, but I need another set of eyes.

We're using a small app that's being deployed from a cluster master, to the indexers, with these three configuration files:

inputs.conf (yes, port 5140 is intentional, the load balancer handles the port translation)

 

[udp:5140]
disabled = 0
connection_host = ip
source = syslog
sourcetype = syslog

 

props.conf:

 

[source::udp:5140]
TRANSFORMS = override_index_f5, override_sourcetype_f5

 

transforms.conf:

 

[override_index_f5]
SOURCE_KEY = _raw
REGEX = (.*)f5-svc-ip=(.*)
DEST_KEY = _MetaData:Index
FORMAT = f5_connlog

[override_sourcetype_f5]
SOURCE_KEY = _raw
REGEX = (.*)f5-svc-ip=(.*)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::f5:connection-log

 

The intent of the above is to take events that look like this:

Jul 14 09:22:33 10.24.43.13 LOCAL1.INFO: Jul 14 2021 09:22:33 f5-svc-ip=1.2.3.201, f5-svc-port=636, externalip=2.3.4.91, externalport=13703, internalip=5.6.7.9, internalport=13703

and route them to the "f5_connlog" index with the "f5:connection-log" sourcetype. Instead, these events are landing in the "main" index (since no other index is specified), with the "syslog" sourcetype. I assume that's happening because the events aren't matching, but the regex I'm using is about as simple as can be.

(Obviously, once I figure out what I'm doing wrong, there will be more transforms, but this is a small simple test case.)

So, wise folks, what am I overlooking?

As a related question, is it possible to perform multiple actions on a single match? (In the above, I'm using the same source_key and same regex, so is it possible to combine the sourcetype and index transforms into a single stanza? I know they're two separate things, but it just feels slightly redundant to have to run the same regex twice.)

Labels (3)
0 Karma
1 Solution

dsmith
Path Finder

Following up to myself here: The problem was with my trying to be a bit too clever in inputs.conf. By specifying the source and sourcetype there, the transforms listed in props.conf were never triggered. So all my events had the source and sourcetype I explicitly specified, and landed in the default index (main).

This will be turning into a blog post in the near future, but the short version is "don't tell Splunk to make assumptions because Splunk will do exactly what you tell it to do". Here's our corrected inputs.conf:

[udp:5140]
disabled = 0
connection_host = ip

View solution in original post

0 Karma

dsmith
Path Finder

Following up to myself here: The problem was with my trying to be a bit too clever in inputs.conf. By specifying the source and sourcetype there, the transforms listed in props.conf were never triggered. So all my events had the source and sourcetype I explicitly specified, and landed in the default index (main).

This will be turning into a blog post in the near future, but the short version is "don't tell Splunk to make assumptions because Splunk will do exactly what you tell it to do". Here's our corrected inputs.conf:

[udp:5140]
disabled = 0
connection_host = ip
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Friends don't let friends send syslog directly to Splunk.  Avoid it.  Consider replacing the old syslog server with the Splunk Connect for Syslog (SC4S) app.  It uses syslog-ng in a container to receive syslog and forward it to Splunk.  SC4S knows about many syslog sources and can set the sourcetype appropriately.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dsmith
Path Finder

Yes, but then I'd have to build a whole container infrastructure. 🙂 If I can use my existing large, highly-redundant Splunk environment, and my existing load balancers, that feels like a much wiser choice of limited IT budget.

(Aside from that, and this is diverging wildly off-topic for my original question, how does SC4S work in terms of redundancy and availability? It looks like it's ultimately a single point of failure, whereas using load balancers and sending traffic to my large cluster of indexers is likely to be more highly available and to lose fewer messages over time.)

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, you have to stand up a container or two, depending on the load, but SC4S does the rest for you.  No more having to worry about mapping sources, sourcetypes, etc.  Check the docs at https://splunk-connect-for-syslog.readthedocs.io/en/latest/

---
If this reply helps you, Karma would be appreciated.
0 Karma

dsmith
Path Finder

Let's just assume I don't want to build multiple new servers, or learn a new load-balancer technology, or Docker or Kubernetes. 🙂 

(The containers wouldn't know my sourcetype anyway, these logs are something I literally created myself and are specific to my environment. So I'd have to do the same kind of transforms anyway, just somewhere else.)

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...