Getting Data In

Using props/transforms to assign sourcetype and extract fields?

gowen
Path Finder

We have various 514/udp sources that all get mashed in under sourcetype "syslog". I'd like to break some of these out and do some specific extraction. Can a sourcetype be assigned using transforms.conf and then (as the new sourcetype) be operated on within props.conf?

So, let's say I have this in props.conf:

[source::udp:514]
TRANSFORMS-set_sourcetype_cisco = set_sourcetype_cisco

Which references this in transforms.conf:

[set_sourcetype_cisco]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.1\.1$
FORMAT = sourcetype::cisco
DEST_KEY = MetaData:Sourcetype

Can I then have something like this further down in props.conf?

[cisco]
EXTRACT-ip_proto,src_address,src_port,etc = "list 101 denied (?<ip_proto>[a-zA-Z]+) (?<src_address>d+.d+.d+.d+)((?<src_port>d+)) -> (?<dst_address>d+.d+.d+.d+)((?<dst_port>d+))"

in order to extract data from these lines after they've been tagged as sourcetype 'cisco'?

Any thoughts appreciated. I must say, I'm kind of surprised that extractors for Cisco aren't cooked in or easily available. The Cisco Security Suite app doesn't seem to cover routers/switches.

Update - does not appear to work. Props.conf contains:

[source::udp:514]
TRANSFORMS-set_sourcetype_514 = set_sourcetype_f5, set_sourcetype_cisco

# This isn't working
#[cisco]
# But this does
[host::208.70.177.252]
# Which implies to me that props isn't taking advantage of the sourcetype
# transform above?
TIME_PREFIX = ^\d+:\s+\d+:
TIME_FORMAT = %b %e %H:%d:%m.%3N

And transforms.conf is correctly setting the sourcetype like this:

[set_sourcetype_cisco]
REGEX = ^(\d+:\s+\d+:\s+\w{3}\s+\d+\s+\d+:\d{2}:\d{2}\.\d{3}\s+\w+:\s+%[^:]*:)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::cisco

I know the sourcetype is being rewritten because I get it in search results. If I try to parse the timestamp by triggering on [cisco], the timestamps aren't parsed. If I try to parse the same records by triggering on [host::IP ADDRESS], that works.

aharvey7
New Member

Anyone else running into this issue? This is exactly what I need to do

0 Karma

xpac
SplunkTrust
SplunkTrust

Basically - the initial sourcetype determines the props.conf rules that are being applied to the data at index time.
Therefore, you can rewrite the sourcetype at index-time, but Splunk will not use index-time rules for that new sourcetype. It will however use search-time rules for that new sourcetype.
Therefore, you either need to get data in with the right sourcetype from the very beginning - best practice is not to let Splunk receive on port 514, but a syslog server like syslog-ng, that writes the data to disk, split by hostname/IP of sender.
You can then built proper file monitors for every device and assign them the proper sourcetype. 🙂

0 Karma

Drainy
Champion

Yes! I do exactly this at the moment and the trick is the order in which you apply things.
So, the first thing I do is apply an initial sourcetype to my input in inputs.conf. E.g syslog.
In props I then define a syslog stanza as [syslog]. The first thing I do is then do a host re-write to pull the host out correctly and then do a sourcetype re-assignment, much like you have in your example.

Then you can reference the new sourcetype to do any search time extractions as you please 🙂

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...