Getting Data In

Using props/transforms to assign sourcetype and extract fields?

gowen
Path Finder

We have various 514/udp sources that all get mashed in under sourcetype "syslog". I'd like to break some of these out and do some specific extraction. Can a sourcetype be assigned using transforms.conf and then (as the new sourcetype) be operated on within props.conf?

So, let's say I have this in props.conf:

[source::udp:514]
TRANSFORMS-set_sourcetype_cisco = set_sourcetype_cisco

Which references this in transforms.conf:

[set_sourcetype_cisco]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.1\.1$
FORMAT = sourcetype::cisco
DEST_KEY = MetaData:Sourcetype

Can I then have something like this further down in props.conf?

[cisco]
EXTRACT-ip_proto,src_address,src_port,etc = "list 101 denied (?<ip_proto>[a-zA-Z]+) (?<src_address>d+.d+.d+.d+)((?<src_port>d+)) -> (?<dst_address>d+.d+.d+.d+)((?<dst_port>d+))"

in order to extract data from these lines after they've been tagged as sourcetype 'cisco'?

Any thoughts appreciated. I must say, I'm kind of surprised that extractors for Cisco aren't cooked in or easily available. The Cisco Security Suite app doesn't seem to cover routers/switches.

Update - does not appear to work. Props.conf contains:

[source::udp:514]
TRANSFORMS-set_sourcetype_514 = set_sourcetype_f5, set_sourcetype_cisco

# This isn't working
#[cisco]
# But this does
[host::208.70.177.252]
# Which implies to me that props isn't taking advantage of the sourcetype
# transform above?
TIME_PREFIX = ^\d+:\s+\d+:
TIME_FORMAT = %b %e %H:%d:%m.%3N

And transforms.conf is correctly setting the sourcetype like this:

[set_sourcetype_cisco]
REGEX = ^(\d+:\s+\d+:\s+\w{3}\s+\d+\s+\d+:\d{2}:\d{2}\.\d{3}\s+\w+:\s+%[^:]*:)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::cisco

I know the sourcetype is being rewritten because I get it in search results. If I try to parse the timestamp by triggering on [cisco], the timestamps aren't parsed. If I try to parse the same records by triggering on [host::IP ADDRESS], that works.

aharvey7
New Member

Anyone else running into this issue? This is exactly what I need to do

0 Karma

xpac
SplunkTrust
SplunkTrust

Basically - the initial sourcetype determines the props.conf rules that are being applied to the data at index time.
Therefore, you can rewrite the sourcetype at index-time, but Splunk will not use index-time rules for that new sourcetype. It will however use search-time rules for that new sourcetype.
Therefore, you either need to get data in with the right sourcetype from the very beginning - best practice is not to let Splunk receive on port 514, but a syslog server like syslog-ng, that writes the data to disk, split by hostname/IP of sender.
You can then built proper file monitors for every device and assign them the proper sourcetype. 🙂

0 Karma

Drainy
Champion

Yes! I do exactly this at the moment and the trick is the order in which you apply things.
So, the first thing I do is apply an initial sourcetype to my input in inputs.conf. E.g syslog.
In props I then define a syslog stanza as [syslog]. The first thing I do is then do a host re-write to pull the host out correctly and then do a sourcetype re-assignment, much like you have in your example.

Then you can reference the new sourcetype to do any search time extractions as you please 🙂

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...