topic Re: Using props/transforms to assign sourcetype and extract fields? in Getting Data In

Using props/transforms to assign sourcetype and extract fields?

gowen — Thu, 12 Jul 2012 21:08:35 GMT

We have various 514/udp sources that all get mashed in under sourcetype "syslog". I'd like to break some of these out and do some specific extraction. Can a sourcetype be assigned using transforms.conf and then (as the new sourcetype) be operated on within props.conf?

So, let's say I have this in props.conf:

[source::udp:514]
TRANSFORMS-set_sourcetype_cisco = set_sourcetype_cisco

Which references this in transforms.conf:

[set_sourcetype_cisco]
SOURCE_KEY = MetaData:Host
REGEX = ^host::192\.168\.1\.1$
FORMAT = sourcetype::cisco
DEST_KEY = MetaData:Sourcetype

Can I then have something like this further down in props.conf?

[cisco]
EXTRACT-ip_proto,src_address,src_port,etc = "list 101 denied (?<ip_proto>[a-zA-Z]+) (?<src_address>d+.d+.d+.d+)((?<src_port>d+)) -> (?<dst_address>d+.d+.d+.d+)((?<dst_port>d+))"

in order to extract data from these lines after they've been tagged as sourcetype 'cisco'?

Any thoughts appreciated. I must say, I'm kind of surprised that extractors for Cisco aren't cooked in or easily available. The Cisco Security Suite app doesn't seem to cover routers/switches.

Update - does not appear to work. Props.conf contains:

[source::udp:514]
TRANSFORMS-set_sourcetype_514 = set_sourcetype_f5, set_sourcetype_cisco

# This isn't working
#[cisco]
# But this does
[host::208.70.177.252]
# Which implies to me that props isn't taking advantage of the sourcetype
# transform above?
TIME_PREFIX = ^\d+:\s+\d+:
TIME_FORMAT = %b %e %H:%d:%m.%3N

And transforms.conf is correctly setting the sourcetype like this:

[set_sourcetype_cisco]
REGEX = ^(\d+:\s+\d+:\s+\w{3}\s+\d+\s+\d+:\d{2}:\d{2}\.\d{3}\s+\w+:\s+%[^:]*:)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::cisco

I know the sourcetype is being rewritten because I get it in search results. If I try to parse the timestamp by triggering on [cisco], the timestamps aren't parsed. If I try to parse the same records by triggering on [host::IP ADDRESS], that works.

Re: Using props/transforms to assign sourcetype and extract fields?

Drainy — Fri, 13 Jul 2012 06:19:35 GMT

Yes! I do exactly this at the moment and the trick is the order in which you apply things.
So, the first thing I do is apply an initial sourcetype to my input in inputs.conf. E.g syslog.
In props I then define a syslog stanza as [syslog]. The first thing I do is then do a host re-write to pull the host out correctly and then do a sourcetype re-assignment, much like you have in your example.

Then you can reference the new sourcetype to do any search time extractions as you please 🙂

Re: Using props/transforms to assign sourcetype and extract fields?

aharvey7 — Wed, 19 Oct 2016 14:49:26 GMT

Anyone else running into this issue? This is exactly what I need to do

Re: Using props/transforms to assign sourcetype and extract fields?

xpac — Mon, 30 Apr 2018 21:44:00 GMT

Basically - the initial sourcetype determines the props.conf rules that are being applied to the data at index time.
Therefore, you can rewrite the sourcetype at index-time, but Splunk will not use index-time rules for that new sourcetype. It will however use search-time rules for that new sourcetype.
Therefore, you either need to get data in with the right sourcetype from the very beginning - best practice is not to let Splunk receive on port 514, but a syslog server like syslog-ng, that writes the data to disk, split by hostname/IP of sender.
You can then built proper file monitors for every device and assign them the proper sourcetype. 🙂