topic Re: Is my sourcetype override messing up my field extraction, or am I? in Splunk Search

Is my sourcetype override messing up my field extraction, or am I?

gpullis — Thu, 02 Jun 2011 01:15:59 GMT

My sourcetype override is working, but my field extractions are not.

props.conf

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_barracuda_sf

[barracuda_sf]
KV_MODE=none
REPORT-bsf = bsf_scan, bsf_send, bsf_recv

transforms.conf

[set_sourcetype_barracuda_sf]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(10.1.5.49|10.1.5.50)[\w\.\-]*\]?\s
FORMAT = sourcetype::barracuda_sf
DEST_KEY = MetaData:Sourcetype

[bsf_scan]
REGEX = (?:[^\s\n]*\s){5}([\w/]*)\[(\d*)\]:\s(.*\]|127.0.0.1)\s([\w\d-]*)\s(\d*)\s(\d*)\s(SCAN)\s(.*)
FORMAT = barracuda_process::$2  barracuda_pid::$3 client_ip::$4 message_id::$5 start_time::$6 end_time::$7 service::$8 info::$9

[bsf_send]
REGEX = (?:[^\s\n]*\s){5}([\w/]*)\[(\d*)\]:\s(.*\]|127.0.0.1)\s([\w\d-]*)\s(\d*)\s(\d*)\s(SCAN)\s(.*)
FORMAT = barracuda_process::$2  barracuda_pid::$3 client_ip::$4 message_id::$5 start_time::$6 end_time::$7 service::$8 info::$9

[bsf_recv]
REGEX = (?:[^\s\n]*\s){5}([\w/]*)\[(\d*)\]:\s(.*\]|127.0.0.1)\s([\w\d-]*)\s(\d*)\s(\d*)\s(SEND)\s(.*)
FORMAT = barracuda_process::$2  barracuda_pid::$3 client_ip::$4 message_id::$5 start_time::$6 end_time::$7 service::$8 info::$9

Re: Is my sourcetype override messing up my field extraction, or am I?

lguinn2 — Thu, 02 Jun 2011 01:34:55 GMT

I think you have been caught by way that stanzas in props.conf are processed; Splunk only makes one pass. You probably shouldn't count on the transformed sourcetype to be available for use in the second stanza.

But there is an easy cure for your problem. You can eliminate the second stanza altogether, unless you already have some barracuda_sf events from some other input.

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_barracuda_sf
KV_MODE=none
REPORT-bsf = bsf_scan, bsf_send, bsf_recv

[barracuda_sf]
KV_MODE=none
REPORT-bsf = bsf_scan, bsf_send, bsf_recv

Re: Is my sourcetype override messing up my field extraction, or am I?

gpullis — Thu, 02 Jun 2011 01:50:26 GMT

My concern would be that my REPORT and KV_MODE keywords would affect all of my syslog stuff.

Maybe this is another example of why one shouldn't pump syslog directly into Splunk? 😕

Re: Is my sourcetype override messing up my field extraction, or am I?

gpullis — Thu, 02 Jun 2011 02:13:26 GMT

I tried putting REPORT-bsf = bsf_scan, bsf_send, bsf_recv in my [source::udp:514], but unfortunately I still didn't get my field extractions.

Re: Is my sourcetype override messing up my field extraction, or am I?

gpullis — Thu, 02 Jun 2011 02:27:10 GMT

Here's another thing, isn't my sourcetype override happening at index time and my field extractions happening at search time?

Re: Is my sourcetype override messing up my field extraction, or am I?

dwaddle — Thu, 02 Jun 2011 03:08:52 GMT

If you're going to pump syslog directly into Splunk, there is nothing at all wrong with defining multiple syslog ports on a per-sourcetype basis. Use (for example) 5140 for barracuda, 5141 for VMWare ESXi, 5142 for Cisco ASA, etc ...

Re: Is my sourcetype override messing up my field extraction, or am I?

Starlette — Thu, 02 Jun 2011 05:49:18 GMT

Are the extracts not working at all?
I ran into some troubles as well with one big syslogfeed on udp so now i use a sysloghost with a forwarder (rolled files monitor) and push this as one sourcetype to the indexer (splunksyslog).
There i use exact the same method as you are using ( making 8+ sourcetype overrides) and have dozens of fields extracts on those new ones.
So I am not sure if this is working only for cooked data,,,my concern was the load ( 50G a day,so wanted a store and forward before parsing)
I def. want to test your setup cause I have some planned deployments with this as well!

Re: Is my sourcetype override messing up my field extraction, or am I?

Starlette — Thu, 02 Jun 2011 05:57:14 GMT

Here's another thing, isn't my sourcetype override happening at index time and my field extractions happening at search time? :
yeah thats what i think,,,btw why the KV_MODE=none setting
??