Splunk Search

Is my sourcetype override messing up my field extraction, or am I?

gpullis
Communicator

My sourcetype override is working, but my field extractions are not.

props.conf

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_barracuda_sf

[barracuda_sf]
KV_MODE=none
REPORT-bsf = bsf_scan, bsf_send, bsf_recv

transforms.conf

[set_sourcetype_barracuda_sf]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?)\.\w+\s+)*\[?(10.1.5.49|10.1.5.50)[\w\.\-]*\]?\s
FORMAT = sourcetype::barracuda_sf
DEST_KEY = MetaData:Sourcetype

[bsf_scan]
REGEX = (?:[^\s\n]*\s){5}([\w/]*)\[(\d*)\]:\s(.*\]|127.0.0.1)\s([\w\d-]*)\s(\d*)\s(\d*)\s(SCAN)\s(.*)
FORMAT = barracuda_process::$2  barracuda_pid::$3 client_ip::$4 message_id::$5 start_time::$6 end_time::$7 service::$8 info::$9

[bsf_send]
REGEX = (?:[^\s\n]*\s){5}([\w/]*)\[(\d*)\]:\s(.*\]|127.0.0.1)\s([\w\d-]*)\s(\d*)\s(\d*)\s(SCAN)\s(.*)
FORMAT = barracuda_process::$2  barracuda_pid::$3 client_ip::$4 message_id::$5 start_time::$6 end_time::$7 service::$8 info::$9

[bsf_recv]
REGEX = (?:[^\s\n]*\s){5}([\w/]*)\[(\d*)\]:\s(.*\]|127.0.0.1)\s([\w\d-]*)\s(\d*)\s(\d*)\s(SEND)\s(.*)
FORMAT = barracuda_process::$2  barracuda_pid::$3 client_ip::$4 message_id::$5 start_time::$6 end_time::$7 service::$8 info::$9
0 Karma

Starlette
Contributor

Are the extracts not working at all?
I ran into some troubles as well with one big syslogfeed on udp so now i use a sysloghost with a forwarder (rolled files monitor) and push this as one sourcetype to the indexer (splunksyslog).
There i use exact the same method as you are using ( making 8+ sourcetype overrides) and have dozens of fields extracts on those new ones.
So I am not sure if this is working only for cooked data,,,my concern was the load ( 50G a day,so wanted a store and forward before parsing)
I def. want to test your setup cause I have some planned deployments with this as well!

0 Karma

lguinn2
Legend

I think you have been caught by way that stanzas in props.conf are processed; Splunk only makes one pass. You probably shouldn't count on the transformed sourcetype to be available for use in the second stanza.

But there is an easy cure for your problem. You can eliminate the second stanza altogether, unless you already have some barracuda_sf events from some other input.

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_barracuda_sf
KV_MODE=none
REPORT-bsf = bsf_scan, bsf_send, bsf_recv

[barracuda_sf]
KV_MODE=none
REPORT-bsf = bsf_scan, bsf_send, bsf_recv
0 Karma

Starlette
Contributor

Here's another thing, isn't my sourcetype override happening at index time and my field extractions happening at search time? :
yeah thats what i think,,,btw why the KV_MODE=none setting
??

0 Karma

dwaddle
SplunkTrust
SplunkTrust

If you're going to pump syslog directly into Splunk, there is nothing at all wrong with defining multiple syslog ports on a per-sourcetype basis. Use (for example) 5140 for barracuda, 5141 for VMWare ESXi, 5142 for Cisco ASA, etc ...

0 Karma

gpullis
Communicator

Here's another thing, isn't my sourcetype override happening at index time and my field extractions happening at search time?

0 Karma

gpullis
Communicator

I tried putting REPORT-bsf = bsf_scan, bsf_send, bsf_recv in my [source::udp:514], but unfortunately I still didn't get my field extractions.

0 Karma

gpullis
Communicator

My concern would be that my REPORT and KV_MODE keywords would affect all of my syslog stuff.

Maybe this is another example of why one shouldn't pump syslog directly into Splunk? 😕

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...