topic Re: REGEX filter in transforms.conf file setting question in Splunk Enterprise

REGEX filter in transforms.conf file setting question

erictodor — Mon, 08 Jan 2018 22:07:20 GMT

We're forwarding events to a 3rd party. In our transforms.conf file, the filter looks like the following

REGEX = .

For some reason, this filter capture names without any hyphens. Here's what I'm talking about

Success - Computer
Failure - Co-m-puter

We have computer names with the '-' in them but they don't get captured. Is there a better wild card string that can be used to capture all computer names, regardless of what characters are in them.

Thanks!

Re: REGEX filter in transforms.conf file setting question

micahkemp — Mon, 08 Jan 2018 22:13:34 GMT

REGEX = . isn't capturing names at all. It's not capturing anything, and it matches anything. I think you're looking at the wrong transform.

Re: REGEX filter in transforms.conf file setting question

richgalloway — Mon, 08 Jan 2018 22:28:30 GMT

If you share some sample data and your transforms.conf settings we can help better.

Re: REGEX filter in transforms.conf file setting question

erictodor — Tue, 29 Sep 2020 17:35:07 GMT

Here's some additional info. Let me know if this is helpful.

Thank you

Props.conf

[host::*]
TRANSFORMS-main = test_eventlogs

Transforms.conf

[test_eventlogs]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = windowsworkstationsmain

Outputs.conf

[syslog:windowsworkstationsmain]
server = my.destination.com
sendCookedData = true

Re: REGEX filter in transforms.conf file setting question

micahkemp — Mon, 08 Jan 2018 22:40:22 GMT

That configuration tells Splunk where to send the data (in this case by syslog to my.destination.com). That is uninvolved in the parsing of the data.

Can you include props.conf (from the search head, not the forwarder) for the sourcetype in question?

Re: REGEX filter in transforms.conf file setting question

erictodor — Mon, 08 Jan 2018 22:47:31 GMT

This is from the search head.

Re: REGEX filter in transforms.conf file setting question

erictodor — Tue, 29 Sep 2020 17:35:12 GMT

This is the complete props.conf

[set_sourcetype_to_stash]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::stash

[extract_spent]
REGEX = (?P\d+)ms$

[eliminate-eventcodes]
REGEX = EventCode=(0000)
DEST_KEY = queue
FORMAT = nullQueue

[wstn_eventlogs]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = windowsworkstationsmain

Re: REGEX filter in transforms.conf file setting question

richgalloway — Tue, 09 Jan 2018 12:54:28 GMT

The REGEX string "." matches any single character, which is probably not what you want.
If you share some sample data and let us know what you want extracted from it we can probably help create a working regex.

Re: REGEX filter in transforms.conf file setting question

erictodor — Tue, 09 Jan 2018 15:11:04 GMT

Basically, we have computer names that are whole words without any non-characters in the name (i.e. Computer01) and we have names with non-characters in them (i.e. Computer-01). It looks like the filter only matches the 1st, names with only characters and numbers. I need to capture all names, including the ones with '-' in them.

Any help would be appreciated.

Re: REGEX filter in transforms.conf file setting question

micahkemp — Tue, 09 Jan 2018 15:23:00 GMT

The issue is we still haven't seen the regex that is matching computer names. None of the transforms you posted above show that extraction.

Can you show the props.conf configuration for the sourcetype of this data that is being improperly extracted?

Re: REGEX filter in transforms.conf file setting question

erictodor — Tue, 29 Sep 2020 17:35:35 GMT

This is what I have. I'm assuming the text below represents the computer name filter.

--Eric

[WinEventLog:Security]
TRANSFORMS-t1 = eliminate-eventcodes

[CSC_FileMon]
DATETIME_CONFIG = CURRENT
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1

[host::*]
TRANSFORMS-main = wstn_eventlogs

Re: REGEX filter in transforms.conf file setting question

micahkemp — Tue, 09 Jan 2018 15:50:51 GMT

When you search that data in Splunk, what does it show is the sourcetype for the events with the mis-parsed computer name? If it's CSC_FileMon then the above shows the props for it, but there is no extraction defined, which means Splunk may be grabbing the computer name via KV_MODE = auto (which looks for key=value in the events).

Please include some sample events that have computer name, both where it is properly extracted and where it is not.

Re: REGEX filter in transforms.conf file setting question

erictodor — Tue, 09 Jan 2018 16:03:22 GMT

I'm sorry, the limits of my Splunk knowledge is being hit by your question. I don't know how to query Splunk to see the output of the data stream to the 3rd party. I'm looking at 3rd party tool itself and looking at the data coming in. By looking at that data, I can see the computer names that are being matched. The pattern I've discerned is what I've posted before about the computer names. Those with letters and numbers are fine. Anything with something else in it doesn't reach the 3rd party.

Is there a way I can write a query to see the output to compare and provide the data you're looking for ?

(and thank you for your patience with me on this question)

Re: REGEX filter in transforms.conf file setting question

micahkemp — Tue, 09 Jan 2018 16:05:40 GMT

So the question here is why is a 3rd party tool not parsing the logs correctly?

Your config above looks correct to forward logs, and doesn't do any other parsing or formatting. If a 3rd party tool isn't parsing as you'd expect that sounds like an issue with that other tool.

Re: REGEX filter in transforms.conf file setting question

erictodor — Tue, 09 Jan 2018 16:13:51 GMT

That's a possibility. When I'm looking at the 3rd party feed, I'm assuming I'm looking at the feed before it's parsed and when I look at the raw data, I don't see all computer names. Is there a way to mimic this props.conf filter in a Splunk query so I can see what is being captured?

I guess the question is, IS Splunk sending entries from all hosts or is it sending a sub-set of hosts? I can't verify this from the Splunk side (yet?). If I can verify that it is, I can then focus on the 3rd party for not parsing the entries correctly but right now I don't know if the data out is good.

I hope that makes sense.

Re: REGEX filter in transforms.conf file setting question

micahkemp — Tue, 09 Jan 2018 16:17:42 GMT

Splunk's concept of host is the machine that created or sent the logs. Your configuration tells Splunk to forward all of those hosts' logs to this other tool.

The other concept of host you appear to be using is the computer name referenced in the log, and there is no pattern matching against that based on what you have posted from your configs. The REGEX = . line means "match everything, seriously."

Re: REGEX filter in transforms.conf file setting question

erictodor — Tue, 09 Jan 2018 16:27:21 GMT

Thank you for the distinction in definitions. Helps with explaining the issue more. This configuration is what we want Splunk to do. To forward all logs to our 3rd party. In these logs are computer names (my definition of host I was using). While the REGEX = . means match everything, is there any way to double verify that everything, is indeed, being forwarded?