topic Re: regex in transform.conf to extract hostname after equal to sign in Getting Data In

regex in transform.conf to extract hostname after equal to sign

dbashyam — Wed, 28 Nov 2018 11:40:47 GMT

Hi, I need help in extracting the hostname after equal to sign in the transform.conf file. The string pattern is like this

cs1=host-name-01-02

I tried the pattern but it seems to not work.
REGEX = .+cs1=(\S+)

could someone help?

Thanks,
Dinesh

Re: regex in transform.conf to extract hostname after equal to sign

kamlesh_vaghela — Wed, 28 Nov 2018 12:22:49 GMT

@dbashyam

I think cs1 should extract automatically. Can you please share full sample event ? So we can work on right path.

Re: regex in transform.conf to extract hostname after equal to sign

dbashyam — Tue, 29 Sep 2020 22:08:37 GMT

Hi, below is the sample.

Re: regex in transform.conf to extract hostname after equal to sign

FrankVl — Wed, 28 Nov 2018 13:23:13 GMT

What does the rest of your props and transforms look like for this (maybe the issue isn't with the regex itself).

For the regex: that .+ is not very useful, try without it.

Re: regex in transform.conf to extract hostname after equal to sign

dbashyam — Wed, 28 Nov 2018 13:27:09 GMT

Hi,

the props.conf contains

[dbnetworks:monitor:file]
TRANSFORMS-DBN = DBNsyslog

the transforms.conf contains

[DBNsyslog]
REGEX = .+cs1=(\S+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Re: regex in transform.conf to extract hostname after equal to sign

FrankVl — Wed, 28 Nov 2018 13:31:29 GMT

I don't see anything obviously wrong with that. Perhaps try without the .+.

Also:
did you restart the respective splunk instance after adding this config?
You realize this will only apply to newly ingested events?
What does your splunk architecture look like for this data feed and where did you deploy this config in that architecture?

Re: regex in transform.conf to extract hostname after equal to sign

joebisesi — Wed, 28 Nov 2018 14:29:26 GMT

Have you tried using the Extract New Fields? If you go into "Extract New Fields" select the event that you want, select Regular Expression, then next. Highlight the text that you want extracted, give it a test name. You can then select 'Show Regular Expression', and it will give you a pretty good idea of what Splunk is looking for as far as Regular Expressions go. I have found this method to work most of the time. Using sites like regex101 is useful, although I have put Regex's that I pulled from the above method, not return the same results in regex101. You might have to tweak the regex that you get from the above method, but usually the tweaking is minimal.

Hope this helps.

Re: regex in transform.conf to extract hostname after equal to sign

FrankVl — Wed, 28 Nov 2018 14:41:12 GMT

Not sure if I would really agree with that suggestion. Yes it can give some inspiration, but the regexes generated that way are often overly complex and not the most sensible solution.

It is worth investing some time in learning regex and writing your own, by understanding the data structure, rather than relying on automated tools to generate stuff like this for you.

For example in this case, this is what Splunk comes up with: ^(?:[^=\n]*=){2}(?P<host>[^ ]+), while cs1=(\S+) should do just fine.

Re: regex in transform.conf to extract hostname after equal to sign

joebisesi — Wed, 28 Nov 2018 15:01:52 GMT

I agree that the regexes generated that way are overly complex. Although sometimes, at least what I have found, is the ones that are generated will work in Splunk, and the regexes that are you can create and test in other environments don't work. In addition, by generating the overly complex ones, you can put them in something like regex101 and you can start to understand regex and how to work with the data.

One question for you. Do you have or know of a good site to learn regex? I have not found any that I would recommend. I have learned by taking generated regexes and breaking them down. So I'm sure I have several regexes that are not written in the best way.

Re: regex in transform.conf to extract hostname after equal to sign

FrankVl — Wed, 28 Nov 2018 15:09:10 GMT

I also mostly learned by doing and trying to understand regexes created by others / found in splunkbase apps etc. So no, I don't have a site that is good for learning. Although regex101 does also have content (bottom right corner) explaining the various regex concepts, but that is a bit limited in the amount of explanation and examples given.

Re: regex in transform.conf to extract hostname after equal to sign

joebisesi — Wed, 28 Nov 2018 15:20:09 GMT

That's pretty much what I do

Re: regex in transform.conf to extract hostname after equal to sign

dbashyam — Sun, 02 Dec 2018 12:37:33 GMT

thanks @FrankVl your suggestion of removing the .+ did the trick.

Re: regex in transform.conf to extract hostname after equal to sign

FrankVl — Mon, 03 Dec 2018 08:39:25 GMT

That's good to hear! I converted my comment to an answer, so you can mark it as accepted.