Getting Data In

regex in transform.conf to extract hostname after equal to sign

dbashyam
Explorer

Hi, I need help in extracting the hostname after equal to sign in the transform.conf file. The string pattern is like this

cs1=host-name-01-02

I tried the pattern but it seems to not work.
REGEX = .+cs1=(\S+)

could someone help?

Thanks,
Dinesh

0 Karma

joebisesi
Path Finder

Have you tried using the Extract New Fields? If you go into "Extract New Fields" select the event that you want, select Regular Expression, then next. Highlight the text that you want extracted, give it a test name. You can then select 'Show Regular Expression', and it will give you a pretty good idea of what Splunk is looking for as far as Regular Expressions go. I have found this method to work most of the time. Using sites like regex101 is useful, although I have put Regex's that I pulled from the above method, not return the same results in regex101. You might have to tweak the regex that you get from the above method, but usually the tweaking is minimal.

Hope this helps.

0 Karma

FrankVl
Ultra Champion

Not sure if I would really agree with that suggestion. Yes it can give some inspiration, but the regexes generated that way are often overly complex and not the most sensible solution.

It is worth investing some time in learning regex and writing your own, by understanding the data structure, rather than relying on automated tools to generate stuff like this for you.

For example in this case, this is what Splunk comes up with: ^(?:[^=\n]*=){2}(?P<host>[^ ]+), while cs1=(\S+) should do just fine.

0 Karma

joebisesi
Path Finder

I agree that the regexes generated that way are overly complex. Although sometimes, at least what I have found, is the ones that are generated will work in Splunk, and the regexes that are you can create and test in other environments don't work. In addition, by generating the overly complex ones, you can put them in something like regex101 and you can start to understand regex and how to work with the data.

One question for you. Do you have or know of a good site to learn regex? I have not found any that I would recommend. I have learned by taking generated regexes and breaking them down. So I'm sure I have several regexes that are not written in the best way.

0 Karma

FrankVl
Ultra Champion

I also mostly learned by doing and trying to understand regexes created by others / found in splunkbase apps etc. So no, I don't have a site that is good for learning. Although regex101 does also have content (bottom right corner) explaining the various regex concepts, but that is a bit limited in the amount of explanation and examples given.

0 Karma

joebisesi
Path Finder

That's pretty much what I do

0 Karma

FrankVl
Ultra Champion

What does the rest of your props and transforms look like for this (maybe the issue isn't with the regex itself).

For the regex: that .+ is not very useful, try without it.

0 Karma

dbashyam
Explorer

Hi,

the props.conf contains

[dbnetworks:monitor:file]
TRANSFORMS-DBN = DBNsyslog

the transforms.conf contains

[DBNsyslog]
REGEX = .+cs1=(\S+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

0 Karma

FrankVl
Ultra Champion

I don't see anything obviously wrong with that. Perhaps try without the .+.

Also:
did you restart the respective splunk instance after adding this config?
You realize this will only apply to newly ingested events?
What does your splunk architecture look like for this data feed and where did you deploy this config in that architecture?

dbashyam
Explorer

thanks @FrankVl your suggestion of removing the .+ did the trick.

0 Karma

FrankVl
Ultra Champion

That's good to hear! I converted my comment to an answer, so you can mark it as accepted.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@dbashyam

I think cs1 should extract automatically. Can you please share full sample event ? So we can work on right path.

0 Karma

dbashyam
Explorer

Hi, below is the sample.

Nov 28 10:11:06 fwrw dbn: arf:0|data|data|0.0.1|1|repeat_event|7|cs1Label=system identifier cs1=MM63-GH-CG-A-0101 system_identifier=DB63-GH-CG-A-0101 externalId=10043284 cnt=63784 rt=12345 start=123456 destinationServiceName=dummy cn1Label=statement identifier

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...