topic Re: extract field from the source's file path and make it the host field in Splunk Search

extract field from the source's file path and make it the host field

nmohammed — Fri, 22 Apr 2016 18:59:37 GMT

Hi,

I am trying to extract the field from the log file path which includes the actual host. currently, the host field is populated with the third segment of the log file path that is the clientid field. But what we want is the actual host name . we are currently indexing from a shared mount which is the reason not able to capture the actual host name

Example source field with log file name

/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log.

current host field - DE123456

expected host field - SC9VEABE1092

can someone guide me, how to achieve this?

Thanks

Re: extract field from the source's file path and make it the host field

cramasta — Fri, 22 Apr 2016 19:18:22 GMT

on your indexer you can try something like this. these changes require a restart of the indexer

#props.conf
[source::/emp_logs-sc9/loaner/DE123456/*]
TRANSFORMS-hostFromSource=hostFromSource


#transforms.conf
[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=.*\/.*?\..*?\.(\w+)
FORMAT = $1
DEST_KEY= MetaData:Host

The regex really just needs to be something that has a capturing group of what you want the hostname to be when ran against the source. There may be a better regular expression depending on what the rest of your logs source paths look like.

Re: extract field from the source's file path and make it the host field

cramasta — Fri, 22 Apr 2016 19:21:37 GMT

one day Ill eventually figure out how to get formatting to work on this site.

Re: extract field from the source's file path and make it the host field

nmohammed — Fri, 22 Apr 2016 19:46:03 GMT

Thanks cramasta,

all my log files are of the same format. the clientid and hostname in the source changes.

/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log.

DE123456 is actually the clientid

but in the fourth segment "EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log" the hostname is SC9VEABE1092. This is what we want to replace the host field with.

Re: extract field from the source's file path and make it the host field

cramasta — Fri, 22 Apr 2016 19:50:48 GMT

so it seem like you get the basic idea here, i just captured the wrong part of the group. All you have to do is change the regex to capture that group. Also updated my original answer

.*\/.*?\..*?\.(\w+)

Re: extract field from the source's file path and make it the host field

cramasta — Fri, 22 Apr 2016 19:54:12 GMT

or something like this for your regex. theres more than one way to write the regex depending on what the sources look like.
EmpServer\.\w+\.(\w+)

Re: extract field from the source's file path and make it the host field

cramasta — Tue, 26 Apr 2016 03:27:31 GMT

did this end up working? if so please mark this as accepted. thanks

Re: extract field from the source's file path and make it the host field

nmohammed — Tue, 29 Sep 2020 09:34:22 GMT

I tried this out, but did not work in props and transforms settings with regex. It works with rex in search though.

Props.conf:

[source::/emp_logs*/.../*.log]
TRANSFORMS-hostFromSource=hostFromSource

transforms.conf :

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.\w+.(\w+)
FORMAT = $1
DEST_KEY= MetaData:Host

and here is the rex search which worked in over-riding the host field

index="emp_logs" | rex field=source "EmpServer.[A-Za-z0-9]*.(?P[^.]+)"

Re: extract field from the source's file path and make it the host field

cramasta — Thu, 28 Apr 2016 16:00:18 GMT

Did you put this setting on your indexers?

Does using the regex that you are putting in your props.conf work in search with rex?

Re: extract field from the source's file path and make it the host field

cramasta — Thu, 28 Apr 2016 16:08:50 GMT

Try this, didnt think its needed buy maybe it is

Props.conf:

[source::/emp_logs*/.../*.log]
TRANSFORMS-hostFromSource=hostFromSource

transforms.conf :

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.\w+.(\w+)
FORMAT = host::$1
DEST_KEY= MetaData:Host

Re: extract field from the source's file path and make it the host field

nmohammed — Tue, 29 Sep 2020 09:34:27 GMT

I tried this.. did not work.
transforms.conf

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.[A-Za-z0-9]*.([^.]+)
FORMAT = $1
DEST_KEY= MetaData:Host

Let me try the updated one

Re: extract field from the source's file path and make it the host field

cramasta — Thu, 28 Apr 2016 17:01:40 GMT

Ok try it with the last update I provided which adds
FORMAT = host::$1

Re: extract field from the source's file path and make it the host field

nmohammed — Tue, 29 Sep 2020 09:34:40 GMT

tried this.. still doesn't work

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.[A-Za-z0-9]*.([^.]+)
FORMAT = host::$1
DEST_KEY= MetaData:Host

Re: extract field from the source's file path and make it the host field

cramasta — Thu, 28 Apr 2016 18:41:31 GMT

and you put this on the indexers where the data is being collected?

Can you also try hardcoding in a specific current source in your props.conf to make sure we its not the wildcards/ ... thats throwing it off.
[source::/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2016-04-30-X.log]

Re: extract field from the source's file path and make it the host field

nmohammed — Tue, 10 May 2016 18:16:04 GMT

HI Cramasta,

Sorry was out for while, But I did try this, still not working.