Splunk Search

extract field from the source's file path and make it the host field

Contributor

Hi,

I am trying to extract the field from the log file path which includes the actual host. currently, the host field is populated with the third segment of the log file path that is the clientid field. But what we want is the actual host name . we are currently indexing from a shared mount which is the reason not able to capture the actual host name

Example source field with log file name

/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log. 

current host field - DE123456

expected host field - SC9VEABE1092

can someone guide me, how to achieve this?

Thanks

0 Karma

Builder

on your indexer you can try something like this. these changes require a restart of the indexer

#props.conf
[source::/emp_logs-sc9/loaner/DE123456/*]
TRANSFORMS-hostFromSource=hostFromSource


#transforms.conf
[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=.*\/.*?\..*?\.(\w+)
FORMAT = $1
DEST_KEY= MetaData:Host

The regex really just needs to be something that has a capturing group of what you want the hostname to be when ran against the source. There may be a better regular expression depending on what the rest of your logs source paths look like.

0 Karma

Contributor

tried this.. still doesn't work

[hostFromSource]
SOURCEKEY = MetaData:Source
REGEX=EmpServer.[A-Za-z0-9]*.([^.]+)
FORMAT = host::$1
DEST
KEY= MetaData:Host

0 Karma

Builder

and you put this on the indexers where the data is being collected?

Can you also try hardcoding in a specific current source in your props.conf to make sure we its not the wildcards/ ... thats throwing it off.
[source::/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2016-04-30-X.log]

0 Karma

Contributor

HI Cramasta,

Sorry was out for while, But I did try this, still not working.

0 Karma

Contributor

I tried this out, but did not work in props and transforms settings with regex. It works with rex in search though.

Props.conf:

[source::/emp_logs/.../.log]
TRANSFORMS-hostFromSource=hostFromSource

transforms.conf :

[hostFromSource]
SOURCEKEY = MetaData:Source
REGEX=EmpServer.\w+.(\w+)
FORMAT = $1
DEST
KEY= MetaData:Host

and here is the rex search which worked in over-riding the host field

index="emp_logs" | rex field=source "EmpServer.[A-Za-z0-9]*.(?P[^.]+)"

0 Karma

Builder

Did you put this setting on your indexers?

Does using the regex that you are putting in your props.conf work in search with rex?

0 Karma

Builder

Try this, didnt think its needed buy maybe it is

Props.conf:

[source::/emp_logs*/.../*.log]
TRANSFORMS-hostFromSource=hostFromSource

transforms.conf :

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.\w+.(\w+)
FORMAT = host::$1
DEST_KEY= MetaData:Host
0 Karma

Contributor

I tried this.. did not work.
transforms.conf

[hostFromSource]
SOURCEKEY = MetaData:Source
REGEX=EmpServer.[A-Za-z0-9]*.([^.]+)
FORMAT = $1
DEST
KEY= MetaData:Host

Let me try the updated one

0 Karma

Builder

Ok try it with the last update I provided which adds
FORMAT = host::$1

0 Karma

Contributor

Thanks cramasta,

all my log files are of the same format. the clientid and hostname in the source changes.

/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log.

DE123456 is actually the clientid

but in the fourth segment "EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log" the hostname is SC9VEABE1092. This is what we want to replace the host field with.

0 Karma

Builder

did this end up working? if so please mark this as accepted. thanks

0 Karma

Builder

so it seem like you get the basic idea here, i just captured the wrong part of the group. All you have to do is change the regex to capture that group. Also updated my original answer

.*\/.*?\..*?\.(\w+)

0 Karma

Builder

or something like this for your regex. theres more than one way to write the regex depending on what the sources look like.
EmpServer\.\w+\.(\w+)

0 Karma

Builder

one day Ill eventually figure out how to get formatting to work on this site.

0 Karma