How to overwrite the host field value with dvc field value ?

[Hemnaath]

Hi All, I have a request from the client to overwrite the host field value with the dvc field value from the interesting field in splunk.

example :

index = firewall host=test01 sourcetype=opsec | table host dvc_host

We could see that host = test01 and dvc_host = test02.xxxx.com

Actual requirement:
We want overwrite the "host" field for logs to use to value for the "dvc" field = test02.xxx.com instead of test01 and also wanted to remove ".xxxx.com " so that "test02.xxxxx.com" is written to the "host" field as "test02".

Kindly guide me how to overwrite host value = test01 with dvc filed=test02.

[mayurr98 · Accepted solution]

hey try this on the indexer

props.conf (indexer)

 [<sourcetype>]
 TRANSFORMS-host_override = host_override

transforms.conf (indexer)

[host_override]
REGEX = orig=([^\.|\|]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[Hemnaath]

Hi Mayurr, thanks for your effort on this, I have not pushed this change yet, since we have three different source types configured to fetch data. In this case can I configure the props.conf and transforms.conf like this.

Props.conf
[opsec]
TRANSFORMS-host_override1 = host_override1

[opsec:vpn]
TRANSFORMS-host_override2 = host_override2

[opsec:audit]
TRANSFORMS-host_override3 = host_override3

Transforms.conf
[host_override1]
REGEX = orig=(?[^(.||)]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[host_override2]
REGEX = orig=(?[^(.||)]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[host_override3]
REGEX = orig=(?[^(.||)]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

Kindly guide me on this.
thanks in advance

[mayurr98]

Try doing for one sourcetype.If it is working for you then do it for all.

[Hemnaath]

Hi mayurr, we had pushed above changes but it did not work, so kindly guide me how /where to check Error/Info for this sourcetype and also help me to fix this issue.

thanks in advance.

[Hemnaath]

Hi Mayurr, Can you please guide me on this on how to overwrite host value = test01 with dvc filed=test02.

[FrankVl]

Not sure if it is the reason for things not working, but that fist | character should be removed from your regex and the . should be escaped I think. Regex should be: orig=([^\.|]+) to read the hostname after the orig= key until the first . or |.

[Hemnaath]

Hi Frank, thanks for your effort on this, i am not too good in Regex started learning now. Mayuur provided me a query with a Regex and it worked when we executed the query from search head but when the same was used in the props.conf and transforms.conf in Heavy forwarder instance it did not work.

Query Details:

index=firewall sourcetype="opsec:vpn" "orig" | rex field=_raw "orig=(?<host>[^(\.|\|)]+)" | table host dvc

So can I push the below change with the regex you had posted in the comments.
Props.conf
[opsec:vpn]
TRANSFORMS-host_override = host_override

Transforms.conf
[host_override1]
REGEX = orig=([^.|]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

Kindly guide me on this.

[Hemnaath]

Hi Frank, Today I had pushed the above configuration to all the HF instances where splunk first sees the data, but it did not work, not sure what is going wrong in the configuration, its not over ridding the host value with the dvc field value.

index=firewall sourcetype="opsec:vpn" "orig" | rex field=_raw "orig=(?[^.|]+)" | table host dvc

But when the same regex is placed in the search query its working perfectly, I could see that host value is getting over written of the dvc field value.

So could please guide me how to sort this out, as it is pending for very long time.

[mayurr98]

try doing this at search time and see if it is working or not:
go to Fields » Field extractions » Add new
put Name,sourcetype and regex

orig=(?<host>[^(\.|\|)]+)

[Hemnaath]

Hi Mayurr,
I had test the below stanza along with the regex in my test machine and it was working fine, could see host over ride was happening.

I had also found why it was not working in Prod. In Prod the Field Extraction for these sourcetype where missing completely [opsec:vpn] [opsec:smartdefense] and there was another set of stanza configured in search head instances and the stanza mentioned in props for these sourcetype was different. So in test Environment, I had copy/pasted the stanza that are relevant to these sourcetype from search head along with the below Props and transforms.conf.

Props.conf: 
[opsec:vpn]
TRANSFORMS-host_override = host_override 

Transforms.conf:
 [host_override]
 REGEX = orig=(?<host>[^(\.|\|)]+)
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

[FrankVl]

You're missing some "" around the regex 🙂

Also: perhaps better post this as a new question, if you still need further support with this new topic.

[Hemnaath]

Hi FrankVI thanks for your support, I have created a new post with the regex requirement.

https://answers.splunk.com/answers/618695/need-a-help-in-regex-to-over-ride-the-host-value-w.html?mi...

Kindly guide me on the regex.

[micahkemp]

This config needs to go on the indexers or first heavy forwarders that see the events.

props.conf:

[<sourcetype>]
TRANSFORMS-orig_as_host = orig_as_host

transforms.conf:

[orig_as_host]
REGEX = orig=([^|]+)\|
DEST_KEY = MetaData:Host
FORMAT = host::$1

Note: you must use the name of the sourcetype the event comes in with where I have <sourcetype> above. Specifically you can't change the sourcetype via TRANSFORM and then have that new sourcetypes TRANSFORMS also run.

Also, this is a change for events that come in after the change is made. This will not affect events that have already been indexed.

[Hemnaath]

Hi Micahkemp, Based on the below query I could see there are four sourcetype which are fetching the information from the destination.

index=firewall sourcetype="opsec*"  "orig"  | rex field=_raw "orig=(?<host>[^(\.|\|)]+)"

Sourcetype :
[opsec]
[opsec:smartdefense]
[opsec:vpn]
[opsec:audit]

Props.conf:

[opsec]
TRANSFORMS-orig_as_host = orig_as_host
transforms.conf:
[orig_as_host]
 REGEX = orig=([^|]+)\|
 DEST_KEY = MetaData:Host

Props.conf 
[opsec:vpn]
TRANSFORMS-orig_as_host = orig_as_host

transforms.conf:
[orig_as_host]
 REGEX = orig=([^|]+)\|
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

Kindly guide me whether the above props.conf and transforms are correct.

[micahkemp]

But do the events initially come in as all of those sourcetypes, or are they rewritten at search or index time? My guess (and that's all it is, a guess) is they come in as opsec, so you may only need:

props.conf

[opsec]
TRANSFORMS-orig_as_host

transforms.conf

 [orig_as_host]
 REGEX = orig=([^|]+)\|
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

[Hemnaath]

Hi Micahkemp, After implementing this changes in splunk HF instance where the parsing is happening , the events stopped ingesting into splunk, when tried to search for the output , I got zero result.

Props.conf:
[opsec]
TRANSFORMS-orig_as_host

Transforms.conf:
[orig_as_host]
REGEX = orig=([^|]+)|
DEST_KEY = MetaData:Host
FORMAT = host::$1

index=firwall sourcetype="opsec*"

Zero result.

Similarly before applying the changes, I tried to execute the query with the regex which you had mentioned in your comment and it was throwing an error.

index=firewall  sourcetype="opsec*" | rex field=_raw "orig=([^|]+)\|"  | table host dvc

Error in 'rex' command: The regex 'orig=([^|]+)\|' does not extract anything. It should specify at least one named group. Format: (?<name>...).

The search job has failed due to an error. You may be able view the job in the Job Inspector.

Kindly guide me on this please..

[micahkemp]

I can't see how the changes you implemented could have resulted in missing data.

As for the rex warning, rex needs named capture groups ((?<name>...)), but index-time transforms can't use named capture groups, instead using numbered (which are automatically numbered when using parentheses in the regex).

After making this change, was new data indexed for you to search on?

[Hemnaath]

Hi Micahkemp, thanks for your effort on this. we had just copy pasted the below props.conf and transforms.conf along with other stanza's and pushed to the Heavy Forwarder instances via deployment server and reload the DP instances.

Props.conf:
[opsec]
TRANSFORMS-orig_as_host

Transforms.conf:
[orig_as_host]
REGEX = orig=([^|]+)|
DEST_KEY = MetaData:Host
FORMAT = host::$1

The moment it was pushed to the prod, we could not see any new events being ingested into splunk, we had waited for an hour but no luck, then decided to test the regex in the splunk console and got the above Error.

index=firwall sourcetype="opsec*" | rex field=_raw "orig=([^|]+)|" | table host dvc

But It was working fine when I use the below search string

index=firewall sourcetype="opsec:vpn" "orig" | rex field=_raw "orig=(?[^(.||)]+)"
| table host dvc

So could you please guide me on this, to fix the issue.

thanks in advance.

[Hemnaath]

Hi Micahkemp, could you please guide me on this issue.

thanks in advance.

[Hemnaath]

Hi Micahkemp, as we are unable to over write host" field for logs to use to value for the "dvc" field . Kindly guide me on this issue.