topic Re: How to overwrite the host field value with dvc field value ? in Getting Data In

How to overwrite the host field value with dvc field value ?

Hemnaath — Tue, 30 Jan 2018 17:54:51 GMT

Hi All, I have a request from the client to overwrite the host field value with the dvc field value from the interesting field in splunk.

example :

index = firewall host=test01 sourcetype=opsec | table host dvc_host

We could see that host = test01 and dvc_host = test02.xxxx.com

Actual requirement:
We want overwrite the "host" field for logs to use to value for the "dvc" field = test02.xxx.com instead of test01 and also wanted to remove ".xxxx.com " so that "test02.xxxxx.com" is written to the "host" field as "test02".

Kindly guide me how to overwrite host value = test01 with dvc filed=test02.

Re: How to overwrite the host field value with dvc field value ?

493669 — Tue, 30 Jan 2018 18:02:36 GMT

Hi @Hemnaath,
Try this:

index = firewall host=test01 sourcetype=opsec | table host dvc_host|rex field=dvc_host"(?<host>\w+)"

Try this run anywhere search:

|makeresults|eval  host = "test01", dvc_host = "test02.xxxx.com"|rex field=dvc_host"(?<host>\w+)"

Hope this helps!

Re: How to overwrite the host field value with dvc field value ?

micahkemp — Tue, 30 Jan 2018 21:02:40 GMT

Do you want to change the indexed value of host, or just the value at search time? If the former you would need to use an index-time transform to set the value of MetaData:Host. From the transforms.conf doc:

[hostoverride]
DEST_KEY = MetaData:Host
REGEX = \s(\w*)$
FORMAT = host::$1

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Wed, 31 Jan 2018 12:10:03 GMT

Hi thanks for your effort on this, but it did not fetch the result as expected. After executing the above query I had got the below output.

 index=firwall | rex field=dvc "(?<host>\w+)" | table host dvc 

     host            dvc
      test01        test02
      test02 

      test01         test03 
      test03   

 Event details:

 time=1517397957|loc=10718231|fileid=1517392517|action=decrypt|**orig=test02.xxxx.com**|i/f_dir=inbound|i/f_name=bond0.470|has_accounting=0|uuid=<5a71a7c5,00000005,30f08e0a,c0000000>|product=VPN-1 & FireWall-1|inzone=External|outzone=Internal|rule=250|rule_uid={DAE6-F7DD-4167-BCAC-1DE4B472}|rule_name=DNS|service_id=domain-udp|src=10.x.x.x|s_port=577|dst=dip02.xxxx.com|service=domain-udp|proto=udp|scheme:=IKE|methods:=ESP: AES-128 + SHA1 + PFS (group 2)|peer gateway=VPN_AWS_Gateway_Prod2|community=vpn-xxxxe799|fw_subproduct=VPN-1|vpn_feature_name=VPN|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={0181B41F-6A86-E04C-8E1E-38146FBFD921};**mgmt=test01**;date=1517158905;policy_name=Global-2]|origin_sic_name=CN=test02.xxxx.com,O=dron01.xxxx.com.evv25

And in the selected field host=test01,test02, test03 etc

Note: test01 is the source from where the data is ingested into splunk and other hosts values are found in the events.

Need to over ride the host=test01 when the data is getting indexed itself. So kindly guide me how to fix this issue.

Re: How to overwrite the host field value with dvc field value ?

493669 — Wed, 31 Jan 2018 12:17:38 GMT

can you provide the output of

index=firwall | rex field=dvc_host "(?<host>\w+)" | table host dvc dvc_host

As here regex is applied on dvc_host field

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Wed, 31 Jan 2018 12:25:58 GMT

Hi Micahkemp, thanks for your effort on this, yes i need change the value of the host. currently host=test01 is the source from where the data is being ingested in to splunk and other host details are found in the event data.

With the help of the forum, I had executed the below query to overwrite the "host" field for logs to use to value for the "dvc" field and also to remove ".xxxx.com " and write to the "host" field without xxxx.com.
But it did not work as expected, it included the other host details test02,test03 etc under the host field along with the host=test01.

index=firwall sourcetype="opsec:vpn" | rex field=dvc  "(?<host>[^\.]+)" | table host dvc 

     host            dvc
      test01        test02
      test02 

      test01         test03 
      test03   

Event details:
time=1517397957|loc=10718231|fileid=1517392517|action=decrypt|**orig=test02.xxxx.com**|i/f_dir=inbound|i/f_name=bond0.470|has_accounting=0|uuid=<5a71a7c5,00000005,30f08e0a,c0000000>|product=VPN-1 & FireWall-1|inzone=External|outzone=Internal|rule=250|rule_uid={DAE6-F7DD-4167-BCAC-1DE4B472}|rule_name=DNS|service_id=domain-udp|src=10.x.x.x|s_port=577|dst=dip02.xxxx.com|service=domain-udp|proto=udp|scheme:=IKE|methods:=ESP: AES-128 + SHA1 + PFS (group 2)|peer gateway=VPN_AWS_Gateway_Prod2|community=vpn-xxxxe799|fw_subproduct=VPN-1|vpn_feature_name=VPN|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={0181B41F-6A86-E04C-8E1E-38146FBFD921};**mgmt=test01**;date=1517158905;policy_name=Global-2]|origin_sic_name=CN=test02.xxxx.com,O=dron01.xxxx.com.evv25

Getting the other host details under selected fields, along with the host=test01

Note: test01 is the source from where the data is ingested into splunk and other hosts values are found in the events.

Need to over ride the host=test01 when the data is getting indexed itself and in the host field we should get only the host values orig=test02.xxxx.com from the event data.

Props.conf Details:

[opsec:vpn]
KV_MODE          = none

REPORT-0policy_id_tag_for_opsec     = policy_id_tag_for_opsec,mgmt_for_opsec,
REPORT-action_as_threat_emulation_action = action_as_threat_emulation_action
REPORT-auto_kv_for_opsec            = auto_kv_for_opsec

FIELDALIAS-dvc_for_opsec       = orig as dvc

There are lot many EVAL and FIELDALIAS in the props for this sourcetype.

Transforms.conf:

[opsec_sourcetype_vpn]
REGEX = fw_subproduct\=VPN-1
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::opsec:vpn

[action_as_threat_emulation_action]
REGEX = action\=([^|]+)
FORMAT = te_action::$1

[auto_kv_for_opsec]
REGEX = ([^|=]*)\b:?=([^|]*)
FORMAT = $1::$2

[policy_id_tag_for_opsec]
REGEX  = __policy_id_tag\=([^|]+)
FORMAT = policy_id_tag::$1

Kindly guide me how to overwrite host value = test01 with dvc filed=test02.

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Wed, 31 Jan 2018 15:06:41 GMT

hey regex got applied to the field dvc and i have already pasted the output in my previous comment.

Re: How to overwrite the host field value with dvc field value ?

micahkemp — Thu, 01 Feb 2018 16:33:22 GMT

This config needs to go on the indexers or first heavy forwarders that see the events.

props.conf:

[<sourcetype>]
TRANSFORMS-orig_as_host = orig_as_host

transforms.conf:

[orig_as_host]
REGEX = orig=([^|]+)\|
DEST_KEY = MetaData:Host
FORMAT = host::$1

Note: you must use the name of the sourcetype the event comes in with where I have <sourcetype> above. Specifically you can't change the sourcetype via TRANSFORM and then have that new sourcetypes TRANSFORMS also run.

Also, this is a change for events that come in after the change is made. This will not affect events that have already been indexed.

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Thu, 01 Feb 2018 17:24:27 GMT

Hi Micahkemp, Based on the below query I could see there are four sourcetype which are fetching the information from the destination.

index=firewall sourcetype="opsec*"  "orig"  | rex field=_raw "orig=(?<host>[^(\.|\|)]+)"

Sourcetype :
[opsec]
[opsec:smartdefense]
[opsec:vpn]
[opsec:audit]

Props.conf:

[opsec]
TRANSFORMS-orig_as_host = orig_as_host
transforms.conf:
[orig_as_host]
 REGEX = orig=([^|]+)\|
 DEST_KEY = MetaData:Host

Props.conf 
[opsec:vpn]
TRANSFORMS-orig_as_host = orig_as_host

transforms.conf:
[orig_as_host]
 REGEX = orig=([^|]+)\|
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

Kindly guide me whether the above props.conf and transforms are correct.

Re: How to overwrite the host field value with dvc field value ?

micahkemp — Thu, 01 Feb 2018 17:38:13 GMT

But do the events initially come in as all of those sourcetypes, or are they rewritten at search or index time? My guess (and that's all it is, a guess) is they come in as opsec, so you may only need:

props.conf

[opsec]
TRANSFORMS-orig_as_host

transforms.conf

 [orig_as_host]
 REGEX = orig=([^|]+)\|
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Tue, 29 Sep 2020 17:56:54 GMT

Hi Micahkemp, After implementing this changes in splunk HF instance where the parsing is happening , the events stopped ingesting into splunk, when tried to search for the output , I got zero result.

Props.conf:
[opsec]
TRANSFORMS-orig_as_host

Transforms.conf:
[orig_as_host]
REGEX = orig=([^|]+)|
DEST_KEY = MetaData:Host
FORMAT = host::$1

index=firwall sourcetype="opsec*"

Zero result.

Similarly before applying the changes, I tried to execute the query with the regex which you had mentioned in your comment and it was throwing an error.

index=firewall  sourcetype="opsec*" | rex field=_raw "orig=([^|]+)\|"  | table host dvc

Error in 'rex' command: The regex 'orig=([^|]+)\|' does not extract anything. It should specify at least one named group. Format: (?<name>...).

The search job has failed due to an error. You may be able view the job in the Job Inspector.

Kindly guide me on this please..

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Fri, 02 Feb 2018 14:55:53 GMT

Hi Micahkemp, could you please guide me on this issue.

thanks in advance.

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Fri, 02 Feb 2018 16:39:19 GMT

Hi Micahkemp, as we are unable to over write host" field for logs to use to value for the "dvc" field . Kindly guide me on this issue.

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Fri, 02 Feb 2018 18:00:13 GMT

hi micahkemp, could you please guide me on this .

Re: How to overwrite the host field value with dvc field value ?

micahkemp — Fri, 02 Feb 2018 18:28:28 GMT

This is still on my list, but I'm working several other things right now as well.

Re: How to overwrite the host field value with dvc field value ?

mayurr98 — Fri, 02 Feb 2018 18:44:51 GMT

hey try this on the indexer

props.conf (indexer)

 [<sourcetype>]
 TRANSFORMS-host_override = host_override

transforms.conf (indexer)

[host_override]
REGEX = orig=([^\.|\|]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Mon, 05 Feb 2018 12:00:29 GMT

Hi Micahkemp, could you please guide me to fix this issue. I am still unable to fix this issue.

thanks in advance.

Re: How to overwrite the host field value with dvc field value ?

Hemnaath — Tue, 29 Sep 2020 17:57:20 GMT

Hi Mayurr, thanks for your effort on this, I have not pushed this change yet, since we have three different source types configured to fetch data. In this case can I configure the props.conf and transforms.conf like this.

Props.conf
[opsec]
TRANSFORMS-host_override1 = host_override1

[opsec:vpn]
TRANSFORMS-host_override2 = host_override2

[opsec:audit]
TRANSFORMS-host_override3 = host_override3

Transforms.conf
[host_override1]
REGEX = orig=(?[^(.||)]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[host_override2]
REGEX = orig=(?[^(.||)]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[host_override3]
REGEX = orig=(?[^(.||)]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

Kindly guide me on this.
thanks in advance

Re: How to overwrite the host field value with dvc field value ?

mayurr98 — Mon, 05 Feb 2018 15:28:12 GMT

Try doing for one sourcetype.If it is working for you then do it for all.

Re: How to overwrite the host field value with dvc field value ?

micahkemp — Mon, 05 Feb 2018 16:46:45 GMT

I can't see how the changes you implemented could have resulted in missing data.

As for the rex warning, rex needs named capture groups ((?<name>...)), but index-time transforms can't use named capture groups, instead using numbered (which are automatically numbered when using parentheses in the regex).

After making this change, was new data indexed for you to search on?