How can I get regex to over ride the host value with another field value from an interesting field?

[Hemnaath]

Hi All, Need a help in regex for doing the host over ride with dvc_host field value from the interesting fields for a firewall based on the sourcetype.

1) For the first source type paloalto:network:traffic, I am getting the below events details

index=firewall sourcetype="paloalto:network:traffic" 

Event details:

Feb 15 10:21:12 test01pano.xxxxxx.com 1,2018/02/15 10:21:12,012501001041,TRAFFIC,end,1,2018/02/15 10:21:11,10.x.x.x,10.x.x.x,0.0.0.0,0.0.0.0,Foundation Services,,,dns,vsys2,Data-Center-Admin,Data-Center-Core,ae5.2005,ae5.250,pan_log_forward,2018/02/15 10:21:11,667736,1,50811,53,0,0,0x19,udp,allow,xxx,xx,1xx,2,2018/02/15 10:20:42,0,any,0,6477528057945424450,0x8000000000000000,10.0.0.0-10.x.x.x,10.0.0.0-10.x.x.x,0,1,1,aged-out,17,0,0,0,Data_Center,east01fw,from-policy,,,0,,0,,N/A

Requirement:
We want to overwrite the "host" field for firewall logs to use to value for the "dvc_host" field

host=test01pano.xxxxxx.com should be replaced with the dvc_host value=deast01fw

I had tried this query but it is throwing Error:

index=firewall  sourcetype="paloalto:network:traffic" | rex field=_raw (?<host>\b[^(\.),]+\b(,)+\b(?=from-policy)\b) | table host dvc_host

Error in 'SearchParser': Missing a search command before '^'. Error at position '89' of search query 'search index=firewall sourcetype="paloalto:networ...{snipped} {errorcontext = ?<host>\b[^(\.),]+\b(}'.

But the regex is working fine when tested in the regex101.com , not sure where is the problem.

2) For the second sourcetype paloalto:network:sys, I am getting the below events:

Event Details:

Feb 15 10:45:14 test01pano.xxxxx.com 1,2018/02/15 10:45:14,000702503748,SYSTEM,general,0,2018/02/15 10:45:14,,general,,0,0,general,informational,"Connection to Update server: updates.paloaltonetworks.com completed successfully, initiated by 10.x.x.x",139387,0x0,0,0,0,0,,east01pano

Requirement:
We want overwrite the "host" field for logs to use to value for the "dvc_host" field.

host=test01pano.xxxxxx.com should be replaced with the dvc_host value=east01pano

Kindly guide me in creating the a regex which can over ride the host value to the value of dvc_host field.

[Hemnaath]

Hi Frank, Yes Splunk see the data first in the HF servers, then the data are getting indexed in the indexer instances.

Environment details:

We have 5 heavy forwarder instances, 5 indexer instances, 3 search head cluster members, Deployment server and Deployer server. All the UF forwards the data to the HF and it get parsed and ingested into indexer instances. All the traffic is load balanced by F5 device before entering the splunk instances.

We are using Splunk 6.6.1 version both in prod/test.

Hey I had copy/pasted complete Paloalto-TA-app details in to test environment and also copy pasted some events from prod to test by going to splunk-->settings-->Add data-->uploaded. And data's where mapped as per the sourcetype configured in the prod environment.

I had validated by executing the query without adding the host_over ride stanza first and it did not over ride the host value with the dvc_host field.Then again included the host_over ride stanza and restarted the splunk service and added new set of events data and i could see the host value was getting over ride.

Just now I have checked complete TA details in both HF /Search head instances all have same set of configuration.

Not sure where is the problem now, could you please guide me on this.

[FrankVl]

Are the transforms stanzas named uniquely? Because it looks like you used the same as what you had for your checkpoint opsec host override transforms?

[Hemnaath]

Hi Frank, I had used same set of name for Paloalto app, I mean I had used same stanza which was used in opsec, but mapped to the source type belong to Paloalto Firewall and also changed the regex as per the app.

The same set of props.conf/transforms.conf was used in the test and it worked.

Props.conf:
 [paloalto:network:traffic]
 TRANSFORMS-host_override = host_override

 [paloalto:network:system]
 TRANSFORMS-host_override = host_override1

 Transforms.conf:

   [host_override]
   REGEX = (?:[^,]*,){52}(?<host>\w+)
   DEST_KEY = MetaData:Host
   FORMAT = host::$1

   [host_override1]
   REGEX = (?<host>(\w+)$)
   DEST_KEY = MetaData:Host
   FORMAT = host::$1

Hey in search head do we need to configure different set of props/transforms to perform search time extraction ?

Kindly guide me on this..

[Hemnaath]

Hi Frank, it worked after changing the name of the stanza, but now got in to another problem the regex is not working as expected , host field value is getting over written but not with the dvc_host value instead its taking the value like ping,dns,ssh,not,splunk etc..

Props.conf:
[paloalto:network:traffic]
TRANSFORMS-host_override = hostname_override

Transforms.conf:

[hostname_override]
REGEX = (?:[^,]*,){52}(?<host>\w+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

But the same regex when executed in the search portal getting the required output.

index=firewall sourcetype="paloalto:network:traffic" |rex "(?:[^,]*,){52}(?\w+)" | table host dvc_host

Kindly guide me on this, not sure where is the problem now.

[FrankVl]

Are you getting different results between the transforms and the search time rex command for the exact same events? Or do you again have some events that are not in the expected format, causing the {52} to be incorrect?

It would really help if you do a bit more troubleshooting yourself and come with some more concrete info on what is and what isn't working. It's very hard to do this remotely just based on some info you are sharing here. Also: it is much more beneficial for yourself if you really make sure you understand the config you are using as that also enables you to troubleshoot it better, needing less support and not needing to ask basically the same questions over and over.

[Hemnaath]

Hi Frank, I am extremely sorry about this, yes I can understand the pain of supporting based on some information.

Are you getting different results between the transforms and the search time rex command for the exact same events?

Yes I am getting different output for the same events, between the transforms and the search time rex.

Will it be better to share the complete props/transforms.conf in the comment session.
so that it will give you a better picture how other stanzas are configured.

[Hemnaath]

Hi Frank,

After anchoring the regex based upon the log type (traffic,system,config) instead of end of the line and placing the sourcetype before the renaming of the first set of source type in transforms.conf ,as it won't be able to parse pipeline again.

Props.conf : 
TRANSFORMS-sourcetype = pan_traffic_host, pan_system_host, pan_config_host, pan_client_vpn_host,  pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_endpoint

Transforms.conf:
#hostname_override

[pan_traffic_host]
REGEX= TRAFFIC,(?:[^,]*,){48}(?<host>\w+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[pan_system_host]
REGEX = SYSTEM,(?:[^,]*,){18}(?<host>\w+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[pan_config_host]
REGEX = CONFIG,(?:[^,]*,){17}(?<host>\w+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[pan_client_vpn_host]
REGEX = Client_VPN,(?:[^,]*,){0}(?<host>\w+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

The above stanza where used to over write the host field value with dvc_host from interesting field value.

thanks for your support.

[jacauc]

The above values didn't work for me for some reason,

When I used for example:

SYSTEM,(?:[^,]*,){18}([^,]*)

....it worked - (same REGEX as yours, but with the suffix after the "}" being different)

[FrankVl]

But in your test environment, did you also still have the opsec transforms active? Because those need to be unique and if they are not, that will cause splunk to apply the wrong transform potentially.

So I would suggest that you give the transforms stanzas unique names and then try again.

[Hemnaath]

No, After testing the opsec, I had removed the opsec related configuration from the test environment. And also un-installed the splunk application and re-installed it again.

Let me try a different name for this Paloalto.

[FrankVl]

Just upload it to some image host. Be sure you're not publishing stuff you're not supposed to of course. If you're working with production data you might want to mask/blur some bits first.

Then just post the public url of it in a comment (or use the image button in the messsage editor here to include the image)