topic Re: rex field command in props.conf file in Splunk Dev

rex field command in props.conf file

pgabo66 — Tue, 16 Apr 2024 17:33:32 GMT

The event.url field stores all the urls found in the logs, I want to create a new field called url_domain that only captures the domain of the urls stored in event.url, temporarily what I do is from the search write the following:

| rex field=event.url "^(?:https?:\/\/)?(?:www[0-9]*\.)?(?)(?<url_domain>[^\n:\/]+)"

What should I add in the props.conf so that this instruction is fixed for the sourcetype "sec-web"?

Re: rex field command in props.conf file

bowesmana — Mon, 15 Apr 2024 22:52:28 GMT

You can do this in the UI - go to Settings->Fields-Field Transformations and add the regex and the field you want to extract from and then in Field Extractions add a new Extraction using transforms and reference the Field Transformation.

This will translate to something like this in props/transforms conf files

In transforms.conf you will need

[url_domain] CLEAN_KEYS = 0 REGEX = ^(?:https?:\/\/)?(?:www[0-9]*\.)?(?)(?<url_domain>[^\n:\/]+) SOURCE_KEY = event.url

In props.conf

[sec-web] REPORT-file_name = url_domain

Re: rex field command in props.conf file

pgabo66 — Mon, 15 Apr 2024 22:58:55 GMT

So with the "SOURCE_KEY = event.url" what I do is call the field where I want to get the information from?
In my case it would be the urls stored there.

Re: rex field command in props.conf file

bowesmana — Mon, 15 Apr 2024 23:03:03 GMT

In your rex example you said

| rex field=event.url ...

that is why SOURCE_KEY is event.url - as that is where the urls are coming from right?

Your rex example indicated you are extracting the url into a field called url_domain, which is also what is in the transforms.

Re: rex field command in props.conf file

gcusello — Tue, 16 Apr 2024 05:24:25 GMT

Hi @pgabo66 ,

you have to create a new field associating it to your sourcetype and using this rule:

^(?:https?:\/\/)?(?:www[0-9]*\.)?(?)(?<url_domain>[^\n:\/]+) in event.url

in the field extraction.

Ciao.

Giuseppe

Re: rex field command in props.conf file

pgabo66 — Tue, 16 Apr 2024 16:28:57 GMT

In the props.conf example, when it says "REPORT-file_name = url_domain", what should I replace file_name with?

I'll stay tuned, thank you very much.

Re: rex field command in props.conf file

pgabo66 — Tue, 16 Apr 2024 22:53:51 GMT

I did what you explained to me but it still doesn't work, when I check the zscaler logs apun the url_domain field does not appear.

It is important to mention that I am implementing this from a custom app for zsacaler.

Re: rex field command in props.conf file

pgabo66 — Tue, 16 Apr 2024 22:55:45 GMT

Do you have an example of how the props.conf would look like with that rule?

I've tried several sentences but it still doesn't take it.

Re: rex field command in props.conf file

bowesmana — Wed, 17 Apr 2024 00:08:37 GMT

REPORT-url_domain

It's the name of the field you want to assign the result to.

Re: rex field command in props.conf file

isoutamo — Wed, 17 Apr 2024 08:52:11 GMT

here is order how those are managed in search time https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Searchtimeoperationssequence

You should ensure that this field has defined before you can use those e.g. in transforms.conf. E.g. if you are using ALIAS-field1 on props.conf you cannot use that field1 as a SOURCE_KEY in props.conf. In this kind of situation you should extract that information from _raw instead of field which has defined on later phase of input sequence.

I'm not sure about your event.url field is same as this TA has defined or not. If it's then you can see in props.conf that it has defined like

EVAL-url = Host+URL

and if this is your event.url field then it didn't exists yet when you try to use it on transforms.conf.

r. Ismo

Re: rex field command in props.conf file

bowesmana — Wed, 17 Apr 2024 22:40:18 GMT

It is probably because your field looks like it has come from JSON and based on the link provided by @isoutamo , means that the field extractions are happening at stage 4, whereas your REPORT extraction is happening at stage 3, therefore the field does not exist.

You could try creating a calculated field using an eval replace expression to remove the non-domain part.

You can try this in standard SPL by experimenting with your regex using

| eval domain=replace('event.url', "(?:https?:\/\/)?(?:www[0-9]*\.)?(?)([^\n:\/]+)", "\1")

That is NOT correct above, as I am not sure what the replacement token \1 should be with all the brackets and capturing/non-capturing groups, but you can experiment with regex101.com

Re: rex field command in props.conf file

pgabo66 — Fri, 26 Apr 2024 20:45:23 GMT

Hi @isoutamo and @bowesmana,

I have tried the ways shared by you but it still doesn't work it's like Splunk doesn't read the transforms.conf I checked the logs of the index=_internal but I don't see any errors related to it.