topic Re: Coalesce and CIM Compliant Fields in Getting Data In

Coalesce and CIM Compliant Fields

BP9906 — Tue, 29 Sep 2020 07:22:29 GMT

Hello,
From a reporting perspective, I have apache logs in a company standard format. Due to load balancing configuration, we have 3 possible fields where the source ip is noted.

These fields are as follows:
clientip (standard source IP field)
X_FORWARDED_IP (x-forwarded-for http header)
ns_client_ip (load balancer's view of source ip)

Since all 3 fields exist in sourcetype=access_combined (apache) logs, how do I coalesce the fields to "src" to make it CIM compliant?

I will mention that Apache logs a hyphen "-" for null field values for the above too.

Thanks for your help.

Re: Coalesce and CIM Compliant Fields

domenico_perre — Fri, 25 Sep 2015 21:19:21 GMT

Is there a time where all fields will be the same?

Re: Coalesce and CIM Compliant Fields

BP9906 — Tue, 29 Sep 2020 07:22:34 GMT

The X_FORWARDED_IP commonly matches clientip or SOURCE_IP, but depending on the load balancer configuration either clientip is "-" or SOURCE_IP is "-", they never match.

Re: Coalesce and CIM Compliant Fields

domenico_perre — Fri, 25 Sep 2015 21:33:03 GMT

Regex is your friend .

Create a search time field extraction for the following

\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}

This will grab the IP address. click extract new fields and then I prefer to write My own regex.

Re: Coalesce and CIM Compliant Fields

domenico_perre — Fri, 25 Sep 2015 21:52:16 GMT

Looks like the regex is being changed when I post, here is a working one

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Try this regex as a search time extraction.

Re: Coalesce and CIM Compliant Fields

BP9906 — Tue, 29 Sep 2020 07:22:39 GMT

The Regex is fine for IP addresses, but there's no if logic to assess which IP to use since its common to have both clientip and X_FORWARDED_IP present.

Re: Coalesce and CIM Compliant Fields

domenico_perre — Sat, 26 Sep 2015 11:00:41 GMT

Can you put an example which may make it easier to understand 🙂

Re: Coalesce and CIM Compliant Fields

woodcock — Mon, 28 Sep 2015 07:03:57 GMT

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

Re: Coalesce and CIM Compliant Fields

BP9906 — Tue, 29 Sep 2015 15:56:45 GMT

Thanks! Does this go on both indexer and search head?

Re: Coalesce and CIM Compliant Fields

maraman_splunk — Wed, 19 Jul 2017 15:04:21 GMT

this is a search time settings so will have no effect on a indexer (but should be in a TA which will be deployed on both SH and IDX)