Coalesce and CIM Compliant Fields

[BP9906]

Hello,
From a reporting perspective, I have apache logs in a company standard format. Due to load balancing configuration, we have 3 possible fields where the source ip is noted.

These fields are as follows:
clientip (standard source IP field)
X_FORWARDED_IP (x-forwarded-for http header)
ns_client_ip (load balancer's view of source ip)

Since all 3 fields exist in sourcetype=access_combined (apache) logs, how do I coalesce the fields to "src" to make it CIM compliant?

I will mention that Apache logs a hyphen "-" for null field values for the above too.

Thanks for your help.

[woodcock · Accepted solution]

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

[BP9906]

Thanks! Does this go on both indexer and search head?

[maraman_splunk]

this is a search time settings so will have no effect on a indexer (but should be in a TA which will be deployed on both SH and IDX)

[domenico_perre]

Looks like the regex is being changed when I post, here is a working one

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Try this regex as a search time extraction.

[BP9906]

The Regex is fine for IP addresses, but there's no if logic to assess which IP to use since its common to have both clientip and X_FORWARDED_IP present.

[domenico_perre]

Can you put an example which may make it easier to understand 🙂

[domenico_perre]

Is there a time where all fields will be the same?

[BP9906]

The X_FORWARDED_IP commonly matches clientip or SOURCE_IP, but depending on the load balancer configuration either clientip is "-" or SOURCE_IP is "-", they never match.

[domenico_perre]

Regex is your friend .

Create a search time field extraction for the following

\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}

This will grab the IP address. click extract new fields and then I prefer to write My own regex.