Getting Data In
Highlighted

Coalesce and CIM Compliant Fields

Builder

Hello,
From a reporting perspective, I have apache logs in a company standard format. Due to load balancing configuration, we have 3 possible fields where the source ip is noted.

These fields are as follows:
clientip (standard source IP field)
XFORWARDEDIP (x-forwarded-for http header)
nsclientip (load balancer's view of source ip)

Since all 3 fields exist in sourcetype=access_combined (apache) logs, how do I coalesce the fields to "src" to make it CIM compliant?

I will mention that Apache logs a hyphen "-" for null field values for the above too.

Thanks for your help.

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Path Finder

Is there a time where all fields will be the same?

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Builder

The XFORWARDEDIP commonly matches clientip or SOURCEIP, but depending on the load balancer configuration either clientip is "-" or SOURCEIP is "-", they never match.

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Path Finder

Regex is your friend .

Create a search time field extraction for the following

\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}

This will grab the IP address. click extract new fields and then I prefer to write My own regex.

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Path Finder

Looks like the regex is being changed when I post, here is a working one

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Try this regex as a search time extraction.

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Builder

The Regex is fine for IP addresses, but there's no if logic to assess which IP to use since its common to have both clientip and XFORWARDEDIP present.

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Path Finder

Can you put an example which may make it easier to understand 🙂

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

View solution in original post

Highlighted

Re: Coalesce and CIM Compliant Fields

Builder

Thanks! Does this go on both indexer and search head?

0 Karma
Highlighted

Re: Coalesce and CIM Compliant Fields

Splunk Employee
Splunk Employee

this is a search time settings so will have no effect on a indexer (but should be in a TA which will be deployed on both SH and IDX)

0 Karma