Getting Data In

Coalesce and CIM Compliant Fields

BP9906
Builder

Hello,
From a reporting perspective, I have apache logs in a company standard format. Due to load balancing configuration, we have 3 possible fields where the source ip is noted.

These fields are as follows:
clientip (standard source IP field)
X_FORWARDED_IP (x-forwarded-for http header)
ns_client_ip (load balancer's view of source ip)

Since all 3 fields exist in sourcetype=access_combined (apache) logs, how do I coalesce the fields to "src" to make it CIM compliant?

I will mention that Apache logs a hyphen "-" for null field values for the above too.

Thanks for your help.

0 Karma
1 Solution

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

View solution in original post

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

View solution in original post

BP9906
Builder

Thanks! Does this go on both indexer and search head?

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

this is a search time settings so will have no effect on a indexer (but should be in a TA which will be deployed on both SH and IDX)

0 Karma

domenico_perre
Path Finder

Looks like the regex is being changed when I post, here is a working one

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Try this regex as a search time extraction.

0 Karma

BP9906
Builder

The Regex is fine for IP addresses, but there's no if logic to assess which IP to use since its common to have both clientip and X_FORWARDED_IP present.

0 Karma

domenico_perre
Path Finder

Can you put an example which may make it easier to understand 🙂

0 Karma

domenico_perre
Path Finder

Is there a time where all fields will be the same?

0 Karma

BP9906
Builder

The X_FORWARDED_IP commonly matches clientip or SOURCE_IP, but depending on the load balancer configuration either clientip is "-" or SOURCE_IP is "-", they never match.

0 Karma

domenico_perre
Path Finder

Regex is your friend .

Create a search time field extraction for the following

\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}

This will grab the IP address. click extract new fields and then I prefer to write My own regex.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!