Getting Data In

Coalesce and CIM Compliant Fields

BP9906
Builder

Hello,
From a reporting perspective, I have apache logs in a company standard format. Due to load balancing configuration, we have 3 possible fields where the source ip is noted.

These fields are as follows:
clientip (standard source IP field)
X_FORWARDED_IP (x-forwarded-for http header)
ns_client_ip (load balancer's view of source ip)

Since all 3 fields exist in sourcetype=access_combined (apache) logs, how do I coalesce the fields to "src" to make it CIM compliant?

I will mention that Apache logs a hyphen "-" for null field values for the above too.

Thanks for your help.

0 Karma
1 Solution

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

View solution in original post

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

BP9906
Builder

Thanks! Does this go on both indexer and search head?

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

this is a search time settings so will have no effect on a indexer (but should be in a TA which will be deployed on both SH and IDX)

0 Karma

domenico_perre
Path Finder

Looks like the regex is being changed when I post, here is a working one

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Try this regex as a search time extraction.

0 Karma

BP9906
Builder

The Regex is fine for IP addresses, but there's no if logic to assess which IP to use since its common to have both clientip and X_FORWARDED_IP present.

0 Karma

domenico_perre
Path Finder

Can you put an example which may make it easier to understand 🙂

0 Karma

domenico_perre
Path Finder

Is there a time where all fields will be the same?

0 Karma

BP9906
Builder

The X_FORWARDED_IP commonly matches clientip or SOURCE_IP, but depending on the load balancer configuration either clientip is "-" or SOURCE_IP is "-", they never match.

0 Karma

domenico_perre
Path Finder

Regex is your friend .

Create a search time field extraction for the following

\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}

This will grab the IP address. click extract new fields and then I prefer to write My own regex.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...