Getting Data In

Coalesce and CIM Compliant Fields

BP9906
Builder

Hello,
From a reporting perspective, I have apache logs in a company standard format. Due to load balancing configuration, we have 3 possible fields where the source ip is noted.

These fields are as follows:
clientip (standard source IP field)
X_FORWARDED_IP (x-forwarded-for http header)
ns_client_ip (load balancer's view of source ip)

Since all 3 fields exist in sourcetype=access_combined (apache) logs, how do I coalesce the fields to "src" to make it CIM compliant?

I will mention that Apache logs a hyphen "-" for null field values for the above too.

Thanks for your help.

0 Karma
1 Solution

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

View solution in original post

woodcock
Esteemed Legend

Inside props.conf do this:

EVAL-src = case((clientip != "-"), clientip,  (X_FORWARDED_IP != "-"), X_FORWARDED_IP, (ns_client_ip != "-"), ns_client_ip)

BP9906
Builder

Thanks! Does this go on both indexer and search head?

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

this is a search time settings so will have no effect on a indexer (but should be in a TA which will be deployed on both SH and IDX)

0 Karma

domenico_perre
Path Finder

Looks like the regex is being changed when I post, here is a working one

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Try this regex as a search time extraction.

0 Karma

BP9906
Builder

The Regex is fine for IP addresses, but there's no if logic to assess which IP to use since its common to have both clientip and X_FORWARDED_IP present.

0 Karma

domenico_perre
Path Finder

Can you put an example which may make it easier to understand 🙂

0 Karma

domenico_perre
Path Finder

Is there a time where all fields will be the same?

0 Karma

BP9906
Builder

The X_FORWARDED_IP commonly matches clientip or SOURCE_IP, but depending on the load balancer configuration either clientip is "-" or SOURCE_IP is "-", they never match.

0 Karma

domenico_perre
Path Finder

Regex is your friend .

Create a search time field extraction for the following

\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}

This will grab the IP address. click extract new fields and then I prefer to write My own regex.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...