Splunk Search
Highlighted

How to normalize field value from two different sourcetypes?

New Member

Suppose I have two sourcetypes:

proxy1_source

in sourcetype=proxy1_source, the field url starts with: "http://"

proxy2_source

in sourcetype=proxy2_source, the field url doesn't start with: "http://"

How do I search for all events in both sourcetypes so that I can table the "url" but all urls in proxy2source must be prepended with "http://"? Also, proxy2source doesn't always have that url field.

I tried this below:

| rex field=url "(?((http|https)://))" |fillnull value="http://" |eval url= scheme + url

but i got output such as:

http://http:// (this keeps showing up)
http://http://www.domain.com/any/thing/ (proxy1_source - redundant http://)

proxy2_source urls look fine with the intended http:// prefix.

thanks in advance.

0 Karma
Highlighted

Re: How to normalize field value from two different sourcetypes?

Legend

Hi splunkb0y,
you could try something like this:

index=your_index sourcetype=proxy1_source OR sourcetype=proxy2_source
| eval url=if(sourcetype="proxy2_source","http://"+url,url)
| ...

Bye.
Giuseppe

Highlighted

Re: How to normalize field value from two different sourcetypes?

Legend

@splunkb0y, try the following:

( sourcetype=proxy1_source OR sourcetype=proxy2_source ) url=* <YourBaseSearch>
| eval url=case(sourcetype="proxy1_source",url,sourcetype="proxy2_source","http://".url)
| <your remaining search>
Highlighted

Re: How to normalize field value from two different sourcetypes?

SplunkTrust
SplunkTrust

I would go the other direction. It is much easier to get rid of "https?:\" than it is to figure out whether you need an s or not when inserting it.

  index=your_index sourcetype=proxy1_source OR sourcetype=proxy2_source
  | rex field=url "^(https?:[\\]*)*(?<url>.*)$"
Highlighted

Re: How to normalize field value from two different sourcetypes?

Motivator

Hey @splunkb0y, welcome to the Answers community! If one of these solutions answered your question, remember to "√Accept" the answer to award karma points 🙂 You can also upvote posts to give points.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.