Splunk Search

How to normalize field value from two different sourcetypes?

splunkb0y
New Member

Suppose I have two sourcetypes:

proxy1_source

in sourcetype=proxy1_source, the field url starts with: "http://"

proxy2_source

in sourcetype=proxy2_source, the field url doesn't start with: "http://"

How do I search for all events in both sourcetypes so that I can table the "url" but all urls in proxy2_source must be prepended with "http://"? Also, proxy2_source doesn't always have that url field.

I tried this below:

| rex field=url "(?((http|https)://))" |fillnull value="http://" |eval url= scheme + url

but i got output such as:

http://http:// (this keeps showing up)
http://http://www.domain.com/any/thing/ (proxy1_source - redundant http://)

proxy2_source urls look fine with the intended http:// prefix.

thanks in advance.

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Hey @splunkb0y, welcome to the Answers community! If one of these solutions answered your question, remember to "√Accept" the answer to award karma points 🙂 You can also upvote posts to give points.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

I would go the other direction. It is much easier to get rid of "https?:\" than it is to figure out whether you need an s or not when inserting it.

  index=your_index sourcetype=proxy1_source OR sourcetype=proxy2_source
  | rex field=url "^(https?:[\\]*)*(?<url>.*)$"

@splunkb0y, try the following:

( sourcetype=proxy1_source OR sourcetype=proxy2_source ) url=* <YourBaseSearch>
| eval url=case(sourcetype="proxy1_source",url,sourcetype="proxy2_source","http://".url)
| <your remaining search>
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

gcusello
Legend

Hi splunkb0y,
you could try something like this:

index=your_index sourcetype=proxy1_source OR sourcetype=proxy2_source
| eval url=if(sourcetype="proxy2_source","http://"+url,url)
| ...

Bye.
Giuseppe

Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...