How to normalize field value from two different sourcetypes?

[splunkb0y]

Suppose I have two sourcetypes:

proxy1_source

in sourcetype=proxy1_source, the field url starts with: "http://"

proxy2_source

in sourcetype=proxy2_source, the field url doesn't start with: "http://"

How do I search for all events in both sourcetypes so that I can table the "url" but all urls in proxy2_source must be prepended with "http://"? Also, proxy2_source doesn't always have that url field.

I tried this below:

| rex field=url "(?((http|https)://))" |fillnull value="http://" |eval url= scheme + url

but i got output such as:

http://http:// (this keeps showing up)
http://http://www.domain.com/any/thing/ (proxy1_source - redundant http://)

proxy2_source urls look fine with the intended http:// prefix.

thanks in advance.

[lfedak_splunk]

Hey @splunkb0y, welcome to the Answers community! If one of these solutions answered your question, remember to "√Accept" the answer to award karma points 🙂 You can also upvote posts to give points.

[DalJeanis]

I would go the other direction. It is much easier to get rid of "https?:\" than it is to figure out whether you need an s or not when inserting it.

  index=your_index sourcetype=proxy1_source OR sourcetype=proxy2_source
  | rex field=url "^(https?:[\\]*)*(?<url>.*)$"

[niketn]

@splunkb0y, try the following:

( sourcetype=proxy1_source OR sourcetype=proxy2_source ) url=* <YourBaseSearch>
| eval url=case(sourcetype="proxy1_source",url,sourcetype="proxy2_source","http://".url)
| <your remaining search>

[gcusello]

Hi splunkb0y,
you could try something like this:

index=your_index sourcetype=proxy1_source OR sourcetype=proxy2_source
| eval url=if(sourcetype="proxy2_source","http://"+url,url)
| ...

Bye.
Giuseppe