Finding original URL where the request came from when http_referrer is not available

[neerajs_81]

Hello, When analyzing web traffic logs, at times the url field does not have a http_referrer field.  We are interested in finding out which URL did the original request came from ?  There is looping involved.
 THis is similar to the post:  https://community.splunk.com/t5/Getting-Data-In/Loop-through-URL-and-http-referrer-to-find-original-...

In the above post, user makes use of a script which I cannot use in my environment.  How to then use the MAP command or any other command to recursively/loop thru the URL field and find out which was the original domain ?

For example:
index=firewall url =malicious-domain.com
Actual flow of traffic:
abc.com  >>> bcd.com  >>  No Http_Referrer field  >> malicious-domain.com  ( http_referrer is <empty>) 

Expected result:
abc.com

[scelikok · Accepted solution]

Hi @neerajs_81,

Only filtering URL should be enough. I could only test with your sample data. You may need to adjust according to indexed data.

index="firewall" sourcetype="proxy" URL="malicious-domain.com"
| map search="search index="firewall" sourcetype="proxy" http_referrer=$URL$ OR http_referrer="-" | eval finalURL=$URL$ " 
| map search="search index="firewall" sourcetype="proxy" http_referrer=$http_referrer$ | eval finalURL=$finalURL$ " 
| search http_referrer="-" 
| dedup _raw 
| rename URL as originalURL 
| table finalURL originalURL

[neerajs_81]

Thank you.  I am getting close  but not there yet.  In my case, the original URL shows many different URLs . There is not one specific originalURL in the results.  There is no way microsoft.com would be responsible for triggering that malicious domain.  Anyways,Appreciate your quick response.

My Results:

finalURL	originalURL
malicious.com	outlook.office365.com
malicious.com	microsoft.com
malicious.com	app.growme.com:443
malicious.com	xxxxxx

[scelikok]

Hi @neerajs_81,

I just replied on your previous question in the previous post.

Please try below;

index="firewall" sourcetype="proxy" 
| map search="search index="firewall" sourcetype="proxy" http_referrer=$URL$ OR http_referrer="-" | eval finalURL=$URL$ " 
| map search="search index="firewall" sourcetype="proxy" http_referrer=$http_referrer$ | eval finalURL=$finalURL$ " 
| search http_referrer="-" 
| dedup _raw 
| rename URL as originalURL 
| table finalURL originalURL