Hello, When analyzing web traffic logs, at times the url field does not have a http_referrer field. We are interested in finding out which URL did the original request came from ? There is looping involved.
THis is similar to the post: https://community.splunk.com/t5/Getting-Data-In/Loop-through-URL-and-http-referrer-to-find-original-...
In the above post, user makes use of a script which I cannot use in my environment. How to then use the MAP command or any other command to recursively/loop thru the URL field and find out which was the original domain ?
For example:
index=firewall url =malicious-domain.com
Actual flow of traffic:
abc.com >>> bcd.com >> No Http_Referrer field >> malicious-domain.com ( http_referrer is <empty>)
Expected result:
abc.com
Hi @neerajs_81,
Only filtering URL should be enough. I could only test with your sample data. You may need to adjust according to indexed data.
index="firewall" sourcetype="proxy" URL="malicious-domain.com"
| map search="search index="firewall" sourcetype="proxy" http_referrer=$URL$ OR http_referrer="-" | eval finalURL=$URL$ "
| map search="search index="firewall" sourcetype="proxy" http_referrer=$http_referrer$ | eval finalURL=$finalURL$ "
| search http_referrer="-"
| dedup _raw
| rename URL as originalURL
| table finalURL originalURL
Hi @neerajs_81,
Only filtering URL should be enough. I could only test with your sample data. You may need to adjust according to indexed data.
index="firewall" sourcetype="proxy" URL="malicious-domain.com"
| map search="search index="firewall" sourcetype="proxy" http_referrer=$URL$ OR http_referrer="-" | eval finalURL=$URL$ "
| map search="search index="firewall" sourcetype="proxy" http_referrer=$http_referrer$ | eval finalURL=$finalURL$ "
| search http_referrer="-"
| dedup _raw
| rename URL as originalURL
| table finalURL originalURL
Thank you. I am getting close but not there yet. In my case, the original URL shows many different URLs . There is not one specific originalURL in the results. There is no way microsoft.com would be responsible for triggering that malicious domain. Anyways,Appreciate your quick response.
My Results:
finalURL | originalURL |
malicious.com | outlook.office365.com |
malicious.com | microsoft.com |
malicious.com | app.growme.com:443 |
malicious.com | xxxxxx |
Hi @neerajs_81,
I just replied on your previous question in the previous post.
Please try below;
index="firewall" sourcetype="proxy"
| map search="search index="firewall" sourcetype="proxy" http_referrer=$URL$ OR http_referrer="-" | eval finalURL=$URL$ "
| map search="search index="firewall" sourcetype="proxy" http_referrer=$http_referrer$ | eval finalURL=$finalURL$ "
| search http_referrer="-"
| dedup _raw
| rename URL as originalURL
| table finalURL originalURL