topic Re: Loop through URL and http_referrer to find original request in Getting Data In

Can I loop through URL and http_referrer to find original request?

bababou — Tue, 17 Jan 2023 15:55:40 GMT

Hi everyone,

I'd like to see the flow from a given final URL, back to original URL the user typed.

In my Web Proxy Logs, I see the following :
_time, src_ip, http_referrer, http_method, URL

For example :
003, 1.1.1.1, htp://www.bbb.com/ads.html, GET, htp://www.ccc.com/ccc.html
002, 1.1.1.1, htp://www.aaa.com/, GET, htp://www.bbb.com/ads.html
001, 1.1.1.1, -, GET, htp://www.aaa.com/

What I want to do is, given the final URL (ccc.com/ccc.html), be able to go back in time, through the pair (http_referrer, URL) and find all the URLs up to the original one (aaa.com) with http_referrer="-".

Sometimes this flow can be spread among 10 different requests mixed in the middle of other web traffic, so this is hard to find by hand.

Programmatically I would do this with one loop, but I cannot find any loops with Splunk.

Can you help me ? Thanks.

Re: Loop through URL and http_referrer to find original request

technoe — Wed, 05 Feb 2014 20:24:43 GMT

How is the data indexed? Maybe you could use a last or first command instead of looping through each one...

Re: Loop through URL and http_referrer to find original request

bababou — Wed, 05 Feb 2014 20:33:29 GMT

What really interests me is the whole path.
In this example : aaa.com -> bbb.com/ads.html -> ccc.com/ccc.html
And not only the first and last requests.

Re: Loop through URL and http_referrer to find original request

somesoni2 — Wed, 05 Feb 2014 20:36:20 GMT

See Splunk's map command which is looping operator.

Re: Loop through URL and http_referrer to find original request

jsie_splunk — Wed, 05 Feb 2014 20:38:56 GMT

When you say "interested" how do you want the data expressed? As a single field containing the full path?

Re: Loop through URL and http_referrer to find original request

bababou — Wed, 05 Feb 2014 20:42:12 GMT

Some kind of "transaction" could also be fine, ideally a table with _time and url.

Re: Loop through URL and http_referrer to find original request

bababou — Thu, 13 Feb 2014 12:54:20 GMT

I solved my problem with an external script :

import splunk.Intersplunk

results, dummyresults, settings = splunk.Intersplunk.getOrganizedResults()

keywords, options = splunk.Intersplunk.getKeywordsAndOptions()
httpref = options.get('url', '-')

newresults = []

for result in results:
    if httpref == '-':
        break
    if result.get('url') == httpref:
        newresults.append(result)
        httpref = result.get('http_referer')

splunk.Intersplunk.outputResults(newresults)

And I call it this way :

... | referer url="htp://www.ccc.com/ccc.html" | table _time, http_referer, url

Re: Loop through URL and http_referrer to find original request

neerajs_81 — Tue, 17 Jan 2023 11:53:45 GMT

Can someone pls assist how to use MAP command or how to search for the original request URL without the external script that was marked as solution ?

Re: Loop through URL and http_referrer to find original request

scelikok — Tue, 17 Jan 2023 12:36:53 GMT

Hi @neerajs_81,

Please try below sample with map command;