Dashboards & Visualizations

Dashboard element with runtime search behaviour

lnn2204
Path Finder

Hi guys,

I'm using ipinfo to check IPs of my system.

 

 

 

<base search>
| stats sum(Download) as Download by DestIP 
| sort 5 -Size
| ipinfo DestIP

 

 

 

The problem is it didn't wait for the final result then call the command "ipinfo", it'll make more request than 5 times, depends on how much DestIP it had.

Are there any solutions for this case?

Labels (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Is this search in a dashboard? I noticed that you have a token. Often, searches in dashboards are executed multiple times as tokens are given different values (searchWhenChanged). Is this perhaps what is going on?

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Firstly Size doesn't exist so the sort won't do anything (of course, this could just be a typo in your example)

Even so, you should only get 5 events after the sort.

Try inserting a table command

<base search>
| stats sum(Download) as Download by DestIP 
| sort 5 -Download
| table DestIP Download
| ipinfo DestIP
0 Karma

lnn2204
Path Finder

the table command was put after "ipinfo" commands, so I think it's totally fine.

And yes, I got just 5 row in the table, but the request was made to Ipinfo API is more than 5. Depends on how many DestIP user surfed.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Even if you think it might be totally fine, it might be worth checking - just saying 😁

0 Karma

lnn2204
Path Finder

Checked it, not work hehe.

Do you have another solutions?

0 Karma

PickleRick
Ultra Champion

Please be a bit more specific because there doesn't seem to be anything wrong with your search.

The "sort 5 - Size" part should give you 5 results and ipinfo should perform calls for those 5 results. If it's making more than one IPinfo API call per result row, that's nothing you can do withouth fiddling with internals of the app.

0 Karma

lnn2204
Path Finder

Ah, you can think about Splunk will do sort 5 ips, but the event will be indexed one by one follow the time of its. And in that moment, the top 5 is the top, but 5s later, the top 5 will be changed, so it will do10 requests instead of 5 request at the final

0 Karma

PickleRick
Ultra Champion

Honestly, I don't understand the issue.

If you run the same search against different sets of data (for example, because of running it for different time ranges), you might get different results. So those "top 5 IPs" might indeed be different.

There are a few possibilities that could let you store some kind of "global state" but firstly you'd have to think what would be the expected behaviour if - for example - you got 5 IPs at the first run of your search and 3 "old" and 2 "new" IPs at next run.

0 Karma

lnn2204
Path Finder

I don't know how to detail my issue.

Like you search this in 5min:

 

| eval RespondSize=round(('RespondSize'/1024)/1024,2)
| search SrcIP="$ipDownloadDetail$" 
| stats sum(RespondSize) as Size by DstIP 
| sort 5 -Size
| ipinfo DstIP
| table DstIP, org, hostname, country, Size

 

And when it had run for 2-3 mins, the top 5 will be set, and when it run to the final, the top 5 will be changed. So when top 5 are set at first time, they will call the API with 5 requests, and when it run to the final stage, another 5 requests will be made

0 Karma

PickleRick
Ultra Champion

I still don't understand. If you change your search, change its conditions or time range, you can get different results. That's how it works. So what's surprising here?

Also if you're running two separate searches with ipinfo command in them - the command will get run separately for each result set.

Again - honestly, I have no idea what other behaviour you expect and why.

lnn2204
Path Finder

Okay, tks for your replying. I'll try to figure it out.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Is this search in a dashboard? I noticed that you have a token. Often, searches in dashboards are executed multiple times as tokens are given different values (searchWhenChanged). Is this perhaps what is going on?

lnn2204
Path Finder

Yeah, I think it's the problem. It's in dashboard, with realtime search and a drilldown which to change the token then this dashboard can run. But my token will not change when it's running. So in my mind now, maybe about the realtime search.

0 Karma

PickleRick
Ultra Champion

So that's a completely different problem. It's not about search as such but about the fact that when you're running a realtime search the results are indeed being updated as the search progresses.

That's "by design". As @ITWhisperer already pointed out, real-time searches are not supposed to be used often. In fact, it's best to avoid them altogether (there are very few use cases when they are indeed justified).

With a normal historical ad-hoc search, ever run of your search will produce a set of results which will be passed to your command which calls external IP and that's it. But remember that if you have a refresh set on your dashboard, every time the dashboard refreshes, the search will get re-run however so your API calls will be performed again. That's simply how it works 🙂

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Realtime searches tie up cpus and should be used sparingly. Does your dashboard need a realtime search? Can you split off the "realtime" part of the dashboard to another dashboard? Can you use short refresh cycles to produce "near realtime" results which you can then tune to an appropriate value? How near to realtime does the dashboard need to be? Do you have someone watching the dashboard every second of every minute of every hour of every day? How quickly do they need to respond to something they see in the dashboard?

lnn2204
Path Finder

Thank you, I'll think about that

0 Karma
Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...