About amomchilov

amomchilov · ‎04-15-2020

I have a dashboard with three radio buttons to select the "Style" of all graphs on my dashboard: "Relative breakdown": An area chart that's stacked 100%, and shows the "other" series "Absolute breakdown": An area chart that's stacked, and shows the "other" series "Raw comparison": A line chart that's not stacked, and hides the "other" series They're defined like so: <label>Style</label> <choice value="style_relative">Relative breakdown</choice> <choice value="style_absolute">Absolute breakdown</choice> <choice value="style_compare">Raw comparison</choice> <change> <condition label="Relative breakdown"> <set token="chartType">area</set> <set token="stackMode">stacked100</set> <set token="useother">t</set> </condition> <condition label="Absolute breakdown"> <set token="chartType">area</set> <set token="stackMode">stacked</set> <set token="useother">t</set> </condition> <condition label="Raw comparison"> <set token="chartType">line</set> <set token="stackMode">default</set> <set token="useother">f</set> </condition> </change> Is it possible to set useother=t on my timechart , and then only/show hide it in the visualization layer? That way toggling doesn't require a full re-run of the search?

amomchilov · ‎04-14-2020

Wow, that's a night and day difference! Whereas before I couldn't squeeze out more than a 30 minute window, this code let me go back over 7 days! I thought timeseries worked like bin and stats together, so I'm surprised there such a big difference. Is untable the culprit? I really know how to interpret the job inspection. The profiler chart shows startup.handoff eating up pretty much all of the time, and there are basically no other big "chunks"

amomchilov · ‎04-13-2020

Thanks man, this worked wonderfully! The min/median/p99 values of this were heavily skewed by the IPs with 0 requests/min (which comprise most of the data points), so I fixed it by popping in a | where count_per_s != 0 . This had a nice side effect of drastically reducing the memory use. Do you know of any others ways to decrease the memory usage of this? For time scales above a few hours i still get EOM errors (using like 30 GB, the limit for us is 3 GB lol).

amomchilov · ‎04-07-2020

I have a dataset of Nginx (a web server) request logs. Each entry contains a client_ip . I want to impose some rate limiting, but I want to see what my current traffic patterns are, so my rate limits don't impede the current regular traffic. There are two rate limit settings available, one expressed as a limit per second, and a limit per minute. I would like to calculate the requests/second rate of each client_ip for each second. I would like to then aggregate (playing around with different aggregation functions, like avg, median, p90, p99, max, etc.) those values per-client_ip values into a timechart . Put another way, I would like to make this timechart have one data point per minute, each of which shows the p99 request/seconds among all the client_ips for that minute. For example, that would give me a per-second rate limit that would make 99% pass, and block the top 1%. I thought this would do it: application="my-app" index="my-index" request client_ip="*" user_agent="*" request="*" kube_pod="web-*" | timechart span=1s count as count_per_sec by client_ip | timechart span=1s avg(count_per_sec) But all of the count_per_sec values come out blank under the "statistics" tab.

amomchilov · ‎04-06-2020

Woo, that worked great!

amomchilov · ‎04-06-2020

Ah, I hadn't looked at that. From the docs it seems promising: Specifies the criteria for including particular data series when a field is given in the . The most common use of this option is to look for spikes in your data rather than overall mass of distribution in series selection. The default value finds the top ten series by area under the curve. Alternately one could replace sum with max to find the series with the ten highest spikes. Essentially the default is the same as specifying where sum in top10. The has no relation to the where command. I'll give it a shot and report back. How do where and limit interact?

amomchilov · ‎04-06-2020

I tried to adapt your comment to fit my case, but it doesn't seem quite right: https://i.imgur.com/GjvilZn.png It seems like a very similar set of series were picked, but with the Y values squished by about 1/5. I posted a related question, could you please take a look if you're free? https://answers.splunk.com/answers/815185

amomchilov · ‎04-06-2020

I'm looking to investigate IP addresses with highest peak loads on our service. Here's my current query: application="my-app" index="my-index" request client_ip="*" user_agent="*" request="*" kube_pod="web-*" | timechart limit=10 useother=f span="5minute" count by client_ip It works, and it's almost what I'm looking for. The documentation states: If a single aggregation is specified, the score is based on the sum of the values in the aggregation for that split-by value. For example, for timechart avg(foo) BY <field> the avg(foo) values are added up for each value of to determine the scores. If I understand this correctly, timeseries is picking the top 10 series whose sum of count s over the time span are the greatest. That is to say, it's picking the 10 top series by greatest integral. Instead, I want to select the 10 top series with the highest peak values (of any time in the timespan). For example, if there's a single data point that shows 10,000 requests per second for a single second, I want to be chosen over another series that shows 10 requests/second for months straight (whose integral would be much greater, but with a much lower max peak). Could you please help me do that?

Posts	8
Solutions	0
Karma Given	4
Karma Received	0
Member Since	‎04-06-2020

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

Hide/show "other" series in a Dashboard without re...

Timechart: p99 requests/min by client

timechart limit: pick top 10 series with the highe...

Hide/show "other" series in a Dashboard without re...

Re: Timechart: p99 requests/min by client

Re: Timechart: p99 requests/min by client

Timechart: p99 requests/min by client

Re: timechart limit: pick top 10 series with the h...

Re: timechart limit: pick top 10 series with the h...

Re: Timechart peaks - replace sum with max

timechart limit: pick top 10 series with the highe...