Splunk Dev

## #machinelearning Why Mean on beta Distribution is negative for Density function

Engager

#machinelearning

Hello,

I am using dist=auto in my Density function and I am getting negative Beta Results. I feel like this is wrong but keep me honest, I would like to understand how Beta distribution is captured  and why the mean is a negative result if I am using 0 to 100% success rate? other distribution I am happy with it (e.g Gaussian KDE and Normal)

|fit DensityFunction MyModelSuccessRate by "HourOfDay,Object" into MyModel2 dist="auto"

Thanks,

Joseph

Engager

Thank you so much for the response @tscroggins. I will validate using your math, though looking at it may suggest not be a negative number but I will definitely doublr check, I will also reach out to our support. Thank you so much.

Influencer

Hi @jasantor,

The implementation is in \$SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit/bin/algos_support/density_function/beta_distribution.py:

1. Sample min(data.shape[0], 10000) elements from field using numpy.random.choice.
2. Normalize sample to [0..1] using (data - data.min()) / (data.max() - data.min()).
3. Fit normalized sample to Beta using scipy.stats.beta.fit.
4. If either alpha <= 0 or beta <= 0, estimate parameters using normalized sample mean and variance.

The return values for scipy.stats.beta.fit are alpha, beta, loc, and scale. MLTK's implementation of dist=beta either misinterprets or mislabels loc and scale as mean and standard deviation, respectively.

You could compute the values yourself:

``````| summary MyModel2
| rex field=other "Alpha: (?<alpha>[^,]+), Beta: (?<beta>.+)"
| eval mean=alpha/(alpha+beta), std=sqrt((alpha*beta)/(pow(alpha+beta,2)*(alpha+beta+1)))``````

However, this will give you the approximate mean and standard deviation of the normalized sample, not the original data.

The dist=beta implementation is a little over four years old now, and something tells me no one has validated it. At the risk of being overly critical, the code looks suspiciously like it was copied from Stack Overflow.

I don't have a personal Splunk support account, so I can't report the issue. If you have support, I recommend opening a support case.

Get Updates on the Splunk Community!

#### Splunk Smartness with Brandon Sternfield | Episode 3

Hello and welcome to another episode of "Splunk Smartness," the interview series where we explore the power of ...

#### Monitoring Postgres with OpenTelemetry

Behind every business-critical application, you’ll find databases. These behind-the-scenes stores power ...

#### Mastering Synthetic Browser Testing: Pro Tips to Keep Your Web App Running Smoothly

To start, if you're new to synthetic monitoring, I recommend exploring this synthetic monitoring overview. In ...