Statistical Analysis using Splunk

[daskuntal]

Hi,
I'm trying to create an ALert mechanism that will be triggered if certain condition is "statistically significant".

Let's say
n = Number of Errors/5 min
p = Probability of n<=5
a = The critical alpha level = 5%
Null Hypothesis:
H0: p <= 5%
H1: p > 5%

Let's say I have a set of data with the # of Errors per5 min, for the last 7 days. e.g.
3:36 -- 3
3:41 -- 0
3:46 -- 0
.
.
.
.
4:00 -- 1
4:05 -- 4

Is there a way for me to configure an alert in Splunk, which will check every 5 mins, if the number of Errors occurred in the last 5 mins is statistically significant, in order to trigger the alert?

Thanks

[daskuntal · Accepted solution]

Here's the most updated answer:
A few modifications to make this a high performing search:
1) Created Summary search to collect data for the Error Conditions
2) Created a query that runs every midnight, and saves a lookup table with all the statistical data for each combination of Error COnditions
3) The actual search that runs every 5 mins, to check if any issues occurred that is a Grubb's Outlier.

summary_alert_cust_affected:

Definition: Summary search to collect data for Customer ID & Exception Type.This search created the summary index called "some_summary_index"

index=SomeIndex sourcetype=SomeSourceType Header.Type=Outbound (Results.ExceptionType!=null AND Results.ExceptionType!="null" AND Results.ExceptionType!=" null") (date_wday!="sunday" AND date_wday!="saturday") (Subject.Cust!=null AND Subject.Cust!="null" AND Subject.Cust!=" null")
| bucket _time span=5m
| sistats dc(Subject.Cust) as UniqCustsAffected, count as numErrors by Application.Channel, Results.ExceptionType, Service.Operation, _time

StartTIme: -11m@m
Finish Time: -1m@m

Average_Std_Dev_Of_Custs_Affected_past_30_days:

Definition: Query to record/save avg & StdDev data for last 30 days, into a csv file for future lookup(instead of doing a subsearch wit ha Summary Search):

index=some_summary_index search_name=summary_alert_cust_affected
(date_wday!="sunday" AND date_wday!="saturday")
| stats dc(Subject.Cust) as UniqCustsAffected, count as numErrors by Application.Channel, Results.ExceptionType, Service.Operation, _time
| stats avg(UniqCustsAffected) as avgCustsAffected, stdev(UniqCustsAffected) as stdevCustsAffected, median(UniqCustsAffected) as medianCustsAffected, perc95(UniqCustsAffected) as perc95CustsAffected, avg(numErrors) as avgErrors, stdev(numErrors) as stdevErrors, median(numErrors) as medianErrors, perc95(numErrors) as perc95Errors, count as numPeriods by Application.Channel, Results.ExceptionType, Service.Operation
| sort - avgCustsAffected, stdevCustsAffected
| outputlookup avg_stdev_lookup.csv

StartTIme: -30d@d
Finish Time: now

Run every night at 00:00 hrs

grubs-outlier-custs-impacted:

Definition:
Actual search that alerts any occurance of Customers impacted in hte last 5 mins, that's a Grubb's Outlier:

index=mobile sourcetype=mobile_elilogs Header.Type=Outbound (Results.ExceptionType!=null AND Results.ExceptionType!="null" AND Results.ExceptionType!=" null") (Subject.Cust!=null AND Subject.Cust!="null" AND Subject.Cust!=" null")
| stats dc(Subject.Cust) as currentCustsAffected, count as currentErrors by Application.Channel, Results.ExceptionType, Service.Operation
| join Application.Channel, Results.ExceptionType, Service.Operation [inputlookup avg_stdev_lookup.csv]
| eval zVal = round(((currentCustsAffected - avgCustsAffected)/stdevCustsAffected), 2)
| lookup alpha_lookup Service.Operation OUTPUT alpha
| eval alpha1=if(alpha=0.01, alpha, "x")
| eval alpha5=if(alpha=0.05, alpha, "x")
| fields - alpha
| lookup g_crit_lookup df as numPeriods, alpha1, alpha5 OUTPUT gcrit
| eval significant = if(numPeriods<=2, "false", if(zVal>gcrit, "true", "false"))
| where significant="true"
| table significant Application.Channel Service.Operation currentCustsAffected avgCustsAffected Results.ExceptionType stdevCustsAffected zVal alpha1 alpha5 gcrit currentErrors avgErrors stdevErrors numPeriods

| sort -zVal

StartTIme: -5m@m
Finish Time: now

alpha_lookup:

With this lookup table you can also dictate the sensitivity of each operation, by changing the alpha value for each.

"Service.Operation",alpha
AutoLogin,"0.05"
SubmitItem,"0.05"
CheckStatus,"0.05"
GetPendingSessions,"0.05"

g_crit_lookup:

This is the flattened z-Lookup table:
Here's the csv version of it:

alpha1,alpha5,df,gcrit
x,"0.05",3,"1.15310"
x,"0.05",4,"1.46250"
x,"0.05",5,"1.67140"
x,"0.05",6,"1.82210"
x,"0.05",7,"1.93810"
x,"0.05",8,"2.03170"
x,"0.05",9,"2.10960"
x,"0.05",10,"2.17610"
x,"0.05",11,"2.23390"
x,"0.05",12,"2.28500"
x,"0.05",13,"2.33050"
x,"0.05",14,"2.37170"
x,"0.05",15,"2.40900"
x,"0.05",16,"2.44330"
x,"0.05",17,"2.47480"
x,"0.05",18,"2.50400"
x,"0.05",19,"2.53120"
x,"0.05",20,"2.55660"
x,"0.05",21,"2.57790"
x,"0.05",22,"2.59910"
x,"0.05",23,"2.62040"
x,"0.05",24,"2.64160"
x,"0.05",25,"2.66290"
x,"0.05",26,"2.67930"
x,"0.05",27,"2.69580"
x,"0.05",28,"2.71220"
x,"0.05",29,"2.72870"
x,"0.05",30,"2.74510"
x,"0.05",31,"2.75730"
x,"0.05",32,"2.76960"
x,"0.05",33,"2.78180"
x,"0.05",34,"2.79410"
x,"0.05",35,"2.80630"
x,"0.05",36,"2.81850"
x,"0.05",37,"2.83080"
x,"0.05",38,"2.84300"
x,"0.05",39,"2.85530"
x,"0.05",40,"2.86750"
x,"0.05",41,"2.87650"
x,"0.05",42,"2.88540"
x,"0.05",43,"2.89440"
x,"0.05",44,"2.90330"
x,"0.05",45,"2.91230"
x,"0.05",46,"2.92120"
x,"0.05",47,"2.93020"
x,"0.05",48,"2.93910"
x,"0.05",49,"2.94810"
x,"0.05",50,"2.95700"
x,"0.05",51,"2.96400"
x,"0.05",52,"2.97100"
x,"0.05",53,"2.97800"
x,"0.05",54,"2.98500"
x,"0.05",55,"2.99200"
x,"0.05",56,"2.99890"
x,"0.05",57,"3.00590"
x,"0.05",58,"3.01290"
x,"0.05",59,"3.01990"
x,"0.05",60,"3.02690"
x,"0.05",61,"3.03260"
x,"0.05",62,"3.03830"
x,"0.05",63,"3.04400"
x,"0.05",64,"3.04970"
x,"0.05",65,"3.05540"
x,"0.05",66,"3.06110"
x,"0.05",67,"3.06680"
x,"0.05",68,"3.07250"
x,"0.05",69,"3.07820"
x,"0.05",70,"3.08390"
x,"0.05",71,"3.08870"
x,"0.05",72,"3.09350"
x,"0.05",73,"3.09830"
x,"0.05",74,"3.10310"
x,"0.05",75,"3.10790"
x,"0.05",76,"3.11270"
x,"0.05",77,"3.11750"
x,"0.05",78,"3.12230"
x,"0.05",79,"3.12710"
x,"0.05",80,"3.13190"
x,"0.05",81,"3.13600"
x,"0.05",82,"3.14020"
x,"0.05",83,"3.14430"
x,"0.05",84,"3.14850"
x,"0.05",85,"3.15260"
x,"0.05",86,"3.15670"
x,"0.05",87,"3.16090"
x,"0.05",88,"3.16500"
x,"0.05",89,"3.16920"
x,"0.05",90,"3.17330"
x,"0.05",91,"3.17690"
x,"0.05",92,"3.18050"
x,"0.05",93,"3.18420"
x,"0.05",94,"3.18780"
x,"0.05",95,"3.19140"
x,"0.05",96,"3.19500"
x,"0.05",97,"3.19860"
x,"0.05",98,"3.20230"
x,"0.05",99,"3.20590"
x,"0.05",100,"3.20950"
x,"0.05",101,"3.21260"
x,"0.05",102,"3.21560"
x,"0.05",103,"3.21870"
x,"0.05",104,"3.22170"
x,"0.05",105,"3.22480"
x,"0.05",106,"3.22780"
x,"0.05",107,"3.23090"
x,"0.05",108,"3.23390"
x,"0.05",109,"3.23700"
x,"0.05",110,"3.24010"
x,"0.05",111,"3.24310"
x,"0.05",112,"3.24620"
x,"0.05",113,"3.24920"
x,"0.05",114,"3.25230"
x,"0.05",115,"3.25530"
x,"0.05",116,"3.25840"
x,"0.05",117,"3.26140"
x,"0.05",118,"3.26450"
x,"0.05",119,"3.26750"
x,"0.05",120,"3.27060"
x,"0.05",121,"3.27310"
x,"0.05",122,"3.27560"
x,"0.05",123,"3.27810"
x,"0.05",124,"3.28060"
x,"0.05",125,"3.28320"
x,"0.05",126,"3.28570"
x,"0.05",127,"3.28820"
x,"0.05",128,"3.29070"
x,"0.05",129,"3.29320"
x,"0.05",130,"3.29570"
x,"0.05",131,"3.29820"
x,"0.05",132,"3.30070"
x,"0.05",133,"3.30320"
x,"0.05",134,"3.30570"
x,"0.05",135,"3.30830"
x,"0.05",136,"3.31080"
x,"0.05",137,"3.31330"
x,"0.05",138,"3.31580"
x,"0.05",139,"3.31830"
x,"0.05",140,"3.32080"
x,"0.05",141,"3.32290"
x,"0.05",142,"3.32510"
x,"0.05",143,"3.32720"
x,"0.05",144,"3.32930"
x,"0.05",145,"3.33140"
x,"0.05",146,"3.33360"
x,"0.05",147,"3.33570"
x,"0.05",148,"3.33780"
x,"0.05",149,"3.33990"
x,"0.05",150,"3.34210"
x,"0.05",151,"3.34420"
x,"0.05",152,"3.34630"
x,"0.05",153,"3.34840"
x,"0.05",154,"3.35060"
x,"0.05",155,"3.35270"
x,"0.05",156,"3.35480"
x,"0.05",157,"3.35690"
x,"0.05",158,"3.35910"
x,"0.05",159,"3.36120"
x,"0.05",160,"3.36330"
x,"0.05",161,"3.36510"
x,"0.05",162,"3.36700"
x,"0.05",163,"3.36880"
x,"0.05",164,"3.37070"
x,"0.05",165,"3.37250"
x,"0.05",166,"3.37430"
x,"0.05",167,"3.37620"
x,"0.05",168,"3.37800"
x,"0.05",169,"3.37990"
x,"0.05",170,"3.38170"
x,"0.05",171,"3.38350"
x,"0.05",172,"3.38540"
x,"0.05",173,"3.38720"
x,"0.05",174,"3.38910"
x,"0.05",175,"3.39090"
x,"0.05",176,"3.39270"
x,"0.05",177,"3.39460"
x,"0.05",178,"3.39640"
x,"0.05",179,"3.39830"
x,"0.05",180,"3.40010"
x,"0.05",181,"3.40170"
x,"0.05",182,"3.40330"
x,"0.05",183,"3.40490"
x,"0.05",184,"3.40660"
x,"0.05",185,"3.40820"
x,"0.05",186,"3.40980"
x,"0.05",187,"3.41140"
x,"0.05",188,"3.41300"
x,"0.05",189,"3.41460"
x,"0.05",190,"3.41630"
x,"0.05",191,"3.41790"
x,"0.05",192,"3.41950"
x,"0.05",193,"3.42110"
x,"0.05",194,"3.42270"
x,"0.05",195,"3.42430"
x,"0.05",196,"3.42590"
x,"0.05",197,"3.42760"
x,"0.05",198,"3.42920"
x,"0.05",199,"3.43080"
x,"0.05",200,"3.43240"
x,"0.05",201,"3.43360"
x,"0.05",202,"3.43480"
x,"0.05",203,"3.43600"
x,"0.05",204,"3.43720"
x,"0.05",205,"3.43840"
x,"0.05",206,"3.43960"
x,"0.05",207,"3.44080"
x,"0.05",208,"3.44200"
x,"0.05",209,"3.44320"
x,"0.05",210,"3.44440"
x,"0.05",211,"3.44560"
x,"0.05",212,"3.44680"
x,"0.05",213,"3.44800"
x,"0.05",214,"3.44920"
x,"0.05",215,"3.45040"
x,"0.05",216,"3.45160"
x,"0.05",217,"3.45280"
x,"0.05",218,"3.45400"
x,"0.05",219,"3.45520"
x,"0.05",220,"3.45640"
x,"0.05",221,"3.45760"
x,"0.05",222,"3.45880"
x,"0.05",223,"3.46000"
x,"0.05",224,"3.46120"
x,"0.05",225,"3.46240"
x,"0.05",226,"3.46360"
x,"0.05",227,"3.46480"
x,"0.05",228,"3.46600"
x,"0.05",229,"3.46720"
x,"0.05",230,"3.46840"
x,"0.05",231,"3.46960"
x,"0.05",232,"3.47080"
x,"0.05",233,"3.47200"
x,"0.05",234,"3.47320"
x,"0.05",235,"3.47440"
x,"0.05",236,"3.47560"
x,"0.05",237,"3.47680"
x,"0.05",238,"3.47800"
x,"0.05",239,"3.47920"
x,"0.05",240,"3.48040"
x,"0.05",241,"3.48160"
x,"0.05",242,"3.48280"
x,"0.05",243,"3.48400"
x,"0.05",244,"3.48520"
x,"0.05",245,"3.48640"
x,"0.05",246,"3.48760"
x,"0.05",247,"3.48880"
x,"0.05",248,"3.49000"
x,"0.05",249,"3.49120"
x,"0.05",250,"3.49250"
x,"0.05",251,"3.49370"
x,"0.05",252,"3.49490"
x,"0.05",253,"3.49610"
x,"0.05",254,"3.49730"
x,"0.05",255,"3.49850"
x,"0.05",256,"3.49970"
x,"0.05",257,"3.50090"
x,"0.05",258,"3.50210"
x,"0.05",259,"3.50330"
x,"0.05",260,"3.50450"
x,"0.05",261,"3.50570"
x,"0.05",262,"3.50690"
x,"0.05",263,"3.50810"
x,"0.05",264,"3.50930"
x,"0.05",265,"3.51050"
x,"0.05",266,"3.51170"
x,"0.05",267,"3.51290"
x,"0.05",268,"3.51410"
x,"0.05",269,"3.51530"
x,"0.05",270,"3.51650"
x,"0.05",271,"3.51770"
x,"0.05",272,"3.51890"
x,"0.05",273,"3.52010"
x,"0.05",274,"3.52130"
x,"0.05",275,"3.52250"
x,"0.05",276,"3.52370"
x,"0.05",277,"3.52490"
x,"0.05",278,"3.52610"
x,"0.05",279,"3.52730"
x,"0.05",280,"3.52850"
x,"0.05",281,"3.52970"
x,"0.05",282,"3.53090"
x,"0.05",283,"3.53210"
x,"0.05",284,"3.53330"
x,"0.05",285,"3.53450"
x,"0.05",286,"3.53570"
x,"0.05",287,"3.53690"
x,"0.05",288,"3.53810"
x,"0.05",289,"3.53930"
x,"0.05",290,"3.54050"
x,"0.05",291,"3.54170"
x,"0.05",292,"3.54290"
x,"0.05",293,"3.54410"
x,"0.05",294,"3.54530"
x,"0.05",295,"3.54650"
x,"0.05",296,"3.54770"
x,"0.05",297,"3.54890"
x,"0.05",298,"3.55010"
x,"0.05",299,"3.55130"
x,"0.05",300,"3.55250"
x,"0.05",301,"3.55330"
x,"0.05",302,"3.55410"
x,"0.05",303,"3.55490"
x,"0.05",304,"3.55580"
x,"0.05",305,"3.55660"
x,"0.05",306,"3.55740"
x,"0.05",307,"3.55820"
x,"0.05",308,"3.55900"
x,"0.05",309,"3.55980"
x,"0.05",310,"3.56060"
x,"0.05",311,"3.56150"
x,"0.05",312,"3.56230"
x,"0.05",313,"3.56310"
x,"0.05",314,"3.56390"
x,"0.05",315,"3.56470"
x,"0.05",316,"3.56550"
x,"0.05",317,"3.56630"
x,"0.05",318,"3.56720"
x,"0.05",319,"3.56800"
x,"0.05",320,"3.56880"
x,"0.05",321,"3.56960"
x,"0.05",322,"3.57040"
x,"0.05",323,"3.57120"
x,"0.05",324,"3.57200"
x,"0.05",325,"3.57290"
x,"0.05",326,"3.57370"
x,"0.05",327,"3.57450"
x,"0.05",328,"3.57530"
x,"0.05",329,"3.57610"
x,"0.05",330,"3.57690"
x,"0.05",331,"3.57770"
x,"0.05",332,"3.57850"
x,"0.05",333,"3.57940"
x,"0.05",334,"3.58020"
x,"0.05",335,"3.58100"
x,"0.05",336,"3.58180"
x,"0.05",337,"3.58260"
x,"0.05",338,"3.58340"
x,"0.05",339,"3.58420"
x,"0.05",340,"3.58510"
x,"0.05",341,"3.58590"
x,"0.05",342,"3.58670"
x,"0.05",343,"3.58750"
x,"0.05",344,"3.58830"
x,"0.05",345,"3.58910"
x,"0.05",346,"3.58990"
x,"0.05",347,"3.59080"
x,"0.05",348,"3.59160"
x,"0.05",349,"3.59240"
x,"0.05",350,"3.59320"
x,"0.05",351,"3.59400"
x,"0.05",352,"3.59480"
x,"0.05",353,"3.59560"
x,"0.05",354,"3.59650"
x,"0.05",355,"3.59730"
x,"0.05",356,"3.59810"
x,"0.05",357,"3.59890"
x,"0.05",358,"3.59970"
x,"0.05",359,"3.60050"
x,"0.05",360,"3.60130"
x,"0.05",361,"3.60220"
x,"0.05",362,"3.60300"
x,"0.05",363,"3.60380"
x,"0.05",364,"3.60460"
x,"0.05",365,"3.60540"
x,"0.05",366,"3.60620"
x,"0.05",367,"3.60700"
x,"0.05",368,"3.60790"
x,"0.05",369,"3.60870"
x,"0.05",370,"3.60950"
x,"0.05",371,"3.61030"
x,"0.05",372,"3.61110"
x,"0.05",373,"3.61190"
x,"0.05",374,"3.61270"
x,"0.05",375,"3.61360"
x,"0.05",376,"3.61440"
x,"0.05",377,"3.61520"
x,"0.05",378,"3.61600"
x,"0.05",379,"3.61680"
x,"0.05",380,"3.61760"
x,"0.05",381,"3.61840"
x,"0.05",382,"3.61920"
x,"0.05",383,"3.62010"
x,"0.05",384,"3.62090"
x,"0.05",385,"3.62170"
x,"0.05",386,"3.62250"
x,"0.05",387,"3.62330"
x,"0.05",388,"3.62410"
x,"0.05",389,"3.62490"
x,"0.05",390,"3.62580"
x,"0.05",391,"3.62660"
x,"0.05",392,"3.62740"
x,"0.05",393,"3.62820"
x,"0.05",394,"3.62900"
x,"0.05",395,"3.62980"
x,"0.05",396,"3.63060"
x,"0.05",397,"3.63150"
x,"0.05",398,"3.63230"
x,"0.05",399,"3.63310"
x,"0.05",400,"3.63390"
x,"0.05",401,"3.63450"
x,"0.05",402,"3.63510"
x,"0.05",403,"3.63570"
x,"0.05",404,"3.63640"
x,"0.05",405,"3.63700"
x,"0.05",406,"3.63760"
x,"0.05",407,"3.63820"
x,"0.05",408,"3.63880"
x,"0.05",409,"3.63940"
x,"0.05",410,"3.64000"
x,"0.05",411,"3.64060"
x,"0.05",412,"3.64130"
x,"0.05",413,"3.64190"
x,"0.05",414,"3.64250"
x,"0.05",415,"3.64310"
x,"0.05",416,"3.64370"
x,"0.05",417,"3.64430"
x,"0.05",418,"3.64490"
x,"0.05",419,"3.64550"
x,"0.05",420,"3.64620"
x,"0.05",421,"3.64680"
x,"0.05",422,"3.64740"
x,"0.05",423,"3.64800"
x,"0.05",424,"3.64860"
x,"0.05",425,"3.64920"
x,"0.05",426,"3.64980"
x,"0.05",427,"3.65050"
x,"0.05",428,"3.65110"
x,"0.05",429,"3.65170"
x,"0.05",430,"3.65230"
x,"0.05",431,"3.65290"
x,"0.05",432,"3.65350"
x,"0.05",433,"3.65410"
x,"0.05",434,"3.65470"
x,"0.05",435,"3.65540"
x,"0.05",436,"3.65600"
x,"0.05",437,"3.65660"
x,"0.05",438,"3.65720"
x,"0.05",439,"3.65780"
x,"0.05",440,"3.65840"
x,"0.05",441,"3.65900"
x,"0.05",442,"3.65960"
x,"0.05",443,"3.66030"
x,"0.05",444,"3.66090"
x,"0.05",445,"3.66150"
x,"0.05",446,"3.66210"
x,"0.05",447,"3.66270"
x,"0.05",448,"3.66330"
x,"0.05",449,"3.66390"
x,"0.05",450,"3.66460"
x,"0.05",451,"3.66520"
x,"0.05",452,"3.66580"
x,"0.05",453,"3.66640"
x,"0.05",454,"3.66700"
x,"0.05",455,"3.66760"
x,"0.05",456,"3.66820"
x,"0.05",457,"3.66880"
x,"0.05",458,"3.66950"
x,"0.05",459,"3.67010"
x,"0.05",460,"3.67070"
x,"0.05",461,"3.67130"
x,"0.05",462,"3.67190"
x,"0.05",463,"3.67250"
x,"0.05",464,"3.67310"
x,"0.05",465,"3.67370"
x,"0.05",466,"3.67440"
x,"0.05",467,"3.67500"
x,"0.05",468,"3.67560"
x,"0.05",469,"3.67620"
x,"0.05",470,"3.67680"
x,"0.05",471,"3.67740"
x,"0.05",472,"3.67800"
x,"0.05",473,"3.67860"
x,"0.05",474,"3.67930"
x,"0.05",475,"3.67990"
x,"0.05",476,"3.68050"
x,"0.05",477,"3.68110"
x,"0.05",478,"3.68170"
x,"0.05",479,"3.68230"
x,"0.05",480,"3.68290"
x,"0.05",481,"3.68360"
x,"0.05",482,"3.68420"
x,"0.05",483,"3.68480"
x,"0.05",484,"3.68540"
x,"0.05",485,"3.68600"
x,"0.05",486,"3.68660"
x,"0.05",487,"3.68720"
x,"0.05",488,"3.68780"
x,"0.05",489,"3.68850"
x,"0.05",490,"3.68910"
x,"0.05",491,"3.68970"
x,"0.05",492,"3.69030"
x,"0.05",493,"3.69090"
x,"0.05",494,"3.69150"
x,"0.05",495,"3.69210"
x,"0.05",496,"3.69270"
x,"0.05",497,"3.69340"
x,"0.05",498,"3.69400"
x,"0.05",499,"3.69460"
x,"0.05",500,"3.69520"
x,"0.05",501,"3.69570"
x,"0.05",502,"3.69620"
x,"0.05",503,"3.69670"
x,"0.05",504,"3.69720"
x,"0.05",505,"3.69770"
x,"0.05",506,"3.69810"
x,"0.05",507,"3.69860"
x,"0.05",508,"3.69910"
x,"0.05",509,"3.69960"
x,"0.05",510,"3.70010"
x,"0.05",511,"3.70060"
x,"0.05",512,"3.70110"
x,"0.05",513,"3.70160"
x,"0.05",514,"3.70210"
x,"0.05",515,"3.70260"
x,"0.05",516,"3.70300"
x,"0.05",517,"3.70350"
x,"0.05",518,"3.70400"
x,"0.05",519,"3.70450"
x,"0.05",520,"3.70500"
x,"0.05",521,"3.70550"
x,"0.05",522,"3.70600"
x,"0.05",523,"3.70650"
x,"0.05",524,"3.70700"
x,"0.05",525,"3.70750"
x,"0.05",526,"3.70790"
x,"0.05",527,"3.70840"
x,"0.05",528,"3.70890"
x,"0.05",529,"3.70940"
x,"0.05",530,"3.70990"
x,"0.05",531,"3.71040"
x,"0.05",532,"3.71090"
x,"0.05",533,"3.71140"
x,"0.05",534,"3.71190"
x,"0.05",535,"3.71240"
x,"0.05",536,"3.71280"
x,"0.05",537,"3.71330"
x,"0.05",538,"3.71380"
x,"0.05",539,"3.71430"
x,"0.05",540,"3.71480"
x,"0.05",541,"3.71530"
x,"0.05",542,"3.71580"
x,"0.05",543,"3.71630"
x,"0.05",544,"3.71680"
x,"0.05",545,"3.71730"
x,"0.05",546,"3.71770"
x,"0.05",547,"3.71820"
x,"0.05",548,"3.71870"
x,"0.05",549,"3.71920"
x,"0.05",550,"3.71970"
x,"0.05",551,"3.72020"
x,"0.05",552,"3.72070"
x,"0.05",553,"3.72120"
x,"0.05",554,"3.72170"
x,"0.05",555,"3.72220"
x,"0.05",556,"3.72260"
x,"0.05",557,"3.72310"
x,"0.05",558,"3.72360"
x,"0.05",559,"3.72410"
x,"0.05",560,"3.72460"
x,"0.05",561,"3.72510"
x,"0.05",562,"3.72560"
x,"0.05",563,"3.72610"
x,"0.05",564,"3.72660"
x,"0.05",565,"3.72710"
x,"0.05",566,"3.72750"
x,"0.05",567,"3.72800"
x,"0.05",568,"3.72850"
x,"0.05",569,"3.72900"
x,"0.05",570,"3.72950"
x,"0.05",571,"3.73000"
x,"0.05",572,"3.73050"
x,"0.05",573,"3.73100"
x,"0.05",574,"3.73150"
x,"0.05",575,"3.73200"
x,"0.05",576,"3.73240"
x,"0.05",577,"3.73290"
x,"0.05",578,"3.73340"
x,"0.05",579,"3.73390"
x,"0.05",580,"3.73440"
x,"0.05",581,"3.73490"
x,"0.05",582,"3.73540"
x,"0.05",583,"3.73590"
x,"0.05",584,"3.73640"
x,"0.05",585,"3.73690"
x,"0.05",586,"3.73730"
x,"0.05",587,"3.73780"
x,"0.05",588,"3.73830"
x,"0.05",589,"3.73880"
x,"0.05",590,"3.73930"
x,"0.05",591,"3.73980"
x,"0.05",592,"3.74030"
x,"0.05",593,"3.74080"
x,"0.05",594,"3.74130"
x,"0.05",595,"3.74180"
x,"0.05",596,"3.74220"
x,"0.05",597,"3.74270"
x,"0.05",598,"3.74320"
x,"0.05",599,"3.74370"
x,"0.05",600,"3.74420"
"0.01",x,3,"1.15460"
"0.01",x,4,"1.49250"
"0.01",x,5,"1.74890"
"0.01",x,6,"1.94420"
"0.01",x,7,"2.09730"
"0.01",x,8,"2.22080"
"0.01",x,9,"2.32310"
"0.01",x,10,"2.40970"
"0.01",x,11,"2.48430"
"0.01",x,12,"2.54940"
"0.01",x,13,"2.60700"
"0.01",x,14,"2.65850"
"0.01",x,15,"2.70490"
"0.01",x,16,"2.74700"
"0.01",x,17,"2.78540"
"0.01",x,18,"2.82080"
"0.01",x,19,"2.85350"
"0.01",x,20,"2.88380"
"0.01",x,21,"2.90880"
"0.01",x,22,"2.93370"
"0.01",x,23,"2.95870"
"0.01",x,24,"2.98360"
"0.01",x,25,"3.00860"
"0.01",x,26,"3.02750"
"0.01",x,27,"3.04630"
"0.01",x,28,"3.06520"
"0.01",x,29,"3.08400"
"0.01",x,30,"3.10290"
"0.01",x,31,"3.11660"
"0.01",x,32,"3.13020"
"0.01",x,33,"3.14390"
"0.01",x,34,"3.15750"
"0.01",x,35,"3.17120"
"0.01",x,36,"3.18490"
"0.01",x,37,"3.19850"
"0.01",x,38,"3.21220"
"0.01",x,39,"3.22580"
"0.01",x,40,"3.23950"
"0.01",x,41,"3.24920"
"0.01",x,42,"3.25890"
"0.01",x,43,"3.26860"
"0.01",x,44,"3.27830"
"0.01",x,45,"3.28810"
"0.01",x,46,"3.29780"
"0.01",x,47,"3.30750"
"0.01",x,48,"3.31720"
"0.01",x,49,"3.32690"
"0.01",x,50,"3.33660"
"0.01",x,51,"3.34410"
"0.01",x,52,"3.35150"
"0.01",x,53,"3.35900"
"0.01",x,54,"3.36640"
"0.01",x,55,"3.37390"
"0.01",x,56,"3.38130"
"0.01",x,57,"3.38880"
"0.01",x,58,"3.39620"
"0.01",x,59,"3.40370"
"0.01",x,60,"3.41110"
"0.01",x,61,"3.41710"
"0.01",x,62,"3.42310"
"0.01",x,63,"3.42910"
"0.01",x,64,"3.43510"
"0.01",x,65,"3.44110"
"0.01",x,66,"3.44700"
"0.01",x,67,"3.45300"
"0.01",x,68,"3.45900"
"0.01",x,69,"3.46500"
"0.01",x,70,"3.47100"
"0.01",x,71,"3.47600"
"0.01",x,72,"3.48100"
"0.01",x,73,"3.48590"
"0.01",x,74,"3.49090"
"0.01",x,75,"3.49590"
"0.01",x,76,"3.50090"
"0.01",x,77,"3.50590"
"0.01",x,78,"3.51080"
"0.01",x,79,"3.51580"
"0.01",x,80,"3.52080"
"0.01",x,81,"3.52500"
"0.01",x,82,"3.52930"
"0.01",x,83,"3.53350"
"0.01",x,84,"3.53780"
"0.01",x,85,"3.54200"
"0.01",x,86,"3.54620"
"0.01",x,87,"3.55050"
"0.01",x,88,"3.55470"
"0.01",x,89,"3.55900"
"0.01",x,90,"3.56320"
"0.01",x,91,"3.56690"
"0.01",x,92,"3.57060"
"0.01",x,93,"3.57430"
"0.01",x,94,"3.57800"
"0.01",x,95,"3.58170"
"0.01",x,96,"3.58540"
"0.01",x,97,"3.58910"
"0.01",x,98,"3.59280"
"0.01",x,99,"3.59650"
"0.01",x,100,"3.60020"
"0.01",x,101,"3.60330"
"0.01",x,102,"3.60640"
"0.01",x,103,"3.60950"
"0.01",x,104,"3.61250"
"0.01",x,105,"3.61560"
"0.01",x,106,"3.61870"
"0.01",x,107,"3.62180"
"0.01",x,108,"3.62490"
"0.01",x,109,"3.62800"
"0.01",x,110,"3.63110"
"0.01",x,111,"3.63410"
"0.01",x,112,"3.63720"
"0.01",x,113,"3.64030"
"0.01",x,114,"3.64340"
"0.01",x,115,"3.64650"
"0.01",x,116,"3.64960"
"0.01",x,117,"3.65260"
"0.01",x,118,"3.65570"
"0.01",x,119,"3.65880"
"0.01",x,120,"3.66190"
"0.01",x,121,"3.66440"
"0.01",x,122,"3.66690"
"0.01",x,123,"3.66940"
"0.01",x,124,"3.67190"
"0.01",x,125,"3.67450"
"0.01",x,126,"3.67700"
"0.01",x,127,"3.67950"
"0.01",x,128,"3.68200"
"0.01",x,129,"3.68450"
"0.01",x,130,"3.68700"
"0.01",x,131,"3.68950"
"0.01",x,132,"3.69200"
"0.01",x,133,"3.69450"
"0.01",x,134,"3.69700"
"0.01",x,135,"3.69960"
"0.01",x,136,"3.70210"
"0.01",x,137,"3.70460"
"0.01",x,138,"3.70710"
"0.01",x,139,"3.70960"
"0.01",x,140,"3.71210"
"0.01",x,141,"3.71420"
"0.01",x,142,"3.71630"
"0.01",x,143,"3.71840"
"0.01",x,144,"3.72050"
"0.01",x,145,"3.72260"
"0.01",x,146,"3.72470"
"0.01",x,147,"3.72680"
"0.01",x,148,"3.72890"
"0.01",x,149,"3.73100"
"0.01",x,150,"3.73320"
"0.01",x,151,"3.73530"
"0.01",x,152,"3.73740"
"0.01",x,153,"3.73950"
"0.01",x,154,"3.74160"
"0.01",x,155,"3.74370"
"0.01",x,156,"3.74580"
"0.01",x,157,"3.74790"
"0.01",x,158,"3.75000"
"0.01",x,159,"3.75210"
"0.01",x,160,"3.75420"
"0.01",x,161,"3.75600"
"0.01",x,162,"3.75780"
"0.01",x,163,"3.75960"
"0.01",x,164,"3.76140"
"0.01",x,165,"3.76330"
"0.01",x,166,"3.76510"
"0.01",x,167,"3.76690"
"0.01",x,168,"3.76870"
"0.01",x,169,"3.77050"
"0.01",x,170,"3.77230"
"0.01",x,171,"3.77410"
"0.01",x,172,"3.77590"
"0.01",x,173,"3.77770"
"0.01",x,174,"3.77950"
"0.01",x,175,"3.78140"
"0.01",x,176,"3.78320"
"0.01",x,177,"3.78500"
"0.01",x,178,"3.78680"
"0.01",x,179,"3.78860"
"0.01",x,180,"3.79040"
"0.01",x,181,"3.79200"
"0.01",x,182,"3.79360"
"0.01",x,183,"3.79510"
"0.01",x,184,"3.79670"
"0.01",x,185,"3.79830"
"0.01",x,186,"3.79990"
"0.01",x,187,"3.80150"
"0.01",x,188,"3.80300"
"0.01",x,189,"3.80460"
"0.01",x,190,"3.80620"
"0.01",x,191,"3.80780"
"0.01",x,192,"3.80940"
"0.01",x,193,"3.81090"
"0.01",x,194,"3.81250"
"0.01",x,195,"3.81410"
"0.01",x,196,"3.81570"
"0.01",x,197,"3.81730"
"0.01",x,198,"3.81880"
"0.01",x,199,"3.82040"
"0.01",x,200,"3.82200"
"0.01",x,201,"3.82320"
"0.01",x,202,"3.82430"
"0.01",x,203,"3.82550"
"0.01",x,204,"3.82670"
"0.01",x,205,"3.82780"
"0.01",x,206,"3.82900"
"0.01",x,207,"3.83020"
"0.01",x,208,"3.83130"
"0.01",x,209,"3.83250"
"0.01",x,210,"3.83370"
"0.01",x,211,"3.83480"
"0.01",x,212,"3.83600"
"0.01",x,213,"3.83710"
"0.01",x,214,"3.83830"
"0.01",x,215,"3.83950"
"0.01",x,216,"3.84060"
"0.01",x,217,"3.84180"
"0.01",x,218,"3.84300"
"0.01",x,219,"3.84410"
"0.01",x,220,"3.84530"
"0.01",x,221,"3.84650"
"0.01",x,222,"3.84760"
"0.01",x,223,"3.84880"
"0.01",x,224,"3.85000"
"0.01",x,225,"3.85110"
"0.01",x,226,"3.85230"
"0.01",x,227,"3.85350"
"0.01",x,228,"3.85460"
"0.01",x,229,"3.85580"
"0.01",x,230,"3.85700"
"0.01",x,231,"3.85810"
"0.01",x,232,"3.85930"
"0.01",x,233,"3.86040"
"0.01",x,234,"3.86160"
"0.01",x,235,"3.86280"
"0.01",x,236,"3.86390"
"0.01",x,237,"3.86510"
"0.01",x,238,"3.86630"
"0.01",x,239,"3.86740"
"0.01",x,240,"3.86860"
"0.01",x,241,"3.86980"
"0.01",x,242,"3.87090"
"0.01",x,243,"3.87210"
"0.01",x,244,"3.87330"
"0.01",x,245,"3.87440"
"0.01",x,246,"3.87560"
"0.01",x,247,"3.87680"
"0.01",x,248,"3.87790"
"0.01",x,249,"3.87910"
"0.01",x,250,"3.88020"
"0.01",x,251,"3.88140"
"0.01",x,252,"3.88260"
"0.01",x,253,"3.88370"
"0.01",x,254,"3.88490"
"0.01",x,255,"3.88610"
"0.01",x,256,"3.88720"
"0.01",x,257,"3.88840"
"0.01",x,258,"3.88960"
"0.01",x,259,"3.89070"
"0.01",x,260,"3.89190"
"0.01",x,261,"3.89310"
"0.01",x,262,"3.89420"
"0.01",x,263,"3.89540"
"0.01",x,264,"3.89660"
"0.01",x,265,"3.89770"
"0.01",x,266,"3.89890"
"0.01",x,267,"3.90010"
"0.01",x,268,"3.90120"
"0.01",x,269,"3.90240"
"0.01",x,270,"3.90350"
"0.01",x,271,"3.90470"
"0.01",x,272,"3.90590"
"0.01",x,273,"3.90700"
"0.01",x,274,"3.90820"
"0.01",x,275,"3.90940"
"0.01",x,276,"3.91050"
"0.01",x,277,"3.91170"
"0.01",x,278,"3.91290"
"0.01",x,279,"3.91400"
"0.01",x,280,"3.91520"
"0.01",x,281,"3.91640"
"0.01",x,282,"3.91750"
"0.01",x,283,"3.91870"
"0.01",x,284,"3.91990"
"0.01",x,285,"3.92100"
"0.01",x,286,"3.92220"
"0.01",x,287,"3.92340"
"0.01",x,288,"3.92450"
"0.01",x,289,"3.92570"
"0.01",x,290,"3.92680"
"0.01",x,291,"3.92800"
"0.01",x,292,"3.92920"
"0.01",x,293,"3.93030"
"0.01",x,294,"3.93150"
"0.01",x,295,"3.93270"
"0.01",x,296,"3.93380"
"0.01",x,297,"3.93500"
"0.01",x,298,"3.93620"
"0.01",x,299,"3.93730"
"0.01",x,300,"3.93850"
"0.01",x,301,"3.93930"
"0.01",x,302,"3.94010"
"0.01",x,303,"3.94080"
"0.01",x,304,"3.94160"
"0.01",x,305,"3.94240"
"0.01",x,306,"3.94320"
"0.01",x,307,"3.94400"
"0.01",x,308,"3.94470"
"0.01",x,309,"3.94550"
"0.01",x,310,"3.94630"
"0.01",x,311,"3.94710"
"0.01",x,312,"3.94790"
"0.01",x,313,"3.94870"
"0.01",x,314,"3.94940"
"0.01",x,315,"3.95020"
"0.01",x,316,"3.95100"
"0.01",x,317,"3.95180"
"0.01",x,318,"3.95260"
"0.01",x,319,"3.95330"
"0.01",x,320,"3.95410"
"0.01",x,321,"3.95490"
"0.01",x,322,"3.95570"
"0.01",x,323,"3.95650"
"0.01",x,324,"3.95720"
"0.01",x,325,"3.95800"
"0.01",x,326,"3.95880"
"0.01",x,327,"3.95960"
"0.01",x,328,"3.96040"
"0.01",x,329,"3.96110"
"0.01",x,330,"3.96190"
"0.01",x,331,"3.96270"
"0.01",x,332,"3.96350"
"0.01",x,333,"3.96430"
"0.01",x,334,"3.96510"
"0.01",x,335,"3.96580"
"0.01",x,336,"3.96660"
"0.01",x,337,"3.96740"
"0.01",x,338,"3.96820"
"0.01",x,339,"3.96900"
"0.01",x,340,"3.96970"
"0.01",x,341,"3.97050"
"0.01",x,342,"3.97130"
"0.01",x,343,"3.97210"
"0.01",x,344,"3.97290"
"0.01",x,345,"3.97360"
"0.01",x,346,"3.97440"
"0.01",x,347,"3.97520"
"0.01",x,348,"3.97600"
"0.01",x,349,"3.97680"
"0.01",x,350,"3.97760"
"0.01",x,351,"3.97830"
"0.01",x,352,"3.97910"
"0.01",x,353,"3.97990"
"0.01",x,354,"3.98070"
"0.01",x,355,"3.98150"
"0.01",x,356,"3.98220"
"0.01",x,357,"3.98300"
"0.01",x,358,"3.98380"
"0.01",x,359,"3.98460"
"0.01",x,360,"3.98540"
"0.01",x,361,"3.98610"
"0.01",x,362,"3.98690"
"0.01",x,363,"3.98770"
"0.01",x,364,"3.98850"
"0.01",x,365,"3.98930"
"0.01",x,366,"3.99000"
"0.01",x,367,"3.99080"
"0.01",x,368,"3.99160"
"0.01",x,369,"3.99240"
"0.01",x,370,"3.99320"
"0.01",x,371,"3.99400"
"0.01",x,372,"3.99470"
"0.01",x,373,"3.99550"
"0.01",x,374,"3.99630"
"0.01",x,375,"3.99710"
"0.01",x,376,"3.99790"
"0.01",x,377,"3.99860"
"0.01",x,378,"3.99940"
"0.01",x,379,"4.00020"
"0.01",x,380,"4.00100"
"0.01",x,381,"4.00180"
"0.01",x,382,"4.00250"
"0.01",x,383,"4.00330"
"0.01",x,384,"4.00410"
"0.01",x,385,"4.00490"
"0.01",x,386,"4.00570"
"0.01",x,387,"4.00640"
"0.01",x,388,"4.00720"
"0.01",x,389,"4.00800"
"0.01",x,390,"4.00880"
"0.01",x,391,"4.00960"
"0.01",x,392,"4.01040"
"0.01",x,393,"4.01110"
"0.01",x,394,"4.01190"
"0.01",x,395,"4.01270"
"0.01",x,396,"4.01350"
"0.01",x,397,"4.01430"
"0.01",x,398,"4.01500"
"0.01",x,399,"4.01580"
"0.01",x,400,"4.01660"
"0.01",x,401,"4.01720"
"0.01",x,402,"4.01780"
"0.01",x,403,"4.01830"
"0.01",x,404,"4.01890"
"0.01",x,405,"4.01950"
"0.01",x,406,"4.02010"
"0.01",x,407,"4.02070"
"0.01",x,408,"4.02130"
"0.01",x,409,"4.02180"
"0.01",x,410,"4.02240"
"0.01",x,411,"4.02300"
"0.01",x,412,"4.02360"
"0.01",x,413,"4.02420"
"0.01",x,414,"4.02480"
"0.01",x,415,"4.02530"
"0.01",x,416,"4.02590"
"0.01",x,417,"4.02650"
"0.01",x,418,"4.02710"
"0.01",x,419,"4.02770"
"0.01",x,420,"4.02830"
"0.01",x,421,"4.02880"
"0.01",x,422,"4.02940"
"0.01",x,423,"4.03000"
"0.01",x,424,"4.03060"
"0.01",x,425,"4.03120"
"0.01",x,426,"4.03180"
"0.01",x,427,"4.03230"
"0.01",x,428,"4.03290"
"0.01",x,429,"4.03350"
"0.01",x,430,"4.03410"
"0.01",x,431,"4.03470"
"0.01",x,432,"4.03530"
"0.01",x,433,"4.03580"
"0.01",x,434,"4.03640"
"0.01",x,435,"4.03700"
"0.01",x,436,"4.03760"
"0.01",x,437,"4.03820"
"0.01",x,438,"4.03880"
"0.01",x,439,"4.03930"
"0.01",x,440,"4.03990"
"0.01",x,441,"4.04050"
"0.01",x,442,"4.04110"
"0.01",x,443,"4.04170"
"0.01",x,444,"4.04230"
"0.01",x,445,"4.04280"
"0.01",x,446,"4.04340"
"0.01",x,447,"4.04400"
"0.01",x,448,"4.04460"
"0.01",x,449,"4.04520"
"0.01",x,450,"4.04570"
"0.01",x,451,"4.04630"
"0.01",x,452,"4.04690"
"0.01",x,453,"4.04750"
"0.01",x,454,"4.04810"
"0.01",x,455,"4.04870"
"0.01",x,456,"4.04920"
"0.01",x,457,"4.04980"
"0.01",x,458,"4.05040"
"0.01",x,459,"4.05100"
"0.01",x,460,"4.05160"
"0.01",x,461,"4.05220"
"0.01",x,462,"4.05270"
"0.01",x,463,"4.05330"
"0.01",x,464,"4.05390"
"0.01",x,465,"4.05450"
"0.01",x,466,"4.05510"
"0.01",x,467,"4.05570"
"0.01",x,468,"4.05620"
"0.01",x,469,"4.05680"
"0.01",x,470,"4.05740"
"0.01",x,471,"4.05800"
"0.01",x,472,"4.05860"
"0.01",x,473,"4.05920"
"0.01",x,474,"4.05970"
"0.01",x,475,"4.06030"
"0.01",x,476,"4.06090"
"0.01",x,477,"4.06150"
"0.01",x,478,"4.06210"
"0.01",x,479,"4.06270"
"0.01",x,480,"4.06320"
"0.01",x,481,"4.06380"
"0.01",x,482,"4.06440"
"0.01",x,483,"4.06500"
"0.01",x,484,"4.06560"
"0.01",x,485,"4.06620"
"0.01",x,486,"4.06670"
"0.01",x,487,"4.06730"
"0.01",x,488,"4.06790"
"0.01",x,489,"4.06850"
"0.01",x,490,"4.06910"
"0.01",x,491,"4.06970"
"0.01",x,492,"4.07020"
"0.01",x,493,"4.07080"
"0.01",x,494,"4.07140"
"0.01",x,495,"4.07200"
"0.01",x,496,"4.07260"
"0.01",x,497,"4.07320"
"0.01",x,498,"4.07370"
"0.01",x,499,"4.07430"
"0.01",x,500,"4.07490"
"0.01",x,501,"4.07540"
"0.01",x,502,"4.07580"
"0.01",x,503,"4.07630"
"0.01",x,504,"4.07680"
"0.01",x,505,"4.07720"
"0.01",x,506,"4.07770"
"0.01",x,507,"4.07820"
"0.01",x,508,"4.07860"
"0.01",x,509,"4.07910"
"0.01",x,510,"4.07960"
"0.01",x,511,"4.08000"
"0.01",x,512,"4.08050"
"0.01",x,513,"4.08090"
"0.01",x,514,"4.08140"
"0.01",x,515,"4.08190"
"0.01",x,516,"4.08230"
"0.01",x,517,"4.08280"
"0.01",x,518,"4.08330"
"0.01",x,519,"4.08370"
"0.01",x,520,"4.08420"
"0.01",x,521,"4.08470"
"0.01",x,522,"4.08510"
"0.01",x,523,"4.08560"
"0.01",x,524,"4.08610"
"0.01",x,525,"4.08650"
"0.01",x,526,"4.08700"
"0.01",x,527,"4.08750"
"0.01",x,528,"4.08790"
"0.01",x,529,"4.08840"
"0.01",x,530,"4.08890"
"0.01",x,531,"4.08930"
"0.01",x,532,"4.08980"
"0.01",x,533,"4.09020"
"0.01",x,534,"4.09070"
"0.01",x,535,"4.09120"
"0.01",x,536,"4.09160"
"0.01",x,537,"4.09210"
"0.01",x,538,"4.09260"
"0.01",x,539,"4.09300"
"0.01",x,540,"4.09350"
"0.01",x,541,"4.09400"
"0.01",x,542,"4.09440"
"0.01",x,543,"4.09490"
"0.01",x,544,"4.09540"
"0.01",x,545,"4.09580"
"0.01",x,546,"4.09630"
"0.01",x,547,"4.09680"
"0.01",x,548,"4.09720"
"0.01",x,549,"4.09770"
"0.01",x,550,"4.09820"
"0.01",x,551,"4.09860"
"0.01",x,552,"4.09910"
"0.01",x,553,"4.09950"
"0.01",x,554,"4.10000"
"0.01",x,555,"4.10050"
"0.01",x,556,"4.10090"
"0.01",x,557,"4.10140"
"0.01",x,558,"4.10190"
"0.01",x,559,"4.10230"
"0.01",x,560,"4.10280"
"0.01",x,561,"4.10330"
"0.01",x,562,"4.10370"
"0.01",x,563,"4.10420"
"0.01",x,564,"4.10470"
"0.01",x,565,"4.10510"
"0.01",x,566,"4.10560"
"0.01",x,567,"4.10610"
"0.01",x,568,"4.10650"
"0.01",x,569,"4.10700"
"0.01",x,570,"4.10750"
"0.01",x,571,"4.10790"
"0.01",x,572,"4.10840"
"0.01",x,573,"4.10880"
"0.01",x,574,"4.10930"
"0.01",x,575,"4.10980"
"0.01",x,576,"4.11020"
"0.01",x,577,"4.11070"
"0.01",x,578,"4.11120"
"0.01",x,579,"4.11160"
"0.01",x,580,"4.11210"
"0.01",x,581,"4.11260"
"0.01",x,582,"4.11300"
"0.01",x,583,"4.11350"
"0.01",x,584,"4.11400"
"0.01",x,585,"4.11440"
"0.01",x,586,"4.11490"
"0.01",x,587,"4.11540"
"0.01",x,588,"4.11580"
"0.01",x,589,"4.11630"
"0.01",x,590,"4.11680"
"0.01",x,591,"4.11720"
"0.01",x,592,"4.11770"
"0.01",x,593,"4.11810"
"0.01",x,594,"4.11860"
"0.01",x,595,"4.11910"
"0.01",x,596,"4.11950"
"0.01",x,597,"4.12000"
"0.01",x,598,"4.12050"
"0.01",x,599,"4.12090"
"0.01",x,600,"4.12140"

[daskuntal]

Correction:

Instead of appendcols, we need to perform "join ErrorCode", otherwise, the merge is not correct. The appendcols will replace the subsearch results with the main result, without doing any "join", and hence the results will be misleading. Hence you need to join the two results with the common field, "ErrorCode"

Note:

1. I've reduced the 2 input Z-Score table with one input Lookup table, hence no need to Z1 & Z2
2. Since the Z-Score table doesn't have results beyond Z=4.09, I've also added another check for Z > 4.09.

Alert Search:

index=someindex sourcetype=sometype environment!="PREPROD*" Type=someType (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
| stats count as currentErrors by ErrorCode
| join ErrorCode [search index=mobile_summary search_name=summary_error_alert
earliest=-7d@d latest=@d (date_wday!="sunday" AND date_wday!="saturday")
| stats count as numErrors by ErrorCode, _time
| stats avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors count as numPeriods by ErrorCode]
| eval zVal = round(((currentErrors - avgErrors)/stdevErrors), 2)
| lookup z_val_lookup zVal OUTPUT pValue1Tail
| eval significant = if(zVal > 4.09 OR pValue1Tail < .05, "true", "false")
| fields + ErrorCode currentErrors avgErrors stdevErrors numPeriods zVal pValue pValue1Tail significant

[andrewkenth]

This is very valuable! Thanks for posting and documenting this so well. Do you have a copy of the z_val_lookup file you used? I'd hate to recreate the wheel.

[daskuntal]

Here's the final answer. I figured out why my summary search wasn't working, it's because instead of search_name="summary_error_alert", we were doing search="summary_error_alert". duh, it took me sometime to compare my other summary searches to figure out what was wrong :).

Nevertheless, here's the final solution. It's not a perfect one, since there are a lot of assumptions:
1) I can't determine the p-value if Z-Score > 4.09, because the table doesn't have data beyond that, & we don't need it either, since beyond that Z-Score, the probability is so low.
2) The Error distribution is Normal
3) I'm avoiding weekends, when gathering summary data, so that my statistics is not skewed, by very low number of errors over the weekends.
4) However, my Alert search runs every day
5) Critical value = 0.05

Summary Search:

Run conditions:

I run it every 5 mins, and it collects the data between "earliest=-10m@m latest=-5m@m".

Query:

index=someindex sourcetype=somelogs environment!="PREPROD*" Type=someType (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
(date_wday!="sunday" AND date_wday!="saturday")
| bucket _time span=5m
| sistats count as numErrors by ErrorCode, _time

Alert Search:

Alert condition:

Runs every 5 mins
Gathers data from the last 5 mins => earliest=-5m@m latest=now
Alerts on a custom condition, when "significant=true" in the search results

Query:

index=someindex sourcetype=somelogs environment!="PREPROD*" Type=someType (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
| stats count as currentErrors by ErrorCode
| appendcols [search index=mobile_summary search_name=summary_error_alert
earliest=-7d@d latest=@d (date_wday!="sunday" AND date_wday!="saturday")
| stats count as numErrors by ErrorCode, _time
| stats avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors count as numPeriods by ErrorCode]
| eval zVal = round(((currentErrors - avgErrors)/stdevErrors/(sqrt(numPeriods))), 2)
| lookup z_val_lookup zVal OUTPUT pValue1Tail
| eval significant = if(zVal > 4.09 OR pValue1Tail < .05, "true", "false")
| fields + ErrorCode currentErrors avgErrors stdevErrors numPeriods zVal pValue pValue1Tail significant

Please let me know if there is any scope for improvement or any issues with the above solution. I'd appreciate any feedback.

Cheers

[daskuntal]

Timechart messed things up, since it's result had time as the column header. The following search works as expected. Now I just need to put the subsearch in a summary indexed search for better performance, when gathering large quantities of data:

index=someindex sourcetype=somelogs environment!="PREPROD*" Type=sometype (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
| stats count as currentErrors by ErrorCode
| appendcols [search index=someindex sourcetype=somelogs environment!="PREPROD*" Type=sometype (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
| bucket _time span=5m
| stats count as numErrors, sum(counter) as numErrors by ErrorCode, _time
| stats avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors count as numPeriods by ErrorCode]
| eval zVal = (currentErrors - avgErrors)/stdevErrors/(sqrt(numPeriods))
| eval Z1 = substr(zVal, 1, 3)
| eval Z2 = substr(zVal, 4, 1)
| lookup z_val_lookup Z1,Z2 OUTPUT P as pValue
| eval significant = if(pValue < .05, "true", "false")
| fields + ErrorCode currentErrors avgErrors stdevErrors numPeriods zVal pValue significant

[daskuntal]

Thanks Guinn,

We are getting close.... but not there yet. I'm having a hard time collecting data from the summary index within a subsearch.

On inspecting my actual alert search, it appeared that I'm getting 0 results for the subsearch:
search index=some_summary search=summary_error_alert earliest=-7d@d latest=@d
| timechart span=5m sum(counter) as numErrors by ErrorCode
| stats count as numPeriods avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors by Errorcode

However, I validated that my summary search(which runs every 5 mins(earliest=-10m@m & latest=-5m@m) does indeed return data once I set the time range for the last 7 days.

What I noticed is that if I change the search to the following(source=summary_error_alert), instead of search=summary_error_alert), then it returns some data. With that in mind, when I run the following section of the search, it only returns two fields, "ErrorCode" & currentErrrs:

index=someindex sourcetype=somesource environment!="someenv*" Type=sometype (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
| stats count as currentErrors by ErrorCode
| appendcols [search index=some_summary search=summary_error_alert earliest=-7d@d latest=@d
| timechart span=5m sum(counter) as numErrors by ErrorCode
| stats count as numPeriods avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors by Errorcode ]

Results:

ErrorCode currentErrors
1 2010 1
2 2011 5
3 2013 6
4 2100 1
5 2102 8
6 2201 7
....
....

I can't put my finger on it, but it's either the timechart/counter values, or the subsearch/summary search, that's screwing things up.

I'm still researching.. if I find the issue, I'll post it. any help is appreciated.

Basically the final result should look something like this:

ErrorCode-currentErrors-avgErrors-stdevErrors-numPeriods-zVal-pValue-significant
2010-1-7.6-12.4-1567-.12-.03-true
2011-5-7.6-12.4-1567-.23-.07-false

I think we need to use eventstats, instead of stats, since the avg, std & numPeriods should be the same value in all the events/rows, and available for all the events/rows, in order to calculate the zVals for each row.

Any help is appreciated..

Kuntal

[lguinn2]

[Okay, I deleted my original dumb answer, and left the second one. This one is the third.]

So your searches are now very close, but I don't think that your summary index is going to give you what you want. A summary indexing search that ends with sistats count as numErrors by ErrorCode
should be retrieved by the following command

index=some_summary search=summary_error_alert earliest=-7d@d latest=@d 
| stats count as numErrors by ErrorCode

And that isn't going to give you the stats that you need. But you can try it. (Run the error_summary search for an hour or two and then try searching the summary index.)

Here is what I think you need:

error_summary:

  index=someindex sourcetype=somesource environment!="someenv*" Type=someType (ErrorCode!="null" AND 
  ErrorCode!=200 AND ErrorCode!=0 ) | eval counter=1 | 
  sitimechart span=5m sum(counter) by ErrorCode

The timechart command will summarize based on the 5-minute intervals and return a series. The stats command will summarize overall and return a single value. I think you need the series.

Alert Search:

index=someindex sourcetype=somesource environment!="someenv*" Type=sometype (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
| stats count as currentErrors by ErrorCode
| appendcols [search index=some_summary search=summary_error_alert earliest=-7d@d latest=@d 
| timechart span=5m sum(counter) as numErrors by ErrorCode
| stats count as numPeriods avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors by Errorcode ]
| eval zVal = (currentErrors - avgErrors)/stdevErrors/(sqrt(numPeriods)) // e,g, z=2.54
| eval zValFirstDec = substr(zVal, 1, 3) //2.5
| eval zValSecDec = "0.0"+substr(zVal, -1)//0.04
| lookup zTable zValFirstDec, zValSecDec OUTPUT pValue
| eval significant = pValue < .05
| fields + currentErrors avgErrors stdevErrors numPeriods zVal significant pValue

The lookup question:

You can totally do a multi-key lookup in Splunk, just as you have written it. But the CSV file needs to be in this format:

zValFirstDec,zValSecDec,pValue

Not in a matrix format.

[daskuntal]

Thanks. I was thinking in the same line. But I'd still prefer to use Summary Indexing instead of a live search, since to gather data for last 7 days will take a very long time. Hence I'd collect the data for every 5 min interval in the Summary Index, and use a separate search to calculate the statistics using the Summary search....

In order to calculate the exact p-value, we need either
1) Integral function in Splunk, or
2) The ability to lookup the Z-Table in SPlunk. Currently, I'm aware of looking up a 2-dimensional table in Splunk, but in order to lookup a Z-Table, we need to match the row & the column values: Is there any way to do that?

e.g. for each X, Z = (X-avg)/stddev
Lookup the p-value from the Z-score table(I'll save that table as a lookup table in Splunk lookup), and then determine if the p-value < alpha(=.05). If it is, for the 1-tailed test, then we can determine, if the result is significant.

Here's what I came up with:

error_summary:

index=someindex sourcetype=somesource environment!="someenv*" Type=someType (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)| sistats count as numErrors by ErrorCode

Alert Search:

index=someindex sourcetype=somesource environment!="someenv*" Type=sometype (ErrorCode!="null" AND ErrorCode!=200 AND ErrorCode!=0)
| stats count as currentErrors by ErrorCode
| appendcols [search index=some_summary search=summary_error_alert earliest=-7d@d latest=@d
| stats avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors count as numPeriods by ErrorCode]
| eval zVal = (currentErrors - avgErrors)/stdevErrors/(sqrt(numPeriods)) // e,g, z=2.54
| eval zValFirstDec = substr(zVal, 1, 3) //2.5
| eval zValSecDec = "0.0"+substr(zVal, -1)//0.04
| lookup zTable zValFirstDec, zValSecDec OUTPUT pValue
| eval significant = if(pValue < .05, "true", "false")
| fields + currentErrors avgErrors stdevErrors numPeriods zVal significant

Now the only thing that I'm not sure of is the Splunk lookup for row & column, in the Z-Table, where you look for the first two values:
"lookup zTable zValFirstDec, zValSecDec OUTPUT pValue"

I don't know if and/or how to do that in Splunk =>

A new question for Splunk: Can we do lookup in a Matrix table with two values(Row value & column value)?

Does that look like something we can do?

[lguinn2]

Bah - my original answer was poor, and I have realized that there is a simpler solution. This calculates a mean number of errors over the last 7 days, then creates an upper bound that is two standard deviations from the mean. If the number of errors is greater than the upper bound, the number is statistically significant.

Try this, no summary index is needed:

your-error-condition | stats count as currentErrors | 
appendcols [search your-error-condition earliest=-7d@d lastest=@d |
timechart span=5m count as errorCount | 
stats avg(errorCount) as avgErrors std(errorCount) as stdevErrors count as numPeriods |
eval maxVal = avgErrors + (stdevErrors * 2) |
eval significant = currentErrors > maxVal |
fields + currentErrors avgErrors stdevErrors numPeriods significant

Note that I am only testing for a statistically significant increase in errors, although using the standard deviation makes this a two-tailed statistical test. There are also a few other statistical functions that might be of interest to you here -

Functions for stats, chart and timechart

Functions for eval and where

These may help you calculate the actual T-test or Chi-Square.

[daskuntal]

Thanks for the response. That sounds like a workaround that we can incorporate. It isn't really a statistical analysis such as a T-Test or a Chi-Test. But for now this is better than simply running an alert with a threshold count. SO, thanks for that.

Note:
I am yet to set it up & collect the data first for the 7 days & then prepare the alert.

However I have a few followup questions:

1. When you say, don't use "'sistats' count" in the summary search, what do you mean? SHold all Summary Indexing be using "sistats". I understand that when I'm using the Summary Indexed search in another search, then I'll use "stats", instead of "sisstats". Am I correct?
2. If I'm not mistaken then the Error Summary should look like "your error condition | sistats count as numErrors"
3. The actual alert search should look like this: your error condition | stats count as currentErrors | appendcols [search index=summary search=ErrorSummary earliest=-7d@d latest=@d | stats avg(numErrors) as avgErrors stdev(numErrors) as stdevErrors count as numPeriods | eval maxVal = avgErrors + (stdevErrors * 2) | eval significant = currentErrors > maxVal | fields + currentErrors avgErrors stdevErrors numPeriods significant

Please let me know if I missed anything ...

I'll report back, once I try the whole thing.

THanks again for yoru help