How to use Transaction and the sum of the same name field in a rangemap and run stats?

slatta — Mon, 10 Aug 2015 15:18:33 GMT

Trying to use the sum of "docCount" in a transaction and use that value for the range and then run statistics by the range. The transaction is working as expected and get the correct logs entries. If I just run the command without worrying about the sum of docCounts the count and stats work as expected, but only get a docCount max of 50. When adding the sum(docCount), the count sums, but lose the ability to run stats on the "duration" field from the log.

//Range counts properly, but no stats

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") | transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false | stats sum(docCount) as doccounter by TSJobId|rangemap field=doccounter "0-5Docs"=0-5, "6-10Docs"=6-10, "11-20Docs"=11-20, "21-30Docs"=21-30, "31-40Docs"=31-40, "41-50Docs"=41-50, "51-60"=51-60, "60-80"=60-80, default=">80Docs" | stats count, avg(duration), p75(duration), p80(duration), p90(duration), p95(duration) by range

//Stats work, but missing the actual docCount

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") | transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false | rangemap field=doccounter "0-5Docs"=0-5, "6-10Docs"=6-10, "11-20Docs"=11-20, "21-30Docs"=21-30, "31-40Docs"=31-40, "41-50Docs"=41-50, "51-60"=51-60, "60-80"=60-80, default=">80Docs" | stats count, avg(duration), p75(duration), p80(duration), p90(duration), p95(duration) by range

Result of Transaction command (Important Fields: TSJobId, docCount, duration):

[2015-08-07 14:31:06,220] DEBUG Component=TSThread="DefaultMessageListenerContainer-9" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:31:07,614] DEBUG Component=TSThread="DefaultMessageListenerContainer-9" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 Starting module=Commit docCount=7
[2015-08-07 14:31:07,614] DEBUG Component=TSThread="pool-2-thread-27" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 Starting module=Commit docCount=50
[2015-08-07 14:31:07,614] DEBUG Component=TSThread="pool-3-thread-20" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 Starting module=Commit docCount=50
[2015-08-07 14:31:31,223] DEBUG Component=TSThread="DefaultMessageListenerContainer-9" TSJobId=20150807_143106_5910 MessageId= REQ_ID=94700114 FinishedJob tsJobType=ingest duration=25004

[2015-08-07 14:26:07,646] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142607_5888 MessageId= REQ_ID=94696492 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:26:08,349] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142607_5888 MessageId= REQ_ID=94696492 Starting module=Commit docCount=23
[2015-08-07 14:26:28,867] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142607_5808 MessageId= REQ_ID=94696492 FinishedJob tsJobType=ingest duration=11221


[2015-08-07 14:25:03,846] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:25:05,343] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 Starting module=Commit docCount=3
[2015-08-07 14:25:05,343] DEBUG Component=TSThread="pool-2-thread-26" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 Starting module=Commit docCount=50
[2015-08-07 14:25:05,343] DEBUG Component=TSThread="pool-3-thread-11" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 Starting module=Commit docCount=50
[2015-08-07 14:25:28,377] DEBUG Component=TSThread="DefaultMessageListenerContainer-7" TSJobId=20150807_142503_5808 MessageId= REQ_ID=94696492 FinishedJob tsJobType=ingest duration=24532

[2015-08-07 14:23:59,434] DEBUG Component=TSThread="DefaultMessageListenerContainer-6" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 StartingJob tsJobType=ingest sync=false
[2015-08-07 14:24:01,094] DEBUG Component=TSThread="DefaultMessageListenerContainer-6" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 Starting module=Commit docCount=1
[2015-08-07 14:24:01,095] DEBUG Component=TSThread="pool-2-thread-22" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 Starting module=Commit docCount=50
[2015-08-07 14:24:24,164] DEBUG Component=TSThread="DefaultMessageListenerContainer-6" TSJobId=20150807_142359_5814 MessageId= REQ_ID=94695906 FinishedJob tsJobType=ingest duration=24730

Re: How to use Transaction and the sum of the same name field in a rangemap and run stats?

somesoni2 — Mon, 10 Aug 2015 17:15:06 GMT

Try something like this

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") | transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false | eventstats sum(docCount) as doccounter by TSJobId| rangemap field=doccounter "0-5Docs"=0-5, "6-10Docs"=6-10, "11-20Docs"=11-20, "21-30Docs"=21-30, "31-40Docs"=31-40, "41-50Docs"=41-50, "51-60"=51-60, "60-80"=60-80, default=">80Docs" | stats count, avg(duration), p75(duration), p80(duration), p90(duration), p95(duration) by range

Re: How to use Transaction and the sum of the same name field in a rangemap and run stats?

lguinn2 — Mon, 10 Aug 2015 17:29:32 GMT

In the second search, you use "doccounter" as the field for the rangemap, but the field does not exist. So the rangemap won't work in the second search. In the first search, you calculated "doccounter" before using it. As a debugging exercise: what do you get when you run -

index=ts* Component=TS ("StartingJob" OR "Starting module=Commit" OR "FinishedJob") 
| transaction TSJobId startswith="StartingJob" endswith="FinishedJob" keepevicted=false 
| stats sum(docCount) as doccounter by TSJobId
| sort -doccounter

Re: How to use Transaction and the sum of the same name field in a rangemap and run stats?

slatta — Mon, 10 Aug 2015 18:05:36 GMT

Changing from stats sum(docCount) to eventstats sum(docCount) seems to have made the difference. Thanks.

topic How to use Transaction and the sum of the same name field in a rangemap and run stats? in Splunk Search

How to use Transaction and the sum of the same name field in a rangemap and run stats?

Re: How to use Transaction and the sum of the same name field in a rangemap and run stats?

Re: How to use Transaction and the sum of the same name field in a rangemap and run stats?

Re: How to use Transaction and the sum of the same name field in a rangemap and run stats?