Installation

_internal index replication causes inaccurate license reports

Path Finder

All,

We noticed something very strange with our reporting. We have recently transitioned to an indexer cluster. We have always had a report that is sent out with yesterdays license usage totals. However, a day after transitioning our reports now show double the license consumption when running searches like : index=_internal source="*license_usage.log" sourcetype=splunkd type="Usage" | stats sum(b) as bytes| eval GB=round(bytes/1024/1024/1024) |fields GB| rename poolszg as "Daily License Quota" GB as "Daily License Quota Used"

Our repfactor and search factor is both set at 2 which correlates with the doubling we see per index.

What seemed to correct our reports was disabling repFactor = auto and setting it to repFactor=0 for the _internal index found under the _cluster app. Since disabling this, our license consumption now matches the trend we were seeing prior to the cluster move.

Has anyone else had this issue with inaccurate license totals when having repFactor=auto set on _internal index?

Labels (2)
0 Karma
1 Solution

Builder

Hi jordanking1992,

this is how splunk calculates the license usage from _internal:

index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d   | eval _time=_time - 43200 | bin _time span=1d | stats latest(b) AS b by slave, pool, _time | timechart span=1d sum(b) AS "volume" fixedrange=false | join type=outer _time [search index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d | eval _time=_time - 43200 | bin _time span=1d | stats latest(stacksz) AS "stack size" by _time] | fields - _timediff  | foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]

Note the macro set_local_host that limits the results to those from one machine. I assume what you see is not a bug, it's a feature of splunk to store license usage in multiple places for redundancy. Maybe you could adapt the search from the management dashboard for your report in order to be consistent with the number splunk uses internally.

Hope it helps.
Oliver

View solution in original post

0 Karma

Builder

Hi jordanking1992,

this is how splunk calculates the license usage from _internal:

index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d   | eval _time=_time - 43200 | bin _time span=1d | stats latest(b) AS b by slave, pool, _time | timechart span=1d sum(b) AS "volume" fixedrange=false | join type=outer _time [search index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d | eval _time=_time - 43200 | bin _time span=1d | stats latest(stacksz) AS "stack size" by _time] | fields - _timediff  | foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]

Note the macro set_local_host that limits the results to those from one machine. I assume what you see is not a bug, it's a feature of splunk to store license usage in multiple places for redundancy. Maybe you could adapt the search from the management dashboard for your report in order to be consistent with the number splunk uses internally.

Hope it helps.
Oliver

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Hi jordanking1992
if you don't set repFactor=auto you don't use cluster!
Using the License consuption report [Settings -- Licensing -- License Usage Reporting] what's your result?
I see differences with your search.

Bye.
Giuseppe

0 Karma

Path Finder

okay its displaying half of what my search is reporting...aka what is should be. So if we enabled replication of _internal, why is the search above sayings is double what i see in t [Settings -- Licensing -- License Usage Reporting]

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!