How does indexer cluster replication affect license usage?

[geoppspl7]

For about a week, two of our indexers were not replicating to their slaves - oddly this reduced our license usage by half(p2) and we did not really see any events missing. Once the replication of the two indexes was re-enabled - the usage doubled to our usual levels.

I've raised a case with Splunk support, but I can't really get a straight answer. Essentially, what I would like to know is if we are penalized for every replica?

Heavy Forwarders and Universal forwarders are configured to forward events to both indexers so in theory, if an index is not replicating, we should still ingest logs. [we have two indexers and two large indexes: network and security, so one indexer will hold the primary copy of the index and the other the slave copy]

[ddrillic]

Obviously, what you observed doesn't make any sense - Managing Indexers and Clusters of Indexers

1: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Systemrequirements -
says -

[MuS]

Hi geoppspl7,

I don't know why you cannot get a straight answer, because it is pretty straight forward:

All _raw data that hits any indexer will count against your license.
See the docs for more details http://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/TypesofSplunklicenses#Licenses_for_indexer_c...

This is important to you:

Only incoming data counts against the license; replicated data does not.
That said, check if

• you're forwarding the data between the indexers and force it back into the parsing queue (not sure if, but it could get you wired results)
• check if you're instead of load balancing the events over your two indexers are sending a duplicate stream form the UF's, check your outputs.conf

Hope this helps ...

cheers, MuS

[geoppspl7]

Hi MuS,

Thank you for your comments.

The indexes.conf on the master(C:\Program Files\Splunk\etc\master-apps_cluster\local) sets "repFactor=auto", which should result in the index's data to be replicated to other peers in the cluster, and I've not come across any other settings.

As far as the outputs.conf is concerned on the universal forwarders - the way I understand this is that we are not cloning data, but merely balancing between the two indexers:

[tcpout:indexers]
disabled=false
autoLBFrequency=40
server=10.250.156.60:9997,10.250.156.61:9997
useACK=true

Similarly the following is the outputs.conf from a heavy forwarder that handles syslog traffic. Here however I don't see useACK - does that mean that if index replica was disabled = some events will not have been indexed and therefore lost?

[tcpout]
defaultGroup = default-autolb-group

[tcpout-server://10.250.156.60:9997]

[tcpout:default-autolb-group]
disabled = false
server = 10.250.156.60:9997,10.250.156.61:9997

[tcpout-server://10.250.156.61:9997]

[MuS]

the outputs.conf of the HWF is cloning the events to 10.250.156.60 and 10.250.156.61 and at the same time using them in a load balanced config......

Just use it like this :

 [tcpout]
 defaultGroup = default-autolb-group

 [tcpout:default-autolb-group]
 disabled = false
 autoLBFrequency = 30
 server = 10.250.156.60:9997,10.250.156.61:9997

[geoppspl7]

I've changed the file as suggested, but having restarted Splunk on that heavy forwarder I don't see any event count decrease. I run "* source="udp:514" index=network | timechart count" which are ingested via that heavy forwarder...

[rvany]

This is old - but: do you still think that the shown outputs.conf on th HFW is cloning events? If so, why?

To my knowledge a [tcpout-server:<ipaddr>] stanza could be used for server specific configuration. But that's only in addition to the settings done in a server group, i.e. something like [tcpout:my-1st-indexer-group].

[MuS]

This was just something I observed - it could have been a reason why things go weird.

• Check the _internal as long as you have it for the top sending sourcetype or host and try to figure out where the data originates and what path it takes into the indexers.
• Verify all involved conf files using the btool command http://docs.splunk.com/Documentation/Splunk/6.5.1/Troubleshooting/Usebtooltotroubleshootconfiguratio...
• check _internal for any information related to bucket replication.

This is not everything, but gives you a starting point...