Splunk daily license volume not resetting at midnight.

[mgiddens]

Good afternoon.

I am having an issue with my daily license volume used not resetting at midnight.
I have created an outputs.conf file on the license master (also my master node) and restarted the splunk service on that instance using the following example:

Turn off indexing on the forwarder
[indexAndForward]
index = false

[tcpout]
defaultGroup = my_peers_nodes
forwardedindex.filter.disable = true

indexAndForward = false

[tcpout:my_peers_nodes]
server=x.x.x.x:9997,x.x.x.x:9997,x.x.x.x:9997

This is from: https://docs.splunk.com/Documentation/Splunk/7.2.6/Indexer/Forwardmasterdata

This was mentioned as a fix to this issue from another forum if your in distributed deployment, which we are.
Is this all I need for the outputs.conf?
Will I have to wait until midnight when the volume used is supposed to reset before I see if this fixed my issue?
Has anyone else faced this issue and how did you fix it?

Thank you.

mgiddens

[woodcock]

You need to restart the Splunk instance on your License Server and open the firewall to make sure the data is not blocked. That fixes it for us when we see it.

[mgiddens]

Thank you for the response. I have restarted and the ports are open that I am using on both ends. Where exactly to I put the outputs.conf? I have several locations where one exist so is that ok? My indexers have SSL stanzas in their inputs.conf, so I would assume I would put ssl stanzas in the output.conf on the instances I want to forward right? Or does that matter? Do I need to put a blacklist / whitelist stanza or not? I get a message that 'says something to the effect of TCP outprecessor has paused data flow; forwarding to my "indexergroup" has been blocked for 10 seconds. Reviewing receiving end health.' What should I check for that on the receiving end?

Also, does forwarding the data from the license master to indexers really fix the issue I am seeing where my license usage doesn't reset every night to 0?
It stays constant each day and license usage logs states that the only app I had configured was reporting in when data related to that app was being ingested.

Beating my head against a wall on this one.

Thank you.

[woodcock]

Your MC should have the same outputs.conf file as your UFs do so that it sends all of its logs to the Indexers. There is NOTHING special required.

[mgiddens]

Thanks woodcock! I thought about using the same format as what was on the UF's since those were working at one time. I looked in the splunkd.log file and saw this:

ERROR TcpInputProc - Error encountered for connection from src={forwarderip}:43479. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

SSL stanzas do exist on the inputs.conf on the indexer so I assume this means indexers are looking for SSL connection from the MC when it forwards. I have the output.conf config with SSL stanzas as well that it needs. I got this info from the outputs.conf manual and had a previous file that was on the server someone else has created.

So why am I getting the SSL23 error?
I also see the indexers are coming in and out of "quarantine" in one of the logs when the failures to forward occur?
Could that be due to the SSL issues?

I assume there is something wrong with my SSL configs in the conf files.

Any ideas or pointers on what the SSL stanzas should look like in input and output conf file when including the SSL stanzas?

Thanks again.

Mike

[woodcock]

This is a HUGE questions and a completely different one that you asked in this link. Setting up SSL is a major pain in Splunk.

[mgiddens]

Update: I removed an outputs.conf file on the indexers /local and the message no longer shows up. Still don't see the data from the server I am forwarding from when I search the indexers through my search head.