Complex question here.
I have the following set up:
Universal forwarder[20G rotating file] -> Heavy Forwarder[props.conf, transforms.conf] -> Splunk Light [ 5G license ]
I'm tuning the heavy forwarder for filtering the log file (we only need a fraction of the log for analysis). props.conf and transforms.conf seems to be working now, but there was an error in it yesterday, causing it to send EVERYTHING, so we quickly exceeded our 5G Spunk Light limit and saw indexing stop.
In preparation for today's limit, yesterday I edited our props.conf and transforms.conf, but it seems to have filtered out everything. Some tweaking today, and a restart of the heavy forwarder and I see event flows in the Forwarder again... but..
I'm not seeing growth in today's events, I'm seeing growth in yesterday's events
Does the license exceeded cause the universal forwarder to pause? I would have expected events to drop.
Why did the filtered out events on the heavy forwarder's transforms.conf and props.conf not cause indexing to start at the time of the heavy forwarder restart?
How does the Universal forwarder handle log rotation? (Linux logrotate) For now, because the forwarder has not been restarted, I can see that it's reading the old log file, even though it's been rotated out (see output of lsof):
splunkd 30174 root 46r REG 253,2 36499146450 1073741961 /data/log/messages-20160513 (deleted)
Indexing should not have stopped when you exceeded your license:
If you exceed your daily indexing volume on any calendar day, you get a warning. The message persists for fourteen days. You have until midnight to resolve it before it counts against the total number of warnings within the rolling 30-day period.
If you have five or more warnings in a rolling 30-day period, you are in violation of your license. During a license violation period:
Splunk Light continues to index your data.
Search is disabled, except for searches to the _internal index.
Although you cannot search existing or incoming data inputs, you can still use search to troubleshoot the licensing issue.
Search capabilities return when you have fewer than five warnings in the previous 30 days or when you apply a reset license.
Indexing might have stopped for other reasons, for example because your disk filled up. Or it could be that throughput was throttled from the universal forwarder. The UF has a default limit of 256 kbps (https://answers.splunk.com/answers/53138/maximum-traffic-of-a-universal-forwarder.html) that you may need to increase. Or it could be that it just appeared to stop, but in reality was just falling behind.
For whatever reason, if your indexer isn't available, the HF will detect this and back off sending data. This will cause the UF to begin to back off as well, and so none of your data is lost. If you rotate files, Splunk can also detect this, and read from the rotated files.
Great advice as always. Glad to hear I was wrong about the license being the cause... it stopped squarely at 5GB, it looks like it might be one of those confusing coincidences. My filters are strict enough now that I shouldn't exceed the license over the next few days, I'll see how the rotation goes over the next few nights.