Getting Data In

Heavy Forwarder, License Exceeded and Log Rotation

mkallies
Path Finder

Complex question here.

I have the following set up:

Universal forwarder[20G rotating file] -> Heavy Forwarder[props.conf, transforms.conf] -> Splunk Light [ 5G license ]

I'm tuning the heavy forwarder for filtering the log file (we only need a fraction of the log for analysis). props.conf and transforms.conf seems to be working now, but there was an error in it yesterday, causing it to send EVERYTHING, so we quickly exceeded our 5G Spunk Light limit and saw indexing stop.

In preparation for today's limit, yesterday I edited our props.conf and transforms.conf, but it seems to have filtered out everything. Some tweaking today, and a restart of the heavy forwarder and I see event flows in the Forwarder again... but..

I'm not seeing growth in today's events, I'm seeing growth in yesterday's events

Question 1:

Does the license exceeded cause the universal forwarder to pause? I would have expected events to drop.

Question 2:

Why did the filtered out events on the heavy forwarder's transforms.conf and props.conf not cause indexing to start at the time of the heavy forwarder restart?

Question 3:

How does the Universal forwarder handle log rotation? (Linux logrotate) For now, because the forwarder has not been restarted, I can see that it's reading the old log file, even though it's been rotated out (see output of lsof):

 splunkd   30174      root   46r      REG              253,2 36499146450 1073741961 /data/log/messages-20160513 (deleted)
0 Karma
1 Solution

Jeremiah
Motivator

Indexing should not have stopped when you exceeded your license:


http://docs.splunk.com/Documentation/SplunkLight/latest/Installation/Aboutlicensing

Warnings
If you exceed your daily indexing volume on any calendar day, you get a warning. The message persists for fourteen days. You have until midnight to resolve it before it counts against the total number of warnings within the rolling 30-day period.

Violations
If you have five or more warnings in a rolling 30-day period, you are in violation of your license. During a license violation period:

Splunk Light continues to index your data.
Search is disabled, except for searches to the _internal index.
Although you cannot search existing or incoming data inputs, you can still use search to troubleshoot the licensing issue.

Search capabilities return when you have fewer than five warnings in the previous 30 days or when you apply a reset license.


Indexing might have stopped for other reasons, for example because your disk filled up. Or it could be that throughput was throttled from the universal forwarder. The UF has a default limit of 256 kbps (https://answers.splunk.com/answers/53138/maximum-traffic-of-a-universal-forwarder.html) that you may need to increase. Or it could be that it just appeared to stop, but in reality was just falling behind.

For whatever reason, if your indexer isn't available, the HF will detect this and back off sending data. This will cause the UF to begin to back off as well, and so none of your data is lost. If you rotate files, Splunk can also detect this, and read from the rotated files.

View solution in original post

Jeremiah
Motivator

Indexing should not have stopped when you exceeded your license:


http://docs.splunk.com/Documentation/SplunkLight/latest/Installation/Aboutlicensing

Warnings
If you exceed your daily indexing volume on any calendar day, you get a warning. The message persists for fourteen days. You have until midnight to resolve it before it counts against the total number of warnings within the rolling 30-day period.

Violations
If you have five or more warnings in a rolling 30-day period, you are in violation of your license. During a license violation period:

Splunk Light continues to index your data.
Search is disabled, except for searches to the _internal index.
Although you cannot search existing or incoming data inputs, you can still use search to troubleshoot the licensing issue.

Search capabilities return when you have fewer than five warnings in the previous 30 days or when you apply a reset license.


Indexing might have stopped for other reasons, for example because your disk filled up. Or it could be that throughput was throttled from the universal forwarder. The UF has a default limit of 256 kbps (https://answers.splunk.com/answers/53138/maximum-traffic-of-a-universal-forwarder.html) that you may need to increase. Or it could be that it just appeared to stop, but in reality was just falling behind.

For whatever reason, if your indexer isn't available, the HF will detect this and back off sending data. This will cause the UF to begin to back off as well, and so none of your data is lost. If you rotate files, Splunk can also detect this, and read from the rotated files.

mkallies
Path Finder

Great advice as always. Glad to hear I was wrong about the license being the cause... it stopped squarely at 5GB, it looks like it might be one of those confusing coincidences. My filters are strict enough now that I shouldn't exceed the license over the next few days, I'll see how the rotation goes over the next few nights.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...