Getting Data In

Splunk learned app should stop learning / too_small / dispatch.evaluate

FRoth
Contributor

I defined a new input folder that receives gzipped server logs from a scp copy job on our servers.

inputs.conf

[monitor://F:\ssh_incoming\Logs]
disabled = false
recursive = false
host_regex = pl\-([^_]+)_
index = plserver
sourcetype = syslog

The problem is that the props.conf of the learned app grows and grows until dispatching of new searches takes longer and longer until no searches start at all. (Inspect Job shows that dispatch.evaluate takes more than ten minutes to complete)

/learned/local/props.conf

[pl-www1_20160303053001_sudo-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

[pl-www1_20160303053001_system-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

[pl-www1_20160303060001_crond-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

[pl-www1_20160303060001_sshd-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

So as long as logs keep incoming in that folder, the props.conf grows and grows. I even disabled the "learned" App but that didn't solve the problem. The file keeps growing.

I have also tried to set LEARN_SOURCETYPE to false in the props.conf definition for the "syslog" sourcetype.

[syslog]
CHARSET = latin-1
LEARN_SOURCETYPE = false

I am using Splunk 6.2.2 with Enterprise License.

risgupta_splunk
Splunk Employee
Splunk Employee

If all the sourcetypes are marked as {{-too_small}}.

For small files Splunk is unable to determine the type. This can be controlled by:

{noformat:title=props.conf}
[too_small]
PREFIX_SOURCETYPE = false
{noformat}

Could you please try to add a new entry in props.conf with below configuration and restart splunk:

[too_small]
PREFIX_SOURCETYPE = false

This above configuration will not grow the sourcetypes in learned app and .gz file will also read and forwarded by the splunk.

ncsantucci
Path Finder

If you want to disable to learned app see this post:

https://answers.splunk.com/answers/77271/make-splunk-stop-learning-sourcetypes.html

Here is the relevant part of the post:

If you really would like to disable learning, edit $SPLUNK_HOME/etc/apps/learned/local/app.conf and make sure it says this:

[install]
state = disabled

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...