Getting Data In

Splunk learned app should stop learning / too_small / dispatch.evaluate

FRoth
Contributor

I defined a new input folder that receives gzipped server logs from a scp copy job on our servers.

inputs.conf

[monitor://F:\ssh_incoming\Logs]
disabled = false
recursive = false
host_regex = pl\-([^_]+)_
index = plserver
sourcetype = syslog

The problem is that the props.conf of the learned app grows and grows until dispatching of new searches takes longer and longer until no searches start at all. (Inspect Job shows that dispatch.evaluate takes more than ten minutes to complete)

/learned/local/props.conf

[pl-www1_20160303053001_sudo-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

[pl-www1_20160303053001_system-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

[pl-www1_20160303060001_crond-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

[pl-www1_20160303060001_sshd-too_small]
PREFIX_SOURCETYPE = True
SHOULD_LINEMERGE = False
is_valid = True
maxDist = 9999

So as long as logs keep incoming in that folder, the props.conf grows and grows. I even disabled the "learned" App but that didn't solve the problem. The file keeps growing.

I have also tried to set LEARN_SOURCETYPE to false in the props.conf definition for the "syslog" sourcetype.

[syslog]
CHARSET = latin-1
LEARN_SOURCETYPE = false

I am using Splunk 6.2.2 with Enterprise License.

risgupta_splunk
Splunk Employee
Splunk Employee

If all the sourcetypes are marked as {{-too_small}}.

For small files Splunk is unable to determine the type. This can be controlled by:

{noformat:title=props.conf}
[too_small]
PREFIX_SOURCETYPE = false
{noformat}

Could you please try to add a new entry in props.conf with below configuration and restart splunk:

[too_small]
PREFIX_SOURCETYPE = false

This above configuration will not grow the sourcetypes in learned app and .gz file will also read and forwarded by the splunk.

ncsantucci
Path Finder

If you want to disable to learned app see this post:

https://answers.splunk.com/answers/77271/make-splunk-stop-learning-sourcetypes.html

Here is the relevant part of the post:

If you really would like to disable learning, edit $SPLUNK_HOME/etc/apps/learned/local/app.conf and make sure it says this:

[install]
state = disabled

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...