Why is source type stuck on too_small?

[tretrigh]

In our distributed enterprise Splunk environment we have a log file being generated on each Splunk host (indexers, search head, deployment server, etc) located at: /opt/splunk/var/log/splunk/foo.log

By default this gets logged to _internal using the foo-too_small source type.

We now want to change the source type to one we created (my:custom:sourcetype).  I have created the following props.conf file on the deployment server as a custom app and deployed successfully via apply cluster-bundle.  However, new log data is still being associated with the existing source type of foo-too_small.  We also set the local.meta file (under metadata) for permissions.

I have verified this file is making it to the indexers in peer-apps.

[my:custom:sourcetype]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 25

[source::.../var/log/splunk/foo.log]
sourcetype = my:custom:sourcetype

Questions:

1. Why isn't this working?
2. What needs to be done instead to change to a custom source type?

Thank you in advance!

[tretrigh · Accepted solution]

Answering my own question here:

1. Several indexers were not automatically getting the new source type applied for unknown reasons.  I was specifically looking at one which was not.  A reboot of each indexer missing the source type resolved the issue.  A splunkd restart would probably have been sufficient.  All indexers are working as intended.
2. I added the app to each splunk host (SH, deployment server, etc) which defines the new source type.  A debug refresh populated the new source type correctly on each host.  I incorrectly assumed that the app's presence on the indexers would affect the data coming from each of the splunk hosts.

[gcusello]

Hi @tretrigh ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[gcusello]

Hi @tretrigh,

check if in your forwarders, there's some input without sourcetype definition.

You can do it analyzing host and source values.

Assign the correct sourcetype to all your inputs.

ciao.

Giuseppe

[tretrigh]

Thank you for the reply.  Do you have any specific guidance on how to apply the correct source type to our data in our situation?

[gcusello]

Hi @tretrigh,

sometimes (I don't know why) there's some situation when in the add-on isn't defined the sourcetype, so Splunk automatuically assign the sourcetype based on its knowledge and sometimes it cannot find the correct one.

So analyze your logs where there's a too small sourcetype, find the Add-On with that input and manually assign the correct one in the add-on.

Ciao.

Giuseppe

[tretrigh]

Thanks for the reply @gcusello .  In this situation there is no add on.  The log file on each Splunk host is generated by a script we wrote.  We have attempted to manually define the source type for this specific log unsuccessfully.  Do you have any suggestions for how to correctly manually define the source type other than what we've already done?  Thank you for the assistance!

[richgalloway]

Every Splunk input should have a sourcetype assigned to it.

Every sourcetype needs to be defined in a props.conf file.

Every props.conf stanza should have the "Great Eight" attributes, at a minimum.

[mysourcetype]
TIME_PREFIX = 
TIME_FORMAT = 
MAX_TIMESTAMP_LOOKAHEAD = 
TRUNCATE = 
SHOULD_LINEMERGE = false
LINE_BREAKER =
EVENT_BREAKER = 
EVENT_BREAKER_ENABLE = true

Set values for each attribute that correspond to the data being ingested.

[PickleRick]

https://docs.splunk.com/Documentation/Splunk/Latest/Admin/Propsconf#Small_file_settings

[tretrigh]

Thank you for the reply.  I might be missing something obvious, but unsure how any of these settings might help us reassign the source type to something else.  Could you please provide further elaboration?  Thank you!