Directive ***SPLUNK*** ignored when indexing (too-small issue ?)

[LetMeR00t]

Hi everyone,

I'm stuck with an issue I can't understand...

I created an app that use a custom alert action which generate events to log (this is generating a file under $SPLUNK_HOME$/var/spool/). An example of the file could be:

Name: 1664448416_92764.stash_sourcetype1

***SPLUNK*** index="myindex" host="Host1" source="Source1"
==##~~##~~ 1E8N3D4E6V5E7N2T9 ~~##~~##==
{...event...}

I have setup an input.conf which is looking for this file:

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_sourcetype1]
queue = stashparsing
sourcetype = stash_sourcetype1
move_policy = sinkhole
crcSalt = <SOURCE>

Under my props.conf, I have :

[stash_sourcetype1]
TRUNCATE = 0
# only look for ***SPLUNK*** on the first line
HEADER_MODE = firstline
# we can summary index past data, but rarely future data
MAX_DAYS_AGO = 10000
# 5 years difference between two events
MAX_DIFF_SECS_AGO = 155520000
MAX_DIFF_SECS_HENCE = 155520000
TIME_PREFIX = (?m)^\*{3}Common\sAction\sModel\*{3}.*$
MAX_TIMESTAMP_LOOKAHEAD = 25
LEARN_MODEL = false
# break .stash_new custom format into events
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = false
LINE_BREAKER = (\r?\n==##~~##~~ 1E8N3D4E6V5E7N2T9 ~~##~~##==\r?\n)
KV_MODE = json
TRANSFORMS-0parse_cam_header = orig_action_name_for_stash_cam,orig_sid_for_stash_cam,orig_rid_for_stash_cam,sourcetype_for_stash_cam
TRANSFORMS-1sinkhole_cam_header = sinkhole_cam_header

As you can see, I have configured my props.conf to read the first line "***SPLUNK***" in order to recover the index, host and source.

However, it continues to log all logs in the "main" index and use default values for "source" and "host". It's like it's ignoring this directive whereas it should take it into account.

Does someone knows why it's ignoring this directive please ? I can't find so much documentation on this issue...

For your information, I'm working on a standalone version of Splunk Enterprise.

Thank you

EDIT: I've just noticed that my events are indexed using the sourcetype "stash_sourcetype1-too_small", this can be the reason why but why is it adding the "too_small" and how can I prevent it ?

[richgalloway]

The "-too_small" suffix in a sourcetype means Splunk didn't find the sourcetype in props.conf so it tried to guess what the sourcetype is based on the data, but there wasn't enough data to make that guess.

Verify the props.conf file is in the right place on your indexers (and HFs, if you have them).  Use btool to confirm the settings.

splunk btool --debug props list stash_sourcetype1

[LetMeR00t]

Thank you for your reply,

When I'm executing your command, I have the right properties regarding my sourcetype :

C:\Program Files\Splunk\bin>splunk btool --debug props list stash_sourcetype1
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf [stash_sourcetype1]
C:\Program Files\Splunk\etc\system\default\props.conf               ADD_EXTRA_TIME_FIELDS = True
C:\Program Files\Splunk\etc\system\default\props.conf               ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf               AUTO_KV_JSON = true
C:\Program Files\Splunk\etc\system\default\props.conf               BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf BREAK_ONLY_BEFORE_DATE = false
C:\Program Files\Splunk\etc\system\default\props.conf               CHARSET = AUTO
C:\Program Files\Splunk\etc\system\default\props.conf               DATETIME_CONFIG = \etc\datetime.xml
C:\Program Files\Splunk\etc\system\default\props.conf               DEPTH_LIMIT = 1000
C:\Program Files\Splunk\etc\system\default\props.conf               DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf HEADER_MODE = firstline
C:\Program Files\Splunk\etc\system\default\props.conf               LB_CHUNK_BREAKER_TRUNCATE = 2000000
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf LEARN_MODEL = false
C:\Program Files\Splunk\etc\system\default\props.conf               LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf LINE_BREAKER = (\r?\n==##~~##~~  1E8N3D4E6V5E7N2T9 ~~##~~##==\r?\n)
C:\Program Files\Splunk\etc\system\default\props.conf               LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf               MATCH_LIMIT = 100000
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf MAX_DAYS_AGO = 10000
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf MAX_DIFF_SECS_AGO = 155520000
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf MAX_DIFF_SECS_HENCE = 155520000
C:\Program Files\Splunk\etc\system\default\props.conf               MAX_EVENTS = 256
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 25
C:\Program Files\Splunk\etc\system\default\props.conf               MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf               MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf               MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf               SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf               SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf               SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf               SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf               SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf               SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf SHOULD_LINEMERGE = false
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf TIME_PREFIX = (?m)^\*{3}Common\sAction\sModel\*{3}.*$
C:\Program Files\Splunk\etc\system\default\props.conf               TRANSFORMS =
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf TRANSFORMS-0parse_cam_header = orig_action_name_for_stash_cam,orig_sid_for_stash_cam,orig_rid_for_stash_cam,sourcetype_for_stash_cam
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf TRANSFORMS-1sinkhole_cam_header = sinkhole_cam_header
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf TRUNCATE = 0
C:\Program Files\Splunk\etc\apps\myApp\local\props.conf description = A custom sourcetype for The Watch logs
C:\Program Files\Splunk\etc\system\default\props.conf               detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\system\default\props.conf               maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf               priority =
C:\Program Files\Splunk\etc\system\default\props.conf               sourcetype =
C:\Program Files\Splunk\etc\system\default\props.conf               termFrequencyWeightedDist = false

I see no issue on this configuration.

You say that it tries to guest the sourcetype but the inputs.conf is clearly indicating which sourcetype to use for such kind of files, isn't it ?

Moreover, I tried to execute my script by using the default value (stash_common_action_model) and it's working as expected

[LetMeR00t · Accepted solution]

I indirectly answer to myself.

I found the issue...

As I'm on a Windows workstation, I need to put the equivalent path in my inputs.conf :

[batch://$SPLUNK_HOME/var/spool/splunk/...stash_sourcetype1]
queue       = stashparsing
sourcetype  = stash_sourcetype1
move_policy = sinkhole
crcSalt     = <SOURCE>

[batch://$SPLUNK_HOME\var\spool\splunk\...stash_sourcetype1]
queue       = stashparsing
sourcetype  = stash_sourcetype1
move_policy = sinkhole
crcSalt     = <SOURCE>

And now it's working as expected ... I think that it was just looking for any file in the spool/splunk by default and that a generic rule applied because no modular input was existing.

Thank you for your help