Does a new sourcetype stanza need to exist in a place besides the Forwarder before ingesting?

[thisissplunk]

I was under the impression I could define sourcetypes in props.conf on the forwarder, which would then send that data and the sourcetype information to the indexers. It looks like it does this, at least for the name of the sourcetype, but the rest of the fields are not being taken into account (such at timestamp lookahead).

Am I wrong? Do I need to create the sourcetype on the master and push first, before forwarding the data?

[BenTan · Accepted solution]

The parsing of data happens either on the Splunk Indexer or the Splunk Heavy Forwarder.

Splunk Universal Forwarder does not handle parsing of data except in cases where parameter such as INDEXED_EXTRACTIONS has been used for structured data (CSV, JSON etc..).

If the sourcetype specified in the inputs.conf on the Splunk UF was not declared in the props.conf in the Splunk Indexer or Splunk HF, the attributes of the sourcetype will take all the default props.conf settings (LINE_BREAKER, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD etc...).

Hence, it is Splunk Best Practice to create the sourcetype with the appropriate attributes catered for the particular data in the props.conf on Splunk Indexer or Splunk Heavy Forwarder before the input of data has begun.

I would recommended going through some of the data pipeline related documentations to have a better understanding of how data are being processed in Splunk.

Components and the data pipeline
http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Componentsofadistributedenvironment

How data moves through Splunk deployments: The data pipeline
https://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Datapipeline

HowIndexingWorks
http://wiki.splunk.com/Community:HowIndexingWorks

[thisissplunk]

Thanks. This makes sense.

[richgalloway]

Sourcetypes need to be defined in props.conf on the indexers and, if you have them, heavy forwarders.