Getting Data In

Does a new sourcetype stanza need to exist in a place besides the Forwarder before ingesting?

Builder

I was under the impression I could define sourcetypes in props.conf on the forwarder, which would then send that data and the sourcetype information to the indexers. It looks like it does this, at least for the name of the sourcetype, but the rest of the fields are not being taken into account (such at timestamp lookahead).

Am I wrong? Do I need to create the sourcetype on the master and push first, before forwarding the data?

0 Karma
1 Solution

Path Finder

The parsing of data happens either on the Splunk Indexer or the Splunk Heavy Forwarder.

Splunk Universal Forwarder does not handle parsing of data except in cases where parameter such as INDEXED_EXTRACTIONS has been used for structured data (CSV, JSON etc..).

If the sourcetype specified in the inputs.conf on the Splunk UF was not declared in the props.conf in the Splunk Indexer or Splunk HF, the attributes of the sourcetype will take all the default props.conf settings (LINEBREAKER, TIMEFORMAT, MAXTIMESTAMPLOOKAHEAD etc...).

Hence, it is Splunk Best Practice to create the sourcetype with the appropriate attributes catered for the particular data in the props.conf on Splunk Indexer or Splunk Heavy Forwarder before the input of data has begun.

I would recommended going through some of the data pipeline related documentations to have a better understanding of how data are being processed in Splunk.

Components and the data pipeline
http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Componentsofadistributedenvironment

How data moves through Splunk deployments: The data pipeline
https://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Datapipeline

HowIndexingWorks
http://wiki.splunk.com/Community:HowIndexingWorks

View solution in original post

Path Finder

The parsing of data happens either on the Splunk Indexer or the Splunk Heavy Forwarder.

Splunk Universal Forwarder does not handle parsing of data except in cases where parameter such as INDEXED_EXTRACTIONS has been used for structured data (CSV, JSON etc..).

If the sourcetype specified in the inputs.conf on the Splunk UF was not declared in the props.conf in the Splunk Indexer or Splunk HF, the attributes of the sourcetype will take all the default props.conf settings (LINEBREAKER, TIMEFORMAT, MAXTIMESTAMPLOOKAHEAD etc...).

Hence, it is Splunk Best Practice to create the sourcetype with the appropriate attributes catered for the particular data in the props.conf on Splunk Indexer or Splunk Heavy Forwarder before the input of data has begun.

I would recommended going through some of the data pipeline related documentations to have a better understanding of how data are being processed in Splunk.

Components and the data pipeline
http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Componentsofadistributedenvironment

How data moves through Splunk deployments: The data pipeline
https://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Datapipeline

HowIndexingWorks
http://wiki.splunk.com/Community:HowIndexingWorks

View solution in original post

Builder

Thanks. This makes sense.

0 Karma

SplunkTrust
SplunkTrust

Sourcetypes need to be defined in props.conf on the indexers and, if you have them, heavy forwarders.

---
If this reply helps you, an upvote would be appreciated.