Getting Data In

Does a new sourcetype stanza need to exist in a place besides the Forwarder before ingesting?

thisissplunk
Builder

I was under the impression I could define sourcetypes in props.conf on the forwarder, which would then send that data and the sourcetype information to the indexers. It looks like it does this, at least for the name of the sourcetype, but the rest of the fields are not being taken into account (such at timestamp lookahead).

Am I wrong? Do I need to create the sourcetype on the master and push first, before forwarding the data?

0 Karma
1 Solution

BenTan
Path Finder

The parsing of data happens either on the Splunk Indexer or the Splunk Heavy Forwarder.

Splunk Universal Forwarder does not handle parsing of data except in cases where parameter such as INDEXED_EXTRACTIONS has been used for structured data (CSV, JSON etc..).

If the sourcetype specified in the inputs.conf on the Splunk UF was not declared in the props.conf in the Splunk Indexer or Splunk HF, the attributes of the sourcetype will take all the default props.conf settings (LINE_BREAKER, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD etc...).

Hence, it is Splunk Best Practice to create the sourcetype with the appropriate attributes catered for the particular data in the props.conf on Splunk Indexer or Splunk Heavy Forwarder before the input of data has begun.

I would recommended going through some of the data pipeline related documentations to have a better understanding of how data are being processed in Splunk.

Components and the data pipeline
http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Componentsofadistributedenvironment

How data moves through Splunk deployments: The data pipeline
https://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Datapipeline

HowIndexingWorks
http://wiki.splunk.com/Community:HowIndexingWorks

View solution in original post

BenTan
Path Finder

The parsing of data happens either on the Splunk Indexer or the Splunk Heavy Forwarder.

Splunk Universal Forwarder does not handle parsing of data except in cases where parameter such as INDEXED_EXTRACTIONS has been used for structured data (CSV, JSON etc..).

If the sourcetype specified in the inputs.conf on the Splunk UF was not declared in the props.conf in the Splunk Indexer or Splunk HF, the attributes of the sourcetype will take all the default props.conf settings (LINE_BREAKER, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD etc...).

Hence, it is Splunk Best Practice to create the sourcetype with the appropriate attributes catered for the particular data in the props.conf on Splunk Indexer or Splunk Heavy Forwarder before the input of data has begun.

I would recommended going through some of the data pipeline related documentations to have a better understanding of how data are being processed in Splunk.

Components and the data pipeline
http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Componentsofadistributedenvironment

How data moves through Splunk deployments: The data pipeline
https://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Datapipeline

HowIndexingWorks
http://wiki.splunk.com/Community:HowIndexingWorks

View solution in original post

thisissplunk
Builder

Thanks. This makes sense.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Sourcetypes need to be defined in props.conf on the indexers and, if you have them, heavy forwarders.

---
If this reply helps you, an upvote would be appreciated.
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!