Getting Data In

Does a new sourcetype stanza need to exist in a place besides the Forwarder before ingesting?

thisissplunk
Builder

I was under the impression I could define sourcetypes in props.conf on the forwarder, which would then send that data and the sourcetype information to the indexers. It looks like it does this, at least for the name of the sourcetype, but the rest of the fields are not being taken into account (such at timestamp lookahead).

Am I wrong? Do I need to create the sourcetype on the master and push first, before forwarding the data?

0 Karma
1 Solution

BenTan
Path Finder

The parsing of data happens either on the Splunk Indexer or the Splunk Heavy Forwarder.

Splunk Universal Forwarder does not handle parsing of data except in cases where parameter such as INDEXED_EXTRACTIONS has been used for structured data (CSV, JSON etc..).

If the sourcetype specified in the inputs.conf on the Splunk UF was not declared in the props.conf in the Splunk Indexer or Splunk HF, the attributes of the sourcetype will take all the default props.conf settings (LINE_BREAKER, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD etc...).

Hence, it is Splunk Best Practice to create the sourcetype with the appropriate attributes catered for the particular data in the props.conf on Splunk Indexer or Splunk Heavy Forwarder before the input of data has begun.

I would recommended going through some of the data pipeline related documentations to have a better understanding of how data are being processed in Splunk.

Components and the data pipeline
http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Componentsofadistributedenvironment

How data moves through Splunk deployments: The data pipeline
https://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Datapipeline

HowIndexingWorks
http://wiki.splunk.com/Community:HowIndexingWorks

View solution in original post

BenTan
Path Finder

The parsing of data happens either on the Splunk Indexer or the Splunk Heavy Forwarder.

Splunk Universal Forwarder does not handle parsing of data except in cases where parameter such as INDEXED_EXTRACTIONS has been used for structured data (CSV, JSON etc..).

If the sourcetype specified in the inputs.conf on the Splunk UF was not declared in the props.conf in the Splunk Indexer or Splunk HF, the attributes of the sourcetype will take all the default props.conf settings (LINE_BREAKER, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD etc...).

Hence, it is Splunk Best Practice to create the sourcetype with the appropriate attributes catered for the particular data in the props.conf on Splunk Indexer or Splunk Heavy Forwarder before the input of data has begun.

I would recommended going through some of the data pipeline related documentations to have a better understanding of how data are being processed in Splunk.

Components and the data pipeline
http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Componentsofadistributedenvironment

How data moves through Splunk deployments: The data pipeline
https://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Datapipeline

HowIndexingWorks
http://wiki.splunk.com/Community:HowIndexingWorks

thisissplunk
Builder

Thanks. This makes sense.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Sourcetypes need to be defined in props.conf on the indexers and, if you have them, heavy forwarders.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...