Knowledge Management

sourcetype question and confusion

rchittip
Path Finder

I would like to understand the sourcetype usage scenario in splunk for forwarders, Indexers and search head.

In my environment I have all individual dedicated splunk servers.

Search Head - 2
Indexers (In load balance) -5
Universal Forwarders - 3k

I usually configure sourcetype in forwarders inputs.conf using my deployment server.

Scenario 1:
Using above approach, can I go for the field extraction at search head without creating the same sourcetype on the searchhead?

Scenario 2:
Let's say I have few fields configured with sourcetype on the searchhead. what if the case if I delete the sourcetype on searchhead but the same sourcetype is being forwarded from UF. Should I loose all my field extractions?

Scenario 3:
If I need to do masking on logs, should I have to create same sourcetype on the Indexers?

Request to suggest the best practice in creating the sourcetype in splunk components searchhead , Indexers and forwarder.

Thanks,
Ramu Chittiprolu

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

Here is how sourcetype is used in each of Splunk components

Universal Forwarders - This does monitoring of data and tag the data with the sourcetype defined in the inputs.conf file. Since you're using a sourcetype already, you're good

*Indexers * - With topology of yours, the Indexers will do the data parsing and indexing. The sourcetype definition, as configured in props.conf file on Indexer, will be used to process the events, which includes event breaking, timestamp parsing , index-time field extraction, data masking etc . If you're using a custom sourcetype (not using built-in sourcetypes), then it's recommended that you configure your indexers to have sourcetype definitions. Without this Splunk will use it's default setting to parse the data, which can be, based on your data, can be inefficient.

Search Head - All search-time field extraction for a sourcetype would happen here, so they should be setup here (in props.conf with or without transforms.conf).

Scenario 1:
Using above approach, can I go for the field extraction at search head without creating the same sourcetype on the searchhead?
[Ans: Not sure I get it. A field extraction (on Search Head) is tied to a sourcetype/source/host. So you can create it for source/host without creating stanza for the sourcetype (not sure if this is what you were looking for)]

Scenario 2:
Let's say I have few fields configured with sourcetype on the searchhead. what if the case if I delete the sourcetype on searchhead but the same sourcetype is being forwarded from UF. Should I loose all my field extractions?
[Ans: If Field extraction are setup at sourcetype level on Search Head, you'll loose them when you delete them]

Scenario 3:
If I need to do masking on logs, should I have to create same sourcetype on the Indexers?
[Ans: As mentioned before, data masking happens at Indexer level, so you've create necessary configuration on indexer to do your masking]

View solution in original post

pruthvikrishnap
Contributor

Hi Ramu,

if you are planning to rename any sourcetypes on searchhead, or planning to work on differently named source types same at search time below documentation gives you more insight.

https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/Renamesourcetypes

somesoni2
Revered Legend

Here is how sourcetype is used in each of Splunk components

Universal Forwarders - This does monitoring of data and tag the data with the sourcetype defined in the inputs.conf file. Since you're using a sourcetype already, you're good

*Indexers * - With topology of yours, the Indexers will do the data parsing and indexing. The sourcetype definition, as configured in props.conf file on Indexer, will be used to process the events, which includes event breaking, timestamp parsing , index-time field extraction, data masking etc . If you're using a custom sourcetype (not using built-in sourcetypes), then it's recommended that you configure your indexers to have sourcetype definitions. Without this Splunk will use it's default setting to parse the data, which can be, based on your data, can be inefficient.

Search Head - All search-time field extraction for a sourcetype would happen here, so they should be setup here (in props.conf with or without transforms.conf).

Scenario 1:
Using above approach, can I go for the field extraction at search head without creating the same sourcetype on the searchhead?
[Ans: Not sure I get it. A field extraction (on Search Head) is tied to a sourcetype/source/host. So you can create it for source/host without creating stanza for the sourcetype (not sure if this is what you were looking for)]

Scenario 2:
Let's say I have few fields configured with sourcetype on the searchhead. what if the case if I delete the sourcetype on searchhead but the same sourcetype is being forwarded from UF. Should I loose all my field extractions?
[Ans: If Field extraction are setup at sourcetype level on Search Head, you'll loose them when you delete them]

Scenario 3:
If I need to do masking on logs, should I have to create same sourcetype on the Indexers?
[Ans: As mentioned before, data masking happens at Indexer level, so you've create necessary configuration on indexer to do your masking]

ddrillic
Ultra Champion

What are you trying to achieve?

-- Using above approach, can I go for the field extraction at search head without creating the same sourcetype on the searchhead?

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...