topic Re: Where are Source type definitions stored in Distributed environment? in Getting Data In

Where are Source type definitions stored in Distributed environment?

iparitosh — Thu, 04 Jul 2019 04:40:50 GMT

Where are Source type definitions stored in Distributed environment? and How to manage them?

For example -

When I create a sourcetype using Splunk Web - I am assuming that they are being stored at Search Heads and later somehow migrated to indexers. I can edit them via splunk web UI directly. But Web UI won't show any source types defined in my indexers.

If I want to achieve selective parsing (ex: index only WARN logs), then since event processing occurs at indexer, I will have to define this sourcetype at indexers. And these change won't be visible to me at all since indexers are not accessible via splunk web. ( I am using splunkcloud) This is a major pain point in improving data quality since the sourcetype configuration is not all all visible to us.

What is the best practise, how do i manage my source types especially in case of splunkcloud. Is there any way to keep sourcetypes on indexers and Search Heads synchronised?

Re: Where are Source type definitions stored in Distributed environment?

skalliger — Thu, 04 Jul 2019 05:52:18 GMT

Hi,

regarding sourcetypes: They are stored on the search heads, correct. Sourcetypes are getting indexed in the .tsidx files. That's why the indexers know about them.
Search peers (indexers) get a so-called knowledge bundle with information about the search from the Search Head(s) when a search is being ran.

Regarding parsing: you're right. Everything that needs to be done before indexing an event, has to go on the first instance that does parsing, in your case an indexer (which could also be a Heavy Forwarder).
For an overview, Aplura has a nice cheat sheet about "where to put props": https://www.aplura.com/wp-content/uploads/where_to_put_props.pdf

Skalli

Re: Where are Source type definitions stored in Distributed environment?

iparitosh — Wed, 30 Sep 2020 01:10:45 GMT

Thank you for sharing the document. I have one doubt.

Most of my sourcetypes are defined at search head via splunk web UI. They use LINE_BREAKER attributes like - BREAK_ONLY_BEFORE, LINE _BREAKER etc.

When I make changes into any of these attributes, I can see changes in events breaking within few minutes.

Now, these LINE BREAKING key:value pairs needs to be utilised by indexer before the events are indexed.

How is it that I make a change on Splunk Web UI and it get's reflected in event processing while the changes should have been made at indexer level. Any idea how it works in backend?

Re: Where are Source type definitions stored in Distributed environment?

skalliger — Fri, 05 Jul 2019 09:46:53 GMT

Are you talking about an on-premise or cloud environment? Actually, the configurations you mentioned have to go to the indexers to take effect.

Skalli

Re: Where are Source type definitions stored in Distributed environment?

woodcock — Sat, 06 Jul 2019 22:20:06 GMT

Sourcetypes are not defined so they are not stored anywhere; any arbitrary string can be used. Each one is simply a field value that is generally hard-coded inside of the inputs.conf file that defines the input. Sometimes a sourcetype is overriden by settings inside of props.conf on the first full instance of Splunk Enterprise that handles the event.

Re: Where are Source type definitions stored in Distributed environment?

iparitosh — Sun, 07 Jul 2019 01:00:56 GMT

What do you mean by “first full value of splunk”?

Re: Where are Source type definitions stored in Distributed environment?

iparitosh — Sun, 07 Jul 2019 01:03:17 GMT

We are on splunkcloud, when I create a sourcetypes with LINEBREAKER setting via splunk UI, does it stored at SH or the INDEXER.

Re: Where are Source type definitions stored in Distributed environment?

woodcock — Sun, 07 Jul 2019 01:40:32 GMT

The spellchecker changed instance to value somehow. I fixed it.