Getting Data In
Highlighted

Where are Source type definitions stored in Distributed environment?

Path Finder

Where are Source type definitions stored in Distributed environment? and How to manage them?

For example -

When I create a sourcetype using Splunk Web - I am assuming that they are being stored at Search Heads and later somehow migrated to indexers. I can edit them via splunk web UI directly. But Web UI won't show any source types defined in my indexers.

If I want to achieve selective parsing (ex: index only WARN logs), then since event processing occurs at indexer, I will have to define this sourcetype at indexers. And these change won't be visible to me at all since indexers are not accessible via splunk web. ( I am using splunkcloud) This is a major pain point in improving data quality since the sourcetype configuration is not all all visible to us.

What is the best practise, how do i manage my source types especially in case of splunkcloud. Is there any way to keep sourcetypes on indexers and Search Heads synchronised?

0 Karma
Highlighted

Re: Where are Source type definitions stored in Distributed environment?

SplunkTrust
SplunkTrust

Hi,

regarding sourcetypes: They are stored on the search heads, correct. Sourcetypes are getting indexed in the .tsidx files. That's why the indexers know about them.
Search peers (indexers) get a so-called knowledge bundle with information about the search from the Search Head(s) when a search is being ran.

Regarding parsing: you're right. Everything that needs to be done before indexing an event, has to go on the first instance that does parsing, in your case an indexer (which could also be a Heavy Forwarder).
For an overview, Aplura has a nice cheat sheet about "where to put props": https://www.aplura.com/wp-content/uploads/where_to_put_props.pdf

Skalli

Highlighted

Re: Where are Source type definitions stored in Distributed environment?

Path Finder

Thank you for sharing the document. I have one doubt.

Most of my sourcetypes are defined at search head via splunk web UI. They use LINEBREAKER attributes like - BREAKONLY_BEFORE, LINE _BREAKER etc.

When I make changes into any of these attributes, I can see changes in events breaking within few minutes.

Now, these LINE BREAKING key:value pairs needs to be utilised by indexer before the events are indexed.

How is it that I make a change on Splunk Web UI and it get's reflected in event processing while the changes should have been made at indexer level. Any idea how it works in backend?

0 Karma
Highlighted

Re: Where are Source type definitions stored in Distributed environment?

SplunkTrust
SplunkTrust

Are you talking about an on-premise or cloud environment? Actually, the configurations you mentioned have to go to the indexers to take effect.

Skalli

0 Karma
Highlighted

Re: Where are Source type definitions stored in Distributed environment?

Path Finder

We are on splunkcloud, when I create a sourcetypes with LINEBREAKER setting via splunk UI, does it stored at SH or the INDEXER.

0 Karma
Highlighted

Re: Where are Source type definitions stored in Distributed environment?

Esteemed Legend

Sourcetypes are not defined so they are not stored anywhere; any arbitrary string can be used. Each one is simply a field value that is generally hard-coded inside of the inputs.conf file that defines the input. Sometimes a sourcetype is overriden by settings inside of props.conf on the first full instance of Splunk Enterprise that handles the event.

0 Karma
Highlighted

Re: Where are Source type definitions stored in Distributed environment?

Path Finder

What do you mean by “first full value of splunk”?

0 Karma
Highlighted

Re: Where are Source type definitions stored in Distributed environment?

Esteemed Legend

The spellchecker changed instance to value somehow. I fixed it.

0 Karma