Getting Data In

Where are Source type definitions stored in Distributed environment?

iparitosh
Path Finder

Where are Source type definitions stored in Distributed environment? and How to manage them?

For example -

When I create a sourcetype using Splunk Web - I am assuming that they are being stored at Search Heads and later somehow migrated to indexers. I can edit them via splunk web UI directly. But Web UI won't show any source types defined in my indexers.

If I want to achieve selective parsing (ex: index only WARN logs), then since event processing occurs at indexer, I will have to define this sourcetype at indexers. And these change won't be visible to me at all since indexers are not accessible via splunk web. ( I am using splunkcloud) This is a major pain point in improving data quality since the sourcetype configuration is not all all visible to us.

What is the best practise, how do i manage my source types especially in case of splunkcloud. Is there any way to keep sourcetypes on indexers and Search Heads synchronised?

0 Karma

woodcock
Esteemed Legend

Sourcetypes are not defined so they are not stored anywhere; any arbitrary string can be used. Each one is simply a field value that is generally hard-coded inside of the inputs.conf file that defines the input. Sometimes a sourcetype is overriden by settings inside of props.conf on the first full instance of Splunk Enterprise that handles the event.

0 Karma

iparitosh
Path Finder

What do you mean by “first full value of splunk”?

0 Karma

woodcock
Esteemed Legend

The spellchecker changed instance to value somehow. I fixed it.

0 Karma

skalliger
SplunkTrust
SplunkTrust

Hi,

regarding sourcetypes: They are stored on the search heads, correct. Sourcetypes are getting indexed in the .tsidx files. That's why the indexers know about them.
Search peers (indexers) get a so-called knowledge bundle with information about the search from the Search Head(s) when a search is being ran.

Regarding parsing: you're right. Everything that needs to be done before indexing an event, has to go on the first instance that does parsing, in your case an indexer (which could also be a Heavy Forwarder).
For an overview, Aplura has a nice cheat sheet about "where to put props": https://www.aplura.com/wp-content/uploads/where_to_put_props.pdf

Skalli

iparitosh
Path Finder

Thank you for sharing the document. I have one doubt.

Most of my sourcetypes are defined at search head via splunk web UI. They use LINE_BREAKER attributes like - BREAK_ONLY_BEFORE, LINE _BREAKER etc.

When I make changes into any of these attributes, I can see changes in events breaking within few minutes.

Now, these LINE BREAKING key:value pairs needs to be utilised by indexer before the events are indexed.

How is it that I make a change on Splunk Web UI and it get's reflected in event processing while the changes should have been made at indexer level. Any idea how it works in backend?

0 Karma

skalliger
SplunkTrust
SplunkTrust

Are you talking about an on-premise or cloud environment? Actually, the configurations you mentioned have to go to the indexers to take effect.

Skalli

0 Karma

iparitosh
Path Finder

We are on splunkcloud, when I create a sourcetypes with LINEBREAKER setting via splunk UI, does it stored at SH or the INDEXER.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.