Where do I configure my various Splunk settings?

s2_splunk — Wed, 09 Sep 2015 23:37:51 GMT

I can never remember where I need to configure my various Splunk settings. Some need to be on the forwarder side, some on the indexers and I even sometimes need them on the search head...

So where do they really need to be?

Re: Where do I configure my various Splunk settings?

s2_splunk — Tue, 29 Sep 2020 07:14:25 GMT

In many environments there are a lot of different Splunk servers performing different roles. For example:

Forwarders
Indexers
Search Heads
Summarizers

When we want Splunk to do something, we can find out which configuration file, what settings, and what values to set in the Administration Manual. However it is not always clear which server the settings need to be on, especially for indexing data, and especially with the props.conf and transforms.conf file settings.

Phases of the Splunk data life cycle
To understand this, we first have to understand the different stages of the data life cycle in Splunk. These main phases for the purposes of understanding configuration are:
This topic is also in the docs here

Input
The '''Input''' phase acquires the raw data stream from its source and annotates it with source-wide ''keys''. The ''keys'' are values that apply to the entire input source overall, and includes the host, source, and sourcetype of the data. The keys may also include values that are used internally by Splunk such as the character encoding of the data stream, and values that can control later processing of the data, such as the index into which the events should be stored.

During this phase, Splunk does not look at the contents of the data stream, so key fields must apply to the entire source, and not to individual events. In fact, at this point, Splunk has no notion of individual events at all, only a stream of data with certain global properties.

Structured Data parsing
Since splunk 6, some source can be parsed for structured data (like headers, or json) and be populated at the forwarder level.
see the docs here
Those settings have to be on the forwarders (and indexers if they monitor files)

Parsing
The '''Parsing''' phases looks at, analyzes, and transforms the data. The parsing phase has many sub-phases:
- Breaking the stream of data into individual lines
- Identifying, parsing, and setting time stamps
- Annotating individual events with metadata copied from the source-wide source, host, sourcetype, and other keys
- Transforming event data and metadata according to Splunk regex transform rules

Indexing
The '''Indexing''' phase takes the events as annotated with metadata and after transformations and writes it into the search index.

Search
Search is probably easier to understand and distinguish from the other phases, but configuration for search is similar to and often combined with that for input and parsing.

Other phases
A couple of other phases and sub-phases:
- Routing
- Jobs
- Expiration
also govern the data life cycle, but for the sake of simplification will not be discussed in this article.

Which Splunk servers go with which phases
Here are how some common Splunk server configurations correspond to these phases:

Universal/Light Forwarder -> Indexer
Input                     -> Parsing, Indexing, Search
================================================================================================
Heavy Forwarder           -> Indexer
Input, Parsing            -> Indexing, Search
================================================================================================
Universal Forwarder       -> Indexer               -> Search Head
Input                     -> Parsing, Indexing     -> Search
================================================================================================
Universal Forwarder       -> Heavy Forwarder       -> Indexer
Input                     -> Parsing               -> Indexing, Search
================================================================================================
Universal Forwarder       -> Heavy Forwarder       -> Indexer           -> Search Head
Input                     -> Parsing               -> Indexing          -> Search

Which configuration parameters go with which phases
This is a non-exhaustive list of which configuration parameters go with which phase. By combining this information with an understanding of which server a phase occurs on, you can determine which server particular settings need to be made on.

Input

inputs.conf
props.conf sourcetype CHARSET NO_BINARY_CHECK detect_trailing_nulls CHECK_METHOD CHECK_FOR_HEADER FIELD_NAMES PREFIX_SOURCETYPE INDEXED_EXTRACTIONS, also called Structured Data Header Extraction (version 6+) See this Blog article ** **LEARN_SOURCETYPE, LEARN_MODEL
wmi.conf
regmon-filters.conf

Parsing

props.conf ** TRUNCATE, LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, and all other line merging settings** ** TZ, DATETIME_CONFIG, TIME_FORMAT, TIME_PREFIX, and all other time extraction settings and rules** ** TRANSFORMS* which includes per-event queue filtering, per-event index assignment, per-event routing. Applied in the order defined** ** SEDCMD* ** ** MORE_THAN*, LESS_THAN* **
transforms.conf ** stanzas referenced by a TRANSFORMS* clause in props.conf ** ** LOOKAHEAD, DEST_KEY, WRITE_META, DEFAULT_VALUE, REPEAT_MATCH **
datetime.xml

Indexing

props.conf ** SEGMENTATION* **
indexes.conf
segmenters.conf
multikv.conf

Search

props.conf (note that this is the order in which these occur)
** rename **
** EXTRACT* **
** REPORT* **
** KV_MODE **
** FIELDALIAS* **
** EVAL* (version 5+) **
** LOOKUP* **
transforms.conf
** stanzas referenced by a REPORT* clause in props.conf **
** filename, external_cmd, and all other lookup-related settings **
** FIELDS, DELIMS **
** MV_ADD **
lookup files in the lookups folders
search and lookup scripts in the bin folders
search commands and lookup scripts
savedsearches.conf
eventtypes.conf
tags.conf
commands.conf
alert_actions.conf
macros.conf
fields.conf
transactiontypes.conf
multikv.conf

Other

There are some settings that don't work well in a distributed Splunk environment. These tend to be exceptional and include: - props.conf CHECK_FOR_HEADER, LEARN_MODEL, maxDist. These are created in the parsing phase, but they require generated configurations to be moved to the search phase configuration location. Note that, starting with V6.1, props.conf might have to go on the UFs. From dev: With 6.1, the structured data props.conf are happening at monitoring time therefore the props.conf has also to be on the forwarders. For most up to date information, please always consult the Splunk documentation at [our website][4].

Re: Where do I configure my various Splunk settings?

Steve_G_ — Wed, 09 Sep 2015 23:42:05 GMT

It is all documented here: http://docs.splunk.com/Documentation/Splunk/6.2.5/Admin/Configurationparametersandthedatapipeline

Re: Where do I configure my various Splunk settings?

MuS — Wed, 09 Sep 2015 23:42:35 GMT

Hi sievert [Splunk],

IMHO the best source is this wiki article http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

cheers, MuS

Re: Where do I configure my various Splunk settings?

MuS — Wed, 09 Sep 2015 23:43:27 GMT

That's nice as well 🙂

Re: Where do I configure my various Splunk settings?

s2_splunk — Thu, 10 Sep 2015 02:32:59 GMT

Hi MuS,
that's what happens when your question shows up before the answer you are giving at the same time. The goal of this was to drive customers towards answers.splunk.com or docs.splunk.com vs. the Wiki, which may not contain current, up to date information. 😉
docs and answers are here to stay!

Re: Where do I configure my various Splunk settings?

MuS — Thu, 10 Sep 2015 02:39:12 GMT

@Piebob told me the same 😉 Your answer was stuck in the moderation queue.
Should I remove my post?

Re: Where do I configure my various Splunk settings?

s2_splunk — Thu, 10 Sep 2015 02:45:12 GMT

Nah, it's a valid comment. Let's not mess with history... 🙂

Re: Where do I configure my various Splunk settings?

dvb — Fri, 24 Jun 2022 06:41:54 GMT

Why has this page been removed?

Re: Where do I configure my various Splunk settings?

ruman_splunk — Thu, 06 Oct 2022 01:47:01 GMT

wiki.splunk.com was removed. too much old info, no one managing it. you can still grab the old page from wayback: https://web.archive.org/web/20201202100505/https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

Re: Where do I configure my various Splunk settings?

jotne — Thu, 01 Jun 2023 20:29:08 GMT

Updated link:

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline

topic Re: Where do I configure my various Splunk settings? in Getting Data In

Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Input

Parsing

Indexing

Search

Other

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?

Re: Where do I configure my various Splunk settings?