Knowledge Management

data pipeline and configuration files location

ahmedragy922
Explorer

Hello,
i'm confused about where configuration files (Search Head or Indexer) should i modify when i want to do filed extraction ??
or when i want to override sourcetype,source,host , should i do that in forwarder or indexer or search head???
is there any reference that map the configuration files to which data pipeline applies ?? for example : if i want to do field extraction >>> i should do that in Search head and configure props.conf and transforms.conf

i just found those 2 articles but i still confused.
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Configurationparametersandthedatapipeline
https://docs.splunk.com/Documentation/Splunk/7.2.6/Deploy/Datapipeline

Tags (1)
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

ahmedragy922
Explorer

thank you for the answer , but i think i can override sourcetype,index,source and host in inputs.conf in Universal Forwarder , also i can do the same in indexer and Heavy Forwarder.
but i think there is the difference between them , in Universal Forwarder i can just write the index where the data will be stored in indexer but i don't have any power to filter the data as in inputs level splunk can't determine the events. in the opposite in indexer , the splunk can parse the data so i can dynamically override (writing regex to change a subset of data or routing some data to index and other to another index) the sourcetype,index,host,source for the data .
can you correct me if i'm wrong ??

richgalloway
SplunkTrust
SplunkTrust

One can specify sourcetype, index, source, and host in a UF, but since that where the data originates, I wouldn't call it an "override". The rest of your statement is correct.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...