Knowledge Management

data pipeline and configuration files location

ahmedragy922
Explorer

Hello,
i'm confused about where configuration files (Search Head or Indexer) should i modify when i want to do filed extraction ??
or when i want to override sourcetype,source,host , should i do that in forwarder or indexer or search head???
is there any reference that map the configuration files to which data pipeline applies ?? for example : if i want to do field extraction >>> i should do that in Search head and configure props.conf and transforms.conf

i just found those 2 articles but i still confused.
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Configurationparametersandthedatapipeline
https://docs.splunk.com/Documentation/Splunk/7.2.6/Deploy/Datapipeline

Tags (1)
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

ahmedragy922
Explorer

thank you for the answer , but i think i can override sourcetype,index,source and host in inputs.conf in Universal Forwarder , also i can do the same in indexer and Heavy Forwarder.
but i think there is the difference between them , in Universal Forwarder i can just write the index where the data will be stored in indexer but i don't have any power to filter the data as in inputs level splunk can't determine the events. in the opposite in indexer , the splunk can parse the data so i can dynamically override (writing regex to change a subset of data or routing some data to index and other to another index) the sourcetype,index,host,source for the data .
can you correct me if i'm wrong ??

richgalloway
SplunkTrust
SplunkTrust

One can specify sourcetype, index, source, and host in a UF, but since that where the data originates, I wouldn't call it an "override". The rest of your statement is correct.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...