Knowledge Management

data pipeline and configuration files location

ahmedragy922
Explorer

Hello,
i'm confused about where configuration files (Search Head or Indexer) should i modify when i want to do filed extraction ??
or when i want to override sourcetype,source,host , should i do that in forwarder or indexer or search head???
is there any reference that map the configuration files to which data pipeline applies ?? for example : if i want to do field extraction >>> i should do that in Search head and configure props.conf and transforms.conf

i just found those 2 articles but i still confused.
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Configurationparametersandthedatapipeline
https://docs.splunk.com/Documentation/Splunk/7.2.6/Deploy/Datapipeline

Tags (1)
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

ahmedragy922
Explorer

thank you for the answer , but i think i can override sourcetype,index,source and host in inputs.conf in Universal Forwarder , also i can do the same in indexer and Heavy Forwarder.
but i think there is the difference between them , in Universal Forwarder i can just write the index where the data will be stored in indexer but i don't have any power to filter the data as in inputs level splunk can't determine the events. in the opposite in indexer , the splunk can parse the data so i can dynamically override (writing regex to change a subset of data or routing some data to index and other to another index) the sourcetype,index,host,source for the data .
can you correct me if i'm wrong ??

richgalloway
SplunkTrust
SplunkTrust

One can specify sourcetype, index, source, and host in a UF, but since that where the data originates, I wouldn't call it an "override". The rest of your statement is correct.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Operationalizing TDIR: Building a More Resilient, Scalable SOC

Optimizing SOC workflows with a unified, risk-based approach to Threat Detection, Investigation, and Response ...

Almost Too Eventful Assurance: Part 1

Modern IT and Network teams still struggle with too many alerts and isolating issues before they are notified. ...

Demo Day: Strengthen Your SOC with Splunk Enterprise Security 8.1

Today’s threat landscape is more complex than ever. Security operation centers (SOCs) are overwhelmed with ...