Knowledge Management

data pipeline and configuration files location

ahmedragy922
Explorer

Hello,
i'm confused about where configuration files (Search Head or Indexer) should i modify when i want to do filed extraction ??
or when i want to override sourcetype,source,host , should i do that in forwarder or indexer or search head???
is there any reference that map the configuration files to which data pipeline applies ?? for example : if i want to do field extraction >>> i should do that in Search head and configure props.conf and transforms.conf

i just found those 2 articles but i still confused.
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Configurationparametersandthedatapipeline
https://docs.splunk.com/Documentation/Splunk/7.2.6/Deploy/Datapipeline

Tags (1)
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

You might find this page useful: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
Index-time field extraction (fields that will be stored in the indexes) go in heavy forwarders or indexers, whichever touches the data first.
Search-time field extractions (those done during a search) go in search heads.
Overrides of sourcetype, source, or host go in heavy forwarders or indexers, whichever touches the data first.

---
If this reply helps you, Karma would be appreciated.

ahmedragy922
Explorer

thank you for the answer , but i think i can override sourcetype,index,source and host in inputs.conf in Universal Forwarder , also i can do the same in indexer and Heavy Forwarder.
but i think there is the difference between them , in Universal Forwarder i can just write the index where the data will be stored in indexer but i don't have any power to filter the data as in inputs level splunk can't determine the events. in the opposite in indexer , the splunk can parse the data so i can dynamically override (writing regex to change a subset of data or routing some data to index and other to another index) the sourcetype,index,host,source for the data .
can you correct me if i'm wrong ??

richgalloway
SplunkTrust
SplunkTrust

One can specify sourcetype, index, source, and host in a UF, but since that where the data originates, I wouldn't call it an "override". The rest of your statement is correct.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...