Search Time vs Index time

[splunk_noob2022]

I was wondering,

1. We have search time and index time field extractions, so can i push the same props/transforms over SH and indexer cluster for search time field extraction or no need to push the same at indexer since search head will replicate knowledge budle with its search peers.

And do i need to restart both search head and the indexers for these conf files to take effect?

2.  For index time field extraction we usually push the props/transforms over HF and do i need to push the same conf over indexers as well.

And do i need to restart both HF and the indexers for these conf files to take effect?

[DavidHourani · Accepted solution]

Hi @splunk_noob2022 ,

In terms of "when to restart" Splunk, please have a look at the following docs link:

https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/Configurationfilechangesthatrequirerestart#... 

When you make changes to various config files, refer to the link above to confirm whether a restart is needed. Eventually, you'll memorize what needs restarts and what doesn't.

For search time extractions, there's not need to deploy the configs on the indexers since the search heads already share those configs in the knowledge bundle. So search time extractions should go on search heads only.

For index-time extractions, deploy them on all "indexer" nodes that the data traverses. Since HFs cook (process) the data, you will need to have the TAs on any HF that the data uses. If you're unsure of the data path, it's best to have the index time TAs on HFs and IDXs.

I hope this helps!

Cheers,

David

[PickleRick]

1. The knowledge bundle pushed from search peers to indexers contains the stuff needed for searching and is taken into account only during search. So if you push your index-time configuration items onto SHs they won't work on indexers because they're not there. You can, however, push the same configs containing both search and index-time props and transforms because only the ones appropriate at given stage will be applied - search-time configs will simply have no effect on indexers and index-time elements won't work on SHs.

But you don't need to touch your SHs in order to apply index-time configuration items on indexers. And you definitely don't need to restart them.

2. The index-time extractions (as all the index-time processing, except maybe the indexing itself) is done on the first "heavy" (based on the full splunk installation binary, not the UF one) instance in event's path - it needs to be applied in the place proper for your architecture. So if your UFs send to indexers, you need them on indexers, if you send to HFs - on HFs. If some send here, some there - on both layers.

[splunk_noob2022]

@PickleRick 

So, if I understand correctly, for Search-Time field extraction we can push the same (props/transforms) over SH & IDX and no restart is required for IDX or SH.

And for Index-Time field extraction we can use the same (props/transforms) and push them over HF only and should restart the HF for these configurations to take effect.

IDX restart is not required either for Search-Time or Index-Time field extraction.  

[PickleRick]

No, it's a bit more complicated than that.

@DavidHouranialready pointed you to the document describing what needs a restart and what doesn't (also remember that whereas adding configuration items by means of editing files needs restart, adding them by REST calls does not.

Search-time configurations might need a while in order to be applied (so you might need to wait patiently some time before they "kick in").

About what to push where.... well, that's another thing to which there is no single definite best answer. As I always say - "maintainability first", so you want your configuration understandable and predictable. If you're simply pushing an add-on downloaded from Splunkbase, you might want to distribute the same app onto all layers of your environment for the sake of consistency. Only the applicable settings for each layer would get applied where needed.

But if you're developing a custom set of configs - you might split them into index-time props and search-time props and deploy separately.

It really depends on the situation.

And remember that even though in your particular setup the index-time settings might not be used on your data on indexer or search-head layer if the data comes from UF via HF to indexers, but they are gonna be incorporated into the whole configuration set on the splunk instance, it's just not gonna get used on the data.