topic Does the data flow through those same queues at the indexer? in Getting Data In

Does the data flow through those same queues at the indexer?

dokaas_2 — Tue, 14 Feb 2023 19:32:34 GMT

If an HF is used for a intermediate / aggregation tier and the data is parsed, what does the ingestion pipeline look like when it hits the indexer. That is, if the HF does parsing, aggregation, typing, but not indexing, does the data flow through those same queues at the indexer? Or is the data injected directly in the the indexing queue?

Re: HF Intermediate Forwarder

gcusello — Tue, 14 Feb 2023 08:20:09 GMT

Hi @dokaas_2,

if data is passing through HFs they are coocked, so the pipeline is always the same but on Indexers Data arrive already coocked so they don't load Indexers.

Ciao.

Giuseppe

Re: HF Intermediate Forwarder

PickleRick — Tue, 14 Feb 2023 09:14:36 GMT

Of course the point of reference are the invaluable Masa diagrams

https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774

HF is just an indexer which doesn't do local indexing in the IndexerPipeline. So it processes the events up to the indexing queue but then sends the parsed (not cooked! the data is now fully parsed!) to the indexer.

It's not shown on the diagram (because only standalone indexer and UF->idx cases are covered) but as the indexer receives it via S2S tcp input (or http-embedded s2s input) and sees that it has parsed data (the forwarder tells the indexer at the beginning of the connection what kind of data it's sending), it bypasses all local processing and sends the parsed event stream straight to local indexing queue.