Does the data flow through those same queues at the indexer?

[dokaas_2]

If an HF is used for a intermediate / aggregation tier and the data is parsed,  what does the ingestion pipeline look like when it hits the indexer.  That is, if the HF does parsing, aggregation, typing, but not indexing, does the data flow through those same queues at the indexer? Or is the data injected directly in the the indexing queue?

[PickleRick · Accepted solution]

Of course the point of reference are the invaluable Masa diagrams

https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platfor...

HF is just an indexer which doesn't do local indexing in the IndexerPipeline. So it processes the events up to the indexing queue but then sends the parsed (not cooked! the data is now fully parsed!) to the indexer.

It's not shown on the diagram (because only standalone indexer and UF->idx cases are covered) but as the indexer receives it via S2S tcp input (or http-embedded s2s input) and sees that it has parsed data (the forwarder tells the indexer at the beginning of the connection what kind of data it's sending), it bypasses all local processing and sends the parsed event stream straight to local indexing queue.

[gcusello]

Hi @dokaas_2,

if data is passing through HFs they are coocked, so the pipeline is always the same but on Indexers Data arrive already coocked so they don't load Indexers.

Ciao.

Giuseppe