topic Re: Changing the index on forwarded data in Getting Data In

Changing the index on forwarded data

brianm1002 — Wed, 13 Apr 2011 10:46:43 GMT

I have one Splunk indexer that both indexes and forwards the data to a second Splunk indexer. The name of the index is not the same on both indexers. When the data is received by the second indexer, I get the error that it received an event for an unrecognized index.

How can I change the index when it's sent to the second indexer? I've tried adding an index=newindex to the inputs.conf on the second indexer, but it's still trying to use the name of the index from the first indexer.

Brian

Re: Changing the index on forwarded data

gkanapathy — Wed, 13 Apr 2011 11:18:00 GMT

Why not just name the indexes the same? It's not really a good idea to configure Splunk to be able to make this change. Basically, you would need to put:

[splunktcp]
route=has_key:_linebreaker:parsingQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue

into inputs.conf on the second indexer, then send all the received data through a TRANSFORM to set the index key. This can have non-trivial consequences, so I strongly discourage it.

Re: Changing the index on forwarded data

brianm1002 — Wed, 13 Apr 2011 11:36:46 GMT

The first indexer is managed by a different department and I have no control over it. The second indexer is one that I maintain, and I'm currently using multiple indexes to segregate different sets of data based on disk space and retention requirements.

What are some of the consequences of doing as you suggest?

Re: Changing the index on forwarded data

jrodman — Wed, 13 Apr 2011 23:45:46 GMT

Note that reparsing already parsed events means that they will be timestamped and possibly merged a second time, with possibly worse results than the first time.

There are also some performance costs with processing all data again even if it is already processed.