How to route locally indexed events (from the perspective of an indexer) to another environment?

[w531t4]

All --

I'm seeking any advice I can get at this point. A little background. I manage two different user communities (A and B). Community A consists of a universal forwarder aggregator machine, as well as a combination indexer/search head. I believe the topology of community B to be irrelevant.

I've been given the task of taking a subset of events from community A and sending them to community B. To further complicate things, the data that I'm seeking to send to community B is produced by a saved search (using |output csv ) by the single indexer present in community A. The CSV is output to the local filesystem of community A's indexer, and an inputs.conf file monitors that CSV and imports the contents in to a index dedicated to housing only these events.

My attempts as of this point have consisted of specifying the following in community A's indexer outputs.conf file

[tcpout]
indexAndForward = true
forwardedindex.0.whitelist = indexforthisspecificpurpose
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = .*

[tcpout:mytarget]
server=blahip:9997
... ssl config ...

Community A indexer/search head receives a feed of events from community A via a universal forwarder aggregator. These events should all be written and stored to the indexer.

Given this config, I'm not observing the intended result (events only destined to index=indexforthisspecificpurpose). I'm seeing results go to index=os (unix app) and one of our summary indexes. Knowing this, I can say that as far as the link between communities, the connection works. However, the whitelisting/selection of data is not behaving as desired.

Has anyone tried to do something like this? Do you have any suggestions how to accomplish this?

Thanks!

[muebel · Accepted solution]

Hi w531t4, given the peculiar way that the index and forward whitelisting mechanism works, I believe you will have success by doing something like:

forwardedindex.0.blacklist =
forwardedindex.0.whitelist =
forwardedindex.1.blacklist =
forwardedindex.1.whitelist =
forwardedindex.2.whitelist =
forwardedindex.2.blacklist =
forwardedindex.0.blacklist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = indexforthisspecificpurpose

These settings null-out the filters and then reset them. Try this out and let me know if it helps.

[w531t4]

from outputs.conf

 * The filters can start from either whitelist or blacklist. They are tested from forwardedindex.0 to forwardedindex.<max>.
 *** If both forwardedindex.<n>.whitelist and forwardedindex.<n>.blacklist are present for the same value of n, then
   forwardedindex.<n>.whitelist is honored. forwardedindex.<n>.blacklist is ignored in this case.**

after your suggestion, i still receive all locally generated events.

[muebel]

ah, try

forwardedindex.2.whitelist = indexforthisspecificpurpose

[w531t4]

this --- this worked. Thank you Thank you Thank you. I would have never come up with this just by looking at the outputs.conf.spec file.

[muebel]

glad to help 😄