Can the DB Connect Database Lookups use an RPC pool as well?

[sbochniewicz]

With the new version 2.0 I saw the documentation for Outputs and Inputs using the RPC pool to offload the request, can the same be done for the lookups?

This would reduce the need for open database connections from the search head.

[araitz]

The RPC protocol is always used, whether the operations are local or remote. Efficiency isn't my main concern, although I'm sure there is room for improvement in that regard.

An early version of the resource pool architecture had search peers as auto-discovered resource pools, but due to the complexity of dblookups, specifically securely replicating cache from search heads to indexers, we don't have dblookups running on indexers yet.

All lookups, whether scripted or external, only run on indexers when the search is non-transforming (i.e. does not use table, stats, etc), so for transforming searches, you always have to run the lookup on the search head and they will never run on the peer.

As an alternative, you can always dbinput data and make it in to a conventional CSV lookup, which will then be replicated to search peers via normal replication.

[araitz]

This would be tough since you lookups are performed at the search head - we'd have to shuttle search results off to the resource pool, decorate them, then shuttle them back, which would likely make for terrible performance. We'll still give a good think as to how to improve this feature from a performance perspective.

[sbochniewicz]

Depending on how efficient the rpc protocol is, how is this much different than passing work off to the indexers or using external lookups.

[jcoates_splunk · Accepted solution]

sorry, not yet. It's something that we're very interested in figuring out, though.