Can we run parallel writes in lookup file?

[Taruchit]

Hello All,

I have a lookup file which stores data of hosts across multiple indexes. 

I have reports which fetch information of hosts from each index and updates the records in lookup file.

Can I run parallel search for hosts related to each index and thus parallelly update the same lookup file?

Or is there any risk of performance, consistency of data?

Thank you

Taruchit

[isoutamo · Accepted solution]

If you need parallelism then you must use kvstore based lookups not CSV based.

[gcusello]

Hi @Taruchit ,

in addition, you could use a lookup with kv-store so you'll have a key that guarantees the unicity of data.

Only one question: you're trying to use Splunk as a database and Splunk isn't a database, are you sure that you're using the best solution for your requirements?

Ciao.

Giuseppe

[Taruchit]

Hi @gcusello,

Thank you for sharing your inputs.

I have a report which fetches last seen timestamp of hosts across multiple indexes.

I store the results in lookup file, and then use the lookup file as a bounded static source from where we can read the results in other reports and dashboards as required.

It helps me with two scenarios: -

1. If the report that generates results fails because of some reason, and as the result the downstream dashboards and reports that consume the data will also get impacted. And I will need to wait for Operations team to help with the issue or wait until the report runs again and hope that it runs successfully in the next execution.

2. Since I am referring a lookup file, the fetching and searching of records in SPLs written for reports and dashboards get faster.

Please share if you have any views to consider and improve.

Thank you

[gcusello]

Hi @Taruchit ,

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[richgalloway]

The lookup file is written by the search head in one go.  There is no parallelism.