Why is distsearch.conf whitelisting all json files in the Anomali ThreatStream App?

[maciep]

We noticed that the 6.3.1 version of the Anomali Threatstream App for Splunk ships with a distsearch.conf file. That conf includes a replication whitelist for all json files (see below). Assuming that's still in the latest version, could the developer elaborate on the need for that setting? Because it needs to have a much narrower scope than all json files - like maybe this app's dm json files?

It caused us issues because it effectively whitelisted system/replication/ops.json which absolutely shouldn't be part of the search bundle. That file is updated quite often, which resulted in the bundle being pushed quite often which led to bundle replication errors and ultimately incomplete search results.

[replicationWhitelist]
datamodels = .../*.json

[mghocke]

We POCed Threatstream and now that you mentioned it I just looked for it to have a look. There is absolutely no good reason to have this path whitelisted in distsearch.conf. Actually, it is quite intruding. I would remove this setting or make it more precise like
.../threatstream/default/data/model/*.json or some such. We had quite a good line of communication into Anomali to make the app work to our liking. Is that not the case anymore once you purchase their product?

[maciep]

I agree, they were great during the POC. But the POC is over, and I don't think I still have access to them (I have to go through our SOC team for contact). I imagine if we buy the product, the service will remain as good.

[lakshman239]

If its not needed on the Indexers, there is no need for whitelist and sending them there.

[maciep]

I probably should have mentioned that it's also in the community app for threatstream that was created a couple years back...i wonder if they just started with that app when they created their own.

[josh_hart_oath · Accepted solution]

@maciep,

We're reaching out to the vendor today on that, since we've got a vested interest.

Josh

[josh_hart_oath]

Our contact at Anomali responded regarding this app and said that the configuration will be removed in version 6.4 of the app and that it's safe to comment out that line (or as @mghocke mentioned, make it more precise).