Splunk Enterprise Security

Exclude index from ES dataset

chris
Motivator

Hi,

we are currently adding data sources to our Splunk environment. We try our best to make it CIM compliant. We have a dedicated ES search head and we do not want ES to look at this data. How can we make sure that it is excluded from ES. I'd rather not set up new dedicated indexers just for the new data since we would probably loose performance and the setup (and therefore maintenance) will become more complicated.

Thanks,
Chris

0 Karma
1 Solution

maciep
Champion

So you have a search head dedicated for just the ES app? And you have other search heads to use outside of ES? If that's the case, I'd say just don't put the config for your new sources on the ES search head. I think most CIM-compliance happens at search time, so if ES doesn't have the search time config for those new sources, then those fields shouldn't be available for the dm acceleration.

If that's not an option, because maybe the log data is cim compliant with just k/v extractions, then is it in a different index? I believe the latest version of the CIM app allows you to choose which indexes apply to a given data model. So if those new sources are in their own index, just uncheck that index for the datamodel config on the ES box.

Also, in ES you have the option to import apps. By default, if an app is named something like TA* or Splunk* or DA* (etc), it's automatically imported into ES. If you have apps not named that, you can tell ES the name of your app and will import it. Likewise, I believe you can also tell ES not to import an app if it meets the default naming convention. So if your config is in a separate, just tell ES to ignore it.

Hopefully I understood the problem correctly. And there may be better solutions, but that's what comes to mind for me.

View solution in original post

maciep
Champion

So you have a search head dedicated for just the ES app? And you have other search heads to use outside of ES? If that's the case, I'd say just don't put the config for your new sources on the ES search head. I think most CIM-compliance happens at search time, so if ES doesn't have the search time config for those new sources, then those fields shouldn't be available for the dm acceleration.

If that's not an option, because maybe the log data is cim compliant with just k/v extractions, then is it in a different index? I believe the latest version of the CIM app allows you to choose which indexes apply to a given data model. So if those new sources are in their own index, just uncheck that index for the datamodel config on the ES box.

Also, in ES you have the option to import apps. By default, if an app is named something like TA* or Splunk* or DA* (etc), it's automatically imported into ES. If you have apps not named that, you can tell ES the name of your app and will import it. Likewise, I believe you can also tell ES not to import an app if it meets the default naming convention. So if your config is in a separate, just tell ES to ignore it.

Hopefully I understood the problem correctly. And there may be better solutions, but that's what comes to mind for me.

View solution in original post

chris
Motivator

The data has the same format/sourcetype as existing data that is relevant for ES but resides in a different index. I configured the data models in CIM to only include specific Indexes. Thanks a lot.

Regards
Chris

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.