How do I add fields to a data-model from the CIM without rebuilding my data-models?

[LukeMurphey]

I want to add some fields to a data-model that comes with the Common Information Model app but I want to avoid rebuilding my data-models (since rebuilding the data-models is time and resource intensive).

Can this be done?

[LukeMurphey · Accepted solution]

If you want the new field being in all of the data (including the historical data), then you must rebuild your data-models.

If are ok with the field only appearing in new data (i.e. only appearing following the change), then this can be done.

Data-models do not need to be rebuilt when changed when two things are true:

1. acceleration.manual_rebuilds is set to true in datamodels.conf
2. allow_old_summaries=true is used in the searches

By default, CIM sets acceleration.manual_rebuilds to true, and Enterprise Security sets allow_old_summaries in the searches.

See below for some background:

acceleration.manual_rebuilds (datamodels.conf)
The acceleration.manual_rebuilds setting tells Splunk not to rebuild indexes automatically. By default, Splunk wants to make sure that the data-models are complete and therefore will rebuild data-models when the search that was used to populate the data-model is changed.

However, the data-models shipped with the CIM set acceleration.manual_rebuilds to true which means that the data-model will only be rebuilt when someone specifically requests it (via the UI).

allow_old_summaries=true (tstats searches)
tstats supports an option called allow_old_summaries which will allow it to pull back data from summaries were built from a search that differs from the current data-model populating search. Setting allow_old_summaries to true allows the search to retrieve from that data-model even though the data-model may have changed. If set to false, it would ignore this data. This means that adding a new field to the CIM would cause the searches to ignore the data missing the new fields.

Enterprise Security and its related apps set allow_old_summaries to true so that the searches will still retrieve the older data. This means that the searches and dashboards will continue to work with the old data even though the older data doesn't include the new field.