Splunk Search

Lookup Fields not updating in the Datamodel

poojak2579
Explorer

I have built an accelerated datamodel with lookup fields.
There is a report that is scheduled to run everyday to populate the lookup.
The datamodel does not get updated when lookup file is updated. But if I disable acceleration, I can see the fields updated.
Do I have to rebuild the datamodel every time lookup file is updated or it is rebuild automatically?

0 Karma

woodcock
Esteemed Legend

When you accelerate a datamodel, it goes through an additional indexing pass that creates index-time fields (in the case of your lookup-created fields, as metadata) and it is cooked into the tsidx as it is now. If you change the lookup, then anything that is cooked after the change will reflect the change but not the stuff already cooked. You can delete your datamodel acceleration and rebuild it, or you can use tstats on the field that is the key into the lookup and run the lookup after you pull stuff out of your ADM with the | lookup command.

poojak2579
Explorer

Thanks for replying.
I thought the dynamic lookup attribute will pick up the value change because splunk automatically rebuild datamodels when it finds that those models are outdated.
Is there any way I can rebuild the datamodel everyday after the lookup is updated?

0 Karma

woodcock
Esteemed Legend

What do you mean by dynamic lookup attribute? I am unaware of any such thing.

0 Karma

woodcock
Esteemed Legend

How many days of data are you accelerating? If more than a few days, you'll never have it built in enough time to search it before it needs to get rebuilt!

0 Karma

poojak2579
Explorer

I have a summary range of 1 month.
I thought fields in the datamodel are dynamically populated.
Does that mean we can not take benefits of accelerated data model if the lookup is updated every day?
Can we schedule it to rebuld everyday after the lookup is updated?

0 Karma

woodcock
Esteemed Legend

You could figure out the REST API that turns off acceleration and the one that turns it back on and then add those 2 calls to the bottom of your search that updates the lookup so it would look like this:

... | outputlookup foo
| appendpipe [|rest/foo/bar/.../accelerate=0]
| appendpipe [|rest/foo/bar/.../accelerate=1]

This assumes that you are accelerating little enough data that it can get rebuilt before you come in to search it. It also assumes that you are not using continuous searches like ES does, because these will all fail and be "gapped" for the duration of the outage.

0 Karma

poojak2579
Explorer

Thanks for the answer.
I will have to ask my administrator because I don't have right to enable or disable acceleration.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

I believe that the acceleration is created from the model at the time the data model is created or updated. So if you change the lookup, it changes the acceleration that it creates. Rebuilding the data model is the only way to update the acceleration, unless there is a way to trigger the creation of the acceleration that I am unaware of. Hopefully someone will correct me if I'm wrong, so I'm leaving this as a comment for now.

0 Karma

poojak2579
Explorer

Thank you.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...