Duplicate Web Data Model - how to address?

[wryanthomas]

I'm not sure how this happened, but after installing Splunk App for Web Analytics (SAWA) I now have two Web data models and they seem to be conflicting. And the one from SAWA is pre-empting the one from CIM.

What is the best path to addressing this and/or removing the Web Data Model installed by the SAWA app (I need to keep the one for CIM).

[jbjerke_splunk]

Hi

These are really great improvements suggestions. I can certainly update the docs for it and include the deletion of the Web DM. Or at least disable it.

Updating a DM definition is a problem as you will have to rebuild it. For this reason I have not updated the DM definition for the past 5 releases to avoid this. I'm hoping never to have to update the DM again.

I'm thinking that in a future release I will only have the new non-conflicting DM name preconfigured and have existing users upgrade to the new name. This is not fully confirmed and I have no timelines for it.

Thanks for keeping the community alive!

j

[jbjerke_splunk]

Hi

New version of the app is now live which hopefully solve this issue.
https://splunkbase.splunk.com/app/2699

v 2.2.0
- Added an option to use a different data model name than "Web". This caused conflicts with the default CIM datamodel also called Web.
- Made changes to Sites setup dashboard to make it easier.
- Migrated website setup settings to the KV store.
- Added better support for IIS. Now supports ms:iis:auto and ms:iis:default sourcetypes which comes from the official IIS Add-on.
- Updated User agent string parsing to latest version
- Various bug fixes

[wryanthomas]

Thanks again, j.

A couple of minor questions/comments/suggestions regarding the path of cloning the "Web" data model related to future upgrades of the app.

As instructed in the documentation, I've cloned the "Web" data model that came with the app and gave it a new name, "WebAnalytics".

Although it's not stated in your instructions (in documentation), one needs to delete the "Web" data model that comes with the app in order to avoid the name conflict with the "Web" data model from CIM.

Thinking about future upgrades of the app...

When a new version of the app comes out...
1) If/when there are edits to the app-provided "Web" data model -- how will I get those data model enhancements/edits to my custom "WebAnalytics" data model?
2) Won't a future update add the "Web" data model again? (And therefore create a situation where I have to remember to delete it -- maybe after somehow extracting any enhancements and implementing in my custom data model -- to avoid the conflict?)

Some suggestions in case it's useful:
1) See earlier thread where I recommend using "Web" data model from CIM for the pieces it offers, and a supplemental "WebPlus" (?) data model for SAWA so you can avoid this issue. I understand this could be tricky, but managing this issue (for those of us using CIM) seems very problematic.
2) Consider including in your documentation to delete the "Web" data model if needed.
3) Consider including in your documentation -- on upgrades -- how to implement enhancements to your latest version of data model to a version of the data model created to address this conflict.

[wryanthomas]

So... in Splunk Cloud, I can't get the datamodel deleted nor the "title" field changed. A) There is no admin (or sc_admin) interface for editing actual "title" field (stanza name?) of data model; B) they (Splunk Cloud Support) won't edit any conf files in a 3rd party app. And it's confusing because they use "Title" in the GUI for editing the data model -- but that is actually the "displayName". One can confirm this with...

| rest /servicesNS/-/-/datamodel/model
| stats list(title) list(displayName) list(eai:userName) list(disabled) by eai:appName

After cloning, I removed the datasets from the original "Web" data model from the app. That should sidestep "precedence" issues to allow the datasets from the Web data model from CIM to "win" (be available). (I haven't tested to confirm this.)

So... I think I'm good without having to re-package, though I have this empty "WebObsolete" (displayName) data model sitting there in my list ... and I'd rather not re-package the app (to deliver an updated datamodels.conf) because I want still to be able to use the "look for upgrades" feature in App management.

All that to say...
Regarding your comments:
"I can certainly update the docs for it and include the deletion of the Web DM."

Thanks! Though... see above about how actually to go about that (at least in Splunk Cloud).

"Or at least disable it."

I don't think you can disable it. Or -- at least, I don't see any disable/enable directive in datamodels.conf. But perhaps I'm missing it. I guess you could comment things out. But in context of managed Splunk Cloud -- where support won't touch conf files in 3rd party apps -- that won't work/help much.

"I'm thinking that in a future release I will only have the new non-conflicting DM name preconfigured and have existing users upgrade to the new name. This is not fully confirmed and I have no timelines for it."

That would be a huge advantage for dealing with this -- at least in (managed) Splunk Cloud.

"Updating a DM definition is a problem as you will have to rebuild it. For this reason I have not updated the DM definition for the past 5 releases to avoid this. I'm hoping never to have to update the DM again."

I totally empathize with this dilemma. That said -- I'm not sure why rebuilding acceleration is a huge deal. (Unless you're using SVC-based licensing model and have a ton of data in your data model.) I do it all the time. As long as it's in the instructions... that shouldn't be a problem. (IMHO)

Thanks for your help and consideration!!

[obla]

@jbjerke_splunk as someone who needs to change the data model as it conflicts w/ES, do we need to uninstall app 1st as well as current data model or install app over itself and let tool rename current data model?

[jbjerke_splunk]

You can install the app on top. You need to take the additional configuration step to change form the default Web datamodel to something else. It will not happen automatically.

[wryanthomas]

First: Thanks very much for this update and for your attention to this issue. I appreciate it!

I've installed (but not yet set up) ... and I'm looking your instructions, quoted below, and I have the following questions.

1) Did installing app modify the Web Data Model (from CIM) that was already installed? (I suspect not, but am a little confused by "Clone" instructions.") Regardless, will following the instructions to clone it leave the original un-modified?

2) I intend to use the (CIM) Web Data Model for general (other) purposes. It seems that this process of cloning the (Web) Data Model and specifying a different name for the cloned result will mean I have to associate my environment's (many) web logs to both Data Models. This is "ok", but it seems like a (potential) waste of resources (largely redundant -- and potentially quite wasteful re rebuilding Data Models for acceleration). Do you have a plan to make it so that this app (SAWA) can leverage the CIM Web Data Model ... and use a second (supplemental) Data Model for the SAWA needs you require that the CIM Web Data Model doesn't meet?

Quoted instructions:
4. Choose data model and enable data model acceleration

The Splunk App for Web Analytics uses data model acceleration extensively to power the dashboards. The app allows you to select the datamodel name you want to use. By default the app uses the datamodel "Web". I you have a naming conflict for this datamodel (there is a Web data model alread in the CIM app which is slightly different to this one), you can choose to rename it. See additional instructions below for using a custom datamodel name.

Once the lookups in the previous step has completed you should enable acceleration for the data model "Web" (or the custom name you have chosen). The data model can be found under Settings->Data models. Set the summary range appropriately depending on how long you want to keep the data, > 1 Month. The data model is updated every 10 minutes in order for the sessions to get picked up properly. The data model acceleration needs to finish before you will see any data in any dashboard except the "Real-Time" dashboard which uses raw log data as source. That means that you initially might not see data until the data model has finished building. This could initially take many hours depending on how much data it is trying to build.

If Events are showing 0 after install or upgrade you might have to rebuild the data model.

Using a custom datamodel name
Go to the Settings->Data models page and click Edit->Clone for the datamodel Web inside the SplunkAppForWebAnalytics app.
Give the datamodel a new name, set the same name for the title and the ID fields, i.e. "WebAnalytics". Make sure you also set the Permissions to Clone.
Update the settings macro that defines the datamodel name. Use the same name as in the previous step, i.e. "WebAnalytics" or similar. You can find the macro here.

[jbjerke_splunk]

Hi

1) No the app does not change any existing datamodels. The cloning refers to the Web datamodel as defined inside the SAWA app - not the one defined in the CIM app. When you clone it, you will have another datamodel inside SAWA with whatever name you want.
2) This would be great (one main web datamodel and one supplemental), but difficult in practice. The SAWA web datamodel is based on the CIM Web datamodel with added extra fields. It would be difficult to join these up from two datamodels in the same search. I have not done that much research on this yet.

j

[jbjerke_splunk]

Hi guys,

I'm almost finished with a new version that has better support for ms:iis:* sourcetypes and a feature where you can select your own datamodel name. This is configured in a macro provided with the app.

You can look at the source for this here:
https://github.com/johanbjerke/SplunkAppForWebAnalytics

This is the unpackaged version of the app so there are files in the local folders etc. Use at your own risk 🙂

Estimated release in end May or early June.

johan

[wryanthomas]

Thank you!!

[jbjerke_splunk]

Hi

You can rename the datamodel in the app to something else so it is not conflicting and change all references to the datamodel in each saved search and dashboard. The app uses CIM naming convention in the Web datamodel but extended it with an additional 10 fields or so.

j

[wryanthomas]

Thank you. I tried changing the name via Splunk Web, but it only changed "Title" attribute. "modelName" is (was) still "Web". But I'm guessing this can be done by editing .conf files. Can you please specify how this can be done? What files do I need to edit? That'd be question #1.

Question #2:
Since I couldn't find a clear way to address the conflicting Web Data Model, and since I didn't want to customize my (CIM-installed) Web data model anyway (to minimize complexity when it's time to upgrade CIM) ... I just deleted the Splunk App for Web Analytics. My question is... wouldn't it be better for you (developer of Splunk App for Web Analytics) to leverage the CIM Web data model as it is without customizing it ... and then add your own supplemental data model (and related objects/configurations) that gets installed and upgraded with your app? (And, as appropriate, maybe promote your enhancements to be included in CIM?)

[jbjerke_splunk]

Hi

I will incorporate a way to define your own DM name in the next release to avoid this conflict. The changes made to the Web datamodel for this app are quite specific to web analytics so not sure the standard Web datamodel is the perfect fit.

I will keep the community posted on the progress.

j

[wryanthomas]

Hi j. How's this going? Do you have this project in a github repository? Wondering if there's a way I could/should "follow" your progress. Thanks.

[obla]

@wryanthomas I'm also having this issue conflicting with the CIM Web data model. Look forward to your update

[wryanthomas]

Thanks very much, j.

Re "The changes made to the Web datamodel for this app are quite specific to web analytics so not sure the standard Web datamodel is the perfect fit."

That makes sense. What I meant to offer up as a suggestion is...
1) for those data model elements that SAWA needs and that CIM already provides, have (or at least offer path to have) SAWA use the Web data model from CIM. (So a "Web SAWA" data model won't be 90%+ redundant -- and an unnecessary burden on compute resources (rebuilding accelerated data model, etc.) for essentially the same data, and, in some cases, needing to configure and maintain that configuration for two data models for the same data.)
2) for those data model elements (and related configurations) that SAWA needs and are not a good fit for the Web data model from CIM, offer a "Web SAWA" data model that supplements the Web data model of CIM.
3) for those data model elements (and related configurations) not currently offered by the Web data model from CIM but which you feel are a good fit for CIM (perhaps there aren't any?), promote their inclusion in CIM.

Regardless of what you decide -- thanks for taking the time to look at this.

[nickhills]

New best practice dictates that app/add-on data models should not conflict with CIM models.

I suspect that this app may not have passed app-inspect.

I would download the app/unpack/disable/remove the datamodel, then repack, install and contact the vendor to have them update the app on splunkbase.

[wryanthomas]

I went ahead and deleted the app. After a restart, the duplicate Web Data Model is gone.

So... is there a way to install the SAWA app without installing a conflicting Web Data Model? ... and leveraging the one from CIM instead?

[lakshman239]

looking at the docs, https://splunkbase.splunk.com/app/2699/#/details, it only updates existing Web datamodel and doesn't create/clone another one. On your dev, pls re-install the app and follow the steps as in the above link and once the datamodel builds completely, it should be good. On the prod, based on the amount of data in web datamodel, rebuild may take couple of minus to several mins.

[wryanthomas]

Can you point to exactly where in the documentation it states this? I just read through it again and I'm not seeing it. Clarification: I'm not talking about "building" the data model acceleration -- I'm talking about the creation of a Web data model ... when installing the app. When I installed ... I ended up with two data models with the same modelName attribute ("Web") -- because one already existed.