topic Re: Duplicate Web Data Model - how to address? in All Apps and Add-ons

Duplicate Web Data Model - how to address?

wryanthomas — Mon, 11 Mar 2019 17:07:24 GMT

I'm not sure how this happened, but after installing Splunk App for Web Analytics (SAWA) I now have two Web data models and they seem to be conflicting. And the one from SAWA is pre-empting the one from CIM.

What is the best path to addressing this and/or removing the Web Data Model installed by the SAWA app (I need to keep the one for CIM).

Re: Duplicate Web Data Model - how to address?

wryanthomas — Mon, 11 Mar 2019 17:46:32 GMT

I went ahead and deleted the app. After a restart, the duplicate Web Data Model is gone.

So... is there a way to install the SAWA app without installing a conflicting Web Data Model? ... and leveraging the one from CIM instead?

Re: Duplicate Web Data Model - how to address?

lakshman239 — Mon, 11 Mar 2019 17:51:08 GMT

looking at the docs, https://splunkbase.splunk.com/app/2699/#/details, it only updates existing Web datamodel and doesn't create/clone another one. On your dev, pls re-install the app and follow the steps as in the above link and once the datamodel builds completely, it should be good. On the prod, based on the amount of data in web datamodel, rebuild may take couple of minus to several mins.

Re: Duplicate Web Data Model - how to address?

nickhills — Mon, 11 Mar 2019 17:52:42 GMT

New best practice dictates that app/add-on data models should not conflict with CIM models.

I suspect that this app may not have passed app-inspect.

I would download the app/unpack/disable/remove the datamodel, then repack, install and contact the vendor to have them update the app on splunkbase.

Re: Duplicate Web Data Model - how to address?

wryanthomas — Mon, 11 Mar 2019 21:29:16 GMT

Can you point to exactly where in the documentation it states this? I just read through it again and I'm not seeing it. Clarification: I'm not talking about "building" the data model acceleration -- I'm talking about the creation of a Web data model ... when installing the app. When I installed ... I ended up with two data models with the same modelName attribute ("Web") -- because one already existed.

Re: Duplicate Web Data Model - how to address?

jbjerke_splunk — Mon, 11 Mar 2019 21:43:27 GMT

You can rename the datamodel in the app to something else so it is not conflicting and change all references to the datamodel in each saved search and dashboard. The app uses CIM naming convention in the Web datamodel but extended it with an additional 10 fields or so.

Re: Duplicate Web Data Model - how to address?

wryanthomas — Wed, 13 Mar 2019 20:53:04 GMT

Thank you. I tried changing the name via Splunk Web, but it only changed "Title" attribute. "modelName" is (was) still "Web". But I'm guessing this can be done by editing .conf files. Can you please specify how this can be done? What files do I need to edit? That'd be question #1.

Question #2:
Since I couldn't find a clear way to address the conflicting Web Data Model, and since I didn't want to customize my (CIM-installed) Web data model anyway (to minimize complexity when it's time to upgrade CIM) ... I just deleted the Splunk App for Web Analytics. My question is... wouldn't it be better for you (developer of Splunk App for Web Analytics) to leverage the CIM Web data model as it is without customizing it ... and then add your own supplemental data model (and related objects/configurations) that gets installed and upgraded with your app? (And, as appropriate, maybe promote your enhancements to be included in CIM?)

Re: Duplicate Web Data Model - how to address?

jbjerke_splunk — Thu, 21 Mar 2019 13:51:25 GMT

I will incorporate a way to define your own DM name in the next release to avoid this conflict. The changes made to the Web datamodel for this app are quite specific to web analytics so not sure the standard Web datamodel is the perfect fit.

I will keep the community posted on the progress.

Re: Duplicate Web Data Model - how to address?

wryanthomas — Mon, 25 Mar 2019 16:22:32 GMT

Thanks very much, j.

Re "The changes made to the Web datamodel for this app are quite specific to web analytics so not sure the standard Web datamodel is the perfect fit."

That makes sense. What I meant to offer up as a suggestion is...
1) for those data model elements that SAWA needs and that CIM already provides, have (or at least offer path to have) SAWA use the Web data model from CIM. (So a "Web SAWA" data model won't be 90%+ redundant -- and an unnecessary burden on compute resources (rebuilding accelerated data model, etc.) for essentially the same data, and, in some cases, needing to configure and maintain that configuration for two data models for the same data.)
2) for those data model elements (and related configurations) that SAWA needs and are not a good fit for the Web data model from CIM, offer a "Web SAWA" data model that supplements the Web data model of CIM.
3) for those data model elements (and related configurations) not currently offered by the Web data model from CIM but which you feel are a good fit for CIM (perhaps there aren't any?), promote their inclusion in CIM.

Regardless of what you decide -- thanks for taking the time to look at this.

Re: Duplicate Web Data Model - how to address?

obla — Wed, 08 May 2019 18:58:09 GMT

@wryanthomas I'm also having this issue conflicting with the CIM Web data model. Look forward to your update

Re: Duplicate Web Data Model - how to address?

wryanthomas — Mon, 13 May 2019 17:02:50 GMT

Hi j. How's this going? Do you have this project in a github repository? Wondering if there's a way I could/should "follow" your progress. Thanks.

Re: Duplicate Web Data Model - how to address?

jbjerke_splunk — Thu, 16 May 2019 11:31:07 GMT

Hi guys,

I'm almost finished with a new version that has better support for ms:iis:* sourcetypes and a feature where you can select your own datamodel name. This is configured in a macro provided with the app.

You can look at the source for this here:
https://github.com/johanbjerke/SplunkAppForWebAnalytics

This is the unpackaged version of the app so there are files in the local folders etc. Use at your own risk 🙂

Estimated release in end May or early June.

johan

Re: Duplicate Web Data Model - how to address?

wryanthomas — Thu, 16 May 2019 11:41:26 GMT

Thank you!!

Re: Duplicate Web Data Model - how to address?

jbjerke_splunk — Wed, 29 May 2019 09:35:12 GMT

New version of the app is now live which hopefully solve this issue.
https://splunkbase.splunk.com/app/2699

v 2.2.0
- Added an option to use a different data model name than "Web". This caused conflicts with the default CIM datamodel also called Web.
- Made changes to Sites setup dashboard to make it easier.
- Migrated website setup settings to the KV store.
- Added better support for IIS. Now supports ms:iis:auto and ms:iis:default sourcetypes which comes from the official IIS Add-on.
- Updated User agent string parsing to latest version
- Various bug fixes

Re: Duplicate Web Data Model - how to address?

wryanthomas — Wed, 29 May 2019 10:59:26 GMT

First: Thanks very much for this update and for your attention to this issue. I appreciate it!

I've installed (but not yet set up) ... and I'm looking your instructions, quoted below, and I have the following questions.

1) Did installing app modify the Web Data Model (from CIM) that was already installed? (I suspect not, but am a little confused by "Clone" instructions.") Regardless, will following the instructions to clone it leave the original un-modified?

2) I intend to use the (CIM) Web Data Model for general (other) purposes. It seems that this process of cloning the (Web) Data Model and specifying a different name for the cloned result will mean I have to associate my environment's (many) web logs to both Data Models. This is "ok", but it seems like a (potential) waste of resources (largely redundant -- and potentially quite wasteful re rebuilding Data Models for acceleration). Do you have a plan to make it so that this app (SAWA) can leverage the CIM Web Data Model ... and use a second (supplemental) Data Model for the SAWA needs you require that the CIM Web Data Model doesn't meet?

Quoted instructions:
4. Choose data model and enable data model acceleration

The Splunk App for Web Analytics uses data model acceleration extensively to power the dashboards. The app allows you to select the datamodel name you want to use. By default the app uses the datamodel "Web". I you have a naming conflict for this datamodel (there is a Web data model alread in the CIM app which is slightly different to this one), you can choose to rename it. See additional instructions below for using a custom datamodel name.

Once the lookups in the previous step has completed you should enable acceleration for the data model "Web" (or the custom name you have chosen). The data model can be found under Settings->Data models. Set the summary range appropriately depending on how long you want to keep the data, > 1 Month. The data model is updated every 10 minutes in order for the sessions to get picked up properly. The data model acceleration needs to finish before you will see any data in any dashboard except the "Real-Time" dashboard which uses raw log data as source. That means that you initially might not see data until the data model has finished building. This could initially take many hours depending on how much data it is trying to build.

If Events are showing 0 after install or upgrade you might have to rebuild the data model.

Using a custom datamodel name
Go to the Settings->Data models page and click Edit->Clone for the datamodel Web inside the SplunkAppForWebAnalytics app.
Give the datamodel a new name, set the same name for the title and the ID fields, i.e. "WebAnalytics". Make sure you also set the Permissions to Clone.
Update the settings macro that defines the datamodel name. Use the same name as in the previous step, i.e. "WebAnalytics" or similar. You can find the macro here.

Re: Duplicate Web Data Model - how to address?

jbjerke_splunk — Wed, 29 May 2019 13:08:53 GMT

1) No the app does not change any existing datamodels. The cloning refers to the Web datamodel as defined inside the SAWA app - not the one defined in the CIM app. When you clone it, you will have another datamodel inside SAWA with whatever name you want.
2) This would be great (one main web datamodel and one supplemental), but difficult in practice. The SAWA web datamodel is based on the CIM Web datamodel with added extra fields. It would be difficult to join these up from two datamodels in the same search. I have not done that much research on this yet.

Re: Duplicate Web Data Model - how to address?

obla — Wed, 29 May 2019 17:50:05 GMT

@jbjerke_splunk as someone who needs to change the data model as it conflicts w/ES, do we need to uninstall app 1st as well as current data model or install app over itself and let tool rename current data model?

Re: Duplicate Web Data Model - how to address?

jbjerke_splunk — Mon, 03 Jun 2019 16:03:33 GMT

You can install the app on top. You need to take the additional configuration step to change form the default Web datamodel to something else. It will not happen automatically.

Re: Duplicate Web Data Model - how to address?

wryanthomas — Wed, 12 Feb 2020 12:57:57 GMT

Thanks again, j.

A couple of minor questions/comments/suggestions regarding the path of cloning the "Web" data model related to future upgrades of the app.

As instructed in the documentation, I've cloned the "Web" data model that came with the app and gave it a new name, "WebAnalytics".

Although it's not stated in your instructions (in documentation), one needs to delete the "Web" data model that comes with the app in order to avoid the name conflict with the "Web" data model from CIM.

Thinking about future upgrades of the app...

When a new version of the app comes out...
1) If/when there are edits to the app-provided "Web" data model -- how will I get those data model enhancements/edits to my custom "WebAnalytics" data model?
2) Won't a future update add the "Web" data model again? (And therefore create a situation where I have to remember to delete it -- maybe after somehow extracting any enhancements and implementing in my custom data model -- to avoid the conflict?)

Some suggestions in case it's useful:
1) See earlier thread where I recommend using "Web" data model from CIM for the pieces it offers, and a supplemental "WebPlus" (?) data model for SAWA so you can avoid this issue. I understand this could be tricky, but managing this issue (for those of us using CIM) seems very problematic.
2) Consider including in your documentation to delete the "Web" data model if needed.
3) Consider including in your documentation -- on upgrades -- how to implement enhancements to your latest version of data model to a version of the data model created to address this conflict.

Re: Duplicate Web Data Model - how to address?

jbjerke_splunk — Wed, 12 Feb 2020 15:31:26 GMT

These are really great improvements suggestions. I can certainly update the docs for it and include the deletion of the Web DM. Or at least disable it.

Updating a DM definition is a problem as you will have to rebuild it. For this reason I have not updated the DM definition for the past 5 releases to avoid this. I'm hoping never to have to update the DM again.

I'm thinking that in a future release I will only have the new non-conflicting DM name preconfigured and have existing users upgrade to the new name. This is not fully confirmed and I have no timelines for it.

Thanks for keeping the community alive!