Is there a way to define a common data store to create searches and dashboards on top of for various apps?

[ronak]

Hi

I've a business need to integrate Point of Sale systems from various vendors and produce a common dashboard that shows KPIS and panels for example - Total Sales, Total Tax, Total Fees, #orders at 5 minutes interval, Avg Rev/order, Pie chart of various payment types, Bar chart representing sales by product category, column chart representing sales by product type, top 10 items etc..

the challenge is that not all vendors send the data in same format or call the fields same e.g. Vendor1 might call item_name, vendor2 might call product_name , vendor 1 might call sale_amount - vendor3 might call amount_sold

I'm wondering if there is a mechanism, way such that I can define a common model, create queries and dashboard on top of that common model. Various sources feed into that common model in Splunk.

This allows me to make sure that my splunk app is same that I can ship to various clients along with the connector to the POS that the client might have

Any pointers, method to achieve this functionality ?

thanks, ronak

[maciep]

The short answer is to use search-time operations to map those vendor-specific fields into some common model you define. So first come up with the field names you want to use in your Splunk app. Then for each sourcetype (vendor) decide which fields need mapped where. To do those mappings, you can use aliases, evals, field extractions, lookups, etc. All of those will happen at search time.

So for example if you choose to use item_name, then vendor1's field won't need to change. But for the sourcetype of vendor2, you alias product_name as item_name. Now when you search vendor2's sourcetype, both item_name and product_name fields will be there. Your app's searches will just use item_name.

If you need to normalize something like categories, typically a lookup would work - maybe sale type or payment method or something (not real familiar with pos systems). If you need to maybe concatenate a couple fields of one sourcetype to match a field in your common model, use an eval. Etc.

Hope that helps a little

[ChrisG]

Have you looked at the Splunk Common Information Model Add-on (and docs)? It doesn't have a PoS data model, but you might be able to build what you need following the examples there. The situation you describe is exactly what the Common Information Model is designed to address.

If your question is more about the actual storage of saving and retrieving data so your app(s) can use it, then you should read about the app key value store.