topic Re: How to link fields with different names across sources? in Getting Data In

How to link fields with different names across sources?

bnash_splunk — Mon, 04 Apr 2016 22:53:43 GMT

I have two types of transactions, one coming from a mobile app when a push notification is sent, looks approx like this:

TIMESTAMP="2016-03-29 23:39:01" DDSDKAppEventPushNotificationDelivered Msg=536 UID=101053 DEVICE=galaxys6 IP=XXX

and events from a shopping cart system, where that user has placed an order, that look approx like this:

04/04/2016:15:26:15 TRANSPAYMENT SYSTEM=DD LABEL=1 MERCHANTID=XXX REGISTER=1 LOGIN=125451 TICKET=XXX PAID=YES PAYMETHOD=GIFTCARD CUSTLOYALTY=101053 ITEMS=1 SALE=2.73 DISCTYPE=FLAT DISCAMT=0.0 DISCAPPLIED=0

The CUSTLOYALTY field in the shopping cart is the same data as the UID field from the push notifications (if this was a SQL database, I would join tables on those fields). I have created a field alias for those two fields, "CUSTLOYALTY AS UID".

I've got some simple logic to separate the customers that received a push notification, coming from the mobile app log:

eval PromoRcvd = if (Msg>1,"Yes","No")

But when I try to mix fields from the mobile app log and the shopping cart log, the shopping cart fields (in this case, SALE) are always empty:

table UID,Msg,PromoRcvd,sum(SALE)

I thought I didn't need to explicitly join sources in Splunk in order to search across them?

Re: How to link fields with different names across sources?

martin_mueller — Mon, 04 Apr 2016 23:22:12 GMT

For the short version, check out this excellent post by @MuS at http://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-join-append-or-use-of-subsearches.html

For the long version, check out this excellent talk by @sideview at http://wiki.splunk.com/Virtual_.conf (March 2016).

In your example case, you'd do something like this:

search for datasource1 OR datasource2 | stats values(Msg) as Msg values(PromoRcvd) as PromoRcvd sum(SALE) by UID

The variations are endless.

Re: How to link fields with different names across sources?

MuS — Mon, 04 Apr 2016 23:24:58 GMT

Hi bnash [Splunk],

use stats instead of table:

 your base search to get all the events 
 | eval PromoRcvd = if (Msg>1,"Yes","No")
 | stats sum(SALE) AS SALE by UID, Msg, PromoRcvd

Hope this helps ....

cheers, MuS

Re: How to link fields with different names across sources?

bnash_splunk — Mon, 04 Apr 2016 23:42:46 GMT

That still leaves the SALE field blank:

index=pos OR index=mobileapp | eval PromoRcvd = if (Msg>1,"Yes","No") | stats sum(SALE) AS SALE by UID, Msg, PromoRcvd

https://www.dropbox.com/s/lxmkqtys6lf7fkp/Screenshot%202016-04-04%2016.42.33.png?dl=0

Did I do something wrong?

Re: How to link fields with different names across sources?

bnash_splunk — Mon, 04 Apr 2016 23:44:13 GMT

Ok, that does some weird things in the results table but it does look like it linked the fields correctly. But I think it's also creating duplicates in the results?

https://www.dropbox.com/s/l4oghy8kdt3bsch/Screenshot%202016-04-04%2016.43.39.png?dl=0

Re: How to link fields with different names across sources?

martin_mueller — Mon, 04 Apr 2016 23:46:08 GMT

Each UID is listed once, looks fine to me.

If you're referring to the Yes-No-mv-field, I guess there are events for those UIDs with both Yes and No values. What that means depends on your data and requirements.

Re: How to link fields with different names across sources?

MuS — Mon, 04 Apr 2016 23:46:44 GMT

Can you provide some examples for @martin_mueller and I - please ?

Re: How to link fields with different names across sources?

bnash_splunk — Tue, 05 Apr 2016 00:08:34 GMT

Sure thing - examples of what? What data do you need?