Splunk Search

How can I extract a field from a JSON strcuture and combine it in a search with other fields from different JSON structures??

keiran_harris
Path Finder

Hi SPL guru's!

im struggling with how to 1 pluck one field's value from one JSON structure and [2] combine in the same search with a bunch of other fields plucked from a different JSON data structure. Both JSON's share the same source and sourcetype (but i also am facing the same problem across data coming in from other source / sourcetype... i assume the solution will be the same, so thought it best to keep things simple here)

As mentioned, the data is JSON in both cases, but there is no common field with which i can merge a-la 'transaction'
(although i wrote the code that spits out the JSON, so i can alter if necessary )

Let me elaborate by way of a pic:
alt text

I can get some of the way there

source=*getCrypto.py kCryptoDictType=nanopool | streamstats window=2 first(NanoPool.ETHminingBal) as balFirst, last(NanoPool.ETHminingBal) as balLast, | eval rate=balFirst-balLast | eval income=rate*1028 | timechart span=1hr sum(income)

... which gives me a nice working graph,.... but when i try and expand it with an OR in the early part of the search:

source=*getCrypto.py kCryptoDictType=nanopool OR kCryptoDictType=BTCMticker | streamstats window=2 first(NanoPool.ETHminingBal) as balFirst, last(NanoPool.ETHminingBal) as balLast, | eval rate=balFirst-balLast | eval income=rate*1028 | timechart span=1hr sum(income)

i get zero results.
what am i missing?
thanks in advance..
K.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Your streamstats looking at neighbouring events is breaking if you introduce different events between the ones your streamstats is supposed to look at.

I'm not sure why you're looking at the BTCMticker events, your second search doesn't appear to attempt to use any fields from those events, so I'm not 100% certain this will work for whatever you're planning to do... in most cases, adding by kCryptoDictType to the streamstats should fix things.

0 Karma

keiran_harris
Path Finder

thanks for your help martin!

RE your second paragraph (first sentence), sorry, the value "1028" above in my search is hardcoded just to get the search to run without error. This hard-coded 1028 is the value that needs to come in dynamically to this search from JSON events where kCryptoDictType="BTCMticker"

At the start of the search, i need to have kCryptoDictType="nanopool" in order to get the streamstats working (makes sense based on your first paragraph), but if i have that in the opening search, how then am i to get at data within the kCryptoDictType="BTCMticker" JSON data structure? guess im struggling wrapping my head around the pipelining / order of operations.

below is the sample events which might make things clearer. What i would like to do (as an example) is
1. get a rate/hour on the (NanoPool.ETHminingBal)... hence streamstats
2. multiply that rate by (ETH-DATA.lastPrice).

thanks so much for your help.

{
    "NanoPool": {
        "ETHminingBal": 0.08754256, 
        "ETHwallAddr": "............", 
        "hashRate": 221.0, 
        "hashRate1hrAvg": 198.3, 
        "rig1hashRate": 221.0, 
        "rig1hashRate1hrAvg": 198.3, 
        "rig1lastShare": 1515041966.0, 
        "status": true, 
        "zzNANOPjsonDlTime": 0.964663028717041
    }, 
    "kCryptoDictType": "nanopool"
}

{
    "ETH-DATA": {
        "aaCoin": "ETH", 
        "bestAsk": 1360.93, 
        "bestBid": 1360.0, 
        "lastPrice": 1360.93, 
        "timestamp": 1515042544, 
        "volume24h": 10566.006, 
        "zzBTCMjsonDlTime": 0.15702295303344727
    }, 
    "kCryptoDictType": "BTCMticker"
} 

thanks!

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Not quite like that. I'm guessing both types of measurements are done separately, how do you link the mining progress up with the correct price for that progress? Could be "the most recent one" or "the next one" if you only have one type each.
If the measurements are done together it's easiest to log them together into one event.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

How do you determine which BTCMticker value is supposed to be used in conjunction with which nanopool value? I see no ID linking the two together.

0 Karma

keiran_harris
Path Finder

Ahh so i need a common field? Im writing the code thats spitting out the JSON so thats easy to add. For instanace i could add the following to the nanopool structure.
“aaCoin”: “ETH”
Assuming i had that, what would my search have to look like? Transaction?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...