got there with fiddling!  thanks for setting me on the right path.    ... View more

ok tried it, its darn close, but its duplicating that data set, at several levels of the JSON indenting. Any final tweaks? I can re-write my python code if needed to spit out the JSON with different names if that helps. It seems I might have to do that for the TALLY one (that also ends in -AUD)  Thanks again!    ... View more

Amazing. Thank you!  So, given i already have the JSON data streaming in, i assume then all i need is the below at the tail end of my search (and sorry, it was stupid of me not to give you guys text data, sorry for the work you had to replicate it manually from the screenshot - thanks for explaining that!). re your assumption of “sum” i believe all i need actually is “last” . As in, my code (that spits out the JSON) is hitting an API that is like a balance. So im not interested in summing balanced over the search period, but rather just plotting the latest value that arrived to give me the latest balance. So in that case do i just replace “sum” in your second last line with “last”?  finally, do i need the 4th last line (fields….) or is that a hangover of you having to work with my screenshot   | spath path=BINNANCE-BALANCES | spath input=BINNANCE-BALANCES | fields - _raw _time BINNANCE-BALANCES | fields *-AUD | stats sum(*) as * | transpose 0   thanks again. So impressed at turnaround time, esp on a weekend! keiran.  ... View more

Did you solve this @ljalvrdz - im having an identical issue. have tried across splunk enterprise 7.1.2 and over the weekend upgraded to 8.0.3, but still no joy. Any help appreciated, thanks! ... View more

@nikhilafedex - ingesting is the easy part: setup a new (settings -> data inputs -> UDP..... new listener on 2055) + an obviously matching config on your network devices pointing to your splunk instance (on udp 2055). Then the real work begins - making dashboards fromn the ingested data... i never found time to loo p back on this. Wanted to. Just not managed to. ... View more

Hi @akg2019 how did you get on with this? I haven’t had a chance to circle back yet. Keen to understand how you got on. ... View more

sounds good to me! we may have to tweak the logic when you get a graph going based on some of my assumptions (for instance ive been thinking in the netflow bytes_in field is that bytes since last update, or cumulate to that point - like the TCP seqnumber). If you want some same live data, i could look into exporting some for you. ... View more

^ and im not sure if you missed this comment @DavidHourani Does my logic make sense, and if so do your search query's cover it? i will test in thew weekend. ... View more

Hi @akg2019 - i think your problem / question around sFlow is a more fundamental one. Im a long time network engineer so i might be able to shed light on the different datasets. Netflow is an accurate measure of traffic (bits)- actually it was/still is used for many billing systems for instance to track what customer consumed what data. sFlow on the other hand is not. The clue is in the name - its sampled. Its evolution was driven by faster network kit, where netflow (tracking bit count on EVERY session would flatten the CPU of the router). So in sFlow, every so often, might be one packet in 10,000, the sflow process will wake up, peek in at that packet transiting the box at that time and report on that packet, then the process will disappear... and reappear again to check in at the next sampled interval (another 10,000 packets). The logic is that the sampling will be able to roughly report on the transiting traffic.... as big / long-running / high-bandwidth sessions are more likely to be hit upon by the sampling. In sFlow, I dont believe the concept of sessions (src_ip + src_port + dst_ip + dst_port) are tracked, which is necessary for the router to keep track of incrementing bit count like it does in netflow... so the fact that bytes_in field is not present in sFlow makes perfect sense to me. In saying that i know solarwinds etc have developed an interpretation of this data. Graphing it doesnt make sense to me given the above. What does it look like? Ahh just checked, seems its tabular reporting (see https://www.solarwinds.com/topics/sflow-collector) which does make more sense. Those tables do have byte count though hmmm... how wuld they get that.... (checking your sflow packet sample now).... OK there is a seqnumber field - and in TCP at least thats an incremtal count of the bytes transferred so far, and is included in EVERY packet as a running total. So i guess thats how they do it, and thats what you likely need to report on. But that only exists in TCP (UDP for instance does not have this). hope this helps. As for me, i will likely have time to mess around with this stuff on the wekeend. Ever grateful for your assistance here @DavidHourani thanks Keiran. ... View more

Hi guys, bitrate is simple. Its just: end time minus start time which gives you duration. In this case you derive this from successive events where the flow details match . Lets say this is 10secs. Then the bits side - if you have a delta of say 1MB (again just minus earlierbit count from later bit count) in that 10 seconds, you just divide by 10 to get the avg bitrate per second... so 100kbps in this example ... View more

Hi @DavidHourani - really appreciate your assistance here.... attached is a screenshot of some sample data thats as good as any other. Let me know if you need an actual export. Basically (if you didnt know about netflow) the router sends periodical "flow records" back to a reciever - in this case splunk (FYI - each data packet can contain many flow records, and splunk pulls them out as an event per record).... so its a snapshot into what the routers session table is at that moment, inclusive of byte count for those transiting sessions. So if you have a long running TCP session to a DB server for instance, at minute one, it will have a byte count (bytes_in/out) of say 100.... check back 1 minute later, it migth have a byte count of say 1000, indicating 900 more bytes in that last minute. I think the search logic needs to - group like flows by TCP/UDP sessions which is (src_ip + dst_ip + src_port + dest_port)..... - graphing bytes over time. - And then grabbing only say the top 10 flows by byte count. Check the original post for the kind of flow data visualisation over time we are hoping for. ... View more

Hi all, i havent had time to look at this further. My splunk is still ingesting loads of netflow, but i havent started dev on the SPL. Seems lots of people looking for this. @DavidHourani has specifically asked for a new question to be asked on a new post, not quite sure why, its still the same dev problem we need solved, but regardless happy to follow the new thread, just pls link us in here @akg2019 so we know where to follow. Thanks guys! ... View more

Just circling back an updating here. Couldn't get the above to work. So i fixed it by re-writing my code to structure the JSON in a way that splunk would inherently split it, without any tweaking of props.conf. heres what it spits out now, and splu7nk is parsing, just fine: { "BOMPREDapiPollTime": 1546140373.072095, "BOMPREDdate": "2018-12-30T11:44:43+11:00", "BOMPREDdescBrief": "Mostly sunny.", "BOMPREDdescDetail": "Hot and mostly sunny. Winds west to northwesterly 15 to 20 km/h shifting east to northeasterly 15 to 25 km/h in the late morning and early afternoon then becoming light in the late evening.", "BOMPREDfireDanger": "Very High", "BOMPREDiconCode": 3, "BOMPREDrainChance": 10, "BOMPREDtempMax": 32, "BOMPREDuvAlert": "Sun protection 8:30am to 5:20pm, UV Index predicted to reach 13 [Extreme]" } { "BOMPREDapiPollTime": 1546140373.072095, "BOMPREDdate": "2018-12-31T00:00:00+11:00", "BOMPREDdescBrief": "Shower or two.", "BOMPREDdescDetail": "Partly cloudy. Medium (50%) chance of showers, most likely in the evening. The chance of a thunderstorm in the afternoon and evening. Light winds becoming northeasterly 15 to 20 km/h in the evening then becoming light in the late evening.", "BOMPREDiconCode": 11, "BOMPREDrainChance": 50, "BOMPREDrainMM": "0 to 1 mm", "BOMPREDtempMax": 29, "BOMPREDtempMin": 22 } { "BOMPREDapiPollTime": 1546140373.072095, "BOMPREDdate": "2019-01-01T00:00:00+11:00", "BOMPREDdescBrief": "Mostly sunny.", "BOMPREDdescDetail": "Partly cloudy. Slight (20%) chance of a shower. The chance of a thunderstorm in the morning. Light winds becoming northeasterly 15 to 20 km/h during the day then becoming light during the afternoon.", "BOMPREDiconCode": 3, "BOMPREDrainChance": 20, "BOMPREDtempMax": 30, "BOMPREDtempMin": 23 } ... View more

thanks!! that worked. Never used eventstats before - good to know! ... View more

Hi thanks for your help, but sorry if i didnt explian well.... i dont need just a single record.... each time the script runs, it generates 7 JSON events. And the table needs all 7, but only the latest 7. Each batch of 7 JSON events share the same API poll epoch time. Hopefully that clears up things? ... View more