Splunk IT Service Intelligence

Summing values inside mvfield

rvsroe
Explorer

Hi All,

I am trying to substract values (timestamps) of an mv field, but they are of differing lengths;

## example data:
sysmodtime,idnumber,epoch time
05/03/20 12:40 PM,1,1588502400
05/01/20 12:01 AM,1,1588284060
05/01/20 12:02 AM,1,1588284120
05/01/20 12:02 AM,1,1588284120

05/02/20 12:00 PM,2,1588413600
04/02/20 12:00 AM,2,1585778400
04/02/20 01:00 AM,2,1585782000

04/02/20 02:00 AM,3,1585785600

##desired outcome: = new field time-diff at the end:
sysmodtime,idnumber,epoch time,time_diff
05/03/20 12:40 PM,1,1588502400,218340
05/01/20 12:01 AM,1,1588284060,-60
05/01/20 12:02 AM,1,1588284120,0
05/01/20 12:02 AM,1,1588284120,empty

05/02/20 12:00 PM,2,1588413600,2635200
04/02/20 12:00 AM,2,1585778400,-3600
04/02/20 01:00 AM,2,1585782000,empty

04/02/20 02:00 AM,3,1585785600,empty


-------------------------------
The original data is about 200.000 rows long, so we are looking for a structural solution.

Is there a simple way to loop through the timestamp value inside the mvfield and substract it and place it in a new field
Any suggestions would be very welcome,

Cheers,
Roelof

Labels (1)
0 Karma
1 Solution

bowesmana
SplunkTrust
SplunkTrust

I believe you will have to mvexpand out the fields first to get the mv field into separate rows and then run streamstats. Take this for example

| makeresults
| eval _raw="sysmodtime,idnumber,epoch time
05/03/20 12:40 PM,1,1588502400
05/01/20 12:01 AM,1,1588284060
05/01/20 12:02 AM,1,1588284120
05/01/20 12:02 AM,1,1588284120
05/02/20 12:00 PM,2,1588413600
04/02/20 12:00 AM,2,1585778400
04/02/20 01:00 AM,2,1585782000
04/02/20 02:00 AM,3,1585785600"
| multikv
| stats list(*) as * by idnumber
| eval x="So, up to here is your original data"
| eval base=mvzip(sysmodtime,epoch_time)
| fields - _raw linecount sysmodtime epoch_time
| mvexpand base
| rex field=base "(?<sysmodtime>[^,]*),(?<epoch_time>.*)"
| fields - base
| streamstats window=2 range(epoch_time) as time_diff by idnumber
| stats list(*) as * by idnumber
| eval time_diff=mvindex(time_diff,1,-1)

That gives you what you want, but will be expensive to do the mvexpand and re-aggregations, but you can measure that.

As to calculating values inside MV fields, if you have Splunk 8 I think, then mvmap may be able to get you to where you want, I couldn't get it to work though - I've not used it before.

View solution in original post

bowesmana
SplunkTrust
SplunkTrust

I believe you will have to mvexpand out the fields first to get the mv field into separate rows and then run streamstats. Take this for example

| makeresults
| eval _raw="sysmodtime,idnumber,epoch time
05/03/20 12:40 PM,1,1588502400
05/01/20 12:01 AM,1,1588284060
05/01/20 12:02 AM,1,1588284120
05/01/20 12:02 AM,1,1588284120
05/02/20 12:00 PM,2,1588413600
04/02/20 12:00 AM,2,1585778400
04/02/20 01:00 AM,2,1585782000
04/02/20 02:00 AM,3,1585785600"
| multikv
| stats list(*) as * by idnumber
| eval x="So, up to here is your original data"
| eval base=mvzip(sysmodtime,epoch_time)
| fields - _raw linecount sysmodtime epoch_time
| mvexpand base
| rex field=base "(?<sysmodtime>[^,]*),(?<epoch_time>.*)"
| fields - base
| streamstats window=2 range(epoch_time) as time_diff by idnumber
| stats list(*) as * by idnumber
| eval time_diff=mvindex(time_diff,1,-1)

That gives you what you want, but will be expensive to do the mvexpand and re-aggregations, but you can measure that.

As to calculating values inside MV fields, if you have Splunk 8 I think, then mvmap may be able to get you to where you want, I couldn't get it to work though - I've not used it before.

rvsroe
Explorer

This worked perfectly, thank you very much, I've only added a strftime to convert epoch into hours and minutes:

 | eval time_diff=strftime(mvindex(time_diff,1,-1),"%H:%M:%S") 

forum1.PNG

 

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

If I understand your data correctly, you may be able to use streamstats.

... | streamstats window=2 range('epoch timestamp') as time_diff
---
If this reply helps you, Karma would be appreciated.
0 Karma

rvsroe
Explorer

Thanks for the reply, streamstats was indeed helpful, I've accepted the other reply as answer though for its completeness

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...