Getting Data In

How to merge events with identical timestamps into one event, but drop all differing data?

david_rundle_fi
Explorer

I have the following 9 events with the identical timestamps, but differing information:

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, queue_len, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, files_skipped, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, buildup_skips, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, malware_detected, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, scans_canceled, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, scans_completed, null, null, null, null, null, null, null, null, null, null, 1461, 6735, 8101, 3869, 20166, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, spf_reads, null, null, null, null, null, 1401, 6342, 8101, 3869, 19713, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, spf_writes, 1401, 6342, 8101, 3869, 19713, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, spoolc_drops, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 0

I want to drop the event type (spfreads, spfwrites, etc) and the null values, and combine the events into a single event.

How can I do this?

Tags (3)
0 Karma

kristian_kolb
Ultra Champion

Well, there is perhaps a far more attractive option;
Drop the stuff you don't want (the type) by setting it to the same value everywhere, then make use of the stats max() function. You may first need to replace the string 'null' with a real NULL value, if that is what you have. Or perhaps not. At least in my test you don't

your search | eval type = "Combined" | stats max(*) by _time

This should look something like;

2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, Combined, 1401, 6432, 1234, 3424, 7663, 2342, null, null, 8787, 1461, 6735, 8101, 3869, 20166, null, null, null, null, null, null, null, 0, 0, 0, 0, null, null, null, null, null, null, null, null, null, 5435, 123, 0, 6676, null, null, null, null, null, 0

i.e. if there is a field that does not have a value in either of the events, the combined event will still have 'null'. Otherwise, the highest value will take that place.

For presentation purposes you might then want play with fields, replace, table, or rename etc.

Hope this helps,

K

0 Karma

david_rundle_fi
Explorer

Ok, this is a good start - I ran the transaction on the timestamp, as this is a performance stats collection that is running every 5 minutes on multiple devices.(Session ID's)

I now have a single event that is the composite of the 9 event types.

Any way to remove the duplicate null values? (dedup on each field name?)

0 Karma

kristian_kolb
Ultra Champion

You could probably submit some more info, especially on just how you want the combined information to look like. One thing that you might try is the transaction command.

Assuming that the KQ25B6P is some sort of SessionID, perhaps ... | transaction SessionID max_span=1s | might work for you.

/k

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...