Hi Lowell, I implemented the deduplication and sorting functionality in a custom command. Being your experience far greater than mine you won't have any problem to remove the deduplication logic (and maybe suggest any improvement 😉
Syntax:
| mvdedup [[+|-]fieldname ]*
with no parameter: will dedup all the multivalued fields retaining their order
with one or more fieldnames: will dedup those fields retaining their order
with one or more fieldnames prepended by a +|- (no empty space there!): will dedup and sort ascending/descending
| mvdedup -id
Here's the configs:
commands.conf
[mvdedup]
type = python
streaming = true
maxinputs = 500000
run_in_preview = true
enableheader = true
retainsevents = true
generating = false
generates_timeorder = false
supports_multivalues = true
supports_getinfo = true
mvdedup.py
import sys
import splunk.Intersplunk as si
import string
def uniqfy(seq,sortorder=None):
seen = {}
result = []
for item in seq:
if item in seen: continue
seen[item] = 1
result.append(item)
if sortorder=='+':
result.sort()
elif sortorder=='-':
result.sort(reverse=True)
return result
(isgetinfo, sys.argv) = si.isGetInfo(sys.argv)
if isgetinfo:
#outputInfo(streaming, generating, retevs, reqsop, preop, timeorder=False):
si.outputInfo(True, False, True, False, None, False)
sys.exit(0)
results = si.readResults(None, None, True)
fields={}
if len(sys.argv) > 1:
for a in sys.argv[1:]:
a=str(a)
if a[:1] in ['+','-']:
# set sorting order
fields[a[1:]] = a[:1]
else:
# no sorting!
fields[a] = None
else:
# dedup on all the fields in the data
for k in results[0].keys():
fields[k] = None
for i in range(len(results)):
for key in results[i].keys():
if(isinstance(results[i][key], list)):
if key in fields.keys():
results[i][key] = uniqfy(results[i][key],fields[key])
si.outputResults(results)
Any suggestion is more than welcome
... View more