Splunk Search

finding repetition in array of logs

exocore123
Path Finder

I have a field in my logs that contains an array of string elements. Is there a way to detect for repeating strings and parse it differently within the search query for a dashboard?

array = ["string1", "string2","string1"] regardless of however many repetition

into

array = "extra(string1), string2"

0 Karma
1 Solution

woodcock
Esteemed Legend

Like this:

| makeresults 
| eval array="[\"string1\", \"string2\", \"string1\"]
[\"string1\", \"string2\", \"string3\"]
[\"string2\", \"string2\", \"string1\"]"
| makemv delim="
" array
| mvexpand array

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| fields array
| rex field=array mode=sed "s/\"//g s/^\[// s/\]$//"
| eval array=split(array, ",")
| rex field=array mode=sed "s/^\s+//"
| streamstats count AS _serial
| mvexpand array
| stats count BY array _serial
| eval array=if((count=1), array, "Extra(" . array . ")")
| stats values(array) AS array BY _serial

Then maybe you would like to add:

| nomv array
| eval array = "[" . array . "]"

View solution in original post

0 Karma

woodcock
Esteemed Legend

Like this:

| makeresults 
| eval array="[\"string1\", \"string2\", \"string1\"]
[\"string1\", \"string2\", \"string3\"]
[\"string2\", \"string2\", \"string1\"]"
| makemv delim="
" array
| mvexpand array

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| fields array
| rex field=array mode=sed "s/\"//g s/^\[// s/\]$//"
| eval array=split(array, ",")
| rex field=array mode=sed "s/^\s+//"
| streamstats count AS _serial
| mvexpand array
| stats count BY array _serial
| eval array=if((count=1), array, "Extra(" . array . ")")
| stats values(array) AS array BY _serial

Then maybe you would like to add:

| nomv array
| eval array = "[" . array . "]"
0 Karma

exocore123
Path Finder

I am trying to add another stats line at the end of line 19, but seems to return 0 results. stats count by requests, array , do you have any suggestion on how to use this as well as using stats within the same query?

0 Karma

woodcock
Esteemed Legend

The fields array command drops all fields other than array so you need to change line 11 to fields array requests and you also need to change line 19 to stats values(array) AS array first(requests) AS requests BY _serial.

0 Karma

exocore123
Path Finder

stats values(array) AS array first(requests) AS request by _serial with that, my request field is still empty, I think the by _serial possibly?

0 Karma

woodcock
Esteemed Legend

OK, like this:

| makeresults 
| eval array="[\"string1\", \"string2\", \"string1\"]
[\"string1\", \"string2\", \"string3\"]
[\"string2\", \"string2\", \"string1\"]"
| makemv delim="
" array
| mvexpand array
| streamstats count AS results
| eval results=results+10

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| fields array results
| rex field=array mode=sed "s/\"//g s/^\[// s/\]$//"
| eval array=split(array, ",")
| rex field=array mode=sed "s/^\s+//"
| streamstats count AS _serial
| mvexpand array
| stats count first(results) AS results BY array _serial
| eval array=if((count=1), array, "Extra(" . array . ")")
| stats values(array) AS array first(results) AS results BY _serial

exocore123
Path Finder

Hello Woodcock, sorry but to follow up, what happens if I want to remove the repetitions without splitting it, so for example Extra(string1 + string2) for array = ["string1", "string2","string1"]

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi exocore123,
are you speaking about a multi values field that you want to put in different events?
if this is your need you should try makemv and mvexpand commands, something like this:

your_search
| makemv multi_values_field
| mvexpand multi_values_field
| ...

Bye.
Giuseppe

0 Karma

exocore123
Path Finder

It is not really multi-value? More of just a different representation

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi exocore123,
sorry but I don't understand your situation, could you share a sample of your logs and the expected output?
Bye.
Giuseppe

0 Karma

exocore123
Path Finder

I dont care for the number of repetitions, I just want to know if there are repetition of the string within that field called array, then on my tabular dashboard, I want to display it as extra(...) , (rest of the strings). For instance with the example above, say the array within logs are displayed as array=["string1", "string2", "string1"], I want to be able to parse it as "Extra(string1), string2" on my dashboard. Array can have repetitive values, or different values across the board, assume we do not know array's length.

0 Karma

woodcock
Esteemed Legend

Do you have a field called array right now? Does it have 1 value, 2 values, or 3 values for your example?
What EXACTLY would you like to do with array?

0 Karma

HiroshiSatoh
Champion

I think that it is possible to search by string search if you want to search. What exactly do you want to do?
Do you want to know the number of repetitions?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...