topic Re: finding repetition in array of logs in Splunk Search

finding repetition in array of logs

exocore123 — Wed, 21 Jun 2017 03:58:47 GMT

I have a field in my logs that contains an array of string elements. Is there a way to detect for repeating strings and parse it differently within the search query for a dashboard?

array = ["string1", "string2","string1"] regardless of however many repetition

into

array = "extra(string1), string2"

Re: finding repetition in array of logs

HiroshiSatoh — Wed, 21 Jun 2017 04:27:55 GMT

I think that it is possible to search by string search if you want to search. What exactly do you want to do?
Do you want to know the number of repetitions?

Re: finding repetition in array of logs

woodcock — Wed, 21 Jun 2017 04:53:37 GMT

Do you have a field called array right now? Does it have 1 value, 2 values, or 3 values for your example?
What EXACTLY would you like to do with array?

Re: finding repetition in array of logs

exocore123 — Wed, 21 Jun 2017 05:51:20 GMT

I dont care for the number of repetitions, I just want to know if there are repetition of the string within that field called array, then on my tabular dashboard, I want to display it as extra(...) , (rest of the strings). For instance with the example above, say the array within logs are displayed as array=["string1", "string2", "string1"], I want to be able to parse it as "Extra(string1), string2" on my dashboard. Array can have repetitive values, or different values across the board, assume we do not know array's length.

Re: finding repetition in array of logs

gcusello — Wed, 21 Jun 2017 07:07:44 GMT

Hi exocore123,
are you speaking about a multi values field that you want to put in different events?
if this is your need you should try makemv and mvexpand commands, something like this:

your_search
| makemv multi_values_field
| mvexpand multi_values_field
| ...

Bye.
Giuseppe

Re: finding repetition in array of logs

exocore123 — Wed, 21 Jun 2017 07:30:36 GMT

It is not really multi-value? More of just a different representation

Re: finding repetition in array of logs

gcusello — Wed, 21 Jun 2017 07:36:37 GMT

Hi exocore123,
sorry but I don't understand your situation, could you share a sample of your logs and the expected output?
Bye.
Giuseppe

Re: finding repetition in array of logs

woodcock — Wed, 21 Jun 2017 16:59:18 GMT

Like this:

| makeresults 
| eval array="[\"string1\", \"string2\", \"string1\"]
[\"string1\", \"string2\", \"string3\"]
[\"string2\", \"string2\", \"string1\"]"
| makemv delim="
" array
| mvexpand array

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| fields array
| rex field=array mode=sed "s/\"//g s/^\[// s/\]$//"
| eval array=split(array, ",")
| rex field=array mode=sed "s/^\s+//"
| streamstats count AS _serial
| mvexpand array
| stats count BY array _serial
| eval array=if((count=1), array, "Extra(" . array . ")")
| stats values(array) AS array BY _serial

Then maybe you would like to add:

| nomv array
| eval array = "[" . array . "]"

Re: finding repetition in array of logs

exocore123 — Thu, 22 Jun 2017 17:35:28 GMT

I am trying to add another stats line at the end of line 19, but seems to return 0 results. stats count by requests, array , do you have any suggestion on how to use this as well as using stats within the same query?

Re: finding repetition in array of logs

woodcock — Thu, 22 Jun 2017 17:49:09 GMT

The fields array command drops all fields other than array so you need to change line 11 to fields array requests and you also need to change line 19 to stats values(array) AS array first(requests) AS requests BY _serial.

Re: finding repetition in array of logs

exocore123 — Thu, 22 Jun 2017 18:06:19 GMT

stats values(array) AS array first(requests) AS request by _serial with that, my request field is still empty, I think the by _serial possibly?

Re: finding repetition in array of logs

woodcock — Thu, 22 Jun 2017 18:24:09 GMT

OK, like this:

| makeresults 
| eval array="[\"string1\", \"string2\", \"string1\"]
[\"string1\", \"string2\", \"string3\"]
[\"string2\", \"string2\", \"string1\"]"
| makemv delim="
" array
| mvexpand array
| streamstats count AS results
| eval results=results+10

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| fields array results
| rex field=array mode=sed "s/\"//g s/^\[// s/\]$//"
| eval array=split(array, ",")
| rex field=array mode=sed "s/^\s+//"
| streamstats count AS _serial
| mvexpand array
| stats count first(results) AS results BY array _serial
| eval array=if((count=1), array, "Extra(" . array . ")")
| stats values(array) AS array first(results) AS results BY _serial

Re: finding repetition in array of logs

exocore123 — Tue, 27 Jun 2017 19:00:49 GMT

Hello Woodcock, sorry but to follow up, what happens if I want to remove the repetitions without splitting it, so for example Extra(string1 + string2) for array = ["string1", "string2","string1"]