topic Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field? in Splunk Search

Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Wed, 15 Mar 2017 15:25:17 GMT

Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

edit: here's what I'm trying to do
ie, "eval myField=( "value1", "value2", "value3") | stats count by myField"

Where "value1", "value2", "value3" are literal strings.

I want to get a count for how many "value1"s, "value2"s, and "value3"s there are

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

somesoni2 — Wed, 15 Mar 2017 15:31:44 GMT

Could you please post some sudo query on what you want to achieve, possible with some examples?

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

koshyk — Wed, 15 Mar 2017 15:33:40 GMT

Not quite clear from your question, but an example would be helpful.

But just a guess, are you looking for something like this ? https://answers.splunk.com/answers/103700/how-do-i-create-a-field-whose-name-is-the-value-of-another-field-like-backticks-or-eval-in-other-languages.html

|makeresults| eval aKey="Field1" | eval aValue=123 | eval {aKey}=aValue | table aKey,aValue,Field1

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Wed, 15 Mar 2017 15:34:27 GMT

ie, "eval myField=( "value1", "value2", "value3") | stats count by myField"

Where "value1", "value2", "value3" are literal strings.

I want to get a count for how many "value1"s, "value2"s, and "value3"s there are

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

somesoni2 — Wed, 15 Mar 2017 15:53:20 GMT

How are the values for string "value1", "value2"... coming, static/fix string or dynamically?

If they are static/fixed and limited, something like this would work.

...| eval myField=if(match(myField,"value1"),"value1", match(myField,"value2"),"value2", match(myField,"value3"),"value3") | stats count by myField

If they are move in number, you can put them in a lookup table file say myfield_value.csv with column name as myfieldvalue, and try like this

...| lookup myfield_value.csv myfieldvalue as myField OUTPUT myfieldvalue as myField  | stats count by myField

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Wed, 15 Mar 2017 21:11:50 GMT

Assuming that you are just matching strings in the raw events (the strings are not accessed by a field name), then like this:

Your Base Search Here | stats 
[| makeresults 
 | eval errorMsg="value1::value2::INFO" 
 | makemv delim="::" errorMsg
 | format "" "" "" "" "" "" 
 | rex field=search mode=sed "s/\( errorMsg=| OR errorMsg=/ count(eval(searchmatch(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\"))) AS \"\1\"/g"]

If you do indeed have field names that contain these strings, then like this (you might have to change the field name errorMsg to your field name):

Your Base Search Here | stats 
[| makeresults 
| eval errorMsg="value1::value2::value2" 
| makemv delim="::" errorMsg 
| rex field=errorMsg mode=sed "s/[\r\n]//g"
| format "" "" "" "" "" "" 
| rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\")) AS \"\1\"/g"]

You can make this into a macro.

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Thu, 16 Mar 2017 15:47:27 GMT

Now it throws the following error: Error in 'stats' command: You must specify a rename for the aggregation specifier on the dynamically evaluated field 'count(eval( errorMsg="at the below stack trace. Not closed in the same method"))'.

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Thu, 16 Mar 2017 18:28:05 GMT

Try replacing the last line with this:

| rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"(\w+)\"\)\)/\"\1\")) AS \1/g"

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Thu, 16 Mar 2017 18:28:27 GMT

Answer updated to account for spacing variances in format command.

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Thu, 16 Mar 2017 18:37:37 GMT

Still throwing the same error

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Thu, 16 Mar 2017 18:39:56 GMT

This is the full search:

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available" 
) | stats 
 [| makeresults | eval errorMsg="at the below stack trace. Not closed in the same method::
com.digev.fw.exception.GException: Execution of a DB command failed::
com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERR::
com.mongodb.MongoSocketOpenException::
com.mongodb.MongoTimeoutException::
Data truncation::
ERR [DBStatementAndResultSetTracker] PreparedStatementTracker::
Error encountered in WS-Security engine::
Error in creating Prepared statement for the query::
federation member auth token cannot be refreshed::
GC overhead limit exceeded::
Illegal character::
java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer::
java.lang.StackOverflowError::
Log block not closed correctly. Enable log block tracking to see diagnostic information::
Log frame is closed at the below stack trace::
No corresponding startTraceBlock() is seen::
No key found in WSDL for service::
No process found::
No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error::
OutOfMemoryError::
Timeout waiting for idle object::
Unable to initialize SiteMinder agent::
UsageJDBCWriter.writeUsage::
Wsdl does not conform to wsdl schema::
org.elasticsearch.action.UnavailableShardsException::
None of the configured nodes are available::
Cannot get a connection, pool exhausted" | makemv delim="::" errorMsg
 | format "" "" "" "" "" "" | rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"(\w+)\"\)\)/\"\1\")) AS \1/g"]

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Thu, 16 Mar 2017 19:16:46 GMT

Now we are talking. The missing piece was that your values have spaces. See my updated answer; it is tested and working but the field names are TERRIBLE.

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Thu, 16 Mar 2017 19:31:38 GMT

What I'm trying to get is a count of how many times each string appears per unit time. That doesn't seem to be happening when I run the amended search:

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available" 
) | stats 
 [| makeresults | eval errorMsg="at the below stack trace. Not closed in the same method::
com.digev.fw.exception.GException: Execution of a DB command failed::
com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERR::
com.mongodb.MongoSocketOpenException::
com.mongodb.MongoTimeoutException::
Data truncation::
ERR [DBStatementAndResultSetTracker] PreparedStatementTracker::
Error encountered in WS-Security engine::
Error in creating Prepared statement for the query::
federation member auth token cannot be refreshed::
GC overhead limit exceeded::
Illegal character::
java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer::
java.lang.StackOverflowError::
Log block not closed correctly. Enable log block tracking to see diagnostic information::
Log frame is closed at the below stack trace::
No corresponding startTraceBlock() is seen::
No key found in WSDL for service::
No process found::
No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error::
OutOfMemoryError::
Timeout waiting for idle object::
Unable to initialize SiteMinder agent::
UsageJDBCWriter.writeUsage::
Wsdl does not conform to wsdl schema::
org.elasticsearch.action.UnavailableShardsException::
None of the configured nodes are available::
Cannot get a connection, pool exhausted" | makemv delim="::" errorMsg
 | rex field=errorMsg mode=sed "s/[\r\n]//g"| format "" "" "" "" "" "" | rex field=search mode=sed "s/\(| OR / count(eval(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\")) AS \"\1\"/g"] | stats count by errorMsg

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Thu, 16 Mar 2017 20:02:28 GMT

Drop this part:

| stats count by errorMsg

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Thu, 16 Mar 2017 20:04:33 GMT

Why does this not work?

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available"
| stats count by errorMsg

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Fri, 17 Mar 2017 12:40:27 GMT

That produces a table, which doesn't seem to be picking up the counts of the strings. Each string in the table says 0, but that's not accurate

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Fri, 17 Mar 2017 12:41:02 GMT

Because errorMsg isn't a native field in the data. It's one I have to create

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Sat, 18 Mar 2017 22:20:56 GMT

OK, try the updated answer (the top one of the 2).

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

woodcock — Sat, 18 Mar 2017 22:21:49 GMT

Then my other solution ABSOLUTELY POSITIVELY should work (the one that is now the bottom one in the pair of the other answer).

Re: Is there a way to search for a list of strings, and for each match, put that string as the value of the same field?

Lucas_Henry_ — Mon, 20 Mar 2017 14:19:20 GMT

Making progress. The search produces a table with counts for the frequency of each literal string, but the search itself does not seem to produce the errorMsg field itself when searching in Verbose mode. I would like to be able to produce the errorMsg field so I can add it to tables, or grab the errorMsg value for alerts, etc

The full search query is below

index=its_akana* source="/apps/logs/*" host=ent5*ll5app ("at the below stack trace. Not closed in the same method" OR
"Cannot get a connection, pool exhausted" OR
"com.digev.fw.exception.GException: Execution of a DB command failed" OR
"com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERROR" OR
"com.mongodb.MongoSocketOpenException" OR
"com.mongodb.MongoTimeoutException" OR
"Data truncation" OR
"ERROR [DBStatementAndResultSetTracker] PreparedStatementTracker" OR
"Error encountered in WS-Security engine" OR
"Error in creating Prepared statement for the query" OR
"federation member auth token cannot be refreshed" OR
"GC overhead limit exceeded" OR
"Illegal character" OR
"java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer" OR
"java.lang.StackOverflowError" OR
"Log block not closed correctly. Enable log block tracking to see diagnostic information" OR
"Log frame is closed at the below stack trace" OR
"No corresponding startTraceBlock() is seen" OR
"No key found in WSDL for service" OR
"No process found" OR
"No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error" OR
"OutOfMemoryError" OR
"Timeout waiting for idle object" OR
"Unable to initialize SiteMinder agent" OR
"UsageJDBCWriter.writeUsage" OR
"Wsdl does not conform to wsdl schema" OR
"org.elasticsearch.action.UnavailableShardsException" OR
"None of the configured nodes are available" 
) | stats
 [| makeresults | eval errorMsg="at the below stack trace. Not closed in the same method::
com.digev.fw.exception.GException: Execution of a DB command failed::
com.digev.fw.exception.GException: javax.wsdl.WSDLException: WSDLException: faultCode=OTHER_ERR::
com.mongodb.MongoSocketOpenException::
com.mongodb.MongoTimeoutException::
Data truncation::
ERR [DBStatementAndResultSetTracker] PreparedStatementTracker::
Error encountered in WS-Security engine::
Error in creating Prepared statement for the query::
federation member auth token cannot be refreshed::
GC overhead limit exceeded::
Illegal character::
java.lang.NullPointerException at com.soa.jbi.component.http.marshal.impl.OutgoingExchangeInitializer::
java.lang.StackOverflowError::
Log block not closed correctly. Enable log block tracking to see diagnostic information::
Log frame is closed at the below stack trace::
No corresponding startTraceBlock() is seen::
No key found in WSDL for service::
No process found::
No Subject is associated with the call. Only Container identities can invoke this call. Returning authorization error::
OutOfMemoryError::
Timeout waiting for idle object::
Unable to initialize SiteMinder agent::
UsageJDBCWriter.writeUsage::
Wsdl does not conform to wsdl schema::
org.elasticsearch.action.UnavailableShardsException::
None of the configured nodes are available::
Cannot get a connection, pool exhausted" | makemv delim="::" errorMsg
  | format "" "" "" "" "" "" 
  | rex field=search mode=sed "s/\( errorMsg=| OR errorMsg=/ count(eval(searchmatch(/g s/\" count\(/\")) count(/g s/\s*\)  $/))/ s/\"([^\"]+)\"\)\)/\"\1\"))) AS \"\1\"/g"]