Ignore field when searching for arbitrary string(s) but still return its results.

[shawngsharp]

I am trying to do a query that will search for arbitrary strings, but will ignore if the string is/isn't in a specific field. I still want to see the results from that field, though.

Example:

index = my_index AND *widget*
| <ignore>  my_field_42

Whether my_field_42 contains the word "widget" or not should not matter to the search, but it should still show it's field values in the results. 

Result 1:

my_field_1 = "hello world" 
my_field_2 = "AwesomeWidget69" 
...
my_field_42 = "your mom"

Result 2:

my_field_1 = "hello world" 
my_field_23 = "Widgets are cool" 
...
my_field_42 = "Look, a widget!"

[PickleRick]

As I love border cases I'm wondering if @yuanliu @bowesmana and @livehybrid 's solutions handle properly the situation where not all event is parsed into events. In this case you cannot simply iterate over all extracted fields because the string may be contained within a part of the _raw event which is not extracted. So if you only manipulate fields, you won't catch it.

Another thing to consider - automatic lookups and calculated fields.

Nobody said it would be easy 😉

[yuanliu]

I would go with foreach as @livehybrid does, but the code could be simpler.

|foreach *
  [eval <<FIELD>> = if(match(<<FIELD>>, "(?i)widget") OR "<<FIELD>>" == "my_field_42", <<FIELD>>, null())]

Using the same emulation, you get

my_field1	my_field_2	my_field_23	my_field_42
	AwesomeWidget69		your mom
		Widgets are cool	Look, a widget!
			your widget

[bowesmana]

What should happen if the data is

my_field_1 = "hello world" 
my_field_23 = "goodbye my friend" 
...
my_field_42 = "Look, a widget!"

i.e. widget ONLY appears in the field you want to ignore

[shawngsharp]

Great question! In that case, it shouldn't return the result(s).

[livehybrid]

Hi @shawngsharp 

Further to my last post, you could also use:

|foreach * [eval field_matches = mvappend(field_matches, if(match(<<FIELD>>, "(?i)widget"), "<<FIELD>>", null()))]
| eval field_matches=mvfilter(NOT match(field_matches,"my_field_42"))
| where field_matches!=""

Where your string match is inside the match statement, this works by looking in each field and then creating a multi-value field of all the fields which match, then removing my_field_42 and searching where there is one or more fields that match.

|makeresults format=csv data="my_field1, my_field_2, my_field_23, my_field_42
\"hello world\",\"AwesomeWidget69\",\"\",\"your mom\"
\"hello world\",\"\",\"Widgets are cool\",\"Look, a widget!\"
\"hello world\",\"\",\"Some value here\",\"your widget\""
|foreach * [eval field_matches = mvappend(field_matches, if(match(<<FIELD>>, "(?i)widget"), "<<FIELD>>", null()))]
| eval field_matches=mvfilter(NOT match(field_matches,"my_field_42"))
| where field_matches!=""

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[livehybrid]

Hi @shawngsharp 

So I think I know what you're looking for - you dont want it to match if *only* my_field_42 contains the string? So it must be in one of the other fields?

You could try this - Im not sure how performant it will be at scale but working for me:

index=YourIndex *widget*
| tojson 
| eval orig_field_42=json_extract(_raw,"my_field_42") 
| eval _raw=json_delete(_raw,"my_field_42") 
| search *widget* 
| eval _raw=json_set(_raw,"my_field_42",orig_field_42)

This works by temporarily removing the my_field_42 from the results before applying a secondary search - Ive gone with "tojson" which converts all the fields into json object in _raw. 

Below is a sample query if it helps:

|makeresults format=csv data="my_field1, my_field_2, my_field_23, my_field_42
\"hello world\",\"AwesomeWidget69\",\"\",\"your mom\"
\"hello world\",\"\",\"Widgets are cool\",\"Look, a widget!\"
\"hello world\",\"\",\"Some value here\",\"your widget\""
| tojson 
| eval orig_field_42=json_extract(_raw,"my_field_42") 
| eval _raw=json_delete(_raw,"my_field_42") 
| search *widget*

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing