topic Re: Search not returning results in Splunk Search

Why is my Search not returning results?

jymmitch — Fri, 08 Apr 2022 14:48:07 GMT

Here's the text string from the log I'm searching:

store license for Store 123456
2022-04-07 19:17:44,360 ERROR path not found

Here's my splunk search:

index=* host="storelog*" "store license for " |rex field=_raw "Store\s123456\n\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" | stats count by errortext

Why am I getting the following when I search?

No results found.

Re: Search not returning results

newrose — Thu, 07 Apr 2022 20:06:01 GMT

Is the errortext field created by the rex command being populated correctly? One possible explanation could be that the regex is not matching the text in the events.

You can execute the search again without the stats command and see if the errortext field is present for the desired events.

Re: Search not returning results

jymmitch — Thu, 07 Apr 2022 20:11:18 GMT

I took off the stats command and reran the search, but the errortext field is NOT in the events.

Re: Search not returning results

mayurr98 — Thu, 07 Apr 2022 20:23:45 GMT

could you please share more sample events as I do not see any error in your search.

I have tried in this run anywhere search

| makeresults | eval _raw="store license for Store 123456 2022-04-07 19:17:44,360 ERROR path not found" | rex field=_raw "Store\s123456\n\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" | stats count by errortext

From the regex I could see that you are searching for "Store 123456", please add that in the main search instead of in regex.

index=* host="storelog*" "store license for Store 123456"

Also is it a multiline event? that means is timestamp is on new line in raw logs or its just one line?

Re: Search not returning results

jymmitch — Thu, 07 Apr 2022 20:46:15 GMT

Timestamp is on a new line

Re: Search not returning results

jymmitch — Thu, 07 Apr 2022 20:49:26 GMT

I added "Store 123456" to main search and removed it from regex, but still get "no results found."

Re: Search not returning results

newrose — Thu, 07 Apr 2022 20:59:51 GMT

The regex will not work considering the sample events you provided.

If the event starts with the timestamp and is single lined maybe you should try this regex:

| rex field=_raw "\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path"

Re: Search not returning results

jymmitch — Thu, 07 Apr 2022 21:12:00 GMT

There is a timestamp preceding the text in my main search, but there is also a timestamp that follows it. I'm only searching from the main search text forward:

raw log text:
store license for Store 123456
2022-04-07 19:17:44,360 ERROR path not found

index=* host="storelog*" "store license for " |rex field=_raw "Store\s123456\n\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" | stats count by errortext

Based on my search, I would think splunk should recognize the "store license for " text, then recognize the single whitespace, then recognize the "Store\s123456" text in the regex, then recognize the newline (\n), then recognize the date/timestamp, and finally set errortext field to ERROR.

Re: Search not returning results

PickleRick — Thu, 07 Apr 2022 21:42:57 GMT

I won't give you a solution because I tend to avoid such events 😉 but the main problem is most probably the fact that your event is multiline and by default regex matches only over single line.

Try for yourself:

| makeresults 
| eval a="abcd
fghi"
| rex field=a "(?<b>.*)"

You will only get the first line as a results.

You have to fiddle with the single/multiline regex modifiers to make your rex work across the newline. Might work with the (?s) modifier at the beginning of your expression.

Re: Search not returning results

jymmitch — Thu, 07 Apr 2022 21:53:56 GMT

I have other searches that work where the regex matches over two lines. The only difference between those and this one is the fact that this one includes a datestamp (the others don't).

Re: Search not returning results

newrose — Thu, 07 Apr 2022 22:32:39 GMT

If I correctly understood your logic, it isn't exactly how Splunk is interpreting your query, although the selected events will probably be the same.

Your base search,

index=* host="storelog*" "store license for "

will extract all events which have the "store license for " string, including the single whitespace.

Then it will run the rex over all the selected commands, and the regex will try the match starting from the beginning of the event, not from where you stopped from the previous command.

To overcome the newline issue, check if it is possible given your dataset to run something like this:

index=* host="storelog*" "store license for Store 123456" 
| rex field=_raw "\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" 
| stats count by errortext

The "Store 123456" was moved to the main search, and the regex will try to match starting from the timestamp.

Also you should probably look over this data input parameters, as the raw events doesn't look to have the right boundaries from what you showed here.

Re: Search not returning results

PickleRick — Fri, 08 Apr 2022 08:25:03 GMT

It sounds kinda strange since the single/multiline behaviour is repeatable. And while matching regex to a string Splunk doesn't care whether the event contains timestamp, social securiyy number, your shoe size or whatever other data you can have. It's just text - if it fits, it fits.

OK. It's interesting because sometimes indeed (as in my example) Splunk treats regexes as single line, sometimes as multiline.

Anyway, your initial data sample together with your regex works for me.

BTW, your "no results found" might mean that no events matched your search, not that the rex command extracted the field wrongly or not at all.

See the "Inspect Job" and see if you got anything from the initial search.

Re: Search not returning results

jymmitch — Fri, 08 Apr 2022 13:49:53 GMT

That worked!