Why is my Search not returning results?

[jymmitch]

Here's the text string from the log I'm searching:

store license for Store 123456
2022-04-07 19:17:44,360 ERROR path not found

Here's my splunk search:

index=* host="storelog*" "store license for " |rex field=_raw "Store\s123456\n\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" | stats count by errortext

Why am I getting the following when I search?

No results found.

[PickleRick]

I won't give you a solution because I tend to avoid such events 😉 but the main problem is most probably the fact that your event is multiline and by default regex matches only over single line.

Try for yourself:

| makeresults 
| eval a="abcd
fghi"
| rex field=a "(?<b>.*)"

You will only get the first line as a results.

You have to fiddle with the single/multiline regex modifiers to make your rex work across the newline. Might work with the (?s) modifier at the beginning of your expression.

[jymmitch]

I have other searches that work where the regex matches over two lines. The only difference between those and this one is the fact that this one includes a datestamp  (the others don't).

[PickleRick · Accepted solution]

It sounds kinda strange since the single/multiline behaviour is repeatable. And while matching regex to a string Splunk doesn't care whether the event contains timestamp, social securiyy number, your shoe size or whatever other data you can have. It's just text - if it fits, it fits.

OK. It's interesting because sometimes indeed (as in my example) Splunk treats regexes as single line, sometimes as multiline.

Anyway, your initial data sample together with your regex works for me.

BTW, your "no results found" might mean that no events matched your search, not that the rex command extracted the field wrongly or not at all.

See the "Inspect Job" and see if you got anything from the initial search.

[jymmitch]

That worked!

[newrose]

Is the  errortext  field created by the rex command being populated correctly? One possible explanation could be that the regex is not matching the text in the events.

You can execute the search again without the stats command and see if the  errortext  field is present for the desired events.

[jymmitch]

I took off the stats command and reran the search, but the errortext field is NOT in the events.

[mayurr98]

could you please share more sample events as I do not see any error in your search.

I have tried in this run anywhere search

| makeresults 
| eval _raw="store license for Store 123456
2022-04-07 19:17:44,360 ERROR path not found" 
| rex field=_raw "Store\s123456\n\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" 
| stats count by errortext

From the regex I could see that you are searching  for "Store 123456", please add that in the main search instead of in regex.

index=* host="storelog*" "store license for Store 123456"

Also is it a multiline event? that means is timestamp is on new line in raw logs or its just one line?

[newrose]

The regex will not work considering the sample events you provided.

If the event starts with the timestamp and is single lined maybe you should try this regex:

| rex field=_raw "\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path"

[jymmitch]

There is a timestamp preceding the text in my main search, but there is also a timestamp that follows it. I'm only searching from the main search text forward:

raw log text:
store license for Store 123456
2022-04-07 19:17:44,360 ERROR path not found

index=* host="storelog*" "store license for " |rex field=_raw "Store\s123456\n\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" | stats count by errortext

Based on my search, I would think splunk should recognize the "store license for " text, then recognize the single whitespace, then recognize the "Store\s123456" text in the regex, then recognize the newline (\n), then recognize the date/timestamp, and finally set errortext field to ERROR.

[newrose]

If I correctly understood your logic, it isn't exactly how Splunk is interpreting your query, although the selected events will probably be the same.

Your base search,

index=* host="storelog*" "store license for "

will extract all events which have the "store license for " string, including the single whitespace.

Then it will run the rex over all the selected commands, and the regex will try the match starting from the beginning of the event, not from where you stopped from the previous command.

To overcome the newline issue, check if it is possible given your dataset to run something like this:

index=* host="storelog*" "store license for Store 123456" 
| rex field=_raw "\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}\s(?P<errortext>.*)path" 
| stats count by errortext

 The "Store 123456" was moved to the main search, and the regex will try to match starting from the timestamp.

Also you should probably look over this data input parameters, as the raw events doesn't look to have the right boundaries from what you showed here.

[jymmitch]

I added "Store 123456" to main search and removed it from regex, but still get "no results found."

[jymmitch]

Timestamp is on a new line