How to edit my search to match any string containing "PIN_ERR_STORAGE"?

[mmilano]

Here is the search:
index="brm" host="a-brmapp*" source="/opt/portal/pin/7.5/var/cm/cm.pinlog" PIN_ERR_STORAGE

Now, it DOES find two matches. They are as follows:

D Wed Jan 7 17:10:09 2015 a-brmapp1.corp.com cm:37041 cm_child.c(120):1016 1:a-brmpipe1:UnknownProgramName:0:AWT-EventQueue-0:7:1420650609:0 cm_pcm_op_call_stack ... 3 lines omitted ... 0.001977829 .Exit PCM_OP_SEARCH (0x0) - from DM 0.0.0.1 0.002006352 Exit PCM_OP_SEARCH (0x0) PIN_ERR_STORAGE

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

The first match was a test whereby we echoed the string (PIN_ERR_STORAGE) into the log. It matched this one and triggered an alert. The second match is an actual entry in the log file. Running the search manually finds the entry, but, it doesn't trigger an alert.

It seems the only way to get this to work is to include the entire string "errno=PIN_ERR_STORAGE" as follows:
index="brm" host="a-brmapp*" source="/opt/portal/pin/7.5/var/cm/cm.pinlog" "errno=PIN_ERR_STORAGE"

Result:

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

Can anyone please tell me how I can match ANY string containing "PIN_ERR_STORAGE"?

Thank you in advance!

[lguinn2]

You might want to write the search as

index=brm host=a-brmapp* source=/opt/portal/pin/7.5/var/cm/cm.pinlog *pin_err_storage

BTW Splunk searches are case-insensitive.

[mmilano]

This search does not work.
index=brm host=a-brmapp* source=/opt/portal/pin/7.5/var/cm/cm.pinlog *pin_err_storage

It matches both entries in the log file, but fails to trigger an alert.

I can't seem to include a picture. Please click the link to see the result of the above search.

http://s4.photobucket.com/user/mmilano/media/Capture.png.html

[lguinn2]

What are your alert conditions?
AS @jayannah points out, you have to save your search as an alert - with appropriate conditions set - if you want it to be an alert. A search, even saved as a report, will not alert unless it is configured to do so.

[mmilano]

It is configured as an alert, but it is not working.

The log entry we want an alert from is:

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

If we setup the search as:
PIN_ERR_STORAGE source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

No alert comes.

If we set it up as:
"errno=PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

We get the alert.

I don't want to have to include the "errno=" in the search in order for the alert to trigger.

[lguinn2]

Based on the format of the incoming log, you may have to do this

"errno=PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

OR

"*PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

OR, if you have a field named errno:

errno=PIN_ERR_STORAGE source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

The first and third options will be more efficient.

[mmilano]

I think that is the problem. I do have a field named "errno". How do I get rid of it?

[jayannah]

Your 2nd line is the right one to find any events containing string PIN_ERR_STORAGE . Is your question is why there was no alert for the 2nd matching event?

[jayannah]

While configuring alert, did you choose "Once" or "For each Result" for trigger action parameter?

[mmilano]

This is outside my configured searches. I simply open "New Search" and put that line in the search bar. In case it is helpful, the string is the same as in my configured searches and in there I have "Alert Mode" set to "Once per result"