Splunk Search
Highlighted

How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Explorer

Here is the search:
index="brm" host="a-brmapp*" source="/opt/portal/pin/7.5/var/cm/cm.pinlog" PINERRSTORAGE

Now, it DOES find two matches. They are as follows:

D Wed Jan 7 17:10:09 2015 a-brmapp1.corp.com cm:37041 cm_child.c(120):1016 1:a-brmpipe1:UnknownProgramName:0:AWT-EventQueue-0:7:1420650609:0 cm_pcm_op_call_stack ... 3 lines omitted ... 0.001977829 .Exit PCM_OP_SEARCH (0x0) - from DM 0.0.0.1 0.002006352 Exit PCM_OP_SEARCH (0x0) PIN_ERR_STORAGE

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

The first match was a test whereby we echoed the string (PINERRSTORAGE) into the log. It matched this one and triggered an alert. The second match is an actual entry in the log file. Running the search manually finds the entry, but, it doesn't trigger an alert.

It seems the only way to get this to work is to include the entire string "errno=PINERRSTORAGE" as follows:
index="brm" host="a-brmapp*" source="/opt/portal/pin/7.5/var/cm/cm.pinlog" "errno=PINERRSTORAGE"

Result:

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

Can anyone please tell me how I can match ANY string containing "PINERRSTORAGE"?

Thank you in advance!

Tags (3)
0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Builder

Your 2nd line is the right one to find any events containing string PINERRSTORAGE . Is your question is why there was no alert for the 2nd matching event?

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Builder

While configuring alert, did you choose "Once" or "For each Result" for trigger action parameter?

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Explorer

This is outside my configured searches. I simply open "New Search" and put that line in the search bar. In case it is helpful, the string is the same as in my configured searches and in there I have "Alert Mode" set to "Once per result"

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Legend

You might want to write the search as

index=brm host=a-brmapp* source=/opt/portal/pin/7.5/var/cm/cm.pinlog *pin_err_storage

BTW Splunk searches are case-insensitive.

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Explorer

This search does not work.
index=brm host=a-brmapp* source=/opt/portal/pin/7.5/var/cm/cm.pinlog *pinerrstorage

It matches both entries in the log file, but fails to trigger an alert.

I can't seem to include a picture. Please click the link to see the result of the above search.

http://s4.photobucket.com/user/mmilano/media/Capture.png.html

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Legend

What are your alert conditions?
AS @jayannah points out, you have to save your search as an alert - with appropriate conditions set - if you want it to be an alert. A search, even saved as a report, will not alert unless it is configured to do so.

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Explorer

It is configured as an alert, but it is not working.

The log entry we want an alert from is:

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

If we setup the search as:
PINERRSTORAGE source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

No alert comes.

If we set it up as:
"errno=PINERRSTORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

We get the alert.

I don't want to have to include the "errno=" in the search in order for the alert to trigger.

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Legend

Based on the format of the incoming log, you may have to do this

"errno=PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

OR

"*PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

OR, if you have a field named errno:

errno=PIN_ERR_STORAGE source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

The first and third options will be more efficient.

0 Karma
Highlighted

Re: How to edit my search to match any string containing "PIN_ERR_STORAGE"?

Explorer

I think that is the problem. I do have a field named "errno". How do I get rid of it?

0 Karma