Splunk Search

How to edit my search to match any string containing "PIN_ERR_STORAGE"?

mmilano
Explorer

Here is the search:
index="brm" host="a-brmapp*" source="/opt/portal/pin/7.5/var/cm/cm.pinlog" PIN_ERR_STORAGE

Now, it DOES find two matches. They are as follows:

D Wed Jan 7 17:10:09 2015 a-brmapp1.corp.com cm:37041 cm_child.c(120):1016 1:a-brmpipe1:UnknownProgramName:0:AWT-EventQueue-0:7:1420650609:0 cm_pcm_op_call_stack ... 3 lines omitted ... 0.001977829 .Exit PCM_OP_SEARCH (0x0) - from DM 0.0.0.1 0.002006352 Exit PCM_OP_SEARCH (0x0) PIN_ERR_STORAGE

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

The first match was a test whereby we echoed the string (PIN_ERR_STORAGE) into the log. It matched this one and triggered an alert. The second match is an actual entry in the log file. Running the search manually finds the entry, but, it doesn't trigger an alert.

It seems the only way to get this to work is to include the entire string "errno=PIN_ERR_STORAGE" as follows:
index="brm" host="a-brmapp*" source="/opt/portal/pin/7.5/var/cm/cm.pinlog" "errno=PIN_ERR_STORAGE"

Result:

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

Can anyone please tell me how I can match ANY string containing "PIN_ERR_STORAGE"?

Thank you in advance!

Tags (3)
0 Karma

lguinn2
Legend

You might want to write the search as

index=brm host=a-brmapp* source=/opt/portal/pin/7.5/var/cm/cm.pinlog *pin_err_storage

BTW Splunk searches are case-insensitive.

0 Karma

mmilano
Explorer

This search does not work.
index=brm host=a-brmapp* source=/opt/portal/pin/7.5/var/cm/cm.pinlog *pin_err_storage

It matches both entries in the log file, but fails to trigger an alert.

I can't seem to include a picture. Please click the link to see the result of the above search.

http://s4.photobucket.com/user/mmilano/media/Capture.png.html

0 Karma

lguinn2
Legend

What are your alert conditions?
AS @jayannah points out, you have to save your search as an alert - with appropriate conditions set - if you want it to be an alert. A search, even saved as a report, will not alert unless it is configured to do so.

0 Karma

mmilano
Explorer

It is configured as an alert, but it is not working.

The log entry we want an alert from is:

E Wed Jan 7 07:39:32 2015 a-brmapp1.corp.com cm:70115 cm_child.c(120):4723 1:a-brmapp1.corp.com:rax_gen_bill_object:70064:-362808464331:1420616372:250 ... 2 lines omitted ... 0 PIN_FLD_POID POID [0] 0.0.0.1 /procedure -1 0 0 PIN_FLD_ERR_BUF ERR [0] <location=PIN_ERRLOC_DM:4 class=UNKNOWN:0 errno=PIN_ERR_STORAGE:43> <field num=0:0,0 recid=0 reserved=1422 reserved2=0 time(sec:usec)=0:0>

If we setup the search as:
PIN_ERR_STORAGE source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

No alert comes.

If we set it up as:
"errno=PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

We get the alert.

I don't want to have to include the "errno=" in the search in order for the alert to trigger.

0 Karma

lguinn2
Legend

Based on the format of the incoming log, you may have to do this

"errno=PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

OR

"*PIN_ERR_STORAGE" source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

OR, if you have a field named errno:

errno=PIN_ERR_STORAGE source="/opt/portal/pin/7.5/var/cm/cm.pinlog"

The first and third options will be more efficient.

0 Karma

mmilano
Explorer

I think that is the problem. I do have a field named "errno". How do I get rid of it?

0 Karma

jayannah
Builder

Your 2nd line is the right one to find any events containing string PIN_ERR_STORAGE . Is your question is why there was no alert for the 2nd matching event?

0 Karma

jayannah
Builder

While configuring alert, did you choose "Once" or "For each Result" for trigger action parameter?

0 Karma

mmilano
Explorer

This is outside my configured searches. I simply open "New Search" and put that line in the search bar. In case it is helpful, the string is the same as in my configured searches and in there I have "Alert Mode" set to "Once per result"

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...