Splunk Search

How to get the number of errors for each application ?

lsy9891
Engager

Hi,

I'm new to Splunk and so far I've managed to get the number of errors but I do not know for which application? I know we need an index and host but what exactly should be the index? Here's what I tried.

 error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) ) | stats count by error
0 Karma

woodcock
Esteemed Legend

You can get a better view with a search like this:

(index=* OR index=_*) AND (error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) ))
| stats first(_raw) last(_raw) values(host) count BY punct sourcetype index
0 Karma

sixcorners
Observer

In the list of fields on the left side you can click them and click on the "Top Values" report to get a breakdown of the top counts for the field you select. It basically just adds "| top limit=20 x" to the end of your query where x is the field you clicked.
If I want a count of how many errors are coming from each app I would use that to make this query:
severity=ERROR| top limit=20 source

0 Karma

Sukisen1981
Champion

can you paste what values your index,source and surcetype fields contain?
Ideally you should consider naming your indexes after your applications and let them pick different folders, app logs whatever it is from the default source n sourcetypes
Host will typically be your server name , like xxx_prod_1.0 or something like that

0 Karma

lsy9891
Engager

Hi,may I know how to get the index, source and sourcetype fields of the app?

0 Karma

Sukisen1981
Champion

hi @lsy9891
you are using sourcetype=access_* ( 404 OR 500 OR 503 ), which means you are accessing all sourcetypes starting with access
you can use a query like this, for determining your indexes

index="*" 
|stats values(source),values(sourcetype) by index

Run this for last 15 mins (DO NOT RUN FOR ALL TIME / 24 HRS) and you should get all your index, source n sourcetypes listed. You need to choose the relevant ones.

0 Karma

lsy9891
Engager

Hi, I've managed to get the number of errors however, I tried to extract all the application names using regex but some are omitted. This is the rex I used:

rex field=WindowsIdentity "(?P\w+.\w+)". The field I'm supposed to extract is anything after the ISS APPOOL and just before the .monster. For example,

IIS APPPOOL\ jobs.monster.com
IIS APPPOOL\ *hiring.channels *.monster.com_jcm
IIS APPPOOL\ *wwwcs.channels *.monster.com

0 Karma

Sukisen1981
Champion

hi @lsy9891
You have posted a separate question for the rex and that has been answered as well, you can extract these values using the rex below as well | rex field=WindowsIdentity "\\\+(?<Description>.*?)\.+monster"
can you give examples where rex is failing?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!