topic Re: How to get the number of errors for each application ? in Splunk Search

How to get the number of errors for each application ?

lsy9891 — Thu, 22 Aug 2019 08:59:38 GMT

Hi,

I'm new to Splunk and so far I've managed to get the number of errors but I do not know for which application? I know we need an index and host but what exactly should be the index? Here's what I tried.

 error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) ) | stats count by error

Re: How to get the number of errors for each application ?

Sukisen1981 — Wed, 30 Sep 2020 01:51:35 GMT

can you paste what values your index,source and surcetype fields contain?
Ideally you should consider naming your indexes after your applications and let them pick different folders, app logs whatever it is from the default source n sourcetypes
Host will typically be your server name , like xxx_prod_1.0 or something like that

Re: How to get the number of errors for each application ?

lsy9891 — Fri, 23 Aug 2019 13:39:19 GMT

Hi,may I know how to get the index, source and sourcetype fields of the app?

Re: How to get the number of errors for each application ?

Sukisen1981 — Fri, 23 Aug 2019 16:06:04 GMT

hi @lsy9891
you are using sourcetype=access_* ( 404 OR 500 OR 503 ), which means you are accessing all sourcetypes starting with access
you can use a query like this, for determining your indexes

index="*" 
|stats values(source),values(sourcetype) by index

Run this for last 15 mins (DO NOT RUN FOR ALL TIME / 24 HRS) and you should get all your index, source n sourcetypes listed. You need to choose the relevant ones.

Re: How to get the number of errors for each application ?

sixcorners — Fri, 23 Aug 2019 17:05:51 GMT

In the list of fields on the left side you can click them and click on the "Top Values" report to get a breakdown of the top counts for the field you select. It basically just adds "| top limit=20 x" to the end of your query where x is the field you clicked.
If I want a count of how many errors are coming from each app I would use that to make this query:
severity=ERROR| top limit=20 source

Re: How to get the number of errors for each application ?

lsy9891 — Thu, 29 Aug 2019 06:05:44 GMT

Hi, I've managed to get the number of errors however, I tried to extract all the application names using regex but some are omitted. This is the rex I used:

rex field=WindowsIdentity "(?P\w+.\w+)". The field I'm supposed to extract is anything after the ISS APPOOL and just before the .monster. For example,

IIS APPPOOL\ jobs.monster.com
IIS APPPOOL\ *hiring.channels *.monster.com_jcm
IIS APPPOOL\ *wwwcs.channels *.monster.com

Re: How to get the number of errors for each application ?

Sukisen1981 — Thu, 29 Aug 2019 07:10:32 GMT

hi @lsy9891
You have posted a separate question for the rex and that has been answered as well, you can extract these values using the rex below as well | rex field=WindowsIdentity "\\\+(?<Description>.*?)\.+monster"
can you give examples where rex is failing?

Re: How to get the number of errors for each application ?

woodcock — Tue, 03 Sep 2019 14:50:02 GMT

You can get a better view with a search like this:

(index=* OR index=_*) AND (error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) ))
| stats first(_raw) last(_raw) values(host) count BY punct sourcetype index