I'm new to Splunk and so far I've managed to get the number of errors but I do not know for which application? I know we need an index and host but what exactly should be the index? Here's what I tried.
error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) ) | stats count by error
You can get a better view with a search like this:
(index=* OR index=_*) AND (error OR failed OR severe OR ( sourcetype=access_* ( 404 OR 500 OR 503 ) )) | stats first(_raw) last(_raw) values(host) count BY punct sourcetype index
In the list of fields on the left side you can click them and click on the "Top Values" report to get a breakdown of the top counts for the field you select. It basically just adds "| top limit=20 x" to the end of your query where x is the field you clicked.
If I want a count of how many errors are coming from each app I would use that to make this query:
severity=ERROR| top limit=20 source
can you paste what values your index,source and surcetype fields contain?
Ideally you should consider naming your indexes after your applications and let them pick different folders, app logs whatever it is from the default source n sourcetypes
Host will typically be your server name , like xxx_prod_1.0 or something like that
Hi,may I know how to get the index, source and sourcetype fields of the app?
you are using sourcetype=access_* ( 404 OR 500 OR 503 ), which means you are accessing all sourcetypes starting with access
you can use a query like this, for determining your indexes
index="*" |stats values(source),values(sourcetype) by index
Run this for last 15 mins (DO NOT RUN FOR ALL TIME / 24 HRS) and you should get all your index, source n sourcetypes listed. You need to choose the relevant ones.
Hi, I've managed to get the number of errors however, I tried to extract all the application names using regex but some are omitted. This is the rex I used:
rex field=WindowsIdentity "(?P\w+.\w+)". The field I'm supposed to extract is anything after the ISS APPOOL and just before the .monster. For example,
IIS APPPOOL\ jobs.monster.com
IIS APPPOOL\ *hiring.channels *.monster.com_jcm
IIS APPPOOL\ *wwwcs.channels *.monster.com
You have posted a separate question for the rex and that has been answered as well, you can extract these values using the rex below as well
| rex field=WindowsIdentity "\\\+(?<Description>.*?)\.+monster"
can you give examples where rex is failing?