Alerting

How to monitor and capture muliple scenarios in same alert

Nidd
Path Finder

I have a requirement to monitor the below exceptions and send an alert through mail with few fields mentioned below.

Since I'm not able to achieve this, I have created 4 individual alerts and have monitored this. But that isn't right. I wish to capture all these within the same alert.

Below are sample logs.

 

TYPE 1: INVALID USERNAME/PASSWORD

 

 

2021-03-01 03:36:02,233 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR HikariPool - HikariPool-7333 - Exception during pool initialization.
java.sql.SQLException: ORA-01017: invalid username/password; logon denied

 

 

 

TYPE 2: INVALID SERVICE

 

 

2021-03-01 04:18:26,910 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

 

 

 

TYPE 3: INVALID PORT

 

 

2021-03-01 04:43:12,985 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection
com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1
validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection

 

 

 

TYPE 4: INVALID HOST

 

 

2021-03-01 05:02:13,113 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipelin-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified
com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified

 

Below are the fields to capture:

pipeline - Which is : my-pipeline-name

Exception -
Which are :
1. invalid username/password; logon denied,
2. TNS:listener does not currently know of service requested in connect descriptor
3. The Network Adapter could not establish the connection
4. Unknown host specified

 

Please help in achieving this.

Labels (2)
0 Karma

Nidd
Path Finder

Searches I use? Sorry. Didnt get you.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The searches used in the four alerts you already have

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What is wrong with having different alerts for the different conditions?

What are the searches you currently use?

0 Karma

Nidd
Path Finder

@ITWhisperer ..Yes..no issues in having 4 alerts.. But since we get the events from same server and same transaction, monitoring all the events in the same alert itself was what we have been given as a requirement.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK so combine the searches into one using OR conditions where appropriate.

What are the searches you are currently using?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...