Alerting

How to monitor and capture muliple scenarios in same alert

Nidd
Path Finder

I have a requirement to monitor the below exceptions and send an alert through mail with few fields mentioned below.

Since I'm not able to achieve this, I have created 4 individual alerts and have monitored this. But that isn't right. I wish to capture all these within the same alert.

Below are sample logs.

 

TYPE 1: INVALID USERNAME/PASSWORD

 

 

2021-03-01 03:36:02,233 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR HikariPool - HikariPool-7333 - Exception during pool initialization.
java.sql.SQLException: ORA-01017: invalid username/password; logon denied

 

 

 

TYPE 2: INVALID SERVICE

 

 

2021-03-01 04:18:26,910 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

 

 

 

TYPE 3: INVALID PORT

 

 

2021-03-01 04:43:12,985 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipeline-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection
com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1
validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection

 

 

 

TYPE 4: INVALID HOST

 

 

2021-03-01 05:02:13,113 [user:*myemail@temp.com] [pipeline:my-pipeline-name (SCH Test Run)/testRun__1234__temp.com__myemail@temp.com] [runner:] [thread:ProductionPipelineRunnable-testRun__1234__temp.com__myemail@temp.com-my-pipelin-name (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified
com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified

 

Below are the fields to capture:

pipeline - Which is : my-pipeline-name

Exception -
Which are :
1. invalid username/password; logon denied,
2. TNS:listener does not currently know of service requested in connect descriptor
3. The Network Adapter could not establish the connection
4. Unknown host specified

 

Please help in achieving this.

Labels (2)
0 Karma

Nidd
Path Finder

Searches I use? Sorry. Didnt get you.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The searches used in the four alerts you already have

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What is wrong with having different alerts for the different conditions?

What are the searches you currently use?

0 Karma

Nidd
Path Finder

@ITWhisperer ..Yes..no issues in having 4 alerts.. But since we get the events from same server and same transaction, monitoring all the events in the same alert itself was what we have been given as a requirement.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK so combine the searches into one using OR conditions where appropriate.

What are the searches you are currently using?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...