How to generate 2 lines of events based on a condition?

[David_B]

Hello, 

I have a couple splunk columns that looks as follows:

server:incident:incident#:severity

severity

this object is then fed to another system which separates and generates incidents.

Server: hostname

incident: category of incident

incident#: the incident number

sererity: Critical/Warning/Clear

Example:

serverA:zabbix:123456:Warning	Warning
serverA:zabbix:123456:Critical	Critical

The objective is that it generates uniqueness of the incident (if warning, then create a ticket, if Critical then call out)

All works well when with the separate of Critical and Warning alerts, however when one clear is generated, I need to generate two records to look as follows:

serverA:zabbix:123456:Warning	Clear
serverA:zabbix:123456:Critical	Clear

This way, the object that has been sent will get the clear.

Is there a way to achieve this?

Thanks

David

[gcusello]

Hi @David_B,

you have to divide fields from column1 using a regex, something like this:

<your_search>
| rex field=column1 "^([^:]*:){3}(?<severity>\w*)"
| eval column2=if(column2="clear",severity,column2
| table column1 column2

you can test the regex at https://regex101.com/r/KUTS3I/1

Ciao.

Giuseppe

[David_B]

Hi @gcusello 

Thanks for your reply.

Instead of having the outputs as 2 columns, I need to have two rows generated 

For example an serverA has generated an incident  that is a warning (say disk space)

serverA:zabbix:123456:Warning

Warning

the tool picks up the event and generates a ticket.  Lets say nobody has done anything with it.

That disk has now reached critical and escalates the incident. Splunk picks up the event

serverA:zabbix:123456:Critical

Critical

because column1 is unique, the tool picks up the event and calls out the team.  The team then clear the space.

Splunk picks up the event as:

serverA:zabbix:123456:Clear

Clear

However, that is not match the column1 above. What I need is that when a clear is generated, Splunk generates 2 "fake" records that would look as follows:

serverA:zabbix:123456:Warning	Clear
serverA:zabbix:123456:Critical	Clear

So that Column1 matches the initial columns above and the tool will pick up 2 events and clear both records that were generated.

Thanks,

David

[ITWhisperer]

Assuming your fields are called column_1 and column_2, you could try something like this

| rex field=column_1 "(?<Server>[^:]+):(?<Incident>[^:]+):(?<IncidentNumber>[^:]+):(?<Severity>.*)"
| eventstats values(Severity) as AllSeverities by Server Incident IncidentNumber
| eval AllSeverities=if(Severity="Clear",AllSeverities,Severity)
| mvexpand AllSeverities
| eval column_1=Server.":".Incident.":".IncidentNumber.":".AllSeverities
| fields column_1 column_2
| dedup column_1 column_2

[David_B]

Hi @ITWhisperer ,

Thanks for your reply.  I have taken your code and modified it with the correct columns

| table tool host object_class object severity parameter value message support_group
| rex field=object "^([^:]*:){3}(?<severity>\w*)"
| eventstats values(severity) as AllSeverities by host "OEM_ISSUE" value
| eval AllSeverities=if(severity="Clear",AllSeverities,severity)
| mvexpand AllSeverities
| eval object=host.":OEM_ISSUE:".value.":".AllSeverities
| fields object severity
| dedup object severity

I am getting 2 records for the first clear however instead of the 2 rows showing as

serverA:zabbix:123456:Warning	Clear
serverA:zabbix:123456:Critical	Clear

I am getting

serverA:zabbix:123456:Clear	Clear
serverA:zabbix:123456:Critical	Critical

after the first clear severity, I am getting only one record as (different incident id and server for example)

serverB:zabbix:123457:Clear

Clear

Any help is greatly appreciated!

[ITWhisperer]

You are re-using field name severity - you also already seem to have values extracted to fields. What do you have for this

| table tool host object_class object severity parameter value message support_group

[David_B]

Hi @ITWhisperer 

Not sure what you mean by re-using field name severity as Column_1 is the object and column_2 is the severity

Here is what some output looks like for 

| table tool host object_class object severity parameter value message support_group

Ignore the first column (I put that in for explanation purposes)

	tool	host	object_class	object	severity	parameter	value	message	support_group
1	Tool	ServerA	OS_INCIDENT	ServerA.zabbix:1380217:Warning	WARNING	OS_ISSUE_NUM	1380217	ServerA - Disk space is at 80%	OS  Support
2	Tool	ServerA	OS_INCIDENT	ServerA.zabbix:1380217:Critical	CRITICAL	OS_ISSUE_NUM	1380217	CALL OUT - ServerA - Disk Space is at 90%	OS Support
3	Tool	ServerA	OS_INCIDENT	ServerA.zabbix:1380217:Clear	CLEAR	OS_ISSUE_NUM	1380217	ServerA -Disk Space Clear	OS  Support
4	Tool	ServerA	OS_INCIDENT	ServerA.zabbix:1380217:Warning	CLEAR	OS_ISSUE_NUM	1380217	ServerA -Disk Space Clear	OS  Support
5	Tool	ServerA	OS_INCIDENT	ServerA.zabbix:1380217:Critical	CLEAR	OS_ISSUE_NUM	1380217	CALL OUT - ServerA - Disk Space Clear	OS Support

What I am currently getting is 1,2,3, however what I need is 1 and 2, and when I get a result like 3, change it be 4 and 5

Hope that makes sense 🙂