topic Re: Grouping aggregation events into episode. Splunk and solarwinds Integration in Splunk ITSI

Grouping aggregation events into episode. Splunk and solarwinds Integration

pedro_77 — Sun, 07 Jun 2020 00:44:38 GMT

Hi all,

I'm trying to build dashboard in ITSI with active alarms from Solarwinds and other monitoring tools. I'm retrieving alarms from Solar via API (solarwinds addon). One of the next step is to use aggregation policy. I've noticed that whatever i type in 'split events by field' form no episodes are created.

For example i use below fields:
include events: EvenType *
split events by field: orig_AlertActiveID (or AlertActiveID)

Also noticed that when i'm trying to use smart mode, ITSI cannot find any event, even if re-run analyze for 30 days.

How can i troubleshoot this further?
Thanks
Piotr

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

szhou_splunk — Wed, 30 Sep 2020 04:08:44 GMT

Would you check if itsi_event_grouping saved search is running? Are you able to generate episodes without enabling smart mode?

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

pedro_77 — Wed, 30 Sep 2020 04:04:19 GMT

I'm not able to generate episode with or without smart mode.
I see entries in index="itsi_tracked_alerts"

Every minutes i see in internal logs:

index=_internal itsi_event_grouping

02-10-2020 00:38:30.664 +0100 INFO StreamedSearch - Streamed search search starting: search_id=remote_server01_rt_1581291503.609, server=server01, active_searches=3, search='rtlitsearch (index=_internal itsi_event_grouping) | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"', remote_ttl=600, apiStartTime='MIN_TIME', apiEndTime='MIN_TIME', savedsearch_name=""

How can i check in different way that itsi_event_grouping saved search is running?

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

szhou_splunk — Wed, 30 Sep 2020 04:08:52 GMT

You can go to Settings -> Searches, reports and alerts -> Search for "itsi_event_grouping" for all apps. And you can verify if itsi_event_grouping is running or failed.

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

pedro_77 — Mon, 10 Feb 2020 00:28:21 GMT

i see:
status is enabled
Next Scheduled Time: none
Alerts: 0
Sharing: global

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

szhou_splunk — Wed, 30 Sep 2020 04:08:55 GMT

Would you click the "View Recent" link see if it is running?
Also check if the index itsi_grouped_alerts has events.

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

pedro_77 — Mon, 10 Feb 2020 00:52:25 GMT

in Action tab there is no recent view for this search

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

szhou_splunk — Wed, 30 Sep 2020 04:08:58 GMT

How about Activities -> Jobs , and search for itsi_event_grouping?

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

pedro_77 — Wed, 30 Sep 2020 04:05:13 GMT

There is no itsi_event_grouping in JOBS
In search there is no itsi_event_grouping search as scheduled. Next Scheduled Time filed is empty.
I did test installation on Centos and problem is the same. No itsi_event_grouping search as job or scheduled search.

On Linux java version: is openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

Re: Grouping aggregation events into episode. Splunk and solarwinds Integration

pedro_77 — Wed, 30 Sep 2020 04:05:24 GMT

On Windows even worse, in Notable Event Aggregation Policy in preview windows i have error :

Error in 'itsirulesengine' command: External search command exited unexpectedly with non-zero error code 1.

java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) Client VM (build 25.241-b07, mixed mode, sharing

echo %JAVA_HOME%
C:\Program Files (x86)\Java\jre1.8.0_241\

AV disabled