All Apps and Add-ons

Duplicate Alerts

aromanauskas
Path Finder

Install of the SolarWinds add-on has gone pretty smoothly other than dealing with the standard SSL cert issues that it seems everyone with SolarWinds is fighting. Querying for alerts is working but no matter what time frame I set for the Checkpoint value or the interval there are duplicate EventIDs being ingested.

Anyone have any ideas on how this could be happening? The version of Splunk is 7.0.0.

Thanks,
Alex

jamin358
Explorer

Error Log looks like this:
2019-05-22 08:49:17,904 +0000 log_level=WARNING, pid=23205, tid=Thread-4, file=ext.py, func_name=time_str2str, code_line_no=306 | [stanza_name="solarwinds_alert_stanza"] Unable to convert date_string "2019-05-22T08:41:31.5970000" from format "%Y-%m-%dT%H:%M:%S.%f" to "%Y-%m-%d %H:%M:%S.%f", return the original date_string, cause=Traceback (most recent call last):

Conversion of date_string doesn't work properly as there are too many leading zeros in the date string. the parameter %f only allows 6 digits and not 7.

The fix I found that works for this is to edit the Splunk_TA_SolarWinds/bin/solarwinds_alerts.cc.json line 37.
Change from "%Y-%m-%dT%H:%M:%S.%f" to "%Y-%m-%dT%H:%M:%S.%f0".

0 Karma

Majid_Khan
Engager

How did u get rid of the SSL cert issue, i know there could be multiple ways, how did you fix it?

Thanks much in advance,
Majid

blyons_splunk
Splunk Employee
Splunk Employee

Same question!

0 Karma

ehaddad_splunk
Splunk Employee
Splunk Employee

Can you take a sapshot from inputs tab? How many inputs do you have there? Are they all enabled?

0 Karma

stuieordie
New Member

Found this message in the _internal logs

index=_internal sourcetype="splunk_ta_solarwinds_solarwinds_alerts*"

2017-11-13 21:42:21,370 +0000 log_level=WARNING, pid=19675, tid=Thread-4, file=ext.py, func_name=time_str2str, code_line_no=306 | [stanza_name="sw_alerts"] Unable to convert date_string "2017-11-13T02:59:15.0070000" from format "%Y-%m-%dT%H:%M:%S.%f" to "%Y-%m-%d %H:%M:%S.%f", return the original date_string, cause=Traceback (most recent call last): File "/data/opt/splunk/etc/apps/Splunk_TA_SolarWinds/bin/splunk_ta_solarwinds/cloudconnectlib/core/ext.py", line 289, in time_str2str dt = datetime.strptime(date_string, from_format) File "/data/opt/splunk/lib/python2.7/_strptime.py", line 335, in _strptime data_string[found.end():]) ValueError: unconverted data remains: 0

0 Karma

stuieordie
New Member

We were able to resolve the warning message above by editing the following:

Splunk_TA_SolarWinds/bin/solarwinds_alerts.cc.json

Line 37: "%Y-%m-%dT%H:%M:%S.%f0",
Line 38: "%Y-%m-%d %H:%M:%S.%f0"

Unfortunately the add-on is still pulling in hundreds of thousands of duplicate Events

0 Karma

DBattisto
Communicator

Ever figure out why you were getting duplicate logs? I had this configured properly once, but during a recent reinstall I forgot what had been configured.

0 Karma

stuieordie
New Member

Per the Python documentation here, the strptime function for the microseconds field (%f) supports 1-6 characters. It looks like our Solarwinds EventTime field is returning 7 places on the microseconds. I tested this and found that removing 1 digit from the microseconds doesn't generate the error.

from datetime import datetime
dt7 = "2017-11-13T04:22:33.1100000"
dt6 = "2017-11-13T04:22:33.110000”
dt6_object = datetime.strptime(dt6, "%Y-%m-%dT%H:%M:%S.%f")
dt7_object = datetime.strptime(dt7, "%Y-%m-%dT%H:%M:%S.%f")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib64/python2.6/_strptime.py", line 328, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: 0

0 Karma

stuieordie
New Member

Hi,

I work on the same system as OP - I'm linking a screenshot of the just the 1 alerts input we have configured and a stats command showing the duplicate EventIDs we're pulling in...

Thanks
Mark

https://imgur.com/a/2Srbq

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...