Duplicate Alerts

[aromanauskas]

Install of the SolarWinds add-on has gone pretty smoothly other than dealing with the standard SSL cert issues that it seems everyone with SolarWinds is fighting. Querying for alerts is working but no matter what time frame I set for the Checkpoint value or the interval there are duplicate EventIDs being ingested.

Anyone have any ideas on how this could be happening? The version of Splunk is 7.0.0.

Thanks,
Alex

[jamin358]

Error Log looks like this:
2019-05-22 08:49:17,904 +0000 log_level=WARNING, pid=23205, tid=Thread-4, file=ext.py, func_name=time_str2str, code_line_no=306 | [stanza_name="solarwinds_alert_stanza"] Unable to convert date_string "2019-05-22T08:41:31.5970000" from format "%Y-%m-%dT%H:%M:%S.%f" to "%Y-%m-%d %H:%M:%S.%f", return the original date_string, cause=Traceback (most recent call last):

Conversion of date_string doesn't work properly as there are too many leading zeros in the date string. the parameter %f only allows 6 digits and not 7.

The fix I found that works for this is to edit the Splunk_TA_SolarWinds/bin/solarwinds_alerts.cc.json line 37.
Change from "%Y-%m-%dT%H:%M:%S.%f" to "%Y-%m-%dT%H:%M:%S.%f0".

[Majid_Khan]

How did u get rid of the SSL cert issue, i know there could be multiple ways, how did you fix it?

Thanks much in advance,
Majid

[blyons_splunk]

Same question!

[ehaddad_splunk]

Can you take a sapshot from inputs tab? How many inputs do you have there? Are they all enabled?

[stuieordie]

Found this message in the _internal logs

index=_internal sourcetype="splunk_ta_solarwinds_solarwinds_alerts*"

2017-11-13 21:42:21,370 +0000 log_level=WARNING, pid=19675, tid=Thread-4, file=ext.py, func_name=time_str2str, code_line_no=306 | [stanza_name="sw_alerts"] Unable to convert date_string "2017-11-13T02:59:15.0070000" from format "%Y-%m-%dT%H:%M:%S.%f" to "%Y-%m-%d %H:%M:%S.%f", return the original date_string, cause=Traceback (most recent call last): File "/data/opt/splunk/etc/apps/Splunk_TA_SolarWinds/bin/splunk_ta_solarwinds/cloudconnectlib/core/ext.py", line 289, in time_str2str dt = datetime.strptime(date_string, from_format) File "/data/opt/splunk/lib/python2.7/_strptime.py", line 335, in _strptime data_string[found.end():]) ValueError: unconverted data remains: 0

[stuieordie]

We were able to resolve the warning message above by editing the following:

Splunk_TA_SolarWinds/bin/solarwinds_alerts.cc.json

Line 37: "%Y-%m-%dT%H:%M:%S.%f0",
Line 38: "%Y-%m-%d %H:%M:%S.%f0"

Unfortunately the add-on is still pulling in hundreds of thousands of duplicate Events

[DBattisto]

Ever figure out why you were getting duplicate logs? I had this configured properly once, but during a recent reinstall I forgot what had been configured.

[stuieordie]

Per the Python documentation here, the strptime function for the microseconds field (%f) supports 1-6 characters. It looks like our Solarwinds EventTime field is returning 7 places on the microseconds. I tested this and found that removing 1 digit from the microseconds doesn't generate the error.

from datetime import datetime
dt7 = "2017-11-13T04:22:33.1100000"
dt6 = "2017-11-13T04:22:33.110000”
dt6_object = datetime.strptime(dt6, "%Y-%m-%dT%H:%M:%S.%f")
dt7_object = datetime.strptime(dt7, "%Y-%m-%dT%H:%M:%S.%f")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib64/python2.6/_strptime.py", line 328, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: 0

[stuieordie]

Hi,

I work on the same system as OP - I'm linking a screenshot of the just the 1 alerts input we have configured and a stats command showing the duplicate EventIDs we're pulling in...

Thanks
Mark

https://imgur.com/a/2Srbq