All Apps and Add-ons

SNMP Modular Input: Why are two (perhaps more) events getting merged into one event, which results in multivalue fields when they should not exist?

kingsizebk
Path Finder

I have 5 SNMP inputs configured and enabled. They are all similar to this:

[snmp://test1-default]
communitystring = splunk-default
destination = a.b.c.d
do_bulk_get = 0
do_get_subtree = 0
host = test1
index = test
ipv6 = 0
mib_names = PRODUT-STATUS-MIB
object_names = iso.org.dod.internet.private.enterprises.product.Management.Status.StatusCPUUsage.StatusCPUUsageoneMinute, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusMemoryStatus.StatusMemoryStatusUsage, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusSystemUsage.StatusSystemUsageLoad, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusSystemUsage.StatusSystemUsageWorkList, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusTCPSummary.StatusTCPSummaryestablished, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusFreeEncrypted, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusTotalEncrypted, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusFreeTemporary, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusTotalTemporary, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusFreeInternal, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusTotalInternal
port = 161
snmp_mode = attributes
snmp_version = 2C
snmpinterval = 180
sourcetype = dpStatus-default
split_bulk_output = 1
trap_rdns = 0
v3_authProtocol = usmHMACMD5AuthProtocol
v3_privProtocol = usmDESPrivProtocol
disabled = 0

I also have a lookup table that maps the host to an environment (e.g. DEV, STG, PROD) and each host is in exactly one environment.

The test host is in the DEV environment.

Here is one example of how I can see these "merged events" and/or "events with multivalue fields that should not exist":

When I search for index=test host=test environment=STG I get a result and it shows that environment field has both a "DEV" and "STG" value. How can that be? I did check the lookup table and I had someone else double-check it...

0 Karma
1 Solution

woodcock
Esteemed Legend

When constructing lookup tables by hand, it is very easy to modify only part of the pasted entry. When any input value has multiple output values, performing a lookup creates a multi-value field when you generally expect a single-value field. If you then perform any statistical analysis, one event will be interpreted multiple events causing inflation of the incorrect value's accounting. An audit like this should be performed on all should-be-one-to-one-mapped lookup files to identify this problem:

 awk -F, '{print $1}' ${SPLUNK_HOME}/etc/MyApp/lookups/host2location.csv | uniq -d

View solution in original post

0 Karma

woodcock
Esteemed Legend

When constructing lookup tables by hand, it is very easy to modify only part of the pasted entry. When any input value has multiple output values, performing a lookup creates a multi-value field when you generally expect a single-value field. If you then perform any statistical analysis, one event will be interpreted multiple events causing inflation of the incorrect value's accounting. An audit like this should be performed on all should-be-one-to-one-mapped lookup files to identify this problem:

 awk -F, '{print $1}' ${SPLUNK_HOME}/etc/MyApp/lookups/host2location.csv | uniq -d
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...