SNMP Modular Input: Why are two (perhaps more) events getting merged into one event, which results in multivalue fields when they should not exist?

[kingsizebk]

I have 5 SNMP inputs configured and enabled. They are all similar to this:

[snmp://test1-default]
communitystring = splunk-default
destination = a.b.c.d
do_bulk_get = 0
do_get_subtree = 0
host = test1
index = test
ipv6 = 0
mib_names = PRODUT-STATUS-MIB
object_names = iso.org.dod.internet.private.enterprises.product.Management.Status.StatusCPUUsage.StatusCPUUsageoneMinute, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusMemoryStatus.StatusMemoryStatusUsage, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusSystemUsage.StatusSystemUsageLoad, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusSystemUsage.StatusSystemUsageWorkList, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusTCPSummary.StatusTCPSummaryestablished, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusFreeEncrypted, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusTotalEncrypted, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusFreeTemporary, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusTotalTemporary, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusFreeInternal, iso.org.dod.internet.private.enterprises.product.Management.Status.StatusFilesystemStatus.StatusFilesystemStatusTotalInternal
port = 161
snmp_mode = attributes
snmp_version = 2C
snmpinterval = 180
sourcetype = dpStatus-default
split_bulk_output = 1
trap_rdns = 0
v3_authProtocol = usmHMACMD5AuthProtocol
v3_privProtocol = usmDESPrivProtocol
disabled = 0

I also have a lookup table that maps the host to an environment (e.g. DEV, STG, PROD) and each host is in exactly one environment.

The test host is in the DEV environment.

Here is one example of how I can see these "merged events" and/or "events with multivalue fields that should not exist":

When I search for index=test host=test environment=STG I get a result and it shows that environment field has both a "DEV" and "STG" value. How can that be? I did check the lookup table and I had someone else double-check it...

[woodcock · Accepted solution]

When constructing lookup tables by hand, it is very easy to modify only part of the pasted entry. When any input value has multiple output values, performing a lookup creates a multi-value field when you generally expect a single-value field. If you then perform any statistical analysis, one event will be interpreted multiple events causing inflation of the incorrect value's accounting. An audit like this should be performed on all should-be-one-to-one-mapped lookup files to identify this problem:

 awk -F, '{print $1}' ${SPLUNK_HOME}/etc/MyApp/lookups/host2location.csv | uniq -d