Hi rrussell2020,
As a long time Splunker and someone who worked in telco and network monitoring space, I faced the same scenario, and while I agree that Splunk is a powerful tool that can do many things, sometimes it is best to let the upstream tools do what they are good at, and simply provide Spunk a summary so we can do what we are good at by marrying those metrics to the logs we already have.
Case in point. Cacti and snmptrapd.
Cacti is a rock solid SNMP poller that is the grandaddy of snmp polling (rrdtool) and can do a great job of taking care of doing the hard work of snmp collection (Spine still rocks all these years later). We used Cacti as well, and so I ended up creating a Cacti plugin to feed the poller data to cacti in nice clean key value pairs.
http://docs.cacti.net/userplugin:mirage
Then created a Splunk app as a proof of concept, - https://www.splunk.com/blog/2016/01/29/splunk-and-cacti/ - that shows how to then use the Cacti backend DB to enrich the KVPairs and glean the knowledge you are looking for. I am hoping to clean up and enhance the splunk app soon, admittedly its very basic and just gets you going as our goal was to feed ITSI.
I have been having tons of fun with the new version of Cacti that forked in some of the great automation plugins with DBConnect and pulling useful info from Cacti's db too!
As for traps I simply used snmptrapd on a nix box to catch traps and load mibs, then used a forwarder to bring that info in.
So really at the end of the day, a couple forwarders running Cacti with our plugin and running snmptrapd and you have cooked up a pretty awesome collection layer that will get you the best of all the work you have already done in Cacti, nicely enhanced and augmented with Splunk to build advanced analytics, alerting or even feed ITSI!
... View more