Having an issue that Splunk doesn't build my knowledge bundles. My setup: One indexer cluster and two standalone search heads (no SH cluster). Both search heads use indexer discovery and the setup used to work fine. Until recently the knowledge bundle of one of the two search heads stopped getting updated on the indexers. I observe the following: All indexers always have an up to date knowledge bundle from the first search head in /opt/splunk/var/run/searchpeers, while the bundle from the second search head no longer gets updated and is outdated. When running "splunk show bundle-replication-config" on the two search heads, both show an identical config When running "splunk show bundle-replication-status", one search head shows a fully functional replication, while the other search head states "No knowledge bundle replication cycle status is available yet." The search head that shows the error with the replication cycle status has no local knowledge bundle in /opt/splunk/var/run/ (while the other search head indeed has it). Therefore I guess that there's not a problem on the channel between search head and indexer, but some interna on the search head is dysfunctional and no longer builds the bundles in the first place. I did all the usual checks (reboot, filesystem permissions, btool check, ...). On the broken search head, I moved all local apps out of SPLUNK_HOME/etc/apps and emptied SPLUNK_HOME/etc/users and restarted, but the knowledge bundle still wasn't getting build. In log.cfg on the SH I set DistributedBundleReplicationManager, BundleReplicationProvider, ClassicBundleReplicationProvider, CascadingBundleReplicationProvider, RFSBundleReplicationProvider, RFSManager to DEBUG, but this didn't provide any insights. Any ideas about where we could search further? ... View more

Looking for some hints and suggestions about how to implement this: I have incoming log data that contains EAN barcodes (shown as number). A 3rd party API provides me with further details about what's behind the barcode. That's working so far. Already have an external lookup written in Python. However, the API has a rate limit and only allows to submit 10 queries per minute. It also only allows one barcode per query (so no batch lookups possible). Therefore there's no way to do a "live" lookup when the enduser is searching in the data. Therefore I need to pre-lookup the barcode as soon as the events come in and cache them locally to allow searching within enriched data. The rate of incoming events might exceed 10 per minute often during the day. Due to the rate limiting the pre-lookups will be in residue during the day and again catch up during the night when there's less load. Therefore I need some way to build something like a fifo buffer (which survives even when Splunk restarts) and have a constantly running job feeding the barcodes to the lookup. Same barcode might also show up multiple time, so it must verify if a barcode is already in the buffer or if it has already been looked up earlier to avoid multiple lookups. Basically I need something like this: Incoming log data -> verify if barcode is already in the buffer or has already been pre-looked up -> put barcode to fifo buffer -> feed the buffer at a rate of 10/minute to the external lookup command -> write the lookup result to CSV/kvstore so the enduser can search in the data without being rate limited Any hints? ... View more

Have external threat lists to download. With them it is required to send a customized Authorization header. And no, it's not HTTP basic auth. I get a text string by the list provider and the HTTP GET request needs to have a header in the format "Authorization: thisstring". Thus I cannot use the user/password field in the configuration settings of the threat list, as they would be translated into HTTP basic auth. I need to specify the plain Authorization header, without any translation/interpretation applied. Is there any way to do this natively in the Splunk Enterprise Security? As of now, I was using a customized Python script to do the requests. However, would be much nicer having a native feature built into the ES. ... View more