What is your OS? can you use systemd unit file to avoid this? https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/RunSplunkassystemdservice     ... View more

I would recommend you start with : https://docs.splunk.com/Documentation/Splunk/8.0.5/Forwarding/Setuploadbalancingd#Specify_static_or_DNS_lists_for_receiving_indexers And set DNS A records. See Splunk Docs.  I won't go as far as to say never use a LB between UF and idx, cause i know with newer versions of UF and as i have worked with Splunk Eng, there are ways to make UF "LB Friendly" using newer version with the light event breaking UF does, but i would agree that if using traditional splunk2splunk protocol (s2s), just go all the way and rely on the application LB described in the doc above.  holler at me on slack (mattymo) if you want to chat more! ... View more

dont worry ill tie this rope to your waist....dive in, you wont get lost...docs has a guide for how to interpret the data (see link i posted above). Once you have touched the bottom of the pool come back up for some air and let us know what you working with and what you have tried!   Otherwise, check your jobs tab in splunk and look for jobs that are queued or differed. Also it says 1 search, can you not click into it?     ... View more

HI! Before we go further down this rabbit hole 🐰 🕳 , please review the docs here that state "do not insert a LB in the Splunk2Splunk protocol connection".  https://docs.splunk.com/Documentation/Forwarder/8.0.5/Forwarder/Configureloadbalancing#How_load_balancing_works This is a topic i have feelings on, and I'm all for healthy questioning of constraints, but, before we go further, what are you hoping to get from the LB and who is your LB vendor? What does your Splunk Arch look like? we talking many connections, high ingest? Can we use DNS instead? Do you have a mesh solution like istio? ... View more

Can you provide more detail about what exactly your app is doing? Is this a phantom playbook you are building? ... View more

Hey! Few things to qualify and find your issue: - what does your splunk arch look like?  distributed? clustered? (see Splunk Validated Architectures) - usually lookup issues are app/permissions based or can be replication based etc.. - what app is using it, is it a splunkbase app or custom?   ... View more

ok...so the events get picked up and sent to where? any intermediate forwarders in the path to the indexers? what sourcetype are you seeing in the events in splunk UI? ... View more

what sourcetype are you receiving? is it being overridden at the indexer? ... View more

sourcetype is mispelled - "sourceteyp". splunk is likely ignoring it. can you confirm btool does not show the proper sourcetype set? ... View more

It is a free app in Splunkbase, though it does mention it is for Vault Enterprise Users. It leverages opensource collectors. (telgraf and fluentd) More here: https://splunkbase.splunk.com/app/5093/ https://www.hashicorp.com/get/vault-splunk-application/ https://learn.hashicorp.com/vault/monitoring/monitor-telemetry-audit-splunk   ... View more

depending on your environment, the monitoring console will be available at Settings > Monitoring Console https://docs.splunk.com/Documentation/Splunk/8.0.5/DMC/DMCoverview I believe search lag is a symptom of skipped or differed searches...check out the search related stats and review your server resources   ... View more

What OS are you using? Also, what version of Splunk, as systemd may be in play... ... View more

Hi! Can you please share more details, like Splunk version and full data path to indexer? Is this Universal Forwarder to Indexer? Can you try  ./splunk btool inputs list --debug and confirm the forwarder sees your changes?   ... View more

what line?? ... View more

can't you just talk to the humans that do have access to install apps??? Much easier than you re-inventing the wheel. Also based on the question below about why a lookup is necessary, I would recommend you save the scars of learning 😉 Plus once your alert goes nuts...you'll see why the app is so cool ... View more

yep! Honorable mention for meta woot for sure! ... View more

you can also check /etc/collectd.log or your os equivalent path to see what the issue is. I recommend using the easy installer that comes with Splunk app for infrastructure! https://docs.splunk.com/Documentation/InfraApp/2.0.1/Admin/AddDataLinux ... View more