About mdsnmss

mdsnmss · ‎06-30-2020

I've been noticing some bundle distribution errors recently on one of my search heads. This search head is part of a 3 server search head cluster connecting to an indexer cluster. The message I am seeing is this: 06-30-2020 08:04:57.184 -0400 ERROR DistributedBundleReplicationManager - HTTP response code 400 (HTTP/1.1 400 Bad Request). Error applying delta=/opt/splunk/var/run/searchpeers/CD224AB5-4AD5-4680-BA13-01234645B8B6-1593518595-1593518680.delta, searchHead=CD224AB5-4AD5-4680-BA13-01234645B8B6, prevTime=1593518595, prevChksum=730910445532009996, curTime=1593518680, curChksum=3452252417076391070: Error copying /opt/splunk/var/run/searchpeers/CD224AB5-4AD5-4680-BA13-01234645B8B6-1593518595/indexing_tokens to /opt/splunk/var/run/searchpeers/CD224AB5-4AD5-4680-BA13-01234645B8B6-1593518680.2346fcb2e8de1493.tmp/indexing_tokens. 1 errors occurred. Description for first 1: [{operation:"stat'ing source directory", error:"No suc... (message truncated, search splunkd.log for "distribution_error" to see it in full). I haven't had much luck finding anything on this on Answers or just searching for the issue. I've gone to the splunkd.log to search for this message but it actually shows truncated at the source, so there isn't "seeing it in full" there. May need to turn up logging level for it. Just wanted to see if anyone had any thoughts or have run into this before?

mdsnmss · ‎03-11-2020

Hi @galineth. I'm the creator of the Unified Forwarder Monitoring App for Splunk. Is there a specific panel you are seeing the error on or is it all of them? The base search should be in all panels so I'd expect if you see this in one you'd see it in the others. What version of Splunk are you currently on and are you able to run any of these searches from the GUI? | rest /services/deployment/server/clients splunk_server=* | rest /services/deployment/server/clients splunk_server=127.0.0.1 | rest /services/deployment/server/clients It looks like this is the part that your search is erroring out on. It should take splunk_server from the peers you have added as search peers on the host you are searching from.

mdsnmss · ‎02-12-2020

I have a search being executed via script hitting the REST API. Occasionally it will return no results and looking for the associated events in _internal we get the below: Through this we can see that once it hits around 300000ms (5min) the search times out. Anything below it we get data returned as shown by the non-zero values after each 200 status code. I've been looking through the spec files for what setting might be imposing this limit but have not had any luck in finding one that changes this value. I've gone through looking via grep " 300 " /opt/splunk/etc/system/README/*spec as well as other variations of that time format. In addition to this, I have sent arguments with the POST for auto_cancel and ttl and it does not appear to affect this 5 minute timeout. Any thoughts as to where this limit is being imposed?

mdsnmss · ‎01-22-2020

Are you using LDAP authentication? Slow LDAP or having many returned groups can cause issues. Have you always had slowness in the UI or was it only noticed after installing ITSI?

mdsnmss · ‎01-22-2020

Full instances of Splunk send what is "cooked" data. Heavy forwarders are a full instance of Splunk so from the first input where you are monitoring your .txt file you are sending cooked data to the other HF and then on to the IDX. Things such as indexed extractions and filtering need to take place on uncooked data as that is part of the "cooking" process. So the filtering needs to take place at your first HF where the .txt file resides. You can also try setting sendCookedData = false on the first HF in the process as well. More info on the types of forwarder data is here: https://docs.splunk.com/Documentation/Splunk/8.0.1/Forwarding/Typesofforwarders#Types_of_forwarder_data

mdsnmss · ‎01-21-2020

I'm going through something similar in my environment and it just takes some planning for a switchover. It also depends on how you have your certificates configured. If you have a unique certificate for each forwarder than it is certainly much more painful. A common configuration is to use wildcard certs for forwarders and unique for all servers. Create new certs alongside the old ones and update paths in your configs to point to the new certs without restarting Splunk. Then it is a matter of updating the deployment server to push new certs to each forwarder while also rolling indexers and search heads with CA changes simultaneously allowing those configs to take effect. Likely a brief outage would be required but shouldn't be extensive. You can also temporarily disable the SSL settings while the switch is being made. It definitely isn't easy and done wrong can break your entire Splunk environment.

mdsnmss · ‎01-21-2020

Sounds like you need a custom search command or external lookup. These links should get you started: External Lookups: https://docs.splunk.com/Documentation/Splunk/8.0.1/Knowledge/Configureexternallookups Custom Search Command Docs: https://docs.splunk.com/Documentation/Splunk/8.0.1/Search/Writeasearchcommand Custom Search Command Dev: https://dev.splunk.com/enterprise/docs/developapps/customsearchcommands/ What kind of details does the script return? What does the script do? An example of an external search command is in Splunk Supporting Add-on for Active (https://splunkbase.splunk.com/app/1151/). Given different arguments it will return LDAP user/object results to supplement results or create lookup tables. More details are likely needed to determine the best way to do this. There isn't really a simple "out of the box" way to do this.

mdsnmss · ‎01-21-2020

Many features are disabled once Splunk is converted to the Free license. It looks like this app relies heavily on alerting and scheduling which is no longer in "Free": https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/MoreaboutSplunkFree.

mdsnmss · ‎01-17-2020

I'd recommend this app: https://splunkbase.splunk.com/app/1866/. It has a lot of good in depth visualizations going into role and individual user accesses.

mdsnmss · ‎01-16-2020

I'd say this answer can be pretty broad. There are multiple ways to collect this sort of information without UFs (for example Windows Event Forwarding for Windows endpoints) but most prefer the UF on endpoints directly. This provides distribution and less of a single point of failure or bottleneck in the collection. It also depends on scale of number of endpoints you want to collect from. I put UFs as part of my standard build. Every system that gets built receives a UF and an add-on that points the UF to a deployment server. Depending on scale you may want multiple deployment servers. From the deployment server it gets a default set of configs like outputs to indexers, inputs to define what to collect, etc. From there you have a default baseline that is easy to add stuff to if a certain host requires additional logs to be collected. If you are looking for config specifics for stuff like SSL I'd start here: 1.) https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPractices.pdf. 2.) https://docs.splunk.com/Documentation/Splunk/8.0.1/Security/AboutsecuringyourSplunkconfigurationwithSSL Once you establish a repeatable method of joining your UF to the rest of your environment it mostly identifying what you want to collect and create baselines.

mdsnmss · ‎01-16-2020

I converted the comment to an answer. Feel free to accept if this resolved your issue or let me know if you have any other questions. Thanks!

mdsnmss · ‎01-15-2020

You can just mark your response here as the answer. That way the question will be marked as resolved and answered.

mdsnmss · ‎01-14-2020

I'd start by checking queues. Queue backup can occur on indexers or forwarders. You can check queues for the indexers by going to your Monitoring Console and going to Indexing-->Performance-->Indexing Performance: Deployment. You can check filling queues on your forwarders with the search: index=_internal group=queue | eval percfull=((current_size_kb/max_size_kb)*100) | search percfull>80 | dedup host, name | table _time host name current_size_kb max_size_kb You can check thruput on your forwarders by searching for these events: INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying... Hosts that throw that event may be candidates for increasing maxKbps in limits.conf. (https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Troubleshootingeventsindexingdelay#Possible_thruput_limits)

mdsnmss · ‎01-14-2020

Have you narrowed your license usage down to a single index in _internal? You have a single index search for real raw but the other search provided gives info for all indexes. index="_internal" source="*metrics.log" group="per_index_thruput" series="myindex" | eval mb=kb/1024 | stats sum(mb) as mb index=myindex | eval b=len(_raw) | stats sum(mb) as mb | eval mb=b/1024/1024 Is it the _internal or the raw search that shows as the higher number? You also want to look at conversion. Len will give you bytes vs. the _internal data that provides it in kb.

mdsnmss · ‎01-14-2020

Your search head doesn't have enough space for the recommended minimum. The dispatch directory is what contain artifacts (such as search results) created by each search that get executed. Depending on time to live of the search this stuff ages out and keeps cycling so there is a fluctuation there. You want to have enough space for your results to reside there though. You can change the warning to appear at a different size if you want to get it closer to max <5GB by changing dispatch_dir_warning_size in limits.conf. The directories you are referencing as the largest/oldest are for datamodel acceleration. You can see your datamodels can be seen in "Settings-->Datamodels". I wouldn't go in and manually delete the directory, but if the datamodel is not used and you want to remove it you can remove the acceleration from the Datamodels page. Alternatively, you can reduce the acceleration duration so it retains a smaller set of data.

mdsnmss · ‎12-03-2019

Looks like the hyperlink added the period onto it and broke the link. I just removed the period from the link and it should work now.

mdsnmss · ‎11-20-2019

Your LDAP is handled within authentication.conf so it can be distributed easily from a cluster master, deployment server, or search head deployer. In case of changes to LDAP I'd recommend configuring it for deployment rather than on each system individually in order to be able to quickly distribute those changes if the need arises.

mdsnmss · ‎11-20-2019

Have you tried resetting the password hash? i haven't tested it in 8 but a good write-up on it is available here: https://www.hurricanelabs.com/splunk-tutorials/splunk-7-1-performing-a-splunk-password-reset

mdsnmss · ‎11-20-2019

What kind of changes are you looking to alert on? If it logs an event you can generate an alert with a scheduled search in Splunk to monitor for those changes. It's mostly finding what indicates a config change, ingesting that into Splunk, and configuring a search to alert based on your alert thresholds. Whether that be Windows registry changes, Wineventlog events, or other application events.

mdsnmss · ‎10-17-2019

Update has been posted! https://splunkbase.splunk.com/app/3805/

Posts	196
Solutions	28
Karma Given	11
Karma Received	43
Member Since	‎05-24-2016

Online Status	Offline
Date Last Visited	‎04-08-2023 04:39 PM

DistributedBundleReplicationManager response code ...

RESTAPI Search Limits TTL

Icon for custom app will not appear. I only get th...

Splunk Analytics for Hadoop not filtering correctl...

Configuring virtual index to reach out to S3 over ...

How to resolve authentication issue with virtual p...

Max time spent in per-result alerts issue

Splunk Upgrade to 7.0 TLS and SSL on Windows Serve...

Field alias or batch renaming with wildcard

(Beta) Proofpoint Email Security App for Splunk co...

DistributedBundleReplicationManager response code ...

Re: REST endpoint error

RESTAPI Search Limits TTL

Re: Splunk UI Slowness - takes time to load

Re: Why my configuration is not working ? (nullQue...

Re: Replace root CA for Splunk cluster

Re: How to run a linux command on remote machine v...

Re: Website Monitoring with Splunk 8.01 free not w...

Re: Is there a way to visualize the roles relation...

Re: Log collection off network devices

Re: Dispatch Manager : The minimum free space reac...

Re: How to understand actual license volume for in...

Re: Index feed is having low throughput

Re: How to understand actual license volume for in...

Re: Dispatch Manager : The minimum free space reac...

Re: how to reset admin password in splunk 8?

Re: LDAP authentication configuration in a distrib...

Re: how to reset admin password in splunk 8?

Re: Does Splunk monitor and alert on changes to se...

Re: Versions above 7.1 for UFMA?