About hexx

hexx · ‎10-12-2016

Thank you for reporting this issue with the Monitoring Console Health-check! Product defect SPL-130183 was filed to fix it in a future release.

hexx · ‎09-13-2016

That's unexpected. I'm not sure what to suggest besides attempting a factory reset of the DMC, which can be done via a dedicated button in the DMC general setup page.

hexx · ‎09-13-2016

Did you manually add that search-head as a search peer on the DMC / Cluster Master instance? This is a required step: Repeat steps 3 and 4 for each search head, deployment server, license master, and nonclustered indexer. Do not add clustered indexers, but you must add clustered search heads.

hexx · ‎09-12-2016

Could I connect the two search heads and then somehow search "through" the search heads to the indexer? No. There is no "proxy-ing" of distributed search. As you dispatch a search to search peers, they will respond with their own results but they will not pass on the search to their own search peers if any are defined.

hexx · ‎08-30-2016

You'll have to regenerate the forwarder asset table with in the setup > forwarder monitoring page - see this documentation page for details.

hexx · ‎08-24-2016

(on a side note, why is the Instance(host) column a large font size than the rest of the table?) I think the idea there is to highlight / underscore the "primary" instance name that the DMC uses to identify instances. It should probably be "Instance (serverName)" that is highlighted, though, as it is that value that we use to populate the "instance" pull-downs. I'll file a bug.

hexx · ‎08-24-2016

does it matter if the hostname listed in the second instance (marigold-ds in this example) is not a real hostname or alias? No, it doesn't matter. The settings we are talking about here represent an arbitrary label for your Splunk instance, which can be completely decorrelated with the hostname of the server that the instance runs on.

hexx · ‎08-24-2016

Duplicate instance name. Ensure each instance has a unique instance (host) name. This message highlights one of the DMC requirements: Make sure that each instance in the deployment (each search head, license master, and so on) has a unique server.conf serverName value and inputs.conf host value. I would hope that the "Learn More" link should take you to a documentation topic that points out this requirement. If not, please let me know and I'll file a bug. In essence, all instances monitored by the DMC need to have different values for both of the following properties that set the instance's name: server.conf / [general] / serverName inputs.conf / [default] / host In your case, you need to edit etc/system/local/inputs.conf on instance "marigold-ds" and set the value of host in the [default] stanza to "marigold-ds", then restart this instance and run the DMC setup again.

hexx · ‎08-24-2016

Regardles of the type of license you are using, it all starts here - http://docs.splunk.com/Documentation/Splunk/6.4.2/DMC/Deploymentsetupsteps

hexx · ‎08-24-2016

Yes, you can.

hexx · ‎08-24-2016

But despite the fact that that makes the search peer configuration happy, it's apparently not enough to make the DMC happy enough to add it. Or at least to let the DMC start monitoring it because the machine name is the same. @mfrost8, this is unexpected. While the DMC requires for the instances it monitors to be uniquely identifiable based on the values of "host" (as defined in inputs.conf / host) and "splunk_server" (as defined in server.conf / serverName) associated with the events they read & return, "machine" (which represents the hostname of the server on which the Splunk instance is running) does not need to be unique. While co-hosting Splunk instances is not something we necessarily recommend, it is supported to monitor co-hosted instances with the DMC. Can you be more specific about the behavior you are seeing?

hexx · ‎08-05-2016

To rebuild the forwarder asset table, you simply have to run the "DMC Forwarder - Build Asset Table" saved search with earliest and latest times that cover the lookback period that you want to consider. To do so from the REST API, you'd have to POST to the saved/searches/{name}/dispatch endpoint for this search - http://docs.splunk.com/Documentation/Splunk/6.4.1/RESTREF/RESTsearch#saved.2Fsearches.2F.7Bname.7D.2Fdispatch

hexx · ‎08-02-2016

Actually, @the_wolverine is THE LADY! 🙂

hexx · ‎05-03-2016

I don't foresee any issues with the setup you're describing - two DMCs in distributed mode should be able to monitor the same deployment independently as long as they are correctly configured individually.

hexx · ‎04-18-2016

This issue is specific to 6.4.0 and is tracked as product defect SPL-118170. This issue will be fixed in an early 6.4.x maintenance release. In the meantime, there is a simple work-around to prevent this error, which only affects one python script - info_gather.py , which splunk diag relies upon: Download the patched info_gather.py from Splunk's box account Use it to replace file $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py You can now run splunk diag again without issues - no restart required! For reference, here's the very small diff between the old file and the patched file: $ diff -u /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py --- /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather-6.4.0.py 2016-03-25 20:05:02.000000000 -0700 +++ /opt/splunk/lib/python2.7/site-packages/splunk/clilib/info_gather.py 2016-04-18 14:36:39.700531894 -0700 @@ -3442,7 +3442,7 @@ combined_apps.update(slave_apps) combined_apps.update(pooled_apps) - for key in combined_apps: + for key in combined_apps.keys(): if key not in unique_app_set: del combined_apps[key]

hexx · ‎04-05-2016

To answer this question: We would like to know what exactly is Splunk UF looking for to determine EOF? ...the context here is specific to reading files from disk with a [monitor] or [batch] data input. Here's how splunkd decides that a it has truly hit EOF (end-of-file) for a file it is reading: splunkd reads until the filesystem indicates that the end of the file has been reached splunkd backs off for 3 seconds (configured in inputs.conf / time_before_close ) splunkd checks the end of the file again - if it hasn't moved, splunkd considers that it has truly hit EOF and can move on from the file. Most notably, it can now end the specific data stream sending events from this file to an indexer. if EOF has moved, the new data is read and splunkd will again wait for time_before_close seconds after the new EOF is found. Do note that this really only applies to Universal Forwarders, as they perform no parsing on the data they read and therefore have to wait for a "true EOF" in order to close a file-based data stream safely (i.e: without risking to cut events in half).

hexx · ‎03-07-2016

The first document is correct: You should be able to set the search-head cluster label from any member instance. You set it once and it will be propagated to all members. Note that this does not indeed include the deployer, for which you will need to set the label manually as it is not really a member of the cluster but rather a supporting instance for it. I have amended the documentation to make this clear.

hexx · ‎02-02-2016

Take a look at the Resource Usage: Machine view.

hexx · ‎02-02-2016

If you are looking to estimate the usage of your license quota, the only source of truth is the events of the license_usage.log file as they are recorded on your license master. The panels of the License Usage view in the Distributed Management Console provide authoritative searches on this matter. Now, if you are looking to estimate your daily indexing throughput (whether the data counted against your license quota or not), I would recommend to leverage the events of group=thruput name=index_thruput in metrics.log, like so: index=_internal group=thruput name=index_thruput | timechart span=1d sum(kb) AS daily_KB Do not attempt to use events of group=per_*_thruput to accurately determine license usage or indexing thruput as those represent a sampled measurement.

hexx · ‎02-02-2016

In the panels of the CPU/Memory Usage view, S.o.S expresses CPU usage per process class as a percentage of one CPU core. This means that a CPU usage of 600% is equivalent to 6 CPU cores used - in this case, 100% of system-wide CPU resources. Virtual memory is typically inclusive of both physical memory (RAM) and swap (disk), which is why it is not unexpected to see it exceed the amount of physical memory installed. Finally, as @javiergn points out, I strongly recommend to migrate from S.o.S to the Distributed Management Console to monitor your Splunk deployment.

hexx · ‎01-28-2016

Make sure that you are looking at this particular view from the cluster master! It is not expected for these properties (and that view in general) to be accurate if consulted from an S.o.S app installed anywhere but on the cluster master.

hexx · ‎01-27-2016

Additionally, do note that the DMC does not currently have the capacity to directly monitor forwarders - this means that you should not attempt to set up any forwarder (heavy-weight or otherwise) as a search peer of the DMC for direct monitoring. The DMC does offer the ability to monitor your forwarders, but it does so indirectly by querying logs recording incoming forwarder traffic by the indexers.

hexx · ‎01-12-2016

The machines or servers that are exposed in the Distributed Management Console's "Resource Usage: Machine" view are those that host a Splunk instance that is monitored by the DMC. Therefore, the only way to add a given machine to the scope of that view is for it to host a Splunk instance and to configure the DMC to monitor it.

hexx · ‎01-08-2016

Ultimately, this depends on the exact nature of the failure that leads the distributed search framework on the DMC to declare the status of some of your peers as "down" intermittently. I think that statusTimeout is a good guess here if you need to pick one timeout setting to extend, but ultimately it would be better to review the details of the peer failure in splunkd.log to understand what timeout led to declaring the peer down (the search-head should know this) and what was going on on the peer itself.

hexx · ‎01-08-2016

Just curious but why are you looking to edit this search? Also, was the search disabled manually?

Posts	721
Solutions	220
Karma Given	1917
Karma Received	2176
Member Since	‎07-02-2010

Online Status	Offline
Date Last Visited	‎05-03-2022 09:33 PM

On Linux, how can I leverage operating system faci...

How do optimizations for field-based searches work...

Why are my real-time alerting searches no longer s...

What is the cause of these socket errors reported ...

Forwarder shows extreme lag or latency when sendin...

When is it appropriate to set followTail to 'true'...

How can I trigger the re-indexing of a single file...

How do I check if my custom TIME_FORMAT is success...

Can the Splunk on Splunk app query a heavy forward...

Time stamp of historical syslog data gets incorrec...

Re: NOT a question: There is a small bug in the he...

Re: Why is my search head missing in Distributed M...

Re: Why is my search head missing in Distributed M...

Re: Can I search a search head from another search...

Re: How to remove "missing" forwarders from the Di...

Re: Why am I unable to add hosts with more than on...

Re: Why am I unable to add hosts with more than on...

Re: Why am I unable to add hosts with more than on...

Re: Can we set up the Distributed Management Conso...

Re: Can we set up the Distributed Management Conso...

Re: Why am I unable to add hosts with more than on...

Re: Is there a REST API call to rebuild the forwar...

Re: Your entry was not saved. The following error ...

Re: Two DMCs on Two different Splunk Deployers mon...

Re: Why am I unable to generate a diag from an ind...

Re: Splunk Universal Forwarder and TCP Data: What ...

Re: SHC cluster Label

Re: Splunk on Splunk not matching to resources on ...

Re: How to estimate today's indexing volume

Re: Splunk on Splunk not matching to resources on ...

Re: After installing S.o.S - Splunk on Splunk in m...

Re: Why does my Splunk 6.3.2 Distributed Managemen...

Re: How can I add more machines in the drop-down m...

Re: DMC Alert - Search Peer Not Responding; How to...

Re: Why can't I update the Distributed Management ...