About nnmiller

nnmiller · ‎09-15-2020

@ridwanahmed Yes, you would need to deploy the app from the SHC deployer.

nnmiller · ‎06-25-2019

splunk/etc/system/replication/ops.json ops.json resides on each of the SHC members. It is a commit history for the changes made to KOs and other configurations on the SH for the time period it covers. It will contain references to the roles you removed until the ops.json window shifts past the date where the changes were made or applied on the SHC members.

nnmiller · ‎05-08-2019

You need to review your job inspector and possibly search.log and determine where the latency is occurring. Also, if you are on 6.6 or later, you need to revert max_searches_per_process back to the default of 500. Many orgs ran into the bug prior to 6.6 which causes slow search start up and still have max_searches_per_process = 1 .

nnmiller · ‎02-01-2019

There's are HUGE caveats to the key routing method to re-parse data: a) unsupported (don't bother opening a case about it) b) untested (e.g. it has worked in the past but is not QA'd so could break at any point) c) applies to ALL inputs on the system you set it up on -- regardless of the stanza it's applied in, so you are essentially relegating an IF to a single input.

nnmiller · ‎06-20-2018

This is normal behavior. An index cluster master is also a search head -- that's how it populates all the cluster master's "indexer clustering" information. A standalone License Master, if added to the monitoring console would also have the role of "Search Head", since that's where the license dashboards are generated.

nnmiller · ‎01-28-2018

Splunk cannot "mitigate" or "remediate" the impact of a kernel patches that, essentially, turn off all CPU optimizations. These flaws are inherent to the architecture of the CPU. Your organization must determine, for your organization , if the potential risk is sufficient to warrant applying the patches. To perform risk analysis, you need to understand the nature of the risk, and then examine how likely those potential risks could occur in your environment. Most organizations do not seem to bother with risk analysis of system vulnerabilities. Meltdown/Spectre is definitely a case where it should be performed.

nnmiller · ‎01-08-2018

I've asked the community on Slack/IRC to weigh in with concrete data pre-/post-mitigation.

nnmiller · ‎12-18-2017

You can't sum these as you have in your query, since they are IOPS per disk. From the /services/server/status/resource-usage/iostats docs page: Access the most recent disk I/O statistics for each disk. This endpoint is currently supported for Linux, Windows, and Solaris. By default this endpoint is updated every 60s seconds. Running: splunk cmd splunkd instrument-resource-usage --debug > some_log_file.log 2>&1 shows that these stats come from /proc/diskstats (HT: blachance_splunk)

nnmiller · ‎11-16-2017

Another method for the init script, ignoring the systemd aspect, requiring your version of su on Linux to have libpam.so linked, is to replace: "/opt/splunk/bin/splunk" start --no-prompt --answer-yes with the following, replacing <splunk_user> with the correct user for your environment: su <splunk_user> -c '/opt/splunk/bin/splunk start --no-prompt --answer-yes' So far I've check RHEL 6 and CentOS 6 and both include the PAM library in the su binary.

nnmiller · ‎08-23-2017

mmodestino, I think you typo'd there... I think we have sufficiently realized why CLI will always be there...heck, even on windows and mac, admin via the gui is the way to go. Don't you mean 'admin via the CLI is the way to go'? 😉

nnmiller · ‎08-22-2017

Reviewing the REST API Reference, it appears that the tutorial docs are incorrect. Per auth/login REST API docs section, the correct command here is (yes, enter username/pass 2x): curl -k -u admin:changeme https://localhost:8089/services/auth/login -d username=admin -d password=changeme Which returns an XML response with a session key: <response> <sessionKey>192fd3e46a31246da7ea7f109e7f95fd</sessionKey> </response>

nnmiller · ‎08-16-2017

Please try using one of the following to work around this issue: Unicode or ASCII version of special characters such as backslash, "<", or, ">". For example: Unicode "\" = \u005c Hex ASCII "\" = \x5c Unicode "<" = u003c Hex ASCII "<" = \x3c Use {X} where "X" is the character count to indicate the number of occurrences of the character. E.g.: #blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe\<{1}" #blacklist = EventCode="1" Message="\.{1}" In SEDCMD these two methods are used as a workaround for oddities with the regex parser, so I am wondering if they work here as well.

nnmiller · ‎08-14-2017

The NIX UI will never be common on *servers for many reasons, most important of which is security. In addition it adds a ridiculous level of dependencies which have the potential to make the system unstable when patching (you do, patch right? see "most important" point), increase resource consumption, add configuration management work, etc. Source: 15+ years as a systems admin.

nnmiller · ‎08-11-2017

Just to clarify: the single ERROR line contains the cause "SPL-90606". This references a bug number in our database.

nnmiller · ‎08-02-2017

Just a note--I tried hexx's method in 6.6.0 and it failed with <msg type="ERROR">You do not have permission to share objects at the system level</msg> even though I was attempting it as admin. Instead, I had to use the methods covered in the documentation on REST ACLs, involving curl .

nnmiller · ‎07-28-2017

Splunk automatically calculates the concurrency. If you change the CPU configuration of VMs, you should verify that Splunk recalculates properly. Calculation: max_searches_per_cpu x number_of_cpus + base_max_searches = max_hist_searches (max_searches_perc / 100) x max_hist_searches = max_hist_scheduled_searches max_rt_search_multiplier x max_hist_searches = max_realtime_searches The default settings are base_max_searches = 6 , max_searches_per_cpu = 1 , max_searches_per = 50 , and max_rt_search_multiplier = 1 . So in your case: ((1 * 8) +6) = 14 = max_hist_searches per SH 4 * 14 = 56 = max search head cluster concurrency ((50 /100) * 14 ) = 7 = max_historical_scheduled_searches per SH 4 * 7 = 28 max search head cluster scheduled search concurrency 1 * 14 = 14 = max_realtime_searches per SH 4 * 14 = 56 = max rt search head cluster concurrency How you have quota enforcement set on the SHC will also affect perceived concurrency at the SHC captain (please see How the cluster enforces quotas. Additionally, if there are communication problems in your SHC that prevent the SHs from reporting back that scheduled jobs have been completed, the captain will not remove their search jobs from the queue, resulting in delays or failure to delegate jobs by the captain.

nnmiller · ‎07-07-2017

You should not be editing back end configuration files manually on your SHC. You should be performing back end edits on the deployer, then pushing the changes to the SHC. Please review the following documentation: Use the deployer to distribute apps and configuration updates How configuration changes propagate across the search head cluster There are isolated cases where you may want to change settings in $SPLUNK_HOME/etc/system/local/ config files, but these are rare, and generally only necessary if there is a problem in the SHC.

nnmiller · ‎06-16-2017

You're welcome. Hat tip goes to jkat54 for pinging me about this question.

nnmiller · ‎06-16-2017

The bug is poorly worded. Basically, Splunk capabilities work exactly as intended. If a user does not have the capability to run schedule searches, even if an admin goes in and sets the restricted user's saved search to a scheduled search, the search won't run. A bug that allowed admin to successfully set a search as scheduled for a user without that capability was fixed. Another customer that was using that "loophole" filed this bug when the loophole stopped working. The work around to get the "loophole" back is by creating a the special "service account", giving ownership of the searches to that "service account", then giving each user that needs to modify those searches read/write permission.

nnmiller · ‎06-09-2017

SSH has pretty bad performance, and therefore, so does scp. Once the network buffer is full, it's slow slow slow. There are a couple of ways around this: a) use the high performance network patches for SSH available here PSC HPN Patches (I use these myself on servers which do large file transfers), or switch to an ftp over SSL implementation, netcat, or similar. For example, I had a customer using straight ftp to constantly transfer high volume firewall logs like this without issue. I'm not even sure the HPN patches would do what is needed in this case, because of connection pauses. If logs are not coming continuously through that pipe, then you're going to have the connection restart overhead throwing off your time_before_close . So maybe add something like the following to /etc/ssh/sshd_config : ClientAliveInterval 120 ClientAliveCountMax 720 I presume there's some sort of security policy to prevent you from just using syslog here?

Posts	85
Solutions	10
Karma Given	142
Karma Received	120
Member Since	‎11-08-2013

Online Status	Offline
Date Last Visited	‎12-02-2021 03:02 PM

How can I determine the lag between when an app's ...

KV Store lookup failing with error about KV store ...

Re: Can the Bloodhound app be used in a cluster en...

Re: What's ops.json in etc/system/replication?

Re: Splunk scheduler - how can I reduce latency? W...

Re: Can I parse some data on one heavy forwarder a...

Re: In a Splunk cluster, is it good practice to ha...

Re: Performance impacts of Spectre/Meltdown mitiga...

Re: Performance impacts of Spectre/Meltdown mitiga...

Re: IOPS reported by bonnie++ and Splunk Monitorin...

Re: RHEL/CentOS 7 & systemD not honoring ulimits

Re: Ubuntu Desktop version install - .deb fails to...

Re: Why do I receive REST API "login failed" messa...

Re: How to blacklist events in WinEventLog that ha...

Re: Ubuntu Desktop version install - .deb fails to...

Re: Why am I getting these failed bucket replicati...

Re: How to change the permissions of a saved searc...

Re: How to adjust concurrent search on a indexer c...

Re: What is the correct way to edit conf files and...

Re: What does this bug mean?

Re: What does this bug mean?

Re: Sending logs over scp to heavy forwarder, why ...