Interesting - running UF v7.1.2 on linux and this issue has come back - was previously fixed -- re-updating inputs.conf scripts to not have any whitespace for the interval (e.g. interval=300 <- always works, interval = 300 <- does not always work) ... View more

So quick followup @aaraneta (hoping you can help) -- I updated the the certs per the 6.6.0 documentation/read-me-first-documentation (via distribution server) on ALL of our UFs/Heavy-Forwarders in an attempt to resolve this and still getting the errors. - we have upgraded our "Heavy" infrastructure remotely (SHs/Indexers/LicServer/Heavy-forwarders, etc) fully to 6.6.0 except for our distribution server (v6.5.2) -->if we cannot talk to UFs instances < 6.6.0 we'll be in a bad place until resolved -- yes we have also remotely upgraded "most-all" of our UFs to v6.6.0, however due to a lag in the cloud with getting our AMI's updated (our lag, not Splunk's) - cannot rely on new-ami-rollout with splunk-6-6.+ for a bit yet, So appreciate any info or feedback to help resolve (no, we have not as yet opened a ticket with support as yet - process tends to less efficient) ... View more

Interesting - this issue is for compatibility for versions < 5.0 -- we are running thousands of UF instances all v 6.1 or greater (most 6.5+) and still seeing the issue. ... View more

fyi -- also in Splunk 6.5.2 ... View more

thanks! -- for us the issue was that we needed to enable "edit_index_cluster" for our LDAP based admin group (splunk v6.5.x) ... View more

We had the same issues UF v6.4.1 (latest at this time), and earlier revs - if we restarted the UF it would go away however eventually come back (all current windows flavors) - we have 1000's of hosts so we see the issue perhaps more than most - we actually built an an alert/ restart of the UFs having the condition by changing evt_resolve_ad_obj from 1 to 0 the issue has been (so-far) not been seen (not sure we like the solution, however better than sporadically getting completely bogus security event Messages) hth ... View more

Interesting sidenote: v6.4.1 Windows UF --- if at least one matching directory does not exist, the UF will peg the CPU on the windows server - as soon as you create one that will match it is fine again. ... View more

unfortunately none as yet - ... View more

any update at this stage? - were seeing this too, typically after a restart (v6.3.1) ... View more

you'll have to trust me as unclear how to upload an image to a response without sending to it a website or other first - reading it right now (ran same exact query as listed above) , 142,529,121,751.359985352 (startup.handoff) as indicated - do not want to get into digging into our environment specifically - unless there is something insanely wrong with ours.. .possible, however others are reporting this too (google is our friend) so suspect not - however open to all possibilities of-course thanks -tom ... View more

Thanks - here's the confusion (and trying to avoid drilling into our environment, examining search-logs etc, rather to better understand startup.handoff and how to tune it if applicable (conf variables to dig into etc), regardless of our results - what does the attribute represent and directions to tune it if relevant. Apologies if missing the boat - allot of folks are asking about this in various ways (google is our friend 🙂 taking a very simple query, run in fast-mode ("index=iis sourcetype=iis earliest=-65m@m latest=-5m@m|stats count by cs_host") the query took approximately 31 seconds to complete e.g. from the job inspector This search has completed and has returned 2,018 results by scanning 4,439,123 events in 30.905 seconds. We have fourteen (14) indexers running, our startup.handoff for the same search=141,579,974,861.00 doing that math (assuming it is seconds as the output suggests) 141,579,974,861.00 / 14 / 86400 == 1638657 days to run ? (if it's milliseconds take it down a few notches) - so still confused (despite the documentation) as to what startup.handoff really represents? ... View more

We have the same situation - what I ended up doing (short-term fix) was to write another app - that disables the Splunk_TA_windows app via an update to its local/app.conf, then disables itself as well), since we have our own collections of the Windows eventlogs, we cannot use the ones enabled in the Splunk_TA_windows app, so we had to turn it off. As new UFs are enabled they automatically get the "disable-app", unfortunately there is still a stream of several windows events that are lost until the Splunk_TA_windows app is disabled at initial startup. ... View more

Hi thanks for the feeback, Yes, we tried unsetting by setting it to blank and ("unset") - both actually caused iis to no longer be collected on the 6.1.4 (6.x) UFs e.g. In our troubleshooting we distributed a new UFs props.conf to the 6.x UFs - both did not work and actually caused iis to stop being collected on the 6.x UFs. [iis] INDEXED_EXTRACTIONS = and also tried [iis] INDEXED_EXTRACTIONS = unset in both the cases the 6.x UFs actually stopped forwarding the IIS data due this setting being incorrect. Additionally, since the only workaround appears to be using a custom sourcetype on the UFs then changing it at the indexers, the UFs (correctly) re-index all the data since it is a new sourcetype... (blew out our license over the weekend - no choice since alternative is to have randomly munged data where it never was before) There must be a way to turn off INDEXED_EXTRACTIONS so this scenario does not play out with other data-types as well (e.g. access_combined, json, etc) - For Splunk users migrating to 6.x of the UFs (from 5.x and earlier) this is going to pose a huge issue since the INDEXED_EXTRACTIONS settings that are done by default on well known sourcetypes. The munged _raw data from the UFs is seemingly random, in our shop about 0.5% of the rows coming in for iis sourcetype - just enough to get every user lighting torches and gathering pitchforks, since various reports/alerts/queries/etc now have bogus data peppered through them Defaulting INDEXED_EXTRACTIONS on the 6.x UFs for those migrating from earlier versions is not pretty -- the only workarounds seems to be rewrite all of our working field-extractions, to setup a distributed application to update etc/system/default/props.conf on all the UFs (anytime one is spun up), or change the sourcetype name on the UFs and change it back on the indexers thus avoiding re-indexing the data (unless the default setting on the 6.x UFs for INDEXED_EXTRACTIONS can be turned back off somehow?) Appreciate any and all inputs thanks again -tom ... View more

Hi, this does not appear to work? The user-group does have access to _internal (e.g. can run "index=_internal" queries etc) – and the user in question is a member of “other_group” (see below please). Have setup permissions to allow the user-group to have full r/w access to [views/status_index], even restarted the search-head – the user group in question cannot see the dropdown-menu still? e.g. in /etc/apps/search/metadata/local.meta == [views/status_index] access = read : [ admin, other_group ], write : [ admin, other_group ] == Appreciate any further advice. Thanks ... View more