This configuration still works for us on version 8.0.1. Note we do see handshake errors in some cases (not the same error as yours). This is almost always a firewall rule for a small set of hosts that typically works once the FW rule is updated. Can you confirm this is failing for ALL hosts contacting your DS or if it is isolated to a set of hosts.
... View more
We ended up getting this working using DNS load balancing and the setting crossServerChecksum=true . We had to use DNS load balancing to retain the the originating host ip as well. Otherwise the ability to whitelist by IP in DS wouldn't work. Additionally we have upgraded a few times since this post was originally written.
... View more
Curious to know if anyone has a workaround to the situation as the error logs are massive due to this. Perhaps writing those errors to a different bucket is possible although I have not tried that. Also potentially adding a filter on the bucket may work as well.
... View more
As an FYI. This issue appears to be back in ES 5.2.2 at a minimum. I have not checked versions between 5.1.0 where the issue was supposedly fixed and 5.2.2.
... View more
I had the exact same issue you had or are having. I found that the next event to be ingested had a bad date prior to 1/1/1970 and that field is mapped to the _time field in Splunk. I updated the appropriate input in /var/lib/splunk/modinputs/server/splunk_app_db_connect to force the input to skip that record, restarted splunk and everything started working again.
... View more
In our environment a rolling restart was not required. A restart of the Search Head the had the warning message in the original post corrected the issue (at least for now).
... View more
We are having the same issue where our data model acceleration runs fine and then stops working. I was curious if any further details were found regarding this or if there was a splunk issue already existing.
... View more
This issue does not occur in DBX 3.0.x so in 3.1.x the columns from T2 all need to specified as opposed to using the "*" option to return all rows from T2.
... View more
If you are using a index cluster ensure that your search head is configured to talk to the cluster master and using mode=searchhead in server.conf (please see docs for all details). Do not add the indexers in the index cluster as search peers.
... View more
Adding additional detail. After finding the offending bucket as suggested by Rajpal we simply did the following.
Ensured our cluster had recovered and was meeting search and replication factor.
stopped splunk on the bad indexer.
backed up the offending bucket directory.
Deleted the entire bucket directory.
Restarted splunk.
Since the cluster was essentially recovered simply removing the directory completely did not cause issues.
Eric.
... View more
We have setup an ELB in AWS however when the UF contacts the DS the ELB's IP address and DNS are replaced so all of the hosts in the forwarder management are the same. We're you able to resolve this issue as well?
... View more
I see this issue with large JSON events in version 6.4.0. This could simply be a limit reached but not sure which limit. What limits.conf adjustments were made? My core fields of host, sourcetype and source all disappear and not displayed even when clicking the show all fields option.
... View more
FYI. Had this same issue but with the Splunk_TA_oracle on a linux box. Physically deleting the app on the client server and restarting the client did the trick. This was a search head so the issue is not limited to the UF. This was on 6.3.0.
... View more
If you are planning on forwarding data to RSA in syslog then in RSA set “rfc3164hdr_enable” to “true” on the VLC. This allows RSA to pull the host value from the syslog event as opposed to pulling it from the tcp header which is the default.
... View more