Hi, recent versions of syslog-ng Premium Edition can send log messages to Splunk HEC directly. syslog-ng also has a wildcard file source to monitor files and directories for log messages.
... View more
The problem here ended up being that we had a useACK = true in an outputs.conf file without a stanza tag above it. Therefore, it applied to all output, including the [syslog] output. A syslog server will not send back an ACK. Splunk will wait 2 seconds for each event and then send the event anyway (based on our observations). We added a [tcpout] above the "useACK = true" setting so it would apply only to tcpout and not to syslog output and that fixed this.
Early in the troubleshooting, we did hit on this setting. We added a useACK = false to the syslog stanza, but that still doesn't disable the useACK apparently. I've even explicitly tried again to set that to false under the syslog stanza but it doesn't seem to matter. If it's set to true globally, that seems to take affect.
I still can't explain why maybe 1 out of 10 times we restarted, it would work just fine even though this config error was still present.
Thanks to Jack Herod from Splunk support for finally finding this configuration error. If you're at .conf, I owe you a beer.
... View more
We're hitting the limit here,
we haven't resolved it yet, but this might be at least more info for anyone else coming across the issue...
Update via Splunk support:
The issue is an Enhancement Request : SPL-159073 is that if an Azure SSO user is a member of more than 150 groups a link is sent rather than the group details, and at present Splunk doesn't handle that link. We also encountered some cases where there are more than 100 groups, not 150 as indicated.
There's also a (draft as at time of support ticket) Splunk knowledge article support provided (reproduced with permission):
Title - Splunk - Azure SAML/SSO authentication error "saml response does not contain group information"
Summary - Instructions on how to avoid receiving the error "saml response does not contain group information" when connecting to Splunk with SAML via Azure configured.
Description: To ensure that the token size doesn't exceed HTTP header size limits, Azure AD limits the number of objectIds that it includes in a groups claim. If a user is member of more groups than the overage limit (150 for SAML tokens, 200 for JWT tokens), then Azure AD does not emit the groups claim in the token. Instead, it includes an overage claim in the token that indicates to the application to query the Graph API to retrieve the user's group membership.
How to Identify:
- Splunk is configured for SAML authentication with Azure AD as the Identity Provider
- You attempt to login to Splunk with an Azure AD user which is a member of a SAML group configured in Splunk, and a member of more than 150 Azure AD groups in total (including nested groups).
- You receive the error "saml response does not contain group information" when logging in to Splunk.
Investigation: Enhancement Requests: SPL-134770 SPL-159073
Option 1: Reduce the overall group membership for each connecting Azure AD user to less than 150.
Option 2: Use Azure Application Roles instead of Azure AD Groups for authorization:
Azure - Define the required Application Roles
Splunk - Update the SAML configuration "Role Alias" to use: http://schemas.microsoft.com/ws/2008/06/identity/claims/role
Splunk - Create SAML groups to match the Application Role names.
Using Application Roles: https://social.technet.microsoft.com/wiki/contents/articles/40055.azure-ad-application-roles-essentials.aspx
Details on Azure AD group sending behavior: https://techcommunity.microsoft.com/t5/Azure-Active-Directory-Identity/Azure-Active-Directory-now-with-Group-Claims-and-Application/ba-p/243862
... View more