This is more statement than question, but the community should be advised Splunk Universal Forwarder 9.1.2 and 9.1.5 (those I have used and witnessed this occurring on, I cannot speak to others yet) does not abide by cron syntax 100% of the time. It seems to do so at a very low frequency of occurrence, which may make repeated runs of scripted inputs difficult to notice or detect that it's occurring. I feel it's important the community is aware of this as some scripts may be time sensitive or you expect the scripts to only run once in a reporting period so no deduplicating was added -- amongst other possible circumstances a double-run might cause unintended impacts. In my case I noticed it when doing a limited deploy of a Deployment Server GUID reset script. With only 8 targeted systems scheduled to run once per day it was very easy to notice that it ran twice for multiple assets. Fortunately, I'd designed this script to create a bookmark so it wouldn't run more than once so the UF could run it every day to catch new systems or name changes, so it running more than once wouldn't cause a flood of entries in the DS. In my environment I noticed the Scheduler prematurely ran the scripts roughly 2 seconds before the cron-scheduled hour-minute combination, so the following SPL could find when this occurred: index=_internal sourcetype=splunkd reschedule_ms reschedule_ms<2000 Example: Scripted Input is scheduled to run at "2 7 * * *" (07:02:00), but there was a run at 07:01:58 AND 07:02:00. It ran two seconds early, then reschedules it to occur at the next scheduled interval which happens to be the expected time (two seconds later), and the cycle repeats. It seemed to fluctuate and vary, and sometimes even resolve itself given enough time. It's difficult to know what influences it or why after a correct run the run of milliseconds added is short by ~2000, but it does happen and may be influencing your scripts too.   Reported I opened a ticket with Support, but there's no word on a permanent resolution to this less than 1% affected forwarders issue so far, so I feel letting the community know to look if that's happening -- and if it matters in your environment, because maybe a little duplication doesn't matter -- was important.   Workarounds Changing from cron syntax to number of seconds for the Interval does work as a workaround, but that isn't always what you want -- sometimes you want a specific minute in an hour, not just hourly (whenever). Adding logic into a script to check the time is another possible workaround. ... View more

Add-on: https://splunkbase.splunk.com/app/3662/ Known Affected: 4.8.1 Symptoms: You begin to predominantly see Hexadecimal events in your Cisco FireSIGHT Index/Sourcetype instead of real data, and you see large gaps between events (usually ~10 minutes, the time it takes for it to roll over a file). The 'Source' also ends with '.log.swp' instead of '.log'. Cause: $SPLUNK_HOME/etc/apps/TA-eStreamer/default/inputs.conf  [monitor://$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/data] disabled = 0 source = encore sourcetype = cisco:estreamer:data crcSalt = <SOURCE> The issue I believe is with the bolded line 'source = encore' because 'crcSalt = <SOURCE>' is also specified. Since all files have the same Source, all files have the same crcSalt which is why the actual '.log' is not collected. The '.swp' manages to get collected as Splunk checks the '.log' and since swp is a very short lived file Splunk accidentally collects a lot of garbage unrelated to the actual file contents (sorry Linux Admins for butchering the technical detail). Solution: Edit $SPLUNK_HOME/etc/apps/TA-eStreamer/default/inputs.conf and comment out the Source line, then restart Splunk services.   If someone knows of a way to override (via Local inputs.conf) source back with the filename (which changes frequently) so editing a Default inputs.conf is not necessary, please comment below. Those with the Cisco license allowing TAC Support on this add-on may want to raise this issue with them so they can fix it for new downloads and future versions -- I lack that particular license. Hope this helps someone (I did a search for encore and hex and didn't see any prior conversation on the topic). ... View more

Issue When configured to use Azure SAML on our Enterprise Security search head (no Authentication Extension yet specified) I discovered that Enterprise Security 6.4.0's Incident Review's "Run Adaptive Response" returned "Unexpected token < in JSON at position 0" when attempting to run any response (even Ping) with no data passed to the response. It was an immediate failure. Support noted a HAR showed it was because credentials weren't being passed, and pointed to a lack of AQR Support by Azure as the reason. Backstory While surprising Enterprise Security had a single feature (so far discovered) relying on AQR, the lack of AQR by Azure was not a surprise as I'd been exposed to that when attempting to setup the Secure Gateway as well (which we gave up on as a secondary priority to finishing our installation). That same exposure also led to us discovering in the Secure Gateway documentation that there was a sample script to be used as a SAML Authentication Extension to overcome this lack of Azure support. Unfortunately, at the time, the script didn't seem to work -- after actually looking at its code I could tell why: the Splunk provided sample expected an Azure API Key. Solution WARNING for Production Environments: If you attempt to use the Authentication Extension script be advised that so long as it is enabled and not working your Web Session will timeout after the User Time To Live period regardless of activity because it cannot re-validate your identity (e.g. 3600s by default -- 1 Hour). When it times out your cookie may be well and truly hosed and you'll need to clear cookies & cache to get back to the login page. Worst case scenario, you'll need to edit $SPLUNK_HOME/etc/system/local/authentication.conf manually to comment out or remove getUserInfoTtl, scriptFunctions,scriptPath,scriptSecureArguments,scriptTimeout, then use $SPLUNK_HOME/bin/splunk restart to get back to the login page. Additional Warning: Splunk Support does not support any of the following (won't even try, they'll direct you to your Account Team) so if anything happens you can curse my name, but I take no responsibility etc. etc., but this is the only way anyone (including Splunk's own documentation, see above) has mentioned how to deal with Azure's lack of AQR support. Azure Prerequisites: If you have not already, go to portal.azure.com and under Azure Active Directory > App Registrations  create an App for your Splunk instance. Please see Splunk's documentation regarding setting up SAML -- but be sure to download the certificate and XML file from Azure! That XML file can be uploaded into Splunk's SAML Configuration page to auto-populate almost everything. For those without an Azure API key, we'll use the Client Secret method... In portal.azure.com where SAML was configured for Splunk (Azure Active Directory > App Registrations > All Applications > search for your app name here): Ask your Azure Admin to create a Client Secret under "Certificates & Secrets" Ask your Azure Admin to then add "Microsoft Graph" APIs User.Read.All (vital), Group.Read.All (unconfirmed if needed), and GroupMember.Read.All (unconfirmed if needed) under "API Permissions". Then ask them to provide Admin Consent on the same page (click a button applying these changes, essentially). For those with an Azure API Key, unfortunately I can't provide a lot of detail below). Enterprise Security Command Line: For those with an Azure API Key (may require special permission to request one) use the provided sample script at $SPLUNK_HOME/share/splunk/authScriptSamples/azureScripted.py (confirmed for Splunk 8.1) For those with a Client Secret (assigned to your Splunk SAML application, much easier to acquire) use the script from https://gist.github.com/vprasanth87/5bd091f0eb24c4919b938f0528ee93bc Place a copy of one of the above scripts into the $SPLUNK_HOME/etc/auth/scripts Have a Web Proxy/Gateway? For those with Web Proxies not using a Global Proxy value by some other means:  Open the $SPLUNK_HOME/etc/auth/scripts/azureScripted.py file in a file editor Find this line near the top of the script: USER_ENDPOINT = 'https://graph.microsoft.com/v1.0/users/' Below that, add the following lines: proxies = { "http" : "http://IPADDRESS:PORT", "https" : "http://IPADDRESS:PORT" } Search the py for any reference to "requests" and add "proxies=proxies" as an argument. Example: access_token_response = requests.post(token_url, data=payload, verify=False, allow_redirects=False, headers=headers, proxies=proxies) Want a Debug Log for the Azure Script's Execution? For those wanting to create a log file so you can see what the script is doing and where it's failing: Between the "USER_ENDPOINT =" line and "def getUserInfo(args):" line add: logging.basicConfig(filename='azureScripted.log', level=logging.DEBUG) Then throughout the script you can add logging.LEVEL() to see what is happening at any given point and isolate where the script stops (errors out). Example: logging.info('Header: %s', access_token_response.headers) Example: logging.debug('Token: %s', tokens['access_token']) Implementing the Azure SAML AQR Workaround... For Client Secret users, edit the azureScripted.py file... make the following changes (haven't tested they're mandatory, just know they work) within the "def getUserInfo(args):" definition... Above the line: token_url = "https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token" Add: azure_tenant = args['tenantId'] Add: client_id = args['clientId'] Add: client_secret = args['clientSecret'] Replace the token_url line with: token_url = "https://login.microsoftonline.com/{}/oauth2/v2.0/token".format(azure_tenant) Comment out or remove the following lines: client_id = '${AZURE_SPLUNK_SSO_APP_ID}' client_secret = '{AZURE_SSO_APP_API_KEY}' Then in the Enterprise Security Web UI, go to Settings > Authentication Methods and click the SAML Settings link. Click the SAML Configuration button in the top right. Scroll down until you see the "Authentication Extensions" section header, and click the arrow to expand the section. Script Path: azureScripted.py Script Timeout: if left blank it will default to 10s, this seems to be sufficient Get User Info Time to Live: if left blank it will default to 3600s, this seems to be sufficient Script Functions: getUserInfo Script Secure Arguments: enter the key name below in the left column, value in the right column. For Client Secret, Key: clientId For Client Secret, Key: clientSecret For Client Secret, Key: tenantId For Azure API: azureKey Click Save Validating Open the Enterprise Security app and go to Incident Review Click the down arrow next to any Notable entry you'd like to test with, then click Run Adaptive Response Choose a Response and fill it out, then click Run If the script is working as intended you should get a message about the response being successful instead of complaining about the token. If that works, wait over 1 hour with your session still open/active. Verify it doesn't kick you out or at least not in a way that doesn't require something as easy as click 'Refresh.' This depends on other settings in your environment, but as long as you don't get anything weird like it trying to launch the 'None' app (doesn't exist) or throwing a HTTP 500 response... should be good to go. ... View more