Since upgrading the search heads and indexers to v 6.4 (forwarders are still v6.3) the indexers are now logging in splunkd.log the following: 04-07-2016 11:11:15.221 +1000 ERROR AuditTrailManager - Host="host::ServerNAME" cannot open D:\Program Files\Splunk\var\lib\splunk\persistentstorage\audit\seqnum_host::ServerNAME.dat for write, error="The filename, directory name, or volume label syntax is incorrect.". 04-07-2016 11:11:15.221 +1000 ERROR AuditTrailManager - Failed to save seq_no=289 for host="host::ServerNAME" to disk! This log event is happening for EACH universal forwarder and multiple times. The indexers are Windows2012 and I'm pretty certain Windows ACL's aren't the issue as they have been checked. On each of the (two) indexers in D:\Program Files\Splunk\var\lib\splunk\persistentstorage\audit there is only one file: seqnum_localhost.dat What's going on? How can I fix it? Thanks Cam - Splunk 6.3 architect ... View more

Hello, Trying to import a CSV with dates going back 50+ years (https://www.quandl.com/api/v3/datasets/BCB/UDJIAD1.csv). I have correctly set the props date extraction and it works, except for events after 2010. What's happened is all the "old" dates are now bunched up at 6/7/2010 and haven't extracted properly. I know their is quarantine settings for indexes.conf but that should still display the data correctly it just puts it into different buckets on disk..... Is there a limitation of how 'old' data you can import? How can I fix it? [ dow_jones_index_csv ] CHARSET=UTF-8 INDEXED_EXTRACTIONS=csv KV_MODE=none NO_BINARY_CHECK=true SHOULD_LINEMERGE=false TIMESTAMP_FIELDS=Date TIME_FORMAT=%Y-%m-%d category=Structured description=Comma-separated value format. Set header and other settings in "Delimited Settings" disabled=false pulldown_type=true ... View more

Hello, I am attempting to configure SA-ldapsearch on our Splunk 6.3.1 cluster with search head cluster. I have installed SA-Ldapsearch on the deployer and pushed the bundle, no issue there. I am logging into a particular search head with the intention to configure the domain connections (and eventually copy the config back to the deployer) but attempting to configure the first domain, clicking "Test Connection" it errors with: Connection test for default failed Search | ldaptestconnection domain="default" Result distinguishedName: DC=acme,DC=com Error SSLError at "/opt/splunk/lib/python2.7/ssl.py", line 788 : [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:595)" Testing from our test environment to the same Domain Controller works. So I'm thinking something in our production environment is different. Debugging via ./splunk cmd btool server list --debug doesn't show anything too different outside of the clustering config etc. egrep -i 'security|ssl|cipher|cert|key' /tmp/debug.txt /opt/splunk/etc/system/default/server.conf caCertFile = $SPLUNK_HOME/etc/auth/appsCA.pem /opt/splunk/etc/system/default/server.conf cipherSuite = TLSv1+HIGH:@STRENGTH /opt/splunk/etc/system/default/server.conf sslCommonNameList = apps.splunk.com /opt/splunk/etc/system/local/server.conf pass4SymmKey = $1$<>= /opt/splunk/etc/system/local/server.conf pass4SymmKey = $1<> /opt/splunk/etc/system/default/server.conf encrypt_fields = "server:sslConfig:sslKeysfilePassword", "server:shclustering:pass4SymmKey", "outputs:tcpout:sslPassword", "inputs:SSL:password", "alert_actions:email:auth_password", "server:shclustering:password", "server:clustering:password", "server:clustering:pass4SymmKey", "server:general:pass4SymmKey", "app:credential:password", "passwords:credential:password", "server:deployment:pass4SymmKey", "authentication: :bindDNpassword", "server:kvstore:sslKeysPassword" /opt/splunk/etc/system/local/server.conf pass4SymmKey = $1$<.>= /opt/splunk/etc/system/local/server.conf [sslConfig] /opt/splunk/etc/system/default/server.conf allowSslCompression = true /opt/splunk/etc/system/default/server.conf allowSslRenegotiation = true /opt/splunk/etc/system/local/server.conf caCertFile = ca/RCA-mvmica002.cer /opt/splunk/etc/system/default/server.conf certCreateScript = $SPLUNK_HOME/bin/splunk, createssl, server-cert /opt/splunk/etc/system/default/server.conf cipherSuite = TLSv1+HIGH:@STRENGTH /opt/splunk/etc/system/default/server.conf enableSplunkdSSL = true /opt/splunk/etc/system/local/server.conf requireClientCert = false # The Splunk for Windows Infrastructure App breaks if this is enabled /opt/splunk/etc/system/local/server.conf sendStrictTransportSecurityHeader = true /opt/splunk/etc/system/local/server.conf sslKeysfile = certs/lvmsplunk011.acme.com.pem /opt/splunk/etc/system/local/server.conf sslKeysfilePassword = $1$Yg== /opt/splunk/etc/system/default/server.conf sslVersions = *,-ssl2 /opt/splunk/etc/system/local/server.conf supportSSLV3Only = true /opt/splunk/etc/system/default/server.conf useClientSSLCompression = true /opt/splunk/etc/system/default/server.conf useSplunkdClientSSLCompression = true Any suggestions on where to troubleshoot will be appreciated. ... View more

Hello, I am trying to deploy the Splunk universal forward to Win 2012 R2 servers. Using version : 6.2.0-237341-x64 But it fails instantly with the following message: Universal forwarder setup wizard ended prematurely because of an error. Your system has not been modified. To install this program at a later time, run Setup wizard again. Click the finish button to exit the setup Wizard. The MSI log contains the following: === Logging started: 21/11/2014 12:42:56 === Action 12:42:56: INSTALL. Action start 12:42:56: INSTALL. Action 12:42:56: SetAllUsers. Action start 12:42:56: SetAllUsers. SetAllUsers: Info: Registry setting for current user is not found. SetAllUsers: Info: ALLUSERS value for the existing installation: -1. SetAllUsers: Info: Set ALLUSERS property to 1. SetAllUsers: Info: Leave SetAllUsers: 0x0. Action ended 12:42:56: SetAllUsers. Return value 1. Action 12:42:56: FindRelatedProducts. Searching for related applications Action start 12:42:56: FindRelatedProducts. Action ended 12:42:56: FindRelatedProducts. Return value 1. Action 12:42:56: GetPreviousSettings. Action start 12:42:56: GetPreviousSettings. GetPreviousSettings: Error: DetermineContextForAllProducts failed witht: 0x65b. GetPreviousSettings: Error 0x80004005: Failed to GetInstalledSplunkSettings. GetPreviousSettings: Info: Leave GetPreviousSettings: 0x80004005. CustomAction GetPreviousSettings returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox) Action ended 12:42:57: GetPreviousSettings. Return value 3. Action 12:42:57: FatalError. Action start 12:42:57: FatalError. Error message is then displayed. Then the remain log is produced: Action ended 12:42:59: FatalError. Return value 2. m created a 'Tahoma' font, in 0 character set, of 13 pixels height. Info 2898.For WixUI_Font_Bigger textstyle, the system created a 'Tahoma' font, in 0 character set, of 19 pixels height. Action 12:42:57: FatalError. Dialog created Action ended 12:42:59: INSTALL. Return value 3. Property(C): UpgradeCode = {64B13631-6664-4F23-AFE2-98FCE86920BD} Property(C): SET_ADMIN_USER = 1 Property(C): WixUIRMOption = UseRM Property(C): _UICertFile = UICertFile Property(C): _UIRootCertFile = UIRootCertFile Property(C): _UIMonPath = UIMonPath Property(C): UIUseLocalSystem = 1 Property(C): UIUseBundledTA = 1 Property(C): _UIWinTAPath = UIWinTAPath Property(C): WIXUI_INSTALLDIR = INSTALLDIR Property(C): ALLUSERS = 1 Property(C): ARPNOMODIFY = yes Property(C): ProgramFiles64Folder = C:\Program Files\ Property(C): SourceDir = C:\Windows\ccmcache\4\ Property(C): Manufacturer = Splunk, Inc. Property(C): ProductCode = {0C37FA8A-044D-496A-80A2-39AA95A0B56C} Property(C): ProductLanguage = 1033 Property(C): ProductName = UniversalForwarder Property(C): ProductVersion = 6.2.0.237341 Property(C): ARPPRODUCTICON = WixSplunkIcon Property(C): DefaultUIFont = WixUI_Font_Normal Property(C): WixUI_Mode = InstallDir Property(C): ErrorDialog = ErrorDlg Property(C): SplunkSvcName = SplunkForwarder Property(C): UIShowTADialog = 0 Property(C): UIRecvIdxValid = 0 Property(C): DoNotInstallDrivers = 0 Property(C): SplunkX86Msi = 0 Property(C): UICustomize = 0 Property(C): AGREETOLICENSE = No Property(C): LAUNCHSPLUNK = 1 Property(C): os_OK = 1 Property(C): MSIRESTARTMANAGERCONTROL = Disable Property(C): MSIDISABLERMRESTART = 1 Property(C): MSIRMSHUTDOWN = 2 Property(C): LEGACYDRV = 1 Property(C): AdminProperties = AGREETOLICENSE;CERTFILE;CERTPASSWORD;CLONEPREP;DEPLOYMENT_SERVER;DoNotInstallDrivers;ENABLEADMON;FAILCA;FORCEINSTALLDRIVERS;KEEPSPLUNKHOME;LAUNCHSPLUNK;LEGACYDRV;LOGON_PASSWORD;LOGON_USERNAME;MONITOR_PATH;NEWERVERSIONDETECTED;os_OK;OtherSplunkProductsPresent;PERFMON;PREVPRODUCTCODE;RECEIVING_INDEXER;ROOTCACERTFILE;SET_ADMIN_USER;SPLUNKD_PORT;UIAdmon;UIApplicationLog;UICertFile;UICertPassword;UIConfirmCertPassword;UIConfirmDomainPassword;UIDeplSrv;UIDeplSrvPort;UIDomainAccount;UIDomainPassword;UIForwardedEventsLog;UIMonPath;UINoDeplSrvOrIndexer;UIPerfCpu;UIPerfDisk;UIPerfMemory;UIPerfNetstat;UIRecvIdx;UIRecvIdxPort;UIRootCertFile;UISecurityLog;UISetupLog;UISystemLog;UIWinTAPath;WINDOWS_TA_LOCATION;WINDOWS_TA_VERSION;WINEVENTLOG_APP_ENABLE;WINEVENTLOG_FWD_ENABLE;WINEVENTLOG_SEC_ENABLE;WINEVENTLOG_SET_ENABLE;WINEVENTLOG_SYS_ENABLE Property(C): SecureCustomProperties = ARPNOMODIFY;NEWERVERSIONDETECTED;PREVPRODUCTCODE Property(C): MsiHiddenProperties = LOGON_PASSWORD;SetupServiceConfig Property(C): MsiLogFileLocation = c:\temp\splunk.log Property(C): PackageCode = {E831C453-A05F-46B1-B77A-2FD0420F8735} Property(C): ProductState = 1 Property(C): CURRENTDIRECTORY = C:\Users\edur01\Desktop Property(C): CLIENTUILEVEL = 0 Property(C): CLIENTPROCESSID = 4808 Property(C): PRODUCTLANGUAGE = 1033 Property(C): VersionDatabase = 200 Property(C): VersionMsi = 5.00 Property(C): VersionNT = 603 Property(C): VersionNT64 = 603 Property(C): WindowsBuild = 9600 Property(C): ServicePackLevel = 0 Property(C): ServicePackLevelMinor = 0 Property(C): MsiNTProductType = 3 Property(C): WindowsFolder = C:\Windows\ Property(C): WindowsVolume = C:\ Property(C): System64Folder = C:\Windows\system32\ Property(C): SystemFolder = C:\Windows\SysWOW64\ Property(C): TerminalServer = 1 Property(C): TempFolder = C:\Users\edur01\AppData\Local\Temp\ Property(C): ProgramFilesFolder = C:\Program Files (x86)\ Property(C): CommonFilesFolder = C:\Program Files (x86)\Common Files\ Property(C): CommonFiles64Folder = C:\Program Files\Common Files\ Property(C): AppDataFolder = C:\Users\edur01\AppData\Roaming\ Property(C): FavoritesFolder = C:\Users\edur01\Favorites\ Property(C): NetHoodFolder = C:\Users\edur01\AppData\Roaming\Microsoft\Windows\Network Shortcuts\ Property(C): PersonalFolder = C:\Users\edur01\Documents\ Property(C): PrintHoodFolder = C:\Users\edur01\AppData\Roaming\Microsoft\Windows\Printer Shortcuts\ Property(C): RecentFolder = C:\Users\edur01\AppData\Roaming\Microsoft\Windows\Recent\ Property(C): SendToFolder = C:\Users\edur01\AppData\Roaming\Microsoft\Windows\SendTo\ Property(C): TemplateFolder = C:\ProgramData\Microsoft\Windows\Templates\ Property(C): CommonAppDataFolder = C:\ProgramData\ Property(C): LocalAppDataFolder = C:\Users\edur01\AppData\Local\ Property(C): MyPicturesFolder = C:\Users\edur01\Pictures\ Property(C): AdminToolsFolder = C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Administrative Tools\ Property(C): StartupFolder = C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Startup\ Property(C): ProgramMenuFolder = C:\ProgramData\Microsoft\Windows\Start Menu\Programs\ Property(C): StartMenuFolder = C:\ProgramData\Microsoft\Windows\Start Menu\ Property(C): DesktopFolder = C:\Users\Public\Desktop\ Property(C): FontsFolder = C:\Windows\Fonts\ Property(C): GPTSupport = 1 Property(C): OLEAdvtSupport = 1 Property(C): ShellAdvtSupport = 1 Property(C): MsiAMD64 = 6 Property(C): Msix64 = 6 Property(C): Intel = 6 Property(C): PhysicalMemory = 6141 Property(C): VirtualMemory = 1760 Property(C): AdminUser = 1 Property(C): MsiTrueAdminUser = 1 Property(C): LogonUser = edur01 Property(C): UserSID = S-1-5-21-4275840501-1099104590-2052170957-2787 Property(C): UserLanguageID = 3081 Property(C): ComputerName = QHORD0102 Property(C): SystemLanguageID = 3081 Property(C): ScreenX = 1152 Property(C): ScreenY = 864 Property(C): CaptionHeight = 23 Property(C): BorderTop = 1 Property(C): BorderSide = 1 Property(C): TextHeight = 16 Property(C): TextInternalLeading = 3 Property(C): ColorBits = 32 Property(C): TTCSupport = 1 Property(C): Time = 12:42:59 Property(C): Date = 21/11/2014 Property(C): MsiNetAssemblySupport = 4.0.30319.33440 Property(C): MsiWin32AssemblySupport = 6.3.9600.16384 Property(C): RedirectedDllSupport = 2 Property(C): MsiRunningElevated = 1 Property(C): Privileged = 1 Property(C): USERNAME = Windows User Property(C): DATABASE = C:\Windows\ccmcache\4\splunkforwarder.msi Property(C): OriginalDatabase = C:\Windows\ccmcache\4\splunkforwarder.msi Property(C): SOURCEDIR = C:\Windows\ccmcache\4\ Property(C): VersionHandler = 5.00 Property(C): UILevel = 5 Property(C): ACTION = INSTALL Property(C): EXECUTEACTION = INSTALL === Logging stopped: 21/11/2014 12:42:59 === Trying to install the 6.1.3 forwarder looks more promising (doesn't fail near instantly, but then errors out saying a newer version is already installed. (which is not true) Any help is appreciated. ... View more