With the recent version of Splunk Add-on for Windows, the use of the indexes.conf has been removed. Question i cant figure out the answer to is how the data then knows where to go from the UF to the Index... On my endpoint UF i have the app deployed and the inputs.conf in the \local folder. None of the stanza's have the "index=xxx" line in the config but somehow the data is makings its way to the index. If i want to send data to a custom index how can i? ... View more

Is the transforms.conf working correctly for the 'counter' extract in the Splunk_TA_Windows app? [metric_name_for_perfmon_metrics_store] REGEX = counter=\"?([^\"\r\n]*[^\"\s]) FORMAT = metric_name::$1 WRITE_META = true I am having issues getting the Performance Monitoring dashboard working and my investigations are leading me to think that the 'counter' field is not being populated and it is necessary for the drop-down in the dashboards to work... Testing the regexs via search of the others in the transforms are working fine: perfmon-index eventtype="perfmon_windows" object=* | regex object"=\"?([^\"\r\n][^\"\s])" | regex instance"=\"?([^\"\r\n][^\"\s])" | regex collection"=\"?([^\"\r\n]*[^\"\s])" | table object, instance, collection, counter but the counter extract doesnt work: perfmon-index eventtype="perfmon_windows" object=* | regex counter"=\"?([^\"\r\n]*[^\"\s])" | table counter ... View more

I have a single splunk server environment that i want to use to monitor windows host (which is also a Domain Controller) I have installed the Splunk App for Windows Infrastructure, the Splunk Supporting Add-on for Active Directory and the Splunk Add-on for Microsoft Windows on the SearchHead/Indexer/DeploymentMaster. I am then pushing out the Splunk Add-on for Microsoft Windows to the windows host with inputs.conf enabled in the \local folder via the deployment-apps folder. I run through the setup and configuration of the Splunk App for Windows Infrastructure, build the lookups and migrate to KV store. Data is being indexed and is searchable from the default search app but none of the dashboards in the Splunk App for Windows Infrastructure app are returning anything. No errors in the splunkd log on either of the SH or client. I have a few questions: Where are the lookups stored that get built? When i test a lookup e.g. | inputlookup windows_event_details it returns no results - Why arent these lookups populated? I do know from version 6 the index stanza has been removed, and therefore all data is going into the index = main Could this be the issue? How do i send data to the specific windows indexes? e.g. index=wineventlog ... View more

We have in index cluster: two node index cluster two sites (one index peer in each site) We are seeing an issue whereby rolling restarts of our index cluster is causing indexing issues on the forwarders as the to peers are restarting. One index peer restart is triggered and before its fully operational the second index peer restarts. I think this is because we only have a single index peer in each site and the rolling restart is managed at the site level. We currently have a the following configuration on the Master Node: mode = master multisite=true available_sites=site1,site2 site_replication_factor = origin:1,site1:1,site2:1,total:2 site_search_factor = origin:1,site1:1,site2:1,total:2 replication_factor = 1 search_factor = 1 My intention is to add an additional index peer to each site but maintaining the current replication factor and search factor. The reason for this would be to maintain the data replication factor without increasing the storage requirements substantially but allow us to performing rolling restarts more effectively Are my assumptions and this approach correct? Would i be able to rebalance the data between the old and new index peers? What have i missed form this idea? This is what my environment looks like currently and with the additional index peers. ... View more

Can you modify/create a Splunk action (e.g. send email) to produce an email alert in a more "business user" friendly way? In my example there is a service that will log an error and we have a created an alert that gets sent to a business team (non IT) to inform them that an action is required due to the error Currently we use a standard email alert, "send email", and generate the content of the email in a table format: In most cases this is fine, but we have a requirement to make the email alerts more readable and business friendly, include company logo and a footer. Is this possible in Splunk? Ideally we would want to construct the email alert using fields from the error and then make it look pretty 🙂 e.g ... View more

I am attempting to install the Splunk App for Microsoft Exchange on our search head cluster. I drop the binaries into the C:\Program Files\Splunk\etc\shcluster\apps\splunk_app_microsoft_exchange folder on our deployment server and then push the app to the search head cluster by running the apply shcluster-bundle The app looks to install on the search heads but when you try access the app from the app menu you get the following error on all search heads... If you refresh the page then you occasionally get this error: Any ideas? ... View more

Has anyone had success with running the Splunkd service on a HeavyForwarder using a gMSA (Group Managed Service account)? I am using the gMSA account on the HeavyForwarder already for other services and this is working, but when i try run the Splunkd service as a gMSA i get various issues. The gMSA is also in the local Administrator group. The service starts but there are permission issues... Mongo fails to start, KVStore issues etc... splunkd.log 06-29-2018 10:06:18.557 +0100 ERROR Logger - Failed opening "C:\Program Files\Splunk\var\log\introspection\disk_objects.log": Access is denied. 06-29-2018 10:06:18.557 +0100 ERROR Logger - Failed opening "C:\Program Files\Splunk\var\log\introspection\http_event_collector_metrics.log": Access is denied. 06-29-2018 10:06:18.557 +0100 ERROR Logger - Failed opening "C:\Program Files\Splunk\var\log\introspection\kvstore.log": Access is denied. 06-29-2018 10:06:18.557 +0100 ERROR Logger - Failed opening "C:\Program Files\Splunk\var\log\introspection\resource_usage.log": Access is denied. 06-29-2018 10:08:22.999 +0100 ERROR KVStoreConfigurationProvider - Could not get ping from mongod. 06-29-2018 10:08:22.999 +0100 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed. 06-29-2018 10:08:22.999 +0100 ERROR KVStoreBulletinBoardManager - Failed to start KV Store process. See mongod.log and splunkd.log for details. mongod.log 2018-06-29T09:08:12.347Z I CONTROL [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "C:\Program Files\Splunk\etc\auth\server.pem", PEMKeyPassword: "", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "E98BF268-F1CB-4CF1-945B" }, security: { javascriptEnabled: false, keyFile: "C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } } 2018-06-29T09:08:12.348Z I STORAGE [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\mongod.lock errno:5 Access is denied.. Is a mongod instance already running?, terminating What permissions are necessary for a gMSA? This page talks about service accounts but not specifically about gMSA's Any help would be appreciated. ... View more

Hi all, After successfully installing a POC of the App: Splunk App for Microsoft Exchange in my test environment i went ahead and installed and configured this in production. We run a search head cluster and index cluster, Splunk Version 6.5.2 I have run through the Guided Setup and all looks to be ok. However, there are several errors i am seeing throughout the various dashboards. Error 1: Exchange Service Analyzer - Error in 'map': Did not find value for required attribute 'time'. Error 2: Host Overview - returns data however you see an ! triangle and notified that there are missing lookup tables [IndexServer1] The lookup table 'hostInformation' does not exist. It is referenced by configuration 'source::...(service|process)...'. [IndexServer1] The lookup table 'hostInformation' does not exist. It is referenced by configuration 'source::...Perfmon...'. [IndexServer2] The lookup table 'hostInformation' does not exist. It is referenced by configuration 'source::...(service|process)...'. [IndexServer2] The lookup table 'hostInformation' does not exist. It is referenced by configuration 'source::...Perfmon...'. I have built the look ups again, and rebuilt the Data Models but none of this has helped. Where should the lookup table 'hostInformation' be located? I can only see the 17 default lookups in the app: Any ideas? ... View more

We are monitoring for specific Windows events on our Domain Controllers. Inputs.conf looks like this: [WinEventLog://Security] disabled = 0 index = winevents start_from = oldest current_only = 0 checkpointInterval = 5 whitelist = 4728,4729,4732,4733,4756,4757,4755 Whe I add a test user to a universal security group, I trigger an event in the security log: Log Name: Security Source: Microsoft-Windows-Security-Auditing Date: 8/11/2016 3:22:50 PM Event ID: 4756 Task Category: Security Group Management Level: Information Keywords: Audit Success User: N/A Computer: xxxxxxxxxxxx Description: A member was added to a security-enabled universal group. Subject: Security ID: LONDON\xxxxxxxxxxxx Account Name: xxxxxxx Account Domain: LONDON Logon ID: 0x86xxxx Member: Security ID: LONDON\splunk_test Account Name: CN=Splunk Test Log Generator UK,OU=xxx,OU=Test Area,OU=xxx,OU=OIAL,DC=xxx,DC=xxx,DC=xxx Group: Security ID: LONDON\xxxxxx Account Name: xxxxxxxxxxxx Account Domain: LONDON Additional Information: Privileges: - Event Xml: Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" /> EventID>**4756** etc etc But this event never makes it into the index. No errors on the splunkd log. What am i missing? ... View more