Hi, Not sure when this occurred exactly however all of the indexes with an _ prefix are currently disabled on my indexer (non clustered distributed environment, 1 indexer + 1sh). I did reduce the size of the _internal index a while back which may be related, I have since changed this back and restarted to no avail. splunkd.log does not show any related warnings or errors on restart as far as i can see. see below for end of splunkd.log after restart. The indexes.conf does not specify any disabled params on any of the indexes, how can i re-enable these indexes? 07-18-2018 11:48:36.338 +0100 INFO ProcessTracker - (child_12__Fsck) Fsck - (bloomfilter only) Rebuild for bucket='/opt/splunk/var/lib/splunk/_internaldb/db/db_1531911675_1531910532_8926' took 42.81 milliseconds 07-18-2018 11:48:37.213 +0100 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).' 07-18-2018 11:48:37.214 +0100 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db 07-18-2018 11:48:38.176 +0100 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_internal~8926~620B4469-3CF8-4AF9-B52F-F77683DD529A 07-18-2018 11:48:38.205 +0100 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1' 07-18-2018 11:48:38.205 +0100 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db 07-18-2018 11:48:40.896 +0100 INFO IndexWriter - Creating hot bucket=hot_v1_8927, idx=_internal, event timestamp=1531910771, reason="suitable bucket not found, number of hot buckets=0, max=3" 07-18-2018 11:48:40.896 +0100 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_internal~8927~620B4469-3CF8-4AF9-B52F-F77683DD529A' 07-18-2018 11:48:40.897 +0100 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db ... View more

Hi, I've followed the instructions online and can confirm that the credentials exist in the local/passwords.conf file however I am seeing the following error: ExecProcessor - message from "python /opt/splunk/etc/apps/sophos_central/bin/sophos_events.py" Exception: No credentials have been found Anyone sen this before? ... View more

I've created a bar chart which spans the full width of my monitor when viewing on the dashboard however when I export to PDF it only utilizes about one-third of the page. Is there a way to set it to use more width as it is currently unreadable on the pdf? Cheers Sam ... View more

Hi, I am reviewing the results for the 'ESCU - DNS Query Requests Resolved by Unauthorized DNS Servers - Rule' correlation search in Enterprise Security and when I click on 'View all review activity for this Notable Event' I get no results. I see it pulls rule_id's from the incident_review_lookup however the rule id's it's looking for do not exist. Rule Id's for a different correlation search exist in the lookup but none for this search. Any idea's what populates the lookup with rule id's? or what might prevent it from doing so? Cheers Sam ... View more

Hi, Just setting up the Microsoft Office 365 Reporting Add-on for Splunk, im a global admin in o365 but cant authenticate against the below URL (have tried manually in the browser) 2018-02-20 14:35:09,114 ERROR pid=2418 tid=MainThread file=base_modinput.py:log_error:307 | HTTP Request error: 401 Client Error: Unauthorized for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$format=json&orderby=Received%20asc&$filter=StartDate%20eq%20datetime'2018-02-19T23:00:00Z'%20and%20EndDate%20eq%20datetime'2018-02-20T14:32:08.840 359Z' I guess this is a new restriction on the Microsoft side? is anyone else using this method successfully? There is an alternative (long winded) method detailed in the below doc which uses an Azure playbook and a Splunk HTTP Endpoint Collector: https://www.splunk.com/blog/2017/10/05/splunking-microsoft-cloud-data-part-3.html ... View more

Hi, I've created a custom app on my search head and want to map it to an index on my indexer which is a separate physical server in a distributed deployment(not clustered). I am editing the indexes.conf in the app and dutifully pointing the homepath to $SPLUNK_DB/appname/db, however, it's not on the local filesystem. Does Splunk handle $SPLUNK_DB and realize this actually resides on a remote server? I haven't set $SPLUNK_DB in splunk-launch.conf so I believe it will default to $SPLUNK_HOME/var/lib/splunk which is a local path. Also nothing set in sys env vars. env | grep SPLUNK SPLUNK_HOME=/opt/splunk ... View more