https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414642#M6431 <P>Hi </P> <P>What is the URL you are trying to reach?</P> Sat, 19 Jan 2019 14:56:30 GMT dkeck 2019-01-19T14:56:30Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414641#M6430 <P>I can access the login page, but when I put Userid and Password and click login, the attachement file displays.</P> <P>Splunk Server Environment: AWS Centos6 linux<BR /> Splunk Access Server : AWS WIndows Server 2012 R2 (Both Chrome and Internet Explorer)</P> <P><span class="lia-inline-image-display-wrapper" image-alt="alt text"><img src="https://community.splunk.com/t5/image/serverpage/image-id/6404iE7CDF8E4C5DF2285/image-size/large?v=v2&amp;px=999" role="button" title="alt text" alt="alt text" /></span></P> Sat, 19 Jan 2019 14:37:18 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414641#M6430 htomo12 2019-01-19T14:37:18Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414642#M6431 <P>Hi </P> <P>What is the URL you are trying to reach?</P> Sat, 19 Jan 2019 14:56:30 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414642#M6431 dkeck 2019-01-19T14:56:30Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414643#M6432 <P>the URL is below</P> <P><A href="http://10.0.0.103:8000">http://10.0.0.103:8000</A></P> Sun, 20 Jan 2019 04:20:10 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414643#M6432 htomo12 2019-01-20T04:20:10Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414644#M6433 <P>@htomo12</P> <P>What if you click on <CODE>here</CODE> link on error page?<BR /> What configurations you did?</P> Sun, 20 Jan 2019 05:42:28 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414644#M6433 kamlesh_vaghela 2019-01-20T05:42:28Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414645#M6434 <P>Which version are you on maybe you have this bug</P> <P><A href="http://docs.splunk.com/Documentation/Splunk/6.4.2/ReleaseNotes/KnownIssues#Splunk_Web_and_Home_interface_issues">http://docs.splunk.com/Documentation/Splunk/6.4.2/ReleaseNotes/KnownIssues#Splunk_Web_and_Home_interface_issues</A></P> <P>Or check if you disabled http input app sometimes this could happen then</P> Sun, 20 Jan 2019 08:25:53 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414645#M6434 dkeck 2019-01-20T08:25:53Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414646#M6435 <P>Thanks for reply.</P> <P>version: 7.2.3</P> <P>Which Defect number are you referring?<BR /> <A href="http://docs.splunk.com/Documentation/Splunk/6.4.2/ReleaseNotes/KnownIssues#Splunk_Web_and_Home_interface_issues">http://docs.splunk.com/Documentation/Splunk/6.4.2/ReleaseNotes/KnownIssues#Splunk_Web_and_Home_interface_issues</A></P> <P>How can I chek if you disabled http input app?</P> <P>thanks,</P> Mon, 21 Jan 2019 04:22:22 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414646#M6435 htomo12 2019-01-21T04:22:22Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414647#M6436 <P>if i click "here", the same page display.</P> Mon, 21 Jan 2019 04:22:59 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414647#M6436 htomo12 2019-01-21T04:22:59Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414648#M6437 <P>Can you check any error in _internal logs? <CODE>($SPLUNK_HOME/var/log/splunk)</CODE></P> Tue, 29 Sep 2020 22:51:08 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414648#M6437 p_gurav 2020-09-29T22:51:08Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414649#M6438 <P>The bug seems the be for 6.4.2 so its not with your version.</P> <P>Regarding the http input, if it is disabled it will have in $SPLUNK_HOME/splunk/etc/apps/splunk_httpinput/local/apps.conf a stanza </P> <PRE><CODE>[install] state = disabled </CODE></PRE> Tue, 29 Sep 2020 22:53:02 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414649#M6438 dkeck 2020-09-29T22:53:02Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414650#M6439 <P>@htomo12 </P> <P>Can you check what error is written in splunkd.log (/opt/splunk/var/log/splunk)</P> Mon, 21 Jan 2019 11:42:40 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414650#M6439 ashajambagi 2019-01-21T11:42:40Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414651#M6440 <P>Maybe your http port 8000 is blocked by your firewall. You can try the command below:<BR /> sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent<BR /> sudo firewall-cmd --reload</P> Tue, 22 Jan 2019 09:22:26 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414651#M6440 jaracan 2019-01-22T09:22:26Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414652#M6441 <P>I grepped "WARN" in splunkd.log (/opt/splunk/var/log/splunk)</P> <P>01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/<STRONG>init</STRONG>.py" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:31:44.555 +0900 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 3779. The recommended value is: 16000.<BR /> 01-22-2019 23:31:44.758 +0900 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.<BR /> 01-22-2019 23:31:44.764 +0900 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one.<BR /> 01-22-2019 23:31:44.774 +0900 WARN ulimit - Core file generation disabled.<BR /> 01-22-2019 23:31:44.775 +0900 WARN ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.<BR /> 01-22-2019 23:31:44.887 +0900 WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.<BR /> 01-22-2019 23:31:45.617 +0900 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <BR /> 01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/<STRONG>init</STRONG>.py" did not pass hash-checking due to reason="content mismatch"</P> Tue, 22 Jan 2019 14:37:49 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414652#M6441 htomo12 2019-01-22T14:37:49Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414653#M6442 <P>I grepped "WARN" in splunkd.log (/opt/splunk/var/log/splunk)</P> <P>01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/<STRONG>init</STRONG>.py" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:31:44.555 +0900 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 3779. The recommended value is: 16000.<BR /> 01-22-2019 23:31:44.758 +0900 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.<BR /> 01-22-2019 23:31:44.764 +0900 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one.<BR /> 01-22-2019 23:31:44.774 +0900 WARN ulimit - Core file generation disabled.<BR /> 01-22-2019 23:31:44.775 +0900 WARN ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.<BR /> 01-22-2019 23:31:44.887 +0900 WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.<BR /> 01-22-2019 23:31:45.617 +0900 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <BR /> 01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/<STRONG>init</STRONG>.py" did not pass hash-checking due to reason="content mismatch"</P> Tue, 22 Jan 2019 14:38:39 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414653#M6442 htomo12 2019-01-22T14:38:39Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414654#M6443 <P>Don't grep,just try starting splunk again and paste the latest log entry.</P> Wed, 23 Jan 2019 05:43:13 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414654#M6443 ashajambagi 2019-01-23T05:43:13Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414655#M6444 <P>To ashajambag</P> <P>Please check it.</P> <P>01-22-2019 23:31:51.034 +0900 INFO DatabaseDirectoryManager - idx=<EM>introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'<BR /> 01-22-2019 23:31:51.035 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-22-2019 23:31:51.077 +0900 INFO TelemetryHandler - Telemetry Data Collection has been enabled for app=splunk_instrumentation for categories=License Usage.<BR /> 01-22-2019 23:32:14.908 +0900 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views<BR /> 01-22-2019 23:32:15.704 +0900 INFO KeyManagerLocalhost - Checking for localhost key pair<BR /> 01-22-2019 23:32:15.704 +0900 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem<BR /> 01-22-2019 23:32:15.704 +0900 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem<BR /> 01-22-2019 23:32:15.705 +0900 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem<BR /> 01-22-2019 23:32:15.705 +0900 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem<BR /> 01-22-2019 23:32:15.705 +0900 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem<BR /> 01-22-2019 23:32:46.034 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'<BR /> 01-22-2019 23:32:46.034 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db<BR /> 01-22-2019 23:32:46.034 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'<BR /> 01-22-2019 23:32:46.035 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/</EM><EM>init</EM><EM>.py" did not pass hash-checking due to reason="content mismatch"<BR /> 01-22-2019 23:44:03.439 +0900 FATAL ProcessRunner - Unexpected EOF from process runner child!<BR /> 01-22-2019 23:44:03.439 +0900 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!<BR /> 01-23-2019 15:34:06.942 +0900 INFO ServerConfig - My GUID is 5B2126E1-44B0-458D-8AAC-758821706624<BR /> 01-23-2019 15:34:06.943 +0900 INFO ServerConfig - My server name is "ip-10-0-0-103.ap-northeast-1.compute.internal".<BR /> 01-23-2019 15:34:06.943 +0900 INFO ServerConfig - Found no site defined in server.conf<BR /> 01-23-2019 15:34:06.943 +0900 INFO ServerConfig - My hostname is "ip-10-0-0-103.ap-northeast-1.compute.internal".<BR /> 01-23-2019 15:34:06.954 +0900 INFO ServerConfig - SSL session cache path enabled 0 session timeout on SSL server 300.000<BR /> 01-23-2019 15:34:06.954 +0900 INFO ServerConfig - Setting HTTP server compression state=on<BR /> 01-23-2019 15:34:06.954 +0900 INFO ServerConfig - Setting HTTP client compression state=0 (false)<BR /> 01-23-2019 15:34:06.954 +0900 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 3779. The recommended value is: 16000.<BR /> 01-23-2019 15:34:06.954 +0900 INFO loader - Regex JIT enabled<BR /> 01-23-2019 15:34:06.954 +0900 INFO loader - using CLOCK_MONOTONIC<BR /> 01-23-2019 15:34:06.958 +0900 INFO loader - Splunkd starting (build 06d57c595b80).<BR /> 01-23-2019 15:34:06.958 +0900 INFO loader - System info: Linux, ip-10-0-0-103.ap-northeast-1.compute.internal, 3.10.0-957.1.3.el7.x86_64, #1 SMP Thu Nov 29 14:49:43 UTC 2018, x86_64.<BR /> 01-23-2019 15:34:06.958 +0900 INFO loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 989MB RAM<BR /> 01-23-2019 15:34:06.958 +0900 INFO loader - Maximum number of threads (approximate): 494<BR /> 01-23-2019 15:34:06.958 +0900 INFO loader - Arguments are: "--under-systemd" "--systemd-delegate=yes" "-h" "10.0.0.103" "-p" "8089" "_internal_launch_under_systemd"<BR /> 01-23-2019 15:34:06.958 +0900 INFO loader - Getting configuration data from: /opt/splunk/etc/myinstall/splunkd.xml<BR /> 01-23-2019 15:34:06.959 +0900 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunk/etc/modules<BR /> 01-23-2019 15:34:06.959 +0900 INFO loader - loading modules from /opt/splunk/etc/modules<BR /> 01-23-2019 15:34:06.960 +0900 INFO loader - Writing out composite configuration file: /opt/splunk/var/run/splunk/composite.xml<BR /> 01-23-2019 15:34:06.964 +0900 INFO BundlesSetup - Setup stats for /opt/splunk/etc: wallclock_elapsed_msec=14, cpu_time_used=0.009746, shared_services_generation=1, shared_services_population=1<BR /> 01-23-2019 15:34:06.973 +0900 INFO LicenseMgr - Initing LicenseMgr<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - serverName=ip-10-0-0-103.ap-northeast-1.compute.internal guid=5B2126E1-44B0-458D-8AAC-758821706624<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - connection_timeout=30<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - send_timeout=30<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - receive_timeout=30<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - squash_threshold=2000<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - strict_pool_quota=1<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMConfig - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0<BR /> 01-23-2019 15:34:06.973 +0900 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMStackMgr - closing stack mgr<BR /> 01-23-2019 15:34:06.973 +0900 INFO LMSlaveInfo - all slaves cleared<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStack - Added type=download-trial license, from file=enttrial.lic, to stack=download-trial of group=Trial<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - created stack='download-trial'<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - have to auto-set active stack group='Trial' reason='invalid/missing group id' gidStr='' oldGid=Invalid<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - added pool auto_generated_pool_download-trial to stack download-trial<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - added pool auto_generated_pool_free to stack free<BR /> 01-23-2019 15:34:06.975 +0900 INFO ServerRoles - Declared role=license_master.<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - Initialized hideQuotaWarning = "0"<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - init completed [5B2126E1-44B0-458D-8AAC-758821706624,Trial,runContext_splunkd=true]<BR /> 01-23-2019 15:34:06.975 +0900 INFO LicenseMgr - StackMgr init complete...<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMTracker - Setting default product type='enterprise'<BR /> 01-23-2019 15:34:06.975 +0900 INFO LMTracker - init'ing slaveId=5B2126E1-44B0-458D-8AAC-758821706624 label=ip-10-0-0-103.ap-northeast-1.compute.internal [30,30,self]<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - enabling implicit feature set<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMTracker - attempting to ping master=self from slave=5B2126E1-44B0-458D-8AAC-758821706624<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMSlaveInfo - new slave='5B2126E1-44B0-458D-8AAC-758821706624' created<BR /> 01-23-2019 15:34:06.976 +0900 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Tue Jan 22 23:43:53 2019) &lt; lastRolloverTime(Wed Jan 23 00:00:00 2019), meaning that the master has already rolled over. Ignore slave persisted usage.<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=AWSMarketplace state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=AllowDuplicateKeys state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DataFabricSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DisableQuotaEnforcement state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=FederatedSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=HideQuotaWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SubgroupId state=DISABLED_DUE_TO_LICENSE (featureStatus=2)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - setting masterGuid='5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - changing backwardCompatIsTrial=true<BR /> 01-23-2019 15:34:06.977 +0900 INFO LMTracker - attempting to contact master=self from slave=5B2126E1-44B0-458D-8AAC-758821706624 success<BR /> 01-23-2019 15:34:06.977 +0900 INFO LicenseMgr - Tracker init complete...<BR /> 01-23-2019 15:34:06.980 +0900 INFO loader - Setting SSL configuration.<BR /> 01-23-2019 15:34:06.980 +0900 INFO loader - Server supporting SSL versions TLS1.2<BR /> 01-23-2019 15:34:06.980 +0900 INFO loader - Using cipher suite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256<BR /> 01-23-2019 15:34:06.980 +0900 INFO loader - Using ECDH curves : prime256v1, secp384r1, secp521r1<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 50 parameters: start_from, use_old_eventlog_api, use_threads, thread_wait_time_msec, suppress_checkpoint, suppress_sourcename, suppress_keywords, suppress_type, suppress_task, suppress_opcode, current_only, batch_size, checkpointInterval, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, evt_resolve_ad_ds, evt_ad_cache_disabled, evt_ad_cache_exp, evt_ad_cache_exp_neg, evt_ad_cache_max_entries, evt_sid_cache_disabled, evt_sid_cache_exp, evt_sid_cache_exp_neg, evt_sid_cache_max_entries, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whitelist7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, key, suppress_text, renderXml<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 19 parameters: remoteAddress, process, user, addressFamily, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, sid_cache_disabled, sid_cache_exp, sid_cache_exp_neg, sid_cache_max_entries, disabled, index<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interval, disabled, index<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled, index, printSchema, baseline<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode, samplingInterval, stats, disabled, showZeroValue, useEnglishOnly, formatString<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "powershell2://" with 2 parameters: script, schedule<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "powershell://" with 2 parameters: script, schedule<BR /> 01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "splunktcptoken://" with 1 parameters: token<BR /> 01-23-2019 15:34:07.141 +0900 INFO DS_DC_Common - Initializing the PubSub system.<BR /> 01-23-2019 15:34:07.141 +0900 INFO DS_DC_Common - Initializing core facilities of PubSub system.<BR /> 01-23-2019 15:34:07.150 +0900 INFO DC:DeploymentClient - target-broker clause is missing.<BR /> 01-23-2019 15:34:07.150 +0900 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.<BR /> 01-23-2019 15:34:07.150 +0900 INFO DS_DC_Common - Deployment Client not initialized.<BR /> 01-23-2019 15:34:07.150 +0900 INFO DS_DC_Common - Loading and initializing Deployment Server...<BR /> 01-23-2019 15:34:07.150 +0900 INFO DeploymentServer - Attempting to reload entire DS; reason='init'<BR /> 01-23-2019 15:34:07.151 +0900 INFO DSManager - No serverclasses configured.<BR /> 01-23-2019 15:34:07.155 +0900 INFO DSManager - Loaded count=0 configured SCs<BR /> 01-23-2019 15:34:07.156 +0900 INFO ClientSessionsManager - Initializing ClientSessionsManager<BR /> 01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=deploymentServer/phoneHome/default connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_ds_default listener=0x7f00762f3800<BR /> 01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=deploymentServer/phoneHome/default connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_ds_default listener=0x7f00762f3800<BR /> 01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=deploymentServer/phoneHome/default/metrics connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_ds_default listener=0x7f00762f3800<BR /> 01-23-2019 15:34:07.156 +0900 INFO DeploymentServer - Creating connection to PubSub system.<BR /> 01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=tenantService/handshake connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_tenantService listener=0x7f00762f3e00<BR /> 01-23-2019 15:34:07.156 +0900 INFO DS_DC_Common - Registered REST endpoint for 'broker'.<BR /> 01-23-2019 15:34:07.156 +0900 INFO DS_DC_Common - Deployment Server|Client initialized successfully.<BR /> 01-23-2019 15:34:07.156 +0900 INFO ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 is=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1<BR /> 01-23-2019 15:34:07.156 +0900 INFO ClusteringMgr - clustering disabled<BR /> 01-23-2019 15:34:07.156 +0900 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one.<BR /> 01-23-2019 15:34:07.156 +0900 INFO SHClusterMgr - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10<BR /> 01-23-2019 15:34:07.156 +0900 INFO SHClusterMgr - shpooling disabled<BR /> 01-23-2019 15:34:07.165 +0900 INFO WorkloadManager - Workload management for splunk node=ip-10-0-0-103.ap-northeast-1.compute.internal with guid=5B2126E1-44B0-458D-8AAC-758821706624 has been disabled.<BR /> 01-23-2019 15:34:07.167 +0900 INFO ApplicationLicense - app license disabled by conf setting.<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: virtual address space size: unlimited<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: data segment size: unlimited<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: resident memory size: unlimited<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]<BR /> 01-23-2019 15:34:07.168 +0900 WARN ulimit - Core file generation disabled.<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: data file size: unlimited<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: open files: 65536 files<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: user processes: 3779 processes<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: cpu time: unlimited<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Linux transparent hugepage support, enabled="always" defrag="always"<BR /> 01-23-2019 15:34:07.168 +0900 WARN ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.<BR /> 01-23-2019 15:34:07.168 +0900 INFO ulimit - Linux vm.overcommit setting, value="0"<BR /> 01-23-2019 15:34:07.234 +0900 INFO CertStorageProvider - Updating status from unknown to starting<BR /> 01-23-2019 15:34:07.234 +0900 INFO CertStorageProvider - Updating status from unknown to starting<BR /> 01-23-2019 15:34:07.234 +0900 INFO Rsa2FA - Could not find [externalTwoFactorAuthSettings] in authentication stanza.<BR /> 01-23-2019 15:34:07.238 +0900 INFO IndexerInit - running splunkd specific init<BR /> 01-23-2019 15:34:07.264 +0900 WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.<BR /> 01-23-2019 15:34:07.268 +0900 INFO IndexerService - starting RecreateIndexesThread<BR /> 01-23-2019 15:34:07.268 +0900 INFO IndexerService - indexes.conf - indexThreads param autotuned to=2<BR /> 01-23-2019 15:34:07.268 +0900 INFO IndexerService - indexes.conf - memPoolMB param autotuned to=64MB<BR /> 01-23-2019 15:34:07.268 +0900 INFO MPool - MPool initialized: bytes=67108864<BR /> 01-23-2019 15:34:07.268 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_audit<BR /> 01-23-2019 15:34:07.275 +0900 INFO CMBucketId - CMIndexId: New indexName=_audit inserted, mapping to id=1<BR /> 01-23-2019 15:34:07.337 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.337 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db<BR /> 01-23-2019 15:34:07.349 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_internal<BR /> 01-23-2019 15:34:07.355 +0900 INFO CMBucketId - CMIndexId: New indexName=_internal inserted, mapping to id=2<BR /> 01-23-2019 15:34:07.387 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.387 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db<BR /> 01-23-2019 15:34:07.398 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_introspection<BR /> 01-23-2019 15:34:07.402 +0900 INFO CMBucketId - CMIndexId: New indexName=_introspection inserted, mapping to id=3<BR /> 01-23-2019 15:34:07.421 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.421 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-23-2019 15:34:07.423 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_telemetry<BR /> 01-23-2019 15:34:07.425 +0900 INFO DatabaseDirectoryManager - idx=_telemetry Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_telemetry/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.425 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_telemetry/db<BR /> 01-23-2019 15:34:07.426 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_thefishbucket<BR /> 01-23-2019 15:34:07.427 +0900 INFO DatabaseDirectoryManager - idx=_thefishbucket Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/fishbucket/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.427 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/fishbucket/db<BR /> 01-23-2019 15:34:07.427 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=history<BR /> 01-23-2019 15:34:07.428 +0900 INFO DatabaseDirectoryManager - idx=history Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/historydb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.428 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/historydb/db<BR /> 01-23-2019 15:34:07.429 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=main<BR /> 01-23-2019 15:34:07.429 +0900 INFO DatabaseDirectoryManager - idx=main Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/defaultdb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.430 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/defaultdb/db<BR /> 01-23-2019 15:34:07.430 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=summary<BR /> 01-23-2019 15:34:07.431 +0900 INFO DatabaseDirectoryManager - idx=summary Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/summarydb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'<BR /> 01-23-2019 15:34:07.432 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/summarydb/db<BR /> 01-23-2019 15:34:07.433 +0900 INFO HotDBManager - idx=_audit minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.433 +0900 INFO HotDBManager - idx=_audit Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.433 +0900 INFO HotBucketRoller - found hot bucket='/opt/splunk/var/lib/splunk/audit/db/hot_v1_3'<BR /> 01-23-2019 15:34:07.434 +0900 INFO HotBucketRoller - Trying to force rolling hot db: path='/opt/splunk/var/lib/splunk/audit/db/hot_v1_3'<BR /> 01-23-2019 15:34:07.434 +0900 INFO HotDBManager - idx=_audit Recovered hot: path='hot_v1_3', [id=3, et=1548167504, lt=1548168138]<BR /> 01-23-2019 15:34:07.443 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:07.448 +0900 INFO HotDBManager - idx=_internal minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.448 +0900 INFO HotDBManager - idx=_internal Setting hot mgr params: maxHotSpanSecs=432000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1048576000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.448 +0900 INFO HotBucketRoller - found hot bucket='/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_4'<BR /> 01-23-2019 15:34:07.449 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db<BR /> 01-23-2019 15:34:07.465 +0900 INFO HotBucketRoller - Trying to force rolling hot db: path='/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_4'<BR /> 01-23-2019 15:34:07.465 +0900 INFO HotDBManager - idx=_internal Recovered hot: path='hot_v1_4', [id=4, et=1548167492, lt=1548168224]<BR /> 01-23-2019 15:34:07.465 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:07.467 +0900 INFO HotBucketRoller - Bucket='/opt/splunk/var/lib/splunk/audit/db/db_1548168138_1548167504_3', idx=_audit, newly --all corrupt: reason='Cannot get slices.dat count'<BR /> 01-23-2019 15:34:07.469 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db<BR /> 01-23-2019 15:34:07.480 +0900 INFO OnlineFsck - Scheduled repair fsck; procId=0 idx=_audit bucketName=db_1548168138_1548167504_3 kind='entire bucket'<BR /> 01-23-2019 15:34:07.480 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:07.480 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db<BR /> 01-23-2019 15:34:07.480 +0900 INFO HotBucketRoller - finished moving hot to warm bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624 idx=_audit from=hot_v1_3 to=db_1548168138_1548167504_3 size=98304 caller=init_roll isinit=true selective=false<BR /> 01-23-2019 15:34:07.480 +0900 INFO HotBucketRoller - Rolled hot DB at dir="/opt/splunk/var/lib/splunk/audit/db/hot_v1_3"<BR /> 01-23-2019 15:34:07.480 +0900 INFO HotDBManager - closing hot mgr for idx=_audit<BR /> 01-23-2019 15:34:07.502 +0900 INFO HotDBManager - idx=_introspection minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.502 +0900 INFO HotDBManager - idx=_introspection Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1073741824 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.502 +0900 INFO HotBucketRoller - found hot bucket='/opt/splunk/var/lib/splunk/_introspection/db/hot_v1_3'<BR /> 01-23-2019 15:34:07.503 +0900 INFO HotBucketRoller - Bucket='/opt/splunk/var/lib/splunk/_internaldb/db/db_1548168224_1548167492_4', idx=_internal, newly --all corrupt: reason='count mismatch tsidx=2710 source-metadata=2702'<BR /> 01-23-2019 15:34:07.503 +0900 INFO OnlineFsck - Scheduled repair fsck; procId=1 idx=_internal bucketName=db_1548168224_1548167492_4 kind='entire bucket'<BR /> 01-23-2019 15:34:07.503 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:07.503 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db<BR /> 01-23-2019 15:34:07.503 +0900 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624 idx=_internal from=hot_v1_4 to=db_1548168224_1548167492_4 size=544768 caller=init_roll isinit=true selective=false<BR /> 01-23-2019 15:34:07.503 +0900 INFO HotBucketRoller - Rolled hot DB at dir="/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_4"<BR /> 01-23-2019 15:34:07.503 +0900 INFO HotDBManager - closing hot mgr for idx=_internal<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_telemetry minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_telemetry Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=268435456 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=_telemetry<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_thefishbucket minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_thefishbucket Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=524288000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=_thefishbucket<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=history minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=history Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10485760 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=history<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=main minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=main Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=10 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10737418240 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=main<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=summary minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=summary Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=summary<BR /> 01-23-2019 15:34:07.530 +0900 INFO HotBucketRoller - Trying to force rolling hot db: path='/opt/splunk/var/lib/splunk/_introspection/db/hot_v1_3'<BR /> 01-23-2019 15:34:07.531 +0900 INFO HotDBManager - idx=_introspection Recovered hot: path='hot_v1_3', [id=3, et=1548167505, lt=1548168239]<BR /> 01-23-2019 15:34:07.531 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:07.540 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-23-2019 15:34:07.606 +0900 INFO HotBucketRoller - Bucket='/opt/splunk/var/lib/splunk/_introspection/db/db_1548168239_1548167505_3', idx=_introspection, newly --all corrupt: reason='count mismatch tsidx=539 source-metadata=524'<BR /> 01-23-2019 15:34:07.606 +0900 INFO OnlineFsck - Scheduled repair fsck; procId=2 idx=_introspection bucketName=db_1548168239_1548167505_3 kind='entire bucket'<BR /> 01-23-2019 15:34:07.606 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:07.606 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-23-2019 15:34:07.607 +0900 INFO HotBucketRoller - finished moving hot to warm bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624 idx=_introspection from=hot_v1_3 to=db_1548168239_1548167505_3 size=933888 caller=init_roll isinit=true selective=false<BR /> 01-23-2019 15:34:07.607 +0900 INFO HotBucketRoller - Rolled hot DB at dir="/opt/splunk/var/lib/splunk/_introspection/db/hot_v1_3"<BR /> 01-23-2019 15:34:07.607 +0900 INFO HotDBManager - closing hot mgr for idx=_introspection<BR /> 01-23-2019 15:34:07.609 +0900 INFO IndexerService - Initializing indexes took usec=177155 reloading=false indexes_initialized=8 failed_to_init_indexes=0<BR /> 01-23-2019 15:34:07.620 +0900 INFO IndexerService - adjusting tb licenses<BR /> 01-23-2019 15:34:07.621 +0900 INFO IntrospectionGenerator:disk_objects - Enabled: disk_objects=true indexes=true volumes=true dispatch=true fishbucket=true partitions=true summaries=true<BR /> 01-23-2019 15:34:07.621 +0900 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600.000s<BR /> 01-23-2019 15:34:07.621 +0900 INFO IntrospectionGenerator:disk_objects - Summaries gathering starting; period=1800.000, highfreqency=false<BR /> 01-23-2019 15:34:07.621 +0900 INFO loader - Initializing from configuration<BR /> 01-23-2019 15:34:07.659 +0900 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*<BR /> 01-23-2019 15:34:07.660 +0900 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*<BR /> 01-23-2019 15:34:07.660 +0900 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_internal|_introspection|_telemetry)<BR /> 01-23-2019 15:34:07.730 +0900 INFO IndexProcessor - Initializing: readonly=false reloading=false<BR /> 01-23-2019 15:34:07.730 +0900 INFO IndexProcessor - not starting rt router thread<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_audit minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_audit Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_audit<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_internal minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_internal Setting hot mgr params: maxHotSpanSecs=432000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1048576000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_internal<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_introspection minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_introspection Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1073741824 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_introspection<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_telemetry minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_telemetry Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=268435456 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_telemetry<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_thefishbucket minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_thefishbucket Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=524288000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_thefishbucket<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=history minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=history Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10485760 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.731 +0900 INFO HotDBManager - closing hot mgr for idx=history<BR /> 01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=main minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=main Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=10 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10737418240 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.731 +0900 INFO HotDBManager - closing hot mgr for idx=main<BR /> 01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=summary minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600<BR /> 01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=summary Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000<BR /> 01-23-2019 15:34:07.731 +0900 INFO HotDBManager - closing hot mgr for idx=summary<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_audit, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=188697600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=786432000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=4 idx=_audit<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_internal, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=2592000.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=1048576000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=432000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=5 idx=_internal<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_introspection, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=1209600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=1073741824,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=4 idx=_introspection<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_telemetry, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=63072000.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=268435456,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=_telemetry<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_thefishbucket, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=2419200.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=524288000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=_thefishbucket<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=history, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=604800.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=10485760,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=history<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=main, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=188697600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=10737418240,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=20,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=86400.000,maxHotBuckets=10,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=main<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=summary, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=188697600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=786432000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=summary<BR /> 01-23-2019 15:34:07.731 +0900 INFO IndexProcessor - Initializing indexes took usec=697 reloading=false indexes_initialized=8<BR /> 01-23-2019 15:34:07.732 +0900 INFO TcpInputProc - Registering metrics callback for: tcpin_connections<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval: 0 ms<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval="0 * * * *" is a valid cron schedule<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - cron schedule: "0 * * * *"<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/on_splunk_start.py<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval: run once<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval="0 0 * * *" is a valid cron schedule<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/schedule_delete.py<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - cron schedule: "0 0 * * *"<BR /> 01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py<BR /> 01-23-2019 15:34:07.736 +0900 INFO ExecProcessor - interval: run once<BR /> 01-23-2019 15:34:07.828 +0900 INFO PipelineComponent - Pipeline structuredparsing disabled in default-mode.conf file<BR /> 01-23-2019 15:34:07.936 +0900 INFO ProcessTracker - (child_0</EM><EM>Fsck) Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/audit/db/db_1548168138_1548167504_3' took 350.5 milliseconds<BR /> 01-23-2019 15:34:07.943 +0900 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file<BR /> 01-23-2019 15:34:07.983 +0900 INFO PipelineComponent - Launching the pipelines for set 0.<BR /> 01-23-2019 15:34:08.016 +0900 INFO IndexWriter - Creating hot bucket=hot_v1_4, idx=_audit, event timestamp=1548225246, reason="suitable bucket not found, number of hot buckets=0, max=3"<BR /> 01-23-2019 15:34:08.016 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_audit~4~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:08.017 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db<BR /> 01-23-2019 15:34:08.017 +0900 INFO ServerRoles - Declared role=indexer.<BR /> 01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - TailWatcher initializing...<BR /> 01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.<BR /> 01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.<BR /> 01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.<BR /> 01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/introspection.<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailReader - State transitioning from 1 to 0 (initOrResume).<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailReader - State transitioning from 1 to 0 (initOrResume).<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/introspection.<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailReader - Registering metrics callback for: tailreader0<BR /> 01-23-2019 15:34:08.057 +0900 INFO TailReader - Starting tailreader0 thread<BR /> 01-23-2019 15:34:08.059 +0900 INFO TailReader - Registering metrics callback for: batchreader0<BR /> 01-23-2019 15:34:08.059 +0900 INFO TailReader - Starting batchreader0 thread<BR /> 01-23-2019 15:34:08.061 +0900 INFO loader - Limiting REST HTTP server to 21845 sockets<BR /> 01-23-2019 15:34:08.061 +0900 INFO loader - Limiting REST HTTP server to 164 threads<BR /> 01-23-2019 15:34:08.061 +0900 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <BR /> 01-23-2019 15:34:08.075 +0900 INFO UiHttpListener - Limiting UI HTTP server to 21845 sockets<BR /> 01-23-2019 15:34:08.075 +0900 INFO UiHttpListener - Limiting UI HTTP server to 164 threads<BR /> 01-23-2019 15:34:08.085 +0900 INFO WatchedFile - Will begin reading at offset=1791553 for file='/opt/splunk/var/log/introspection/disk_objects.log'.<BR /> 01-23-2019 15:34:08.099 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/introspection/http_event_collector_metrics.log'.<BR /> 01-23-2019 15:34:08.109 +0900 INFO ProxyConfig - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=<A href="http://host:port" target="_blank">http://host:port</A> in case HTTP proxying needs to be enabled.<BR /> 01-23-2019 15:34:08.109 +0900 INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=<A href="http://host:port" target="_blank">http://host:port</A> in case HTTP proxying needs to be enabled.<BR /> 01-23-2019 15:34:08.109 +0900 INFO ProxyConfig - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.<BR /> 01-23-2019 15:34:08.167 +0900 INFO IndexWriter - Creating hot bucket=hot_v1_4, idx=_introspection, event timestamp=1548225247, reason="suitable bucket not found, number of hot buckets=0, max=3"<BR /> 01-23-2019 15:34:08.168 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_introspection~4~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:08.168 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-23-2019 15:34:08.622 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'<BR /> 01-23-2019 15:34:08.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db<BR /> 01-23-2019 15:34:08.810 +0900 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624<BR /> 01-23-2019 15:34:09.624 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'<BR /> 01-23-2019 15:34:09.624 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db<BR /> 01-23-2019 15:34:09.732 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/btool.log'.<BR /> 01-23-2019 15:34:09.742 +0900 INFO WatchedFile - Will begin reading at offset=40535 for file='/opt/splunk/var/log/splunk/splunkd-utility.log'.<BR /> 01-23-2019 15:34:09.744 +0900 INFO IndexWriter - Creating hot bucket=hot_v1_5, idx=_internal, event timestamp=1548225245, reason="suitable bucket not found, number of hot buckets=0, max=3"<BR /> 01-23-2019 15:34:09.744 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_internal~5~5B2126E1-44B0-458D-8AAC-758821706624'<BR /> 01-23-2019 15:34:09.744 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db<BR /> 01-23-2019 15:34:09.746 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/searchhistory.log'.<BR /> 01-23-2019 15:34:09.777 +0900 INFO WatchedFile - Will begin reading at offset=1854289 for file='/opt/splunk/var/log/splunk/audit.log'.<BR /> 01-23-2019 15:34:09.801 +0900 INFO WatchedFile - Will begin reading at offset=7104 for file='/opt/splunk/var/log/splunk/conf.log'.<BR /> 01-23-2019 15:34:09.813 +0900 INFO WatchedFile - Will begin reading at offset=330016 for file='/opt/splunk/var/log/splunk/mongod.log'.<BR /> 01-23-2019 15:34:09.817 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/license_usage.log'.<BR /> 01-23-2019 15:34:09.819 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/license_usage_summary.log'.<BR /> 01-23-2019 15:34:09.821 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/remote_searches.log'.<BR /> 01-23-2019 15:34:09.871 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/splunkd_stdout.log'.<BR /> 01-23-2019 15:34:09.890 +0900 INFO WatchedFile - Will begin reading at offset=571048 for file='/opt/splunk/var/log/splunk/web_service.log'.<BR /> 01-23-2019 15:34:09.907 +0900 INFO ProcessTracker - (child_1</EM><EM>Fsck) Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_internaldb/db/db_1548168224_1548167492_4' took 924.4 milliseconds<BR /> 01-23-2019 15:34:09.917 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_access.log'.<BR /> 01-23-2019 15:34:09.918 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_error.log'.<BR /> 01-23-2019 15:34:09.920 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_service.log'.<BR /> 01-23-2019 15:34:09.922 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/export_metrics.log'.<BR /> 01-23-2019 15:34:10.237 +0900 INFO CertStorageProvider - Updating status from starting to ready<BR /> 01-23-2019 15:34:10.237 +0900 INFO CertStorageProvider - Updating status from starting to ready<BR /> 01-23-2019 15:34:10.237 +0900 INFO Rsa2FA - Could not find [externalTwoFactorAuthSettings] in authentication stanza.<BR /> 01-23-2019 15:34:10.350 +0900 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (Resource Usage) starting; period=10s<BR /> 01-23-2019 15:34:10.350 +0900 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (IO Statistics) starting; interval=60s<BR /> 01-23-2019 15:34:10.714 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/python.log'.<BR /> 01-23-2019 15:34:11.195 +0900 INFO ProcessTracker - (child_2</EM><EM>Fsck) Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_introspection/db/db_1548168239_1548167505_3' took 256.0 milliseconds<BR /> 01-23-2019 15:34:11.817 +0900 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624<BR /> 01-23-2019 15:34:12.623 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'<BR /> 01-23-2019 15:34:12.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db<BR /> 01-23-2019 15:34:12.812 +0900 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624<BR /> 01-23-2019 15:34:13.046 +0900 INFO TelemetryHandler - Telemetry Data Collection has been enabled for app=splunk_instrumentation for categories=License Usage.<BR /> 01-23-2019 15:34:13.623 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'<BR /> 01-23-2019 15:34:13.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-23-2019 15:34:37.261 +0900 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views<BR /> 01-23-2019 15:34:38.148 +0900 INFO KeyManagerLocalhost - Checking for localhost key pair<BR /> 01-23-2019 15:34:38.151 +0900 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem<BR /> 01-23-2019 15:34:38.151 +0900 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem<BR /> 01-23-2019 15:34:38.152 +0900 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem<BR /> 01-23-2019 15:34:38.152 +0900 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem<BR /> 01-23-2019 15:34:38.152 +0900 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem<BR /> 01-23-2019 15:35:08.623 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'<BR /> 01-23-2019 15:35:08.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db<BR /> 01-23-2019 15:35:08.623 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'<BR /> 01-23-2019 15:35:08.624 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db<BR /> 01-23-2019 15:36:45.074 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"<BR /> 01-23-2019 15:36:45.074 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/</EM><EM>init</EM>_.py" did not pass hash-checking due to reason="content mismatch"</P> Tue, 29 Sep 2020 22:54:46 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414655#M6444 htomo12 2020-09-29T22:54:46Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414656#M6445 <P>Try editing,</P> <PRE><CODE> $Splunk_home/lib/python2.7/site-packages/splunk/rest/__init__.py </CODE></PRE> <P>file and change the value of the following line:</P> <PRE><CODE> SPLUNKD_CONNECTION_TIMEOUT = 30 </CODE></PRE> <P>Refer <A href="https://answers.splunk.com/answers/103751/500-internal-server-error.html">https://answers.splunk.com/answers/103751/500-internal-server-error.html</A> for more detail.</P> <P>Also,try running this : https://[your_splunk]:8089/services/server/status//installed-file-integrity</P> Thu, 24 Jan 2019 10:28:21 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414656#M6445 ashajambagi 2019-01-24T10:28:21Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414657#M6446 <P>Thanks for your reply.</P> <P>I tried it , I got below<BR /> https://[your_splunk]:8089/services/server/status//installed-file-integrity</P> <P><span class="lia-inline-image-display-wrapper" image-alt="alt text"><img src="https://community.splunk.com/t5/image/serverpage/image-id/6403iFF442DF31FFD213B/image-size/large?v=v2&amp;px=999" role="button" title="alt text" alt="alt text" /></span></P> Thu, 24 Jan 2019 14:34:21 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414657#M6446 htomo12 2019-01-24T14:34:21Z https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414658#M6447 <P>did you change the <EM>init</EM>.py?</P> Fri, 25 Jan 2019 05:44:16 GMT https://community.splunk.com/t5/Monitoring-Splunk/500-Internal-Server-Error/m-p/414658#M6447 ashajambagi 2019-01-25T05:44:16Z