500 Internal Server Error

[htomo12]

I can access the login page, but when I put Userid and Password and click login, the attachement file displays.

Splunk Server Environment: AWS Centos6 linux
Splunk Access Server : AWS WIndows Server 2012 R2 (Both Chrome and Internet Explorer)

[htomo12]

Thanks for your reply.

I tried it , I got below
https://[your_splunk]:8089/services/server/status//installed-file-integrity

[ashajambagi]

did you change the init.py?

[htomo12]

To ashajambag

Please check it.

01-22-2019 23:31:51.034 +0900 INFO DatabaseDirectoryManager - idx=introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'
01-22-2019 23:31:51.035 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-22-2019 23:31:51.077 +0900 INFO TelemetryHandler - Telemetry Data Collection has been enabled for app=splunk_instrumentation for categories=License Usage.
01-22-2019 23:32:14.908 +0900 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
01-22-2019 23:32:15.704 +0900 INFO KeyManagerLocalhost - Checking for localhost key pair
01-22-2019 23:32:15.704 +0900 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-22-2019 23:32:15.704 +0900 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-22-2019 23:32:15.705 +0900 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-22-2019 23:32:15.705 +0900 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-22-2019 23:32:15.705 +0900 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-22-2019 23:32:46.034 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
01-22-2019 23:32:46.034 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
01-22-2019 23:32:46.034 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
01-22-2019 23:32:46.035 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:44:03.439 +0900 FATAL ProcessRunner - Unexpected EOF from process runner child!
01-22-2019 23:44:03.439 +0900 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!
01-23-2019 15:34:06.942 +0900 INFO ServerConfig - My GUID is 5B2126E1-44B0-458D-8AAC-758821706624
01-23-2019 15:34:06.943 +0900 INFO ServerConfig - My server name is "ip-10-0-0-103.ap-northeast-1.compute.internal".
01-23-2019 15:34:06.943 +0900 INFO ServerConfig - Found no site defined in server.conf
01-23-2019 15:34:06.943 +0900 INFO ServerConfig - My hostname is "ip-10-0-0-103.ap-northeast-1.compute.internal".
01-23-2019 15:34:06.954 +0900 INFO ServerConfig - SSL session cache path enabled 0 session timeout on SSL server 300.000
01-23-2019 15:34:06.954 +0900 INFO ServerConfig - Setting HTTP server compression state=on
01-23-2019 15:34:06.954 +0900 INFO ServerConfig - Setting HTTP client compression state=0 (false)
01-23-2019 15:34:06.954 +0900 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 3779. The recommended value is: 16000.
01-23-2019 15:34:06.954 +0900 INFO loader - Regex JIT enabled
01-23-2019 15:34:06.954 +0900 INFO loader - using CLOCK_MONOTONIC
01-23-2019 15:34:06.958 +0900 INFO loader - Splunkd starting (build 06d57c595b80).
01-23-2019 15:34:06.958 +0900 INFO loader - System info: Linux, ip-10-0-0-103.ap-northeast-1.compute.internal, 3.10.0-957.1.3.el7.x86_64, #1 SMP Thu Nov 29 14:49:43 UTC 2018, x86_64.
01-23-2019 15:34:06.958 +0900 INFO loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 989MB RAM
01-23-2019 15:34:06.958 +0900 INFO loader - Maximum number of threads (approximate): 494
01-23-2019 15:34:06.958 +0900 INFO loader - Arguments are: "--under-systemd" "--systemd-delegate=yes" "-h" "10.0.0.103" "-p" "8089" "_internal_launch_under_systemd"
01-23-2019 15:34:06.958 +0900 INFO loader - Getting configuration data from: /opt/splunk/etc/myinstall/splunkd.xml
01-23-2019 15:34:06.959 +0900 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunk/etc/modules
01-23-2019 15:34:06.959 +0900 INFO loader - loading modules from /opt/splunk/etc/modules
01-23-2019 15:34:06.960 +0900 INFO loader - Writing out composite configuration file: /opt/splunk/var/run/splunk/composite.xml
01-23-2019 15:34:06.964 +0900 INFO BundlesSetup - Setup stats for /opt/splunk/etc: wallclock_elapsed_msec=14, cpu_time_used=0.009746, shared_services_generation=1, shared_services_population=1
01-23-2019 15:34:06.973 +0900 INFO LicenseMgr - Initing LicenseMgr
01-23-2019 15:34:06.973 +0900 INFO LMConfig - serverName=ip-10-0-0-103.ap-northeast-1.compute.internal guid=5B2126E1-44B0-458D-8AAC-758821706624
01-23-2019 15:34:06.973 +0900 INFO LMConfig - connection_timeout=30
01-23-2019 15:34:06.973 +0900 INFO LMConfig - send_timeout=30
01-23-2019 15:34:06.973 +0900 INFO LMConfig - receive_timeout=30
01-23-2019 15:34:06.973 +0900 INFO LMConfig - squash_threshold=2000
01-23-2019 15:34:06.973 +0900 INFO LMConfig - strict_pool_quota=1
01-23-2019 15:34:06.973 +0900 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-23-2019 15:34:06.973 +0900 INFO LMConfig - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
01-23-2019 15:34:06.973 +0900 INFO LMConfig - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
01-23-2019 15:34:06.973 +0900 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-23-2019 15:34:06.973 +0900 INFO LMStackMgr - closing stack mgr
01-23-2019 15:34:06.973 +0900 INFO LMSlaveInfo - all slaves cleared
01-23-2019 15:34:06.975 +0900 INFO LMStack - Added type=download-trial license, from file=enttrial.lic, to stack=download-trial of group=Trial
01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - created stack='download-trial'
01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - have to auto-set active stack group='Trial' reason='invalid/missing group id' gidStr='' oldGid=Invalid
01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - added pool auto_generated_pool_download-trial to stack download-trial
01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
01-23-2019 15:34:06.975 +0900 INFO ServerRoles - Declared role=license_master.
01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - Initialized hideQuotaWarning = "0"
01-23-2019 15:34:06.975 +0900 INFO LMStackMgr - init completed [5B2126E1-44B0-458D-8AAC-758821706624,Trial,runContext_splunkd=true]
01-23-2019 15:34:06.975 +0900 INFO LicenseMgr - StackMgr init complete...
01-23-2019 15:34:06.975 +0900 INFO LMTracker - Setting default product type='enterprise'
01-23-2019 15:34:06.975 +0900 INFO LMTracker - init'ing slaveId=5B2126E1-44B0-458D-8AAC-758821706624 label=ip-10-0-0-103.ap-northeast-1.compute.internal [30,30,self]
01-23-2019 15:34:06.976 +0900 INFO LMTracker - enabling implicit feature set
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.976 +0900 INFO LMTracker - attempting to ping master=self from slave=5B2126E1-44B0-458D-8AAC-758821706624
01-23-2019 15:34:06.976 +0900 INFO LMSlaveInfo - new slave='5B2126E1-44B0-458D-8AAC-758821706624' created
01-23-2019 15:34:06.976 +0900 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Tue Jan 22 23:43:53 2019) < lastRolloverTime(Wed Jan 23 00:00:00 2019), meaning that the master has already rolled over. Ignore slave persisted usage.
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=AWSMarketplace state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=AllowDuplicateKeys state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DataFabricSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DisableQuotaEnforcement state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=FederatedSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=HideQuotaWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SubgroupId state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
01-23-2019 15:34:06.977 +0900 INFO LMTracker - setting masterGuid='5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:06.977 +0900 INFO LMTracker - changing backwardCompatIsTrial=true
01-23-2019 15:34:06.977 +0900 INFO LMTracker - attempting to contact master=self from slave=5B2126E1-44B0-458D-8AAC-758821706624 success
01-23-2019 15:34:06.977 +0900 INFO LicenseMgr - Tracker init complete...
01-23-2019 15:34:06.980 +0900 INFO loader - Setting SSL configuration.
01-23-2019 15:34:06.980 +0900 INFO loader - Server supporting SSL versions TLS1.2
01-23-2019 15:34:06.980 +0900 INFO loader - Using cipher suite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
01-23-2019 15:34:06.980 +0900 INFO loader - Using ECDH curves : prime256v1, secp384r1, secp521r1
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 50 parameters: start_from, use_old_eventlog_api, use_threads, thread_wait_time_msec, suppress_checkpoint, suppress_sourcename, suppress_keywords, suppress_type, suppress_task, suppress_opcode, current_only, batch_size, checkpointInterval, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, evt_resolve_ad_ds, evt_ad_cache_disabled, evt_ad_cache_exp, evt_ad_cache_exp_neg, evt_ad_cache_max_entries, evt_sid_cache_disabled, evt_sid_cache_exp, evt_sid_cache_exp_neg, evt_sid_cache_max_entries, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whitelist7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, key, suppress_text, renderXml
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 19 parameters: remoteAddress, process, user, addressFamily, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, sid_cache_disabled, sid_cache_exp, sid_cache_exp_neg, sid_cache_max_entries, disabled, index
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interval, disabled, index
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled, index, printSchema, baseline
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode, samplingInterval, stats, disabled, showZeroValue, useEnglishOnly, formatString
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "powershell2://" with 2 parameters: script, schedule
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "powershell://" with 2 parameters: script, schedule
01-23-2019 15:34:07.133 +0900 INFO SpecFiles - Found external scheme definition for stanza "splunktcptoken://" with 1 parameters: token
01-23-2019 15:34:07.141 +0900 INFO DS_DC_Common - Initializing the PubSub system.
01-23-2019 15:34:07.141 +0900 INFO DS_DC_Common - Initializing core facilities of PubSub system.
01-23-2019 15:34:07.150 +0900 INFO DC:DeploymentClient - target-broker clause is missing.
01-23-2019 15:34:07.150 +0900 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-23-2019 15:34:07.150 +0900 INFO DS_DC_Common - Deployment Client not initialized.
01-23-2019 15:34:07.150 +0900 INFO DS_DC_Common - Loading and initializing Deployment Server...
01-23-2019 15:34:07.150 +0900 INFO DeploymentServer - Attempting to reload entire DS; reason='init'
01-23-2019 15:34:07.151 +0900 INFO DSManager - No serverclasses configured.
01-23-2019 15:34:07.155 +0900 INFO DSManager - Loaded count=0 configured SCs
01-23-2019 15:34:07.156 +0900 INFO ClientSessionsManager - Initializing ClientSessionsManager
01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=deploymentServer/phoneHome/default connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_ds_default listener=0x7f00762f3800
01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=deploymentServer/phoneHome/default connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_ds_default listener=0x7f00762f3800
01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=deploymentServer/phoneHome/default/metrics connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_ds_default listener=0x7f00762f3800
01-23-2019 15:34:07.156 +0900 INFO DeploymentServer - Creating connection to PubSub system.
01-23-2019 15:34:07.156 +0900 INFO PubSubSvr - Subscribed: channel=tenantService/handshake connectionId=connection_10.0.0.103_8089_ip-10-0-0-103.ap-northeast-1.compute.internal_direct_tenantService listener=0x7f00762f3e00
01-23-2019 15:34:07.156 +0900 INFO DS_DC_Common - Registered REST endpoint for 'broker'.
01-23-2019 15:34:07.156 +0900 INFO DS_DC_Common - Deployment Server|Client initialized successfully.
01-23-2019 15:34:07.156 +0900 INFO ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 is=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true rrt=restart dft=180 abt=600 sbs=1
01-23-2019 15:34:07.156 +0900 INFO ClusteringMgr - clustering disabled
01-23-2019 15:34:07.156 +0900 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one.
01-23-2019 15:34:07.156 +0900 INFO SHClusterMgr - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10
01-23-2019 15:34:07.156 +0900 INFO SHClusterMgr - shpooling disabled
01-23-2019 15:34:07.165 +0900 INFO WorkloadManager - Workload management for splunk node=ip-10-0-0-103.ap-northeast-1.compute.internal with guid=5B2126E1-44B0-458D-8AAC-758821706624 has been disabled.
01-23-2019 15:34:07.167 +0900 INFO ApplicationLicense - app license disabled by conf setting.
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: virtual address space size: unlimited
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: data segment size: unlimited
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: resident memory size: unlimited
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
01-23-2019 15:34:07.168 +0900 WARN ulimit - Core file generation disabled.
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: data file size: unlimited
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: open files: 65536 files
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: user processes: 3779 processes
01-23-2019 15:34:07.168 +0900 INFO ulimit - Limit: cpu time: unlimited
01-23-2019 15:34:07.168 +0900 INFO ulimit - Linux transparent hugepage support, enabled="always" defrag="always"
01-23-2019 15:34:07.168 +0900 WARN ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.
01-23-2019 15:34:07.168 +0900 INFO ulimit - Linux vm.overcommit setting, value="0"
01-23-2019 15:34:07.234 +0900 INFO CertStorageProvider - Updating status from unknown to starting
01-23-2019 15:34:07.234 +0900 INFO CertStorageProvider - Updating status from unknown to starting
01-23-2019 15:34:07.234 +0900 INFO Rsa2FA - Could not find [externalTwoFactorAuthSettings] in authentication stanza.
01-23-2019 15:34:07.238 +0900 INFO IndexerInit - running splunkd specific init
01-23-2019 15:34:07.264 +0900 WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
01-23-2019 15:34:07.268 +0900 INFO IndexerService - starting RecreateIndexesThread
01-23-2019 15:34:07.268 +0900 INFO IndexerService - indexes.conf - indexThreads param autotuned to=2
01-23-2019 15:34:07.268 +0900 INFO IndexerService - indexes.conf - memPoolMB param autotuned to=64MB
01-23-2019 15:34:07.268 +0900 INFO MPool - MPool initialized: bytes=67108864
01-23-2019 15:34:07.268 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_audit
01-23-2019 15:34:07.275 +0900 INFO CMBucketId - CMIndexId: New indexName=_audit inserted, mapping to id=1
01-23-2019 15:34:07.337 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.337 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db
01-23-2019 15:34:07.349 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_internal
01-23-2019 15:34:07.355 +0900 INFO CMBucketId - CMIndexId: New indexName=_internal inserted, mapping to id=2
01-23-2019 15:34:07.387 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.387 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
01-23-2019 15:34:07.398 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_introspection
01-23-2019 15:34:07.402 +0900 INFO CMBucketId - CMIndexId: New indexName=_introspection inserted, mapping to id=3
01-23-2019 15:34:07.421 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.421 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-23-2019 15:34:07.423 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_telemetry
01-23-2019 15:34:07.425 +0900 INFO DatabaseDirectoryManager - idx=_telemetry Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_telemetry/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.425 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_telemetry/db
01-23-2019 15:34:07.426 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=_thefishbucket
01-23-2019 15:34:07.427 +0900 INFO DatabaseDirectoryManager - idx=_thefishbucket Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/fishbucket/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.427 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/fishbucket/db
01-23-2019 15:34:07.427 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=history
01-23-2019 15:34:07.428 +0900 INFO DatabaseDirectoryManager - idx=history Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/historydb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.428 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/historydb/db
01-23-2019 15:34:07.429 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=main
01-23-2019 15:34:07.429 +0900 INFO DatabaseDirectoryManager - idx=main Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/defaultdb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.430 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/defaultdb/db
01-23-2019 15:34:07.430 +0900 INFO DatabaseDirectoryManager - Start-up refreshing bucket manifest index=summary
01-23-2019 15:34:07.431 +0900 INFO DatabaseDirectoryManager - idx=summary Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/summarydb/db', pendingBucketUpdates=0 . Reason='Refreshing manifest at start-up.'
01-23-2019 15:34:07.432 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/summarydb/db
01-23-2019 15:34:07.433 +0900 INFO HotDBManager - idx=_audit minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.433 +0900 INFO HotDBManager - idx=_audit Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.433 +0900 INFO HotBucketRoller - found hot bucket='/opt/splunk/var/lib/splunk/audit/db/hot_v1_3'
01-23-2019 15:34:07.434 +0900 INFO HotBucketRoller - Trying to force rolling hot db: path='/opt/splunk/var/lib/splunk/audit/db/hot_v1_3'
01-23-2019 15:34:07.434 +0900 INFO HotDBManager - idx=_audit Recovered hot: path='hot_v1_3', [id=3, et=1548167504, lt=1548168138]
01-23-2019 15:34:07.443 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:07.448 +0900 INFO HotDBManager - idx=_internal minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.448 +0900 INFO HotDBManager - idx=_internal Setting hot mgr params: maxHotSpanSecs=432000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1048576000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.448 +0900 INFO HotBucketRoller - found hot bucket='/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_4'
01-23-2019 15:34:07.449 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db
01-23-2019 15:34:07.465 +0900 INFO HotBucketRoller - Trying to force rolling hot db: path='/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_4'
01-23-2019 15:34:07.465 +0900 INFO HotDBManager - idx=_internal Recovered hot: path='hot_v1_4', [id=4, et=1548167492, lt=1548168224]
01-23-2019 15:34:07.465 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:07.467 +0900 INFO HotBucketRoller - Bucket='/opt/splunk/var/lib/splunk/audit/db/db_1548168138_1548167504_3', idx=_audit, newly --all corrupt: reason='Cannot get slices.dat count'
01-23-2019 15:34:07.469 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
01-23-2019 15:34:07.480 +0900 INFO OnlineFsck - Scheduled repair fsck; procId=0 idx=_audit bucketName=db_1548168138_1548167504_3 kind='entire bucket'
01-23-2019 15:34:07.480 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:07.480 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db
01-23-2019 15:34:07.480 +0900 INFO HotBucketRoller - finished moving hot to warm bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624 idx=_audit from=hot_v1_3 to=db_1548168138_1548167504_3 size=98304 caller=init_roll isinit=true selective=false
01-23-2019 15:34:07.480 +0900 INFO HotBucketRoller - Rolled hot DB at dir="/opt/splunk/var/lib/splunk/audit/db/hot_v1_3"
01-23-2019 15:34:07.480 +0900 INFO HotDBManager - closing hot mgr for idx=_audit
01-23-2019 15:34:07.502 +0900 INFO HotDBManager - idx=_introspection minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.502 +0900 INFO HotDBManager - idx=_introspection Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1073741824 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.502 +0900 INFO HotBucketRoller - found hot bucket='/opt/splunk/var/lib/splunk/_introspection/db/hot_v1_3'
01-23-2019 15:34:07.503 +0900 INFO HotBucketRoller - Bucket='/opt/splunk/var/lib/splunk/_internaldb/db/db_1548168224_1548167492_4', idx=_internal, newly --all corrupt: reason='count mismatch tsidx=2710 source-metadata=2702'
01-23-2019 15:34:07.503 +0900 INFO OnlineFsck - Scheduled repair fsck; procId=1 idx=_internal bucketName=db_1548168224_1548167492_4 kind='entire bucket'
01-23-2019 15:34:07.503 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:07.503 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
01-23-2019 15:34:07.503 +0900 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624 idx=_internal from=hot_v1_4 to=db_1548168224_1548167492_4 size=544768 caller=init_roll isinit=true selective=false
01-23-2019 15:34:07.503 +0900 INFO HotBucketRoller - Rolled hot DB at dir="/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_4"
01-23-2019 15:34:07.503 +0900 INFO HotDBManager - closing hot mgr for idx=_internal
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_telemetry minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_telemetry Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=268435456 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=_telemetry
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_thefishbucket minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=_thefishbucket Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=524288000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=_thefishbucket
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=history minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=history Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10485760 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=history
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=main minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=main Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=10 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10737418240 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=main
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=summary minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - idx=summary Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.504 +0900 INFO HotDBManager - closing hot mgr for idx=summary
01-23-2019 15:34:07.530 +0900 INFO HotBucketRoller - Trying to force rolling hot db: path='/opt/splunk/var/lib/splunk/_introspection/db/hot_v1_3'
01-23-2019 15:34:07.531 +0900 INFO HotDBManager - idx=_introspection Recovered hot: path='hot_v1_3', [id=3, et=1548167505, lt=1548168239]
01-23-2019 15:34:07.531 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:07.540 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-23-2019 15:34:07.606 +0900 INFO HotBucketRoller - Bucket='/opt/splunk/var/lib/splunk/_introspection/db/db_1548168239_1548167505_3', idx=_introspection, newly --all corrupt: reason='count mismatch tsidx=539 source-metadata=524'
01-23-2019 15:34:07.606 +0900 INFO OnlineFsck - Scheduled repair fsck; procId=2 idx=_introspection bucketName=db_1548168239_1548167505_3 kind='entire bucket'
01-23-2019 15:34:07.606 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Updating bucket, bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:07.606 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-23-2019 15:34:07.607 +0900 INFO HotBucketRoller - finished moving hot to warm bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624 idx=_introspection from=hot_v1_3 to=db_1548168239_1548167505_3 size=933888 caller=init_roll isinit=true selective=false
01-23-2019 15:34:07.607 +0900 INFO HotBucketRoller - Rolled hot DB at dir="/opt/splunk/var/lib/splunk/_introspection/db/hot_v1_3"
01-23-2019 15:34:07.607 +0900 INFO HotDBManager - closing hot mgr for idx=_introspection
01-23-2019 15:34:07.609 +0900 INFO IndexerService - Initializing indexes took usec=177155 reloading=false indexes_initialized=8 failed_to_init_indexes=0
01-23-2019 15:34:07.620 +0900 INFO IndexerService - adjusting tb licenses
01-23-2019 15:34:07.621 +0900 INFO IntrospectionGenerator:disk_objects - Enabled: disk_objects=true indexes=true volumes=true dispatch=true fishbucket=true partitions=true summaries=true
01-23-2019 15:34:07.621 +0900 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600.000s
01-23-2019 15:34:07.621 +0900 INFO IntrospectionGenerator:disk_objects - Summaries gathering starting; period=1800.000, highfreqency=false
01-23-2019 15:34:07.621 +0900 INFO loader - Initializing from configuration
01-23-2019 15:34:07.659 +0900 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
01-23-2019 15:34:07.660 +0900 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
01-23-2019 15:34:07.660 +0900 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_internal|_introspection|_telemetry)
01-23-2019 15:34:07.730 +0900 INFO IndexProcessor - Initializing: readonly=false reloading=false
01-23-2019 15:34:07.730 +0900 INFO IndexProcessor - not starting rt router thread
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_audit minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_audit Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_audit
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_internal minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_internal Setting hot mgr params: maxHotSpanSecs=432000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1048576000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_internal
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_introspection minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_introspection Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=1073741824 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_introspection
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_telemetry minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_telemetry Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=268435456 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_telemetry
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_thefishbucket minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=_thefishbucket Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=524288000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - closing hot mgr for idx=_thefishbucket
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=history minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.730 +0900 INFO HotDBManager - idx=history Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10485760 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.731 +0900 INFO HotDBManager - closing hot mgr for idx=history
01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=main minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=main Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=10 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=10737418240 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.731 +0900 INFO HotDBManager - closing hot mgr for idx=main
01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=summary minHotIdleSecsBeforeForceRoll=auto; initializing, current value=600
01-23-2019 15:34:07.731 +0900 INFO HotDBManager - idx=summary Setting hot mgr params: maxHotSpanSecs=7776000 maxHotBuckets=3 minHotIdleSecsBeforeForceRoll=0 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
01-23-2019 15:34:07.731 +0900 INFO HotDBManager - closing hot mgr for idx=summary
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_audit, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=188697600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=786432000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=4 idx=_audit
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_internal, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=2592000.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=1048576000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=432000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=5 idx=_internal
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_introspection, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=1209600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=1073741824,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=4 idx=_introspection
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_telemetry, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=63072000.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=268435456,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=_telemetry
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=_thefishbucket, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=2419200.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=524288000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=_thefishbucket
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=history, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=604800.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=10485760,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=history
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=main, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=188697600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=10737418240,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=20,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=86400.000,maxHotBuckets=10,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=main
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - idx=summary, Initializing, params='[300,period=60.000,frozenTimePeriodInSecs=188697600.000,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=786432000,optimizeEvery=5.000,syncMeta=true,maxTotalDataSizeMB=500000,maxGlobalDataSizeMB=0,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000.000,maxMetadataEntries=1000000,maxHotIdleSecs=0.000,maxHotBuckets=3,minHotIdleSecsBeforeForceRoll=0.000,quarantinePastSecs=77760000.000,quarantineFutureSecs=2592000.000,maxSliceSize=131072,serviceMetaPeriod=25.000,partialServiceMetaPeriod=0.000,throttleCheckPeriod=15.000,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551.615,maxBloomBackfillBucketAge_secs=2592000.000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,suspendHotRollByDeleteQuery0,tsidxReductionCheckPeriodInSec=600.000,timePeriodInSecBeforeTsidxReduction=604800.000,remoteVolume=,remotePath=,splitByIndexKeys=,dataType=event,serviceInactiveIndexesPeriod=60,tsidxWritingLevel=1,archiver.enableDataArchive=false,archiver.maxDataArchiveRetentionPeriod=0.000]' isSlave=false
01-23-2019 15:34:07.731 +0900 INFO IndexWriter - openDatabases complete currentId=-1 idx=summary
01-23-2019 15:34:07.731 +0900 INFO IndexProcessor - Initializing indexes took usec=697 reloading=false indexes_initialized=8
01-23-2019 15:34:07.732 +0900 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval: 0 ms
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval="0 * * * *" is a valid cron schedule
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - cron schedule: "0 * * * *"
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/on_splunk_start.py
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval: run once
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - interval="0 0 * * *" is a valid cron schedule
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_instrumentation/bin/schedule_delete.py
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - cron schedule: "0 0 * * *"
01-23-2019 15:34:07.735 +0900 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py
01-23-2019 15:34:07.736 +0900 INFO ExecProcessor - interval: run once
01-23-2019 15:34:07.828 +0900 INFO PipelineComponent - Pipeline structuredparsing disabled in default-mode.conf file
01-23-2019 15:34:07.936 +0900 INFO ProcessTracker - (child_0Fsck) Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/audit/db/db_1548168138_1548167504_3' took 350.5 milliseconds
01-23-2019 15:34:07.943 +0900 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-23-2019 15:34:07.983 +0900 INFO PipelineComponent - Launching the pipelines for set 0.
01-23-2019 15:34:08.016 +0900 INFO IndexWriter - Creating hot bucket=hot_v1_4, idx=_audit, event timestamp=1548225246, reason="suitable bucket not found, number of hot buckets=0, max=3"
01-23-2019 15:34:08.016 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_audit~4~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:08.017 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db
01-23-2019 15:34:08.017 +0900 INFO ServerRoles - Declared role=indexer.
01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - TailWatcher initializing...
01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-23-2019 15:34:08.056 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/introspection.
01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
01-23-2019 15:34:08.057 +0900 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-23-2019 15:34:08.057 +0900 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/introspection.
01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-23-2019 15:34:08.057 +0900 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-23-2019 15:34:08.057 +0900 INFO TailReader - Registering metrics callback for: tailreader0
01-23-2019 15:34:08.057 +0900 INFO TailReader - Starting tailreader0 thread
01-23-2019 15:34:08.059 +0900 INFO TailReader - Registering metrics callback for: batchreader0
01-23-2019 15:34:08.059 +0900 INFO TailReader - Starting batchreader0 thread
01-23-2019 15:34:08.061 +0900 INFO loader - Limiting REST HTTP server to 21845 sockets
01-23-2019 15:34:08.061 +0900 INFO loader - Limiting REST HTTP server to 164 threads
01-23-2019 15:34:08.061 +0900 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
01-23-2019 15:34:08.075 +0900 INFO UiHttpListener - Limiting UI HTTP server to 21845 sockets
01-23-2019 15:34:08.075 +0900 INFO UiHttpListener - Limiting UI HTTP server to 164 threads
01-23-2019 15:34:08.085 +0900 INFO WatchedFile - Will begin reading at offset=1791553 for file='/opt/splunk/var/log/introspection/disk_objects.log'.
01-23-2019 15:34:08.099 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/introspection/http_event_collector_metrics.log'.
01-23-2019 15:34:08.109 +0900 INFO ProxyConfig - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
01-23-2019 15:34:08.109 +0900 INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
01-23-2019 15:34:08.109 +0900 INFO ProxyConfig - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.
01-23-2019 15:34:08.167 +0900 INFO IndexWriter - Creating hot bucket=hot_v1_4, idx=_introspection, event timestamp=1548225247, reason="suitable bucket not found, number of hot buckets=0, max=3"
01-23-2019 15:34:08.168 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_introspection~4~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:08.168 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-23-2019 15:34:08.622 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
01-23-2019 15:34:08.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db
01-23-2019 15:34:08.810 +0900 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_audit~3~5B2126E1-44B0-458D-8AAC-758821706624
01-23-2019 15:34:09.624 +0900 INFO DatabaseDirectoryManager - idx=_audit Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'
01-23-2019 15:34:09.624 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db
01-23-2019 15:34:09.732 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/btool.log'.
01-23-2019 15:34:09.742 +0900 INFO WatchedFile - Will begin reading at offset=40535 for file='/opt/splunk/var/log/splunk/splunkd-utility.log'.
01-23-2019 15:34:09.744 +0900 INFO IndexWriter - Creating hot bucket=hot_v1_5, idx=_internal, event timestamp=1548225245, reason="suitable bucket not found, number of hot buckets=0, max=3"
01-23-2019 15:34:09.744 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_internal~5~5B2126E1-44B0-458D-8AAC-758821706624'
01-23-2019 15:34:09.744 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
01-23-2019 15:34:09.746 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/searchhistory.log'.
01-23-2019 15:34:09.777 +0900 INFO WatchedFile - Will begin reading at offset=1854289 for file='/opt/splunk/var/log/splunk/audit.log'.
01-23-2019 15:34:09.801 +0900 INFO WatchedFile - Will begin reading at offset=7104 for file='/opt/splunk/var/log/splunk/conf.log'.
01-23-2019 15:34:09.813 +0900 INFO WatchedFile - Will begin reading at offset=330016 for file='/opt/splunk/var/log/splunk/mongod.log'.
01-23-2019 15:34:09.817 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/license_usage.log'.
01-23-2019 15:34:09.819 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/license_usage_summary.log'.
01-23-2019 15:34:09.821 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/remote_searches.log'.
01-23-2019 15:34:09.871 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/splunkd_stdout.log'.
01-23-2019 15:34:09.890 +0900 INFO WatchedFile - Will begin reading at offset=571048 for file='/opt/splunk/var/log/splunk/web_service.log'.
01-23-2019 15:34:09.907 +0900 INFO ProcessTracker - (child_1Fsck) Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_internaldb/db/db_1548168224_1548167492_4' took 924.4 milliseconds
01-23-2019 15:34:09.917 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_access.log'.
01-23-2019 15:34:09.918 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_error.log'.
01-23-2019 15:34:09.920 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/django_service.log'.
01-23-2019 15:34:09.922 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/export_metrics.log'.
01-23-2019 15:34:10.237 +0900 INFO CertStorageProvider - Updating status from starting to ready
01-23-2019 15:34:10.237 +0900 INFO CertStorageProvider - Updating status from starting to ready
01-23-2019 15:34:10.237 +0900 INFO Rsa2FA - Could not find [externalTwoFactorAuthSettings] in authentication stanza.
01-23-2019 15:34:10.350 +0900 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (Resource Usage) starting; period=10s
01-23-2019 15:34:10.350 +0900 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (IO Statistics) starting; interval=60s
01-23-2019 15:34:10.714 +0900 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/python.log'.
01-23-2019 15:34:11.195 +0900 INFO ProcessTracker - (child_2Fsck) Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_introspection/db/db_1548168239_1548167505_3' took 256.0 milliseconds
01-23-2019 15:34:11.817 +0900 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_internal~4~5B2126E1-44B0-458D-8AAC-758821706624
01-23-2019 15:34:12.623 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'
01-23-2019 15:34:12.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
01-23-2019 15:34:12.812 +0900 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_introspection~3~5B2126E1-44B0-458D-8AAC-758821706624
01-23-2019 15:34:13.046 +0900 INFO TelemetryHandler - Telemetry Data Collection has been enabled for app=splunk_instrumentation for categories=License Usage.
01-23-2019 15:34:13.623 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'
01-23-2019 15:34:13.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-23-2019 15:34:37.261 +0900 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
01-23-2019 15:34:38.148 +0900 INFO KeyManagerLocalhost - Checking for localhost key pair
01-23-2019 15:34:38.151 +0900 INFO KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-23-2019 15:34:38.151 +0900 INFO KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-23-2019 15:34:38.152 +0900 INFO KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
01-23-2019 15:34:38.152 +0900 INFO KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-23-2019 15:34:38.152 +0900 INFO KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
01-23-2019 15:35:08.623 +0900 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
01-23-2019 15:35:08.623 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
01-23-2019 15:35:08.623 +0900 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
01-23-2019 15:35:08.624 +0900 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/db
01-23-2019 15:36:45.074 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"
01-23-2019 15:36:45.074 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/init_.py" did not pass hash-checking due to reason="content mismatch"

[ashajambagi]

Try editing,

 $Splunk_home/lib/python2.7/site-packages/splunk/rest/__init__.py 

file and change the value of the following line:

 SPLUNKD_CONNECTION_TIMEOUT = 30

Refer https://answers.splunk.com/answers/103751/500-internal-server-error.html for more detail.

Also,try running this : https://[your_splunk]:8089/services/server/status//installed-file-integrity

[htomo12]

I grepped "WARN" in splunkd.log (/opt/splunk/var/log/splunk)

01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:31:44.555 +0900 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 3779. The recommended value is: 16000.
01-22-2019 23:31:44.758 +0900 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-22-2019 23:31:44.764 +0900 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one.
01-22-2019 23:31:44.774 +0900 WARN ulimit - Core file generation disabled.
01-22-2019 23:31:44.775 +0900 WARN ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.
01-22-2019 23:31:44.887 +0900 WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
01-22-2019 23:31:45.617 +0900 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py" did not pass hash-checking due to reason="content mismatch"

[jaracan]

Maybe your http port 8000 is blocked by your firewall. You can try the command below:
sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent
sudo firewall-cmd --reload

[ashajambagi]

@htomo12

Can you check what error is written in splunkd.log (/opt/splunk/var/log/splunk)

[htomo12]

I grepped "WARN" in splunkd.log (/opt/splunk/var/log/splunk)

01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:24:32.928 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:31:44.555 +0900 WARN main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 3779. The recommended value is: 16000.
01-22-2019 23:31:44.758 +0900 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-22-2019 23:31:44.764 +0900 WARN SHCConfig - Default pass4symkey is being used. Please change to a random one.
01-22-2019 23:31:44.774 +0900 WARN ulimit - Core file generation disabled.
01-22-2019 23:31:44.775 +0900 WARN ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. Please fix by setting the values for transparent huge pages to "madvise" or preferably "never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.
01-22-2019 23:31:44.887 +0900 WARN IndexerService - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.
01-22-2019 23:31:45.617 +0900 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/etc/system/default/server.conf" did not pass hash-checking due to reason="content mismatch"
01-22-2019 23:34:23.095 +0900 WARN InstalledFilesHashChecker - An installed file="/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py" did not pass hash-checking due to reason="content mismatch"

[ashajambagi]

Don't grep,just try starting splunk again and paste the latest log entry.

[p_gurav]

Can you check any error in _internal logs? ($SPLUNK_HOME/var/log/splunk)

[dkeck]

Which version are you on maybe you have this bug

http://docs.splunk.com/Documentation/Splunk/6.4.2/ReleaseNotes/KnownIssues#Splunk_Web_and_Home_inter...

Or check if you disabled http input app sometimes this could happen then

[dkeck]

The bug seems the be for 6.4.2 so its not with your version.

Regarding the http input, if it is disabled it will have in $SPLUNK_HOME/splunk/etc/apps/splunk_httpinput/local/apps.conf a stanza

[install]
state = disabled

[htomo12]

Thanks for reply.

version: 7.2.3

Which Defect number are you referring?
http://docs.splunk.com/Documentation/Splunk/6.4.2/ReleaseNotes/KnownIssues#Splunk_Web_and_Home_inter...

How can I chek if you disabled http input app?

thanks,

[kamlesh_vaghela]

@htomo12

What if you click on here link on error page?
What configurations you did?

[htomo12]

if i click "here", the same page display.

[dkeck]

Hi

What is the URL you are trying to reach?

[htomo12]

the URL is below

http://10.0.0.103:8000