Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

Why is the Splunk Universal Forwarder sending data to wrong index and, isn't sending all records of a Catalina.out?

sebardgz
New Member
‎04-02-2018 03:20 PM

Hello everyone,

I have a lab in a Ubuntu VM. In this lab, I have the UF and the Splunk E. The forwarder monitors a folder that has a Catalina.out.bk file. The data arrives at Splunk E but it arrives at the Main index and it doesn't get all the records in (Only ~4.8k out of ~18k events)

Here is my inputs.conf from etc/apps/search/local/:

[monitor:///home/c137/Documents/fwrd]
disabled = false
index = idx-vru-test

Here is my input.conf from etc/system/local/:

[monitor:///home/c137/Documents/fwrd]
disabled = false
index = idx-vru-test

I know I have different directories in each inputs.conf, the reason behind it is for testing stuff.

This is my outputs.conf in etc/apps/search/local and etc/system/local:

[tcpout]
defaultGroup = idx-vru-test

[tcpout:idx-vru-test]
server = ubuntu:9997

[tcpout-server://ubuntu:9997]

Attached below a picture of my Splunk indexes:
Splunk E Indexes

And for your ease I added the logs below for debugging: the logs added are splunkd.log and metrics.log from var/logs/splunk:

#splunkd.log

04-02-2018 17:53:18.854 -0400 INFO  PipelineComponent - Performing early shutdown tasks
04-02-2018 17:53:18.854 -0400 INFO  loader - Shutdown HTTPDispatchThread
04-02-2018 17:53:18.854 -0400 INFO  ShutdownHandler - Shutting down splunkd
04-02-2018 17:53:18.854 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Begin"
04-02-2018 17:53:18.859 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_FileIntegrityChecker"
04-02-2018 17:53:18.859 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_JustBeforeKVStore"
04-02-2018 17:53:18.859 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_KVStore"
04-02-2018 17:53:18.859 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Thruput"
04-02-2018 17:53:18.859 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_TcpInput1"
04-02-2018 17:53:18.859 -0400 INFO  TcpInputProc - Running shutdown level 1. Closing listening ports.
04-02-2018 17:53:18.859 -0400 INFO  TcpInputProc - Shutting down listening ports
04-02-2018 17:53:18.859 -0400 INFO  TcpInputProc - Setting up input quiesce timeout for : 90.000 secs
04-02-2018 17:53:19.781 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_ExecSendInitialSigterm"
04-02-2018 17:53:19.781 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
04-02-2018 17:53:19.781 -0400 INFO  TcpOutputProc - begin to shut down auto load balanced connection strategy
04-02-2018 17:53:19.781 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_UdpInput"
04-02-2018 17:53:19.781 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_FifoInput"
04-02-2018 17:53:19.781 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_WinEventLogInput"
04-02-2018 17:53:19.781 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_HttpInput"
04-02-2018 17:53:19.782 -0400 INFO  TcpInputProc - Cleaning up TCP connections
04-02-2018 17:53:19.782 -0400 INFO  TcpInputProc - Shutting down existing connections.
04-02-2018 17:53:19.782 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_CacheManager"
04-02-2018 17:53:19.782 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"
04-02-2018 17:53:19.782 -0400 INFO  TcpInputProc - TCP connection cleanup complete
04-02-2018 17:53:19.782 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"
04-02-2018 17:53:19.782 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_HTTPOutput"
04-02-2018 17:53:19.782 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot"
04-02-2018 17:53:19.782 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_MainThread"
04-02-2018 17:53:19.782 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Exec"
04-02-2018 17:53:19.783 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Tailing"
04-02-2018 17:53:19.783 -0400 INFO  TailingProcessor - Received shutdown signal.
04-02-2018 17:53:19.783 -0400 INFO  TailingProcessor - Will reconfigure input.
04-02-2018 17:53:19.783 -0400 INFO  TailingProcessor -   Calling addFromAnywhere in TailWatcher=0x7f9e6d3fb8b0.
04-02-2018 17:53:19.783 -0400 INFO  TailingProcessor -   Shutting down with TailingShutdownActor=0x7f9e7302d940 and TailWatcher=0x7f9e6d3fb8b0.
04-02-2018 17:53:19.783 -0400 INFO  TailingProcessor -   Pausing TailReader module...
04-02-2018 17:53:19.783 -0400 INFO  TailReader - State transitioning from 0 to 1 (pseudoPause).
04-02-2018 17:53:19.783 -0400 INFO  TailReader - State transitioning from 0 to 1 (pseudoPause).
04-02-2018 17:53:19.783 -0400 INFO  TailingProcessor -   Removing TailWatcher from eventloop...
04-02-2018 17:53:19.807 -0400 INFO  TailingProcessor -     ...removed.
04-02-2018 17:53:19.807 -0400 INFO  TailingProcessor -   Eventloop terminated successfully.
04-02-2018 17:53:19.807 -0400 INFO  TailingProcessor - Signaling shutdown complete.
04-02-2018 17:53:19.807 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_PeerManager"
04-02-2018 17:53:19.807 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailManager"
04-02-2018 17:53:19.807 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailQueueServiceThread"
04-02-2018 17:53:19.807 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_FSChangeMonitor"
04-02-2018 17:53:19.808 -0400 INFO  TailReader - State transitioning from 1 to 2 (signalShutdown).
04-02-2018 17:53:19.808 -0400 INFO  TailReader - Shutting down batch-reader
04-02-2018 17:53:19.808 -0400 INFO  TailReader - State transitioning from 1 to 2 (signalShutdown).
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_FSChangeManagerProcessor"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_DeploymentClient"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_HttpClientPollingThread"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_AsyncQueuedMessageDispatcherThread"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_OfflineFlusher"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Slave"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_SlaveSearch"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Captain"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Select"
04-02-2018 17:53:20.752 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_IdataDO_Collector"
04-02-2018 17:53:20.753 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput2"
04-02-2018 17:53:20.753 -0400 INFO  PipeFlusher - Flushing pipelines...
04-02-2018 17:53:20.753 -0400 INFO  TcpOutputProc - Shutting down auto load balanced connection strategy
04-02-2018 17:53:20.753 -0400 INFO  PipeFlusher - Finished triggering pipeline flush.
04-02-2018 17:53:20.760 -0400 INFO  TcpOutputProc - Auto load balanced connection strategy shutdown finished
04-02-2018 17:53:20.760 -0400 INFO  TcpOutputProc - Received shutdown control key.
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_IndexerService"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Database1"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_LastIndexerLevel"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_AWSMetering"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_TcpInput2"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_SearchDispatch"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_LoadLDAPUsers"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_MetricsManager"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Pipeline"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Queue"
04-02-2018 17:53:20.760 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner"
04-02-2018 17:53:20.761 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_HttpClient"
04-02-2018 17:53:20.761 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_DmcProxyHttpClient"
04-02-2018 17:53:20.761 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Duo2FAHttpClient"
04-02-2018 17:53:20.761 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_ApplicationLicenseChecker"
04-02-2018 17:53:20.761 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_S3ConnectionPoolManager"
04-02-2018 17:53:20.761 -0400 INFO  ShutdownHandler - shutting down level "ShutdownLevel_TelemetryMetricBuffer"
04-02-2018 17:53:20.761 -0400 INFO  ShutdownHandler - Shutdown complete in 1906.8 milliseconds
04-02-2018 17:53:21.751 -0400 INFO  loader - All pipelines finished.
04-02-2018 17:53:24.226 -0400 INFO  ServerConfig - My GUID is 1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED
04-02-2018 17:53:24.226 -0400 INFO  ServerConfig - My server name is "ubuntu".
04-02-2018 17:53:24.226 -0400 INFO  ServerConfig - Found no site defined in server.conf
04-02-2018 17:53:24.227 -0400 INFO  ServerConfig - My hostname is "ubuntu".
04-02-2018 17:53:24.247 -0400 INFO  ServerConfig - SSL session cache path  enabled 0 session timeout on SSL server 300.000
04-02-2018 17:53:24.247 -0400 INFO  ServerConfig - Setting HTTP server compression state=on
04-02-2018 17:53:24.247 -0400 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
04-02-2018 17:53:24.261 -0400 WARN  main - The hard limit of 'processes/threads' is lower than the recommended value. The hard limit is: 7677. The recommended value is: 16000.
04-02-2018 17:53:24.261 -0400 INFO  loader - Regex JIT enabled
04-02-2018 17:53:24.261 -0400 INFO  loader - using CLOCK_MONOTONIC
04-02-2018 17:53:24.262 -0400 INFO  loader - Splunkd starting (build fa31da744b51).
04-02-2018 17:53:24.262 -0400 INFO  loader - System info: Linux, ubuntu, 4.13.0-37-generic, #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018, x86_64.
04-02-2018 17:53:24.262 -0400 INFO  loader - Detected 1 (virtual) CPUs, 1 CPU cores, and 1970MB RAM
04-02-2018 17:53:24.262 -0400 INFO  loader - Maximum number of threads (approximate): 985
04-02-2018 17:53:24.262 -0400 INFO  loader - Arguments are: "-p" "8087" "restart"
04-02-2018 17:53:24.262 -0400 INFO  loader - Getting configuration data from: /opt/splunkforwarder/etc/myinstall/splunkd.xml
04-02-2018 17:53:24.263 -0400 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunkforwarder/etc/modules
04-02-2018 17:53:24.263 -0400 INFO  loader - loading modules from /opt/splunkforwarder/etc/modules
04-02-2018 17:53:24.263 -0400 INFO  loader - Writing out composite configuration file: /opt/splunkforwarder/var/run/splunk/composite.xml
04-02-2018 17:53:24.289 -0400 INFO  ServerRoles - Declared role=universal_forwarder.
04-02-2018 17:53:24.291 -0400 INFO  BundlesSetup - Setup stats for /opt/splunkforwarder/etc: wallclock_elapsed_msec=9, cpu_time_used=0.00538, shared_services_generation=1, shared_services_population=1
04-02-2018 17:53:24.323 -0400 INFO  LicenseMgr - Initing LicenseMgr
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - serverName=ubuntu guid=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - connection_timeout=30
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - send_timeout=30
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - receive_timeout=30
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - squash_threshold=2000
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - strict_pool_quota=1
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
04-02-2018 17:53:24.323 -0400 INFO  LMConfig - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
04-02-2018 17:53:24.323 -0400 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
04-02-2018 17:53:24.323 -0400 INFO  LMStackMgr - closing stack mgr
04-02-2018 17:53:24.326 -0400 INFO  LMSlaveInfo - all slaves cleared
04-02-2018 17:53:24.327 -0400 INFO  LMStack - Added type=forwarder license, from file=splunkforwarder.lic, to stack=forwarder of group=Forwarder
04-02-2018 17:53:24.327 -0400 INFO  LMStackMgr - created stack='forwarder'
04-02-2018 17:53:24.327 -0400 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
04-02-2018 17:53:24.328 -0400 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
04-02-2018 17:53:24.328 -0400 INFO  ServerRoles - Declared role=license_master.
04-02-2018 17:53:24.328 -0400 INFO  LMStackMgr - Initialized hideQuotaWarning = "0"
04-02-2018 17:53:24.328 -0400 INFO  LMStackMgr - init completed [1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED,Forwarder,runContext_splunkd=true]
04-02-2018 17:53:24.328 -0400 INFO  LicenseMgr - StackMgr init complete...
04-02-2018 17:53:24.328 -0400 INFO  LMTracker - Setting default product type='enterprise'
04-02-2018 17:53:24.328 -0400 INFO  LMTracker - init'ing slaveId=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED label=ubuntu [30,30,self]
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - enabling implicit feature set
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - attempting to ping master=self from slave=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED
04-02-2018 17:53:24.330 -0400 INFO  LMSlaveInfo - new slave='1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED' created
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=AWSMarketplace state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=AllowDuplicateKeys state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=DisableQuotaEnforcement state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=HideQuotaWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=SubgroupId state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
04-02-2018 17:53:24.330 -0400 INFO  LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
04-02-2018 17:53:24.331 -0400 INFO  LMTracker - setting masterGuid='1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED'
04-02-2018 17:53:24.331 -0400 INFO  LMTracker - attempting to contact master=self from slave=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED success
04-02-2018 17:53:24.331 -0400 INFO  LicenseMgr - Tracker init complete...
04-02-2018 17:53:24.331 -0400 INFO  loader - Setting SSL configuration.
04-02-2018 17:53:24.331 -0400 INFO  loader - Server supporting SSL versions TLS1.2
04-02-2018 17:53:24.331 -0400 INFO  loader - Using cipher suite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
04-02-2018 17:53:24.331 -0400 INFO  loader - Using ECDH curves : prime256v1, secp384r1, secp521r1
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 50 parameters: start_from, use_old_eventlog_api, use_threads, thread_wait_time_msec, suppress_checkpoint, suppress_sourcename, suppress_keywords, suppress_type, suppress_task, suppress_opcode, current_only, batch_size, checkpointInterval, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, evt_resolve_ad_ds, evt_ad_cache_disabled, evt_ad_cache_exp, evt_ad_cache_exp_neg, evt_ad_cache_max_entries, evt_sid_cache_disabled, evt_sid_cache_exp, evt_sid_cache_exp_neg, evt_sid_cache_max_entries, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whitelist7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, key, suppress_text, renderXml
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 19 parameters: remoteAddress, process, user, addressFamily, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, sid_cache_disabled, sid_cache_exp, sid_cache_exp_neg, sid_cache_max_entries, disabled, index
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interval, disabled, index
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled, index, printSchema, baseline
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "perfmon://" with 12 parameters: object, counters, instances, interval, mode, samplingInterval, stats, disabled, index, showZeroValue, useEnglishOnly, formatString
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "powershell2://" with 2 parameters: script, schedule
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "powershell://" with 2 parameters: script, schedule
04-02-2018 17:53:24.520 -0400 INFO  SpecFiles - Found external scheme definition for stanza "splunktcptoken://" with 1 parameters: token
04-02-2018 17:53:24.528 -0400 WARN  UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts
04-02-2018 17:53:24.530 -0400 INFO  DS_DC_Common - Initializing the PubSub system.
04-02-2018 17:53:24.530 -0400 INFO  DS_DC_Common - Initializing core facilities of PubSub system.
04-02-2018 17:53:24.539 -0400 INFO  DC:DeploymentClient - target-broker clause is missing.
04-02-2018 17:53:24.539 -0400 WARN  DC:DeploymentClient - DeploymentClient explicitly disabled through config.
04-02-2018 17:53:24.539 -0400 INFO  DS_DC_Common - Deployment Client not initialized.
04-02-2018 17:53:24.539 -0400 INFO  DS_DC_Common - Deployment Server not available on a dedicated forwarder.
04-02-2018 17:53:24.539 -0400 INFO  ClusteringMgr - initing clustering with: ht=60.000 rf=3 sf=2 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 icps=25 sfrt=600.000 pe=1 im=0 is=0 mob=5 mor=5 mosr=5 pb=5 rep_port= pptr=10 fznb=10 Empty/Default cluster pass4symmkey=false allow Empty/Default cluster pass4symmkey=true
04-02-2018 17:53:24.539 -0400 INFO  ClusteringMgr - clustering disabled
04-02-2018 17:53:24.539 -0400 WARN  SHCConfig - Default pass4symkey is being used. Please change to a random one.
04-02-2018 17:53:24.539 -0400 INFO  SHClusterMgr - initing shpooling with: ht=60.000 rf=3 ct=60.000 st=60.000 rt=60.000 rct=5.000 rst=5.000 rrt=10.000 rmst=600.000 rmrt=600.000 pe=1 im=0 is=0 mor=5 pb=5 rep_port= pptr=10
04-02-2018 17:53:24.539 -0400 INFO  SHClusterMgr - shpooling disabled
04-02-2018 17:53:24.545 -0400 INFO  ulimit - Limit: virtual address space size: unlimited
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: data segment size: unlimited
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: resident memory size: unlimited
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
04-02-2018 17:53:24.546 -0400 WARN  ulimit - Core file generation disabled.
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: data file size: unlimited
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: open files: 64000 files [hard maximum: 1048576 files]
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: user processes: 7677 processes
04-02-2018 17:53:24.546 -0400 INFO  ulimit - Limit: cpu time: unlimited
04-02-2018 17:53:24.546 -0400 INFO  ApplicationLicense - app license disabled by conf setting.
04-02-2018 17:53:24.546 -0400 INFO  IndexerInit - running splunkd specific init
04-02-2018 17:53:24.549 -0400 INFO  IntrospectionGenerator:disk_objects - Enabled: disk_objects=false indexes=false volumes=false dispatch=false fishbucket=true partitions=false summaries=false
04-02-2018 17:53:24.549 -0400 INFO  IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600.000s
04-02-2018 17:53:24.549 -0400 INFO  loader - Initializing from configuration
04-02-2018 17:53:24.561 -0400 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
04-02-2018 17:53:24.563 -0400 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
04-02-2018 17:53:24.654 -0400 INFO  ChunkedLBProcessor - Initializing the chunked line breaking processor
04-02-2018 17:53:24.655 -0400 INFO  TcpOutputProc - Initializing with fwdtype=lwf
04-02-2018 17:53:24.666 -0400 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
04-02-2018 17:53:24.668 -0400 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
04-02-2018 17:53:24.668 -0400 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_introspection|_internal|_telemetry)
04-02-2018 17:53:24.668 -0400 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to ubuntu:9997
04-02-2018 17:53:24.669 -0400 INFO  TcpOutputProc - tcpout group idx-vru-test using Auto load balanced forwarding
04-02-2018 17:53:24.669 -0400 INFO  TcpOutputProc - Group idx-vru-test initialized with maxQueueSize=512000 in bytes.
04-02-2018 17:53:24.669 -0400 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
04-02-2018 17:53:24.669 -0400 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
04-02-2018 17:53:24.669 -0400 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
04-02-2018 17:53:24.712 -0400 INFO  PipelineComponent - Launching the pipelines for set 0.
04-02-2018 17:53:24.749 -0400 INFO  TailingProcessor - TailWatcher initializing...
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///home/c137/Documents/fwrd.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor:///home/splunk/documents/frwd/.
04-02-2018 17:53:24.750 -0400 INFO  TailReader - State transitioning from 1 to 0 (initOrResume).
04-02-2018 17:53:24.750 -0400 INFO  TailReader - State transitioning from 1 to 0 (initOrResume).
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Adding watch on path: /home/c137/Documents/fwrd.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Adding watch on path: /home/splunk/documents/frwd.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk.
04-02-2018 17:53:24.750 -0400 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
04-02-2018 17:53:24.753 -0400 INFO  loader - Limiting REST HTTP server to 21333 sockets
04-02-2018 17:53:24.753 -0400 INFO  loader - Limiting REST HTTP server to 328 threads
04-02-2018 17:53:24.753 -0400 WARN  X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>
04-02-2018 17:53:24.755 -0400 INFO  TailReader - Registering metrics callback for: tailreader0
04-02-2018 17:53:24.755 -0400 INFO  TailReader - Starting tailreader0 thread
04-02-2018 17:53:24.755 -0400 INFO  TailReader - Registering metrics callback for: batchreader0
04-02-2018 17:53:24.755 -0400 INFO  TailReader - Starting batchreader0 thread
04-02-2018 17:53:24.780 -0400 INFO  WatchedFile - Resetting fd to re-extract header.
04-02-2018 17:53:24.892 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
04-02-2018 17:53:24.904 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
04-02-2018 17:53:24.905 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
04-02-2018 17:53:24.910 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
04-02-2018 17:53:24.912 -0400 INFO  WatchedFile - Will begin reading at offset=200104 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
04-02-2018 17:53:24.913 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
04-02-2018 17:53:24.915 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
04-02-2018 17:53:24.916 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
04-02-2018 17:53:24.917 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
04-02-2018 17:53:24.919 -0400 INFO  WatchedFile - Will begin reading at offset=3553 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
04-02-2018 17:53:24.920 -0400 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
04-02-2018 17:53:24.921 -0400 INFO  WatchedFile - Will begin reading at offset=10937 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
04-02-2018 17:53:24.923 -0400 INFO  WatchedFile - Will begin reading at offset=1353 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
04-02-2018 17:53:24.928 -0400 INFO  TcpOutputProc - Connected to idx=127.0.1.1:9997, pset=0, reuse=0.
04-02-2018 17:53:24.948 -0400 INFO  WatchedFile - Will begin reading at offset=4019773 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
04-02-2018 17:53:54.547 -0400 INFO  ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views

And

#metrics.log

04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000017, tail=0.000026, udpin=0.000000
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=5, cumulative_hits=2682
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=1, files_queued=4, new_files_queued=0, fd_cache_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=55, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=1, abandoned_channels=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=per_host_thruput, series="127.0.0.1", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=per_index_thruput, series="_internal", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.4261380682230032, eps=0.16129421965711366, kb=13.2099609375, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=7191, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=497.59, _tcp_KBps=0.49, _tcp_avg_thruput=1.57, _tcp_Kprocessed=751, _tcp_eps=0.21, kb=14.09
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4261380819697251, instantaneous_eps=0.1612942248602789, average_kbps=0.44710060274983593, total_k_processed=4546, kb=13.2099609375, ev=5
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.4261380682230032, instantaneous_eps=0.16129421965711366, average_kbps=0.4469039020887053, total_k_processed=4544, kb=13.2099609375, ev=5, load_average=0.28
04-02-2018 17:52:50.842 -0400 INFO  Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000016, tail=0.000032, udpin=0.000000
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=6, cumulative_hits=251
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=6, cumulative_hits=251
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=6, cumulative_hits=253
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=6, cumulative_hits=253
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=6, cumulative_hits=253
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=6, cumulative_hits=251
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=1, files_queued=4, new_files_queued=0, fd_cache_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=map, name=pipelineinputchannel, current_size=54, inactive_channels=33, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=0, abandoned_channels=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=per_host_thruput, series="ubuntu", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=per_index_thruput, series="_internal", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.42630046405523603, eps=0.19355528016223547, kb=13.21484375, ev=6, avg_age=0, max_age=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=tcpout_idx-vru-test, max_size=512000, current_size=0, largest_size=7196, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=tcpout_connections, name=idx-vru-test:127.0.1.1:9997:0, sourcePort=8087, destIp=127.0.1.1, destPort=9997, _tcp_Bps=494.52, _tcp_KBps=0.48, _tcp_avg_thruput=0.87, _tcp_Kprocessed=519, _tcp_eps=0.21, kb=14.00
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4263004778073541, instantaneous_eps=0.19355528640617672, average_kbps=0.5274032732382895, total_k_processed=474, kb=13.21484375, ev=6
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.42630046405523603, instantaneous_eps=0.19355528016223547, average_kbps=0.5274026864157091, total_k_processed=474, kb=13.21484375, ev=6, load_average=0.28
04-02-2018 17:52:54.925 -0400 INFO  Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000019, tail=0.000046, udpin=0.000000
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=13, cumulative_hits=2695
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=3, files_queued=17, new_files_queued=0, fd_cache_size=3
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=56, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=1, abandoned_channels=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_host_thruput, series="127.0.0.1", kbps=0.6927282397198792, eps=0.41935417585952844, kb=21.474609375, ev=13, avg_age=0.15384615384615385, max_age=2
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_index_thruput, series="_internal", kbps=0.6927282397198792, eps=0.41935417585952844, kb=21.474609375, ev=13, avg_age=0.15384615384615385, max_age=2
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.00453628315232663, eps=0.06451602705531208, kb=0.140625, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.42685164384740165, eps=0.1612900676382802, kb=13.232421875, ev=5, avg_age=0.4, max_age=2
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.2595762026053572, eps=0.12903205411062416, kb=8.046875, ev=4, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.0017641101147936895, eps=0.06451602705531208, kb=0.0546875, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.00453628315232663, eps=0.06451602705531208, kb=0.140625, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=0.6864278464527588, eps=0.2903221217489043, kb=21.279296875, ev=9, avg_age=0.2222222222222222, max_age=2
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.0017641101147936895, eps=0.06451602705531208, kb=0.0546875, ev=2, avg_age=0, max_age=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=7196, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=5, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=481.70, _tcp_KBps=0.47, _tcp_avg_thruput=1.51, _tcp_Kprocessed=765, _tcp_eps=0.20, kb=14.11
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.4436421775932619, instantaneous_eps=0.354838137357843, average_kbps=0.44701626692097274, total_k_processed=4559, kb=13.7529296875, ev=11
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.6927282397198792, instantaneous_eps=0.38709616233187244, average_kbps=0.447604575234534, total_k_processed=4565, kb=21.474609375, ev=12, load_average=0.42
04-02-2018 17:53:21.842 -0400 INFO  Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:53:24.655 -0400 ERROR Metrics - Metric with name thruput:thruput already registered
04-02-2018 17:53:24.655 -0400 ERROR Metrics - Metric with name thruput:idxSummary already registered
04-02-2018 17:53:24.909 -0400 INFO  StatusMgr - Registering StatusListener StatusMgrLogger
04-02-2018 17:53:24.909 -0400 INFO  StatusMgr - destHost=ubuntu, destIp=127.0.1.1, destPort=9997, eventType=connect_try, publisher=tcpout, sourcePort=8087, statusee=TcpOutputProcessor
04-02-2018 17:53:24.928 -0400 INFO  StatusMgr - destHost=ubuntu, destIp=127.0.1.1, destPort=9997, eventType=connect_done, publisher=tcpout, sourcePort=8087, statusee=TcpOutputProcessor
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=0, largest_size=0, max_size=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=realtime_search_data, system total, drop_count=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=search_concurrency, system total, active_hist_searches=0, active_realtime_searches=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=search_health_metrics, name=compute_search_quota, compute_search_quota_max_ms=0, compute_search_quota_mean_ms=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:53:52.842 -0400 INFO  Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000018, tail=0.000039, udpin=0.000000
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=21, cumulative_hits=2716
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=aggregator, cpu_seconds=0, executes=3, cumulative_hits=41
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=linebreaker, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=metrics, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=readerin, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=regexreplacement, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=sendout, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=utf8, cpu_seconds=0, executes=3, cumulative_hits=39
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=3, files_queued=23, new_files_queued=0, fd_cache_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=map, name=pipelineinputchannel, current_size=76, inactive_channels=55, new_channels=0, removed_channels=0, reclaimed_channels=0, timedout_channels=4, abandoned_channels=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_host_thruput, series="127.0.0.1", kbps=1.0258594489110233, eps=0.6774169073969797, kb=31.8017578125, ev=21, avg_age=0.2857142857142857, max_age=3
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_index_thruput, series="_internal", kbps=1.0258594489110233, eps=0.6774169073969797, kb=31.8017578125, ev=21, avg_age=0.2857142857142857, max_age=3
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.004410266324199086, eps=0.129031791885139, kb=0.13671875, ev=4, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/conf.log", kbps=0.010143612545657899, eps=0.09677384391385424, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.26965628382245843, eps=0.16128973985642373, kb=8.359375, ev=5, avg_age=0.6, max_age=3
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd-utility.log", kbps=0.018617624268583287, eps=0.09677384391385424, kb=0.5771484375, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.7209210344949721, eps=0.09677384391385424, kb=22.3486328125, ev=3, avg_age=1, max_age=3
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.00211062745515242, eps=0.09677384391385424, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.004410266324199086, eps=0.129031791885139, kb=0.13671875, ev=4, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=1.0091949425860138, eps=0.35483742768413223, kb=31.28515625, ev=11, avg_age=0.5454545454545454, max_age=3
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd_conf", kbps=0.010143612545657899, eps=0.09677384391385424, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.00211062745515242, eps=0.09677384391385424, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=tcpout_default-autolb-group, max_size=512000, current_size=0, largest_size=24344, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=tcpout_connections, name=default-autolb-group:127.0.0.1:9997:0, sourcePort=8088, destIp=127.0.0.1, destPort=9997, _tcp_Bps=1498.03, _tcp_KBps=1.46, _tcp_avg_thruput=1.50, _tcp_Kprocessed=809, _tcp_eps=1.00, kb=43.89
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=1.2749452784111537, instantaneous_eps=0.7419329708721955, average_kbps=0.4494740527955077, total_k_processed=4598, kb=39.5234375, ev=23
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=1.0258594489110233, instantaneous_eps=0.5483851155118407, average_kbps=0.44927854429059444, total_k_processed=4596, kb=31.8017578125, ev=17, load_average=0.5
04-02-2018 17:53:52.852 -0400 INFO  Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:53:54.547 -0400 INFO  Metrics - adding new metrics group: tcpout_connections
04-02-2018 17:53:54.549 -0400 INFO  Metrics - adding new metrics group: queue
04-02-2018 17:53:55.262 -0400 INFO  Metrics - group=conf, action=base_initialize, count=1, wallclock_ms_total=9, wallclock_ms_max=9, cpu_total=0.00538, cpu_max=0.00538
04-02-2018 17:53:55.262 -0400 INFO  Metrics - group=deploy-connections, nCurrent=0
04-02-2018 17:53:55.262 -0400 INFO  Metrics - group=executor, name=cachemgr_down, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=4294967295, largest_size=0, max_size=0
04-02-2018 17:53:55.262 -0400 INFO  Metrics - group=executor, name=cachemgr_up, jobs_added=0, jobs_finished=0, current_size=0, smallest_size=4294967295, largest_size=0, max_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=instance, name=instance, instance_roles="universal_forwarder, license_master", index_cluster_label=none, index_cluster_status=non-clustered, license_status=ENABLED, instance_guid=1C5EA7D1-C88A-4EB8-AC64-19C3FE0692ED, server_name=ubuntu
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=search_health_metrics, name=bundle_directory_reaper, bundle_dir_reaper_max_ms=0, bundle_dir_reaper_mean_ms=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=search_health_metrics, name=dispatch_directory_reaper, dispatch_dir_reaper_max_ms=0, dispatch_dir_reaper_mean_ms=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=tpool, name=bundlereplthreadpool, qsize=0, workers=0, qwork_units=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=dutycycle, name=dutycycle, mgmt_httpd=0.000000, reaper=0.000000, tail=0.000113, udpin=0.000001
04-02-2018 17:53:55.263 -0400 INFO  Metrics - adding new metrics group: per_host_thruput
04-02-2018 17:53:55.263 -0400 INFO  Metrics - adding new metrics group: per_index_thruput
04-02-2018 17:53:55.263 -0400 INFO  Metrics - adding new metrics group: per_source_thruput
04-02-2018 17:53:55.263 -0400 INFO  Metrics - adding new metrics group: per_sourcetype_thruput
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=1, cumulative_hits=1
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=1, cumulative_hits=1
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=chunkedlinebreaker, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=tcp-output-light-forwarder, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=thruput, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0, executes=25, cumulative_hits=25
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=aggregator, cpu_seconds=0, executes=6, cumulative_hits=6
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=linebreaker, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=metrics, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=readerin, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=regexreplacement, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=sendout, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=pipeline, name=structuredparsing, processor=utf8, cpu_seconds=0, executes=4, cumulative_hits=4
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=tailingprocessor, name=batchreader0, current_queue_size=0, max_queue_size=0, files_queued=0, new_files_queued=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=20, files_queued=37, new_files_queued=24, fd_cache_size=2
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=map, name=pipelineinputchannel, current_size=31, inactive_channels=10, new_channels=31, removed_channels=0, reclaimed_channels=0, timedout_channels=0, abandoned_channels=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_host_thruput, series="ubuntu", kbps=1.3457583647451972, eps=0.8167906810848301, kb=41.1904296875, ev=25, avg_age=0.16, max_age=3
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_index_thruput, series="_internal", kbps=1.3457583647451972, eps=0.7841190538414369, kb=41.1904296875, ev=24, avg_age=0.16666666666666666, max_age=3
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/audit.log", kbps=0.004466824037182664, eps=0.09801488173017961, kb=0.13671875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/conf.log", kbps=0.010273695285520128, eps=0.09801488173017961, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/metrics.log", kbps=0.31570874177087477, eps=0.16335813621696602, kb=9.6630859375, ev=5, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd-utility.log", kbps=0.018856378614106818, eps=0.09801488173017961, kb=0.5771484375, ev=3, avg_age=0.3333333333333333, max_age=1
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd.log", kbps=0.9943150306768611, eps=0.22870139070375242, kb=30.43359375, ev=7, avg_age=0.42857142857142855, max_age=3
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_source_thruput, series="/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log", kbps=0.0021376943606517037, eps=0.09801488173017961, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunk_audit", kbps=0.004466824037182664, eps=0.09801488173017961, kb=0.13671875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd", kbps=1.3288801510618427, eps=0.490074408650898, kb=40.673828125, ev=15, avg_age=0.26666666666666666, max_age=3
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd_conf", kbps=0.010273695285520128, eps=0.09801488173017961, kb=0.314453125, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=per_sourcetype_thruput, series="splunkd_stderr", kbps=0.0021376943606517037, eps=0.09801488173017961, kb=0.0654296875, ev=3, avg_age=0, max_age=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=tcpout_idx-vru-test, max_size=512000, current_size=527, largest_size=31828, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=aeq, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=aq, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=execprocessorinternalq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=httpinputq, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=indexqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=structuredparsingqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=tcpin_cooked_pqueue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=queue, name=tcpin_queue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=tcpout_connections, name=idx-vru-test:127.0.1.1:9997:0, sourcePort=8087, destIp=127.0.1.1, destPort=9997, _tcp_Bps=1513.93, _tcp_KBps=1.48, _tcp_avg_thruput=1.48, _tcp_Kprocessed=44, _tcp_eps=0.77, kb=44.35
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=1.3417106131409169, instantaneous_eps=0.7514498326047483, average_kbps=1.3395190799790904, total_k_processed=41, kb=41.06640625, ev=23
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=1.3457583647451972, instantaneous_eps=0.7841190538414369, average_kbps=1.3395190799790904, total_k_processed=41, kb=41.1904296875, ev=24, load_average=0.46
04-02-2018 17:53:55.263 -0400 INFO  Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0, instantaneous_eps=0, average_kbps=0, total_k_processed=0, kb=0, ev=0
04-02-2018 17:54:23.842 -0400 INFO  Metrics - group=deploy-connections, nCurrent=0

Any help is appreciated!

Preview file
1 KB
0 Karma
Reply

gcusello
SplunkTrust
SplunkTrust
‎04-02-2018 11:57 PM

Hi sebardgz,
at first you couldn't have in the same VM Splunk Enterprise and Universal Forwarder because there an IP conflict (UF cannot send logs to its same IP address).
In the second time you have two inputs.conf and outputs.conf and it isn't a best practice.
Probably you're taking events from Splunk Enterprise and not from UF!
So I suggest to install Splunk and UF in two VMs and then use a correct configuration.
Bye.
Giuseppe

0 Karma
Reply

chaker
Contributor
‎04-02-2018 05:05 PM

It COULD be your TCPout stanza and index name getting confused. Try creating a new test index, and update your inputs.conf to reflect it.

Otherwise, go over the conf files for typo's relating to the index, ensure you don't have configuration that has high precedence that is over riding.

Do you get any _internal events from that UF?

Also after re reading your post, do you have a sinlge VM wit both Splunk E and Splunk UF installed? If so, you only need Splunk E. You can remove the UF, don't run them both at the same time.

0 Karma
Reply

sebardgz
New Member
‎04-02-2018 07:22 PM

Hey, hey! Yes, I have Splunk E and Splunk UF in the same VM. I'm was asked to do it this way so I could learn how to use Splunk UF and break things in the VM instead of the real instance of Splunk E. The TCPout had the default names before that and I had the same problems. The _internal index increments by 15k events in each restart of the Splunk UF.

0 Karma
Reply

richgalloway
SplunkTrust
SplunkTrust
‎04-02-2018 04:40 PM

Are the UF and Indexer in the same VM or separate ones?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Reply

sebardgz
New Member
‎04-02-2018 07:16 PM

Same VM. I was asked to do it this way so I could learn my way around the UF, you know, break things in the VM before using the real servers.

0 Karma
Reply

richgalloway
SplunkTrust
SplunkTrust
‎04-03-2018 05:34 AM

Using VMs for testing is a good idea, but putting two Splunks on the same machine is too complicated for newbies. Use separate VMs for Spunk and the UF.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Reply
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...