All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

The datamodel accelerated summary is indeed stored in a bucket but it can (but usually isn't) stored on a different piece of storage (in a different directory or on a different volume). But still is ... See more...
The datamodel accelerated summary is indeed stored in a bucket but it can (but usually isn't) stored on a different piece of storage (in a different directory or on a different volume). But still is organized in buckets and each raw data bucket has its corresponding DAS bucket. You're still thinking in terms of just time periods whereas data is stored and rolled by buckets. Buckets can have overlapping time ranges or even one can "contain" another's whole time range. Also, the time range for summaries works a bit differently. The backfill range doesn't produce duplicates. It updates the data (not sure about the gory technical details underneath - maybe it does internally keep some duplicates and just shows the most current generation but effectively it just shows the "current" state) - it's meant as a way to keep the DAS current even if some lagged events are ingested after the initial summary search run had already been done. So don't try to overoptimize too early
@gcusello  Appreciate your reply.... we have indexer clustering environment . However for both indexers and search head we are using only 4 CPU physical cores ..Do  you think that can cause this pro... See more...
@gcusello  Appreciate your reply.... we have indexer clustering environment . However for both indexers and search head we are using only 4 CPU physical cores ..Do  you think that can cause this problem?
Hi @shakti , there's a delay between the event timestamp and the indexing timestamp probably caused by the too high data volume. This could be caused by a queue issue on the Forwarder, by a network... See more...
Hi @shakti , there's a delay between the event timestamp and the indexing timestamp probably caused by the too high data volume. This could be caused by a queue issue on the Forwarder, by a network latency or by a resource provlem (usually storage performance) on your Indexers. You can check queues using a search like the following  index=_internal source=*metrics.log sourcetype=splunkd group=queue | eval name=case(name=="aggqueue","2 - Aggregation Queue", name=="indexqueue", "4 - Indexing Queue", name=="parsingqueue", "1 - Parsing Queue", name=="typingqueue", "3 - Typing Queue", name=="splunktcpin", "0 - TCP In Queue", name=="tcpin_cooked_pqueue", "0 - TCP In Queue") | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) | eval fill_perc=round((curr/max)*100,2) | bin _time span=1m | stats Median(fill_perc) AS "fill_percentage" perc90(fill_perc) AS "90_perc" max(max) AS max max(curr) AS curr by host, _time, name | where (fill_percentage>70 AND name!="4 - Indexing Queue") OR (fill_percentage>70 AND name="4 - Indexing Queue") | sort -_time About resources, did you checked the IOPS of your storage? have the correct number of CPUs? at least, does your network have sufficient bandwidth to support your data volume? Ciao. Giuseppe
Hi @marco_massari11 , good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated by all the contributors
Hi @kranthimutyala2 , good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated by all the contributors
@PickleRick    Thank you very much for your reply. So, what you are saying is that the data model summary index is tsidx, and this index is also stored in the bucket. Are you saying that the life... See more...
@PickleRick    Thank you very much for your reply. So, what you are saying is that the data model summary index is tsidx, and this index is also stored in the bucket. Are you saying that the life cycle in which this tsidx is created and deleted is the same as the bucket rolling rule that the existing index had? (hot/warm -> cold -> frozen ) However, there is something I am still confused about. Is the entire data being summarized every 5 minutes due to bucket summary even though there is overlapping data as much as the range as shown in the picture above?   And is it right to keep the entire thing for the retention period? Or is only the most recent data kept for the retention period?    
if I run the command on my HF is fine, it works. I configured the AD plugin to send events to the indexer. But if I run it on my SH doesn't work. I guess the send event function is not configured pro... See more...
if I run the command on my HF is fine, it works. I configured the AD plugin to send events to the indexer. But if I run it on my SH doesn't work. I guess the send event function is not configured properly, by that I mean this tutorial : The following steps are the same for saving new alerts or editing existing alerts. From the Add Actions menu, select Log event. Add the following event information to configure the alert action. Use plain text or tokens for search, job, or server metadata. Event text Source and sourcetype Host Destination index for the log event. The main index is the default destination. You can specify a different existing index.
1) Can I create KVStore lookup definition in Splunk UI without creating transform.conf file directly via command line? [Yes/No] Yes (Splunk will create a transforms.conf via the Splunk UI) 2) ... See more...
1) Can I create KVStore lookup definition in Splunk UI without creating transform.conf file directly via command line? [Yes/No] Yes (Splunk will create a transforms.conf via the Splunk UI) 2) Will creating KVStore lookup definition in Splunk UI automatically update transform.conf file? [Yes/No] Yes - (This sounds like, if you want update your kvstore definitions with perhaps new fields etc, so yes it will automatically update the transforms.conf)
Hi All, I have installed Linux monitoring extension to get the NFS utilization metrics as per this documentation https://developer.cisco.com/codeexchange/github/repo/Appdynamics/linux-monitoring-ex... See more...
Hi All, I have installed Linux monitoring extension to get the NFS utilization metrics as per this documentation https://developer.cisco.com/codeexchange/github/repo/Appdynamics/linux-monitoring-extension/#readme but post doing necessary changes as per documentation, i have restarted machine-agent. i am getting this error while startup. Please check and let me know the solution. [rinst@vm-64e6db337156cc18f93ef923 logs]$ cat machine-agentstartup.log  my-vm==> [main] 06 May 2024 07:40:06,143  INFO FlexibleX509TrustManager - Using default keystore for SSL certificate validation. my-vm==> [main] 06 May 2024 07:40:06,446  INFO HostIdProvider - Default Host Identifier Resolver using host name for unique host identifier [my-vm] my-vm==> [main] 06 May 2024 07:40:06,549  INFO MachineLicensePropertiesProvider - Detected Virtual CPU Count: 4 my-vm==> [main] 06 May 2024 07:40:06,549  INFO MachineLicensePropertiesProvider - Detected Logical CPU Count: 4 my-vm==> [main] 06 May 2024 07:40:06,549  INFO MachineLicensePropertiesProvider - Detected Physical CPU Count: 4 my-vm==> [system-thread-0] 06 May 2024 07:40:06,970  INFO SecondStageSystem - Starting main system with features Features(features=[dmm, sim], reason=Features.Reason(message=, code=)) my-vm==> [system-thread-0] 06 May 2024 07:40:07,206  INFO SystemAgent - #################################################################################### my-vm==> [system-thread-0] 06 May 2024 07:40:07,206  INFO SystemAgent - Agent Install Directory [/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23] my-vm==> [system-thread-0] 06 May 2024 07:40:07,206  INFO SystemAgent - Using Agent Version [Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] my-vm==> [system-thread-0] 06 May 2024 07:40:07,207  INFO SystemAgent - JVM Runtime:  java.home=/ngs/app/rinst/applejdk-11.0.16.8.1 java.vm.vendor=Apple Inc. java.vm.name=OpenJDK 64-Bit Server VM java.version=11.0.16 java.specification.version=11 java.runtime.version=11.0.16+8-20220720170112 java.io.tmpdir=/tmp user.language=en user.country=US user.variant= Default locale=en_US my-vm==> [system-thread-0] 06 May 2024 07:40:07,207  INFO SystemAgent - OS Runtime:  os.name=Linux os.arch=amd64 os.version=4.18.0-477.51.1.el8_8.x86_64 user.name=rinst user.home=/ngs/app/rinst user.dir=/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23 my-vm==> [system-thread-0] 06 May 2024 07:40:07,207  INFO SystemAgent - JVM Args : -Xmx256m | -Dlog4j.configuration=file:/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/conf/logging/log4j.xml |  my-vm==> [system-thread-0] 06 May 2024 07:40:07,211  INFO SystemAgent - JVM Runtime Name: 2589878@my-vm my-vm==> [system-thread-0] 06 May 2024 07:40:07,211  INFO SystemAgent - JVM PID: 2589878 my-vm==> [system-thread-0] 06 May 2024 07:40:07,211  INFO SystemAgent - Machine Agent is resolving bootstrap info.... my-vm==> [system-thread-0] 06 May 2024 07:40:07,299  INFO SystemAgent - Orchestration is disabled - disabling virtualization resolvers by default. my-vm==> [system-thread-0] 06 May 2024 07:40:07,303  WARN ContainerIdExtractor - Unable to use /proc/self/cgroup for unique hostname, could not locate container ID my-vm==> [system-thread-0] 06 May 2024 07:40:07,305  INFO SystemAgent - Full Agent Registration Info Resolver found system property [appdynamics.agent.create.agent.info.if.missing] for appdynamics.agent.create.agent.info.if.missing [false] my-vm==> [system-thread-0] 06 May 2024 07:40:07,313  INFO SystemAgent - Default Host Identifier Resolver using host name for unique host identifier [my-vm] my-vm==> [system-thread-0] 06 May 2024 07:40:07,315  INFO SystemAgent - Default IP Address Resolver found IP addresses [[fe80:0:0:0:889b:9bd4:a19a:6e79%eth0, 17.182.56.82]] my-vm==> [system-thread-0] 06 May 2024 07:40:07,319  INFO SystemAgent - Full Agent Registration Info Resolver using selfService [false] my-vm==> [system-thread-0] 06 May 2024 07:40:07,319  INFO SystemAgent - Full Agent Registration Info Resolver using ephemeral node setting [false] my-vm==> [system-thread-0] 06 May 2024 07:40:07,319  INFO SystemAgent - Full Agent Registration Info Resolver using application name [null] my-vm==> [system-thread-0] 06 May 2024 07:40:07,320  INFO SystemAgent - Full Agent Registration Info Resolver using tier name [null] my-vm==> [system-thread-0] 06 May 2024 07:40:07,320  INFO SystemAgent - Full Agent Registration Info Resolver using node name [null] my-vm==> [system-thread-0] 06 May 2024 07:40:07,323  INFO SystemAgent - XML Controller Info Resolver found controller host [rins-appd-stg.apple.com] my-vm==> [system-thread-0] 06 May 2024 07:40:07,323  INFO SystemAgent - XML Controller Info Resolver found controller port [443] my-vm==> [system-thread-0] 06 May 2024 07:40:07,326  INFO SystemAgent - XML Agent Account Info Resolver using account name [customer1] my-vm==> [system-thread-0] 06 May 2024 07:40:07,326  INFO SystemAgent - XML Agent Account Info Resolver using account access key [****] my-vm==> [system-thread-0] 06 May 2024 07:40:07,329  INFO SystemAgent - Keystore file /ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/conf/cacerts.jks was not found my-vm==> [system-thread-0] 06 May 2024 07:40:07,340  INFO SystemAgent - Machine Agent resolved bootstrap info! my-vm==> [system-thread-0] 06 May 2024 07:40:07,340  INFO SystemAgent - Creating machine agent scheduler, pool size: 2 my-vm==> [system-thread-0] 06 May 2024 07:40:07,349  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4 my-vm==> [system-thread-0] 06 May 2024 07:40:07,353  INFO SystemAgent - Started Agent Schedulers my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:07,366  INFO DefaultLegacyAgentRegistrationStateManager - Registered machine with machine ID [Optional.of(14786)] my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:07,367  INFO DefaultLegacyAgentRegistrationStateManager - Scheduling System-Agent start... my-vm==> [system-thread-0] 06 May 2024 07:40:07,372  INFO SimAgentRepetitiveLoggingModule - The turnover time for the SIM agent repetitive logger is 5 minutes my-vm==> [system-thread-0] 06 May 2024 07:40:07,372  INFO SimAgentRepetitiveLoggingModule - The cache size for the SIM agent repetitive logger is 1000 my-vm==> [system-thread-0] 06 May 2024 07:40:07,577  INFO Fabric8Client - No Kubernetes was detected. my-vm==> [system-thread-0] 06 May 2024 07:40:07,596  WARN DynamicMonitoringModeTask - Encountered error checking monitoring mode. Will retry in 60 seconds. my-vm==> [system-thread-0] 06 May 2024 07:40:07,596  INFO DefaultLegacyAgentRegistrationStateManager - Starting machine agent... my-vm==> [system-thread-0] 06 May 2024 07:40:07,596  INFO SystemAgent - Starting Machine Agent.... my-vm==> [system-thread-0] 06 May 2024 07:40:07,597  INFO ControllerTimeSkewHandler - Skew Handler is : [enabled]. my-vm==> [system-thread-0] 06 May 2024 07:40:07,634  INFO SystemAgent - Full certificate chain validation performed using default certificate file my-vm==> [system-thread-0] 06 May 2024 07:40:07,682  INFO ManagedMonitorDelegate - Started Agent Metric Generation Service my-vm==> [system-thread-0] 06 May 2024 07:40:07,688  INFO ManagedMonitorDelegate - Event Service is : [enabled]. my-vm==> [system-thread-0] 06 May 2024 07:40:07,697  INFO ManagedMonitorDelegate - Initialized with maxPublishQueueLength [2], aggregationFrequencyInMillis [60000] my-vm==> [system-thread-0] 06 May 2024 07:40:07,699  INFO ManagedMonitorDelegate - Metric Service is : [enabled]. my-vm==> [system-thread-0] 06 May 2024 07:40:07,702  INFO ManagedMonitorDelegate - Started Agent Env Properties Service my-vm==> [system-thread-0] 06 May 2024 07:40:07,704  INFO ManagedMonitorDelegate - Scheduled Continuous Task Monitor with frequency [30000]ms my-vm==> [system-thread-0] 06 May 2024 07:40:07,705  INFO NodeMonitorManager - Not running legacy system-agent monitor because SIM is enabled. my-vm==> [system-thread-0] 06 May 2024 07:40:07,705  INFO NodeMonitorManager - Not running legacy system-agent monitor because SIM is enabled. my-vm==> [system-thread-0] 06 May 2024 07:40:07,709  INFO MonitorConfigReader - Reading monitor config file:/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/monitors/analytics-agent/monitor.xml my-vm==> [system-thread-0] 06 May 2024 07:40:07,720  INFO MonitorConfigReader - os name [linux] version [4.18.0-477.51.1.el8_8.x86_64] my-vm==> [system-thread-0] 06 May 2024 07:40:07,725  INFO NodeMonitorManager - Initializing managed monitor [analytics-agent] my-vm==> [system-thread-0] 06 May 2024 07:40:07,725  INFO ManagedMonitorDelegate - Initializing managed monitor [AppDynamics Analytics Agent] my-vm==> [system-thread-0] 06 May 2024 07:40:07,725  INFO ManagedMonitorDelegate - Executing managed monitor [AppDynamics Analytics Agent], task name [null] my-vm==> [system-thread-0] 06 May 2024 07:40:07,725  INFO ManagedMonitorDelegate - Task [null] for monitor [AppDynamics Analytics Agent] is SCHEDULED my-vm==> [system-thread-0] 06 May 2024 07:40:07,743  INFO InProcessLauncherTask - Found a directory [/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/monitors/analytics-agent/lib] my-vm==> [system-thread-0] 06 May 2024 07:40:07,756  INFO InProcessLauncherTask - Working directory appears to be [/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/monitors/analytics-agent] my-vm==> [system-thread-0] 06 May 2024 07:40:07,758  INFO AnalyticsAgentLauncher - The logs directory property [ad.dw.log.path] has been set to [/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/logs] my-vm==> [system-thread-0] 06 May 2024 07:40:07,758  INFO InProcessLauncherTask - Starting to execute actual task with parameters [{csvMethodArgs=/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/monitors/analytics-agent/conf/analytics-agent.properties, methodName=main, className=com.appdynamics.analytics.agent.AnalyticsAgent}] my-vm==> [system-thread-0] 06 May 2024 07:40:13,348  INFO InProcessLauncherTask - Started [com.appdynamics.analytics.agent.AnalyticsAgent] my-vm==> [system-thread-0] 06 May 2024 07:40:13,349  INFO MonitorConfigReader - Reading monitor config file:/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/monitors/LinuxMonitor/monitor.xml my-vm==> [system-thread-0] 06 May 2024 07:40:13,355  INFO MonitorConfigReader - os name [linux] version [4.18.0-477.51.1.el8_8.x86_64] my-vm==> [system-thread-0] 06 May 2024 07:40:13,357  INFO NodeMonitorManager - Initializing managed monitor [LinuxMonitor] my-vm==> [system-thread-0] 06 May 2024 07:40:13,357  INFO ManagedMonitorDelegate - Initializing managed monitor [LinuxMonitor] my-vm==> [system-thread-0] 06 May 2024 07:40:13,357  INFO ManagedMonitorDelegate - Executing managed monitor [LinuxMonitor], task name [Linux Monitor Run Task] my-vm==> [system-thread-0] 06 May 2024 07:40:13,357  INFO ManagedMonitorDelegate - Task [Linux Monitor Run Task] is periodic my-vm==> [system-thread-0] 06 May 2024 07:40:13,379 ERROR JavaTaskCreator - Could not load/instantiate the Java Task Main class for Java task [Linux Monitor Run Task] java.lang.NoClassDefFoundError: org/apache/log4j/Layout at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?] at java.lang.Class.privateGetDeclaredConstructors(Class.java:3137) ~[?:?] at java.lang.Class.getConstructor0(Class.java:3342) ~[?:?] at java.lang.Class.newInstance(Class.java:556) ~[?:?] at com.singularity.ee.agent.systemagent.task.JavaTaskCreator.createJavaTask(JavaTaskCreator.java:69) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.MonitorTaskRunner.createTask(MonitorTaskRunner.java:75) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.PeriodicTaskRunner.<init>(PeriodicTaskRunner.java:41) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.ManagedMonitorDelegate.setupEnvTask(ManagedMonitorDelegate.java:255) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.ManagedMonitorDelegate.initializeMonitor(ManagedMonitorDelegate.java:212) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.readConfig(NodeMonitorManager.java:178) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.startAllMonitors(NodeMonitorManager.java:265) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.<init>(NodeMonitorManager.java:79) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.AgentMonitorManager.<init>(AgentMonitorManager.java:63) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.setupMonitorManager(Agent.java:492) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.startServices(Agent.java:399) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.SystemAgent.startServices(SystemAgent.java:79) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.start(Agent.java:384) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.appdynamics.agent.sim.legacy.DefaultLegacyAgentRegistrationStateManager$1.run(DefaultLegacyAgentRegistrationStateManager.java:80) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Layout at com.singularity.ee.util.loader.FileSystemClassLoader.findClass(FileSystemClassLoader.java:372) ~[agent-23.4.0-845.jar:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?] at com.singularity.ee.util.loader.FileSystemClassLoader.loadClass(FileSystemClassLoader.java:320) ~[agent-23.4.0-845.jar:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?] ... 24 more my-vm==> [system-thread-0] 06 May 2024 07:40:13,387  WARN ManagedMonitorDelegate - Error executing managed monitor [LinuxMonitor], task name [Linux Monitor Run Task] com.singularity.ee.agent.systemagent.api.exception.TaskInstantiationException: Could not load/instantiate task main class[com.appdynamics.extensions.linux.LinuxMonitor] for task [Linux Monitor Run Task] at com.singularity.ee.agent.systemagent.task.JavaTaskCreator.createJavaTask(JavaTaskCreator.java:90) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.MonitorTaskRunner.createTask(MonitorTaskRunner.java:75) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.PeriodicTaskRunner.<init>(PeriodicTaskRunner.java:41) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.ManagedMonitorDelegate.setupEnvTask(ManagedMonitorDelegate.java:255) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.ManagedMonitorDelegate.initializeMonitor(ManagedMonitorDelegate.java:212) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.readConfig(NodeMonitorManager.java:178) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.startAllMonitors(NodeMonitorManager.java:265) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.<init>(NodeMonitorManager.java:79) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.AgentMonitorManager.<init>(AgentMonitorManager.java:63) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.setupMonitorManager(Agent.java:492) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.startServices(Agent.java:399) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.SystemAgent.startServices(SystemAgent.java:79) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.start(Agent.java:384) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.appdynamics.agent.sim.legacy.DefaultLegacyAgentRegistrationStateManager$1.run(DefaultLegacyAgentRegistrationStateManager.java:80) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] my-vm==> [system-thread-0] 06 May 2024 07:40:13,387  WARN NodeMonitorManager - Could not initialize monitor com.singularity.ee.agent.systemagent.api.exception.TaskInstantiationException: Could not load/instantiate task main class[com.appdynamics.extensions.linux.LinuxMonitor] for task [Linux Monitor Run Task] com.singularity.ee.agent.systemagent.components.monitormanager.exception.MonitorInitializationException: com.singularity.ee.agent.systemagent.api.exception.TaskInstantiationException: Could not load/instantiate task main class[com.appdynamics.extensions.linux.LinuxMonitor] for task [Linux Monitor Run Task] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.ManagedMonitorDelegate.initializeMonitor(ManagedMonitorDelegate.java:217) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.readConfig(NodeMonitorManager.java:178) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.startAllMonitors(NodeMonitorManager.java:265) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.NodeMonitorManager.<init>(NodeMonitorManager.java:79) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.AgentMonitorManager.<init>(AgentMonitorManager.java:63) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.setupMonitorManager(Agent.java:492) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.startServices(Agent.java:399) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.SystemAgent.startServices(SystemAgent.java:79) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.Agent.start(Agent.java:384) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.appdynamics.agent.sim.legacy.DefaultLegacyAgentRegistrationStateManager$1.run(DefaultLegacyAgentRegistrationStateManager.java:80) [machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: com.singularity.ee.agent.systemagent.api.exception.TaskInstantiationException: Could not load/instantiate task main class[com.appdynamics.extensions.linux.LinuxMonitor] for task [Linux Monitor Run Task] at com.singularity.ee.agent.systemagent.task.JavaTaskCreator.createJavaTask(JavaTaskCreator.java:90) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.MonitorTaskRunner.createTask(MonitorTaskRunner.java:75) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.PeriodicTaskRunner.<init>(PeriodicTaskRunner.java:41) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.ManagedMonitorDelegate.setupEnvTask(ManagedMonitorDelegate.java:255) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] at com.singularity.ee.agent.systemagent.components.monitormanager.managed.ManagedMonitorDelegate.initializeMonitor(ManagedMonitorDelegate.java:212) ~[machineagent.jar:Machine Agent v23.2.0.3568 GA compatible with 4.4.1.0 Build Date 2023-02-21 10:37:10] ... 15 more my-vm==> [system-thread-0] 06 May 2024 07:40:13,388  INFO NodeMonitorManager - Directory [/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/monitors/unmanaged] not found, continuing. my-vm==> [system-thread-0] 06 May 2024 07:40:13,388  INFO AgentMonitorManager - Initialized System Monitor Manager with directory [/ngs/app/rinst/machineagent-bundle-64bit-linux-aarch64-23/monitors] my-vm==> [system-thread-0] 06 May 2024 07:40:13,388  INFO SystemAgent - Set up agent monitor manager my-vm==> [system-thread-0] 06 May 2024 07:40:13,389  INFO SystemAgent - Orchestration is disabled - disabling one-way agent transport. The agent will not be able to execute workflow tasks. my-vm==> [system-thread-0] 06 May 2024 07:40:13,394  INFO SystemAgentConfigManager - Scheduling configuration refresh at an interval of 60 seconds my-vm==> [system-thread-0] 06 May 2024 07:40:13,395  INFO SystemAgentConfigManager - Configuration refresh task interval is 60 seconds my-vm==> [system-thread-0] 06 May 2024 07:40:13,395  INFO SystemAgent - Configuration manager successfully configured my-vm==> [system-thread-0] 06 May 2024 07:40:13,397  INFO RunbookHandler - Runbook Operation Execution is : [enabled]. my-vm==> [system-thread-0] 06 May 2024 07:40:13,398  INFO SystemAgent - Started AppDynamics Machine Agent Successfully. my-vm==> [ExtensionStarter-AgentServer] 06 May 2024 07:40:37,516  INFO SystemAgent - Creating machine agent scheduler, pool size: 2 my-vm==> [ExtensionStarter-AgentServer] 06 May 2024 07:40:37,516  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4 my-vm==> [ExtensionStarter-AgentServer] 06 May 2024 07:40:37,517  INFO SystemAgent - Started Agent Schedulers my-vm==> [ExtensionStarter-ServerMonitoring] 06 May 2024 07:40:38,247  INFO ServersExtensionModule - OS is LINUX my-vm==> [ExtensionStarter-ServerMonitoring] 06 May 2024 07:40:38,689  INFO SystemAgent - Creating machine agent scheduler, pool size: 2 my-vm==> [ExtensionStarter-ServerMonitoring] 06 May 2024 07:40:38,690  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4 my-vm==> [ExtensionStarter-ServerMonitoring] 06 May 2024 07:40:38,690  INFO SystemAgent - Started Agent Schedulers my-vm==> [extension-scheduler-pool-0] 06 May 2024 07:40:38,694  INFO ServersDataCollectorManager - There is change in components collection configurations. my-vm==> [ExtensionStarter-ServerMonitoring] 06 May 2024 07:40:38,699  INFO ServersDataCollectorManager - Version of free command : free from procps-ng 3.3.15 my-vm==> [extension-scheduler-pool-0] 06 May 2024 07:40:38,699  INFO ServersDataCollectorManager - Starting data collectors. my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:38,709  INFO AwsTagsScheduledRunner - Started AWS tags scheduled runner to poll for tags my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:38,711  INFO K8sTagsScheduledRunner - Started K8s tags scheduled runner to poll for tags my-vm==> [ExtensionStarter-NetVizExtension] 06 May 2024 07:40:38,852  INFO SystemAgent - Creating machine agent scheduler, pool size: 2 my-vm==> [ExtensionStarter-NetVizExtension] 06 May 2024 07:40:38,852  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4 my-vm==> [ExtensionStarter-NetVizExtension] 06 May 2024 07:40:38,852  INFO SystemAgent - Started Agent Schedulers my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:38,858  INFO NetVizConfigrationListener - Applying conf: NetVizConfiguration(start=false) my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:38,909  INFO NetVizConfigrationListener - NetViz Agent is not running my-vm==> [ExtensionStarter-DockerMonitoring] 06 May 2024 07:40:39,071  INFO SystemAgent - Creating machine agent scheduler, pool size: 2 my-vm==> [ExtensionStarter-DockerMonitoring] 06 May 2024 07:40:39,071  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4 my-vm==> [ExtensionStarter-DockerMonitoring] 06 May 2024 07:40:39,072  INFO SystemAgent - Started Agent Schedulers my-vm==> [ExtensionStarter-DockerMonitoring] 06 May 2024 07:40:39,085  INFO DockerMonitoringModule - Initializing Executor Service for Docker Metric Collection, pool size:  3 my-vm==> [ExtensionStarter-DockerMonitoring] 06 May 2024 07:40:39,087 ERROR CGroupFileSystemRootProvider - Could not find CGroup files in following path(s) : [/sys/fs/cgroup, /cgroup] my-vm==> [ExtensionStarter-DockerMonitoring] 06 May 2024 07:40:39,149  INFO DockerMonitor - Not starting docker monitoring extension my-vm==> [ExtensionStarter-DockerMonitoring] 06 May 2024 07:40:39,149  INFO DockerMonitor - Docker Enabled: false; SIM Enabled: true; MA Plus available: false my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:39,160  INFO DockerMonitorConfigListener - Docker tags is collection is disabled. my-vm==> [ExtensionStarter-CrashGuard] 06 May 2024 07:40:39,178  INFO SystemAgent - Creating machine agent scheduler, pool size: 2 my-vm==> [ExtensionStarter-CrashGuard] 06 May 2024 07:40:39,179  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4 my-vm==> [ExtensionStarter-CrashGuard] 06 May 2024 07:40:39,179  INFO SystemAgent - Started Agent Schedulers my-vm==> [ConfigExecutor-0] 06 May 2024 07:40:39,216  INFO CrashGuardRunner - Not starting crash guard extension because it is disabled.To enable, please update the configuration enabled in CrashGuardConfig.yml. my-vm==> [extension-scheduler-pool-0] 06 May 2024 07:40:39,878  INFO ServersDataCollector - Started servers data collector - DataCollectorConfig(samplingInterval=30000, componentNames=[monitored.process.classes, cpus, networks, load, operating.system, volumes, partitions, memory, availability]). my-vm==> [extension-scheduler-pool-0] 06 May 2024 07:40:39,881  INFO ServersDataCollector - Started servers data collector - DataCollectorConfig(samplingInterval=30001, componentNames=[remote.volumes, partitions]). my-vm==> [extension-scheduler-pool-1] 06 May 2024 07:40:46,515  INFO AwsTagsSupplier - Skip retrieving AWS tags. Server is either not on AWS or cannot connect to AWS services my-vm==> [system-thread-0] 06 May 2024 07:41:07,366  WARN DynamicMonitoringModeTask - Encountered error checking monitoring mode. Will retry in 60 seconds. my-vm==> [AD Thread-Metric Reporter1] 06 May 2024 07:41:07,714  INFO SystemAgent - Full certificate chain validation performed using default certificate file my-vm==> [extension-scheduler-pool-8] 06 May 2024 07:41:08,692  INFO ServersDataCollectorManager - There is change in components collection configurations. my-vm==> [extension-scheduler-pool-8] 06 May 2024 07:41:08,692  INFO ServersDataCollector - Stopped servers data collector - DataCollectorConfig(samplingInterval=30000, componentNames=[monitored.process.classes, cpus, networks, load, operating.system, volumes, partitions, memory, availability]). my-vm==> [extension-scheduler-pool-8] 06 May 2024 07:41:08,692  INFO ServersDataCollector - Stopped servers data collector - DataCollectorConfig(samplingInterval=30001, componentNames=[remote.volumes, partitions]). my-vm==> [extension-scheduler-pool-8] 06 May 2024 07:41:08,692  INFO ServersDataCollectorManager - Starting data collectors. my-vm==> [extension-scheduler-pool-8] 06 May 2024 07:41:08,693  INFO ServersDataCollector - Started servers data collector - DataCollectorConfig(samplingInterval=30000, componentNames=[monitored.process.classes, networks, load, operating.system, availability]). my-vm==> [extension-scheduler-pool-8] 06 May 2024 07:41:08,694  INFO ServersDataCollector - Started servers data collector - DataCollectorConfig(samplingInterval=30001, componentNames=[remote.volumes, partitions]). my-vm==> [extension-scheduler-pool-8] 06 May 2024 07:41:08,694  INFO ServersDataCollector - Started servers data collector - DataCollectorConfig(samplingInterval=3000, componentNames=[cpus, volumes, partitions, memory]). my-vm==> [system-thread-0] 06 May 2024 07:42:07,366  WARN DynamicMonitoringModeTask - Encountered error checking monitoring mode. Will retry in 60 seconds. my-vm==> [AD Thread-Metric Reporter1] 06 May 2024 07:42:07,709  INFO SystemAgent - Full certificate chain validation performed using default certificate file my-vm==> [extension-scheduler-pool-10] 06 May 2024 07:42:38,756  WARN ProcessMonitor - ProcessMonitor::Caught exception during collection and reporting. com.appdynamics.voltron.rest.client.NonRestException: Method: SimProcessesAgentService#updateProcessMetadata(String,List) - Result: 403 Forbidden - content: <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>Apple</center> </body> </html>   at com.appdynamics.voltron.rest.client.VoltronErrorDecoder.decode(VoltronErrorDecoder.java:62) ~[rest-client-1.1.0.187.jar:?] at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:156) ~[feign-core-10.7.4.jar:?] at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80) ~[feign-core-10.7.4.jar:?] at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100) ~[feign-core-10.7.4.jar:?] at com.sun.proxy.$Proxy156.updateProcessMetadata(Unknown Source) ~[?:?] at com.appdynamics.sim.agent.extensions.servers.DoubleBufferedProcessProperties.reportProcesses(DoubleBufferedProcessProperties.java:75) ~[servers-23.2.0.3568.jar:?] at com.appdynamics.sim.agent.extensions.servers.ProcessMonitor.run(ProcessMonitor.java:78) [servers-23.2.0.3568.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] my-vm==> [system-thread-0] 06 May 2024 07:43:07,365  WARN DynamicMonitoringModeTask - Encountered error checking monitoring mode. Will retry in 60 seconds. my-vm==> [AD Thread-Metric Reporter1] 06 May 2024 07:43:07,712  INFO SystemAgent - Full certificate chain validation performed using default certificate file my-vm==> [extension-scheduler-pool-9] 06 May 2024 07:43:38,739  WARN ProcessMonitor - ProcessMonitor::Caught exception during collection and reporting. com.appdynamics.voltron.rest.client.NonRestException: Method: SimProcessesAgentService#updateProcessMetadata(String,List) - Result: 403 Forbidden - content: <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>Apple</center> </body> </html>   at com.appdynamics.voltron.rest.client.VoltronErrorDecoder.decode(VoltronErrorDecoder.java:62) ~[rest-client-1.1.0.187.jar:?] at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:156) ~[feign-core-10.7.4.jar:?] at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:80) ~[feign-core-10.7.4.jar:?] at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100) ~[feign-core-10.7.4.jar:?] at com.sun.proxy.$Proxy156.updateProcessMetadata(Unknown Source) ~[?:?] at com.appdynamics.sim.agent.extensions.servers.DoubleBufferedProcessProperties.reportProcesses(DoubleBufferedProcessProperties.java:75) ~[servers-23.2.0.3568.jar:?] at com.appdynamics.sim.agent.extensions.servers.ProcessMonitor.run(ProcessMonitor.java:78) [servers-23.2.0.3568.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?]
Hi This depends on what you have already in your Splunk. If you want to create KV based lookup with GUI then minimum requirement is that you have at least one collection defined on your instance. An... See more...
Hi This depends on what you have already in your Splunk. If you want to create KV based lookup with GUI then minimum requirement is that you have at least one collection defined on your instance. And this can do only with conf file. If you haven't any collection then you cannot create kv based lookup with GUI. Of course if you have lookup editor app then you can. But even if you have collection defined it's not so simple than just create a new lookup based on it. Usually there is collection per lookup as collection defines used fields in lookup. I think that your best options are: Ask your Splunk Admin install Splunk Lookup Editor and use it Ask your Splunk Admin / KO admin create that collection + lookup for you Create app which contains those and ask from your Splunk Admin that they install it with needed permission for your use case r. Ismo
As I said in a duplicate post, you don't appear to be referencing the timepicker token correctly - try using $timepicker.earliest$ and $timepicker.latest$
Hi How you have defined that input on your HF (DB Connect) node? Are you sure that those events are really twice here instead of you have e.g. set both INDEX EXTRACTIONS on HF side and KV_MODE=JSON ... See more...
Hi How you have defined that input on your HF (DB Connect) node? Are you sure that those events are really twice here instead of you have e.g. set both INDEX EXTRACTIONS on HF side and KV_MODE=JSON on SH side? r. Ismo
Hi one old post for same kind of situation. https://community.splunk.com/t5/Splunk-Enterprise/How-to-dynamically-lookup-filename/m-p/645855 r. Ismo
Hi Have you done this on all SHC members? Configure each search head cluster member as a search head on the indexer cluster. Use the CLI splunk edit cluster-config command. For example: https://do... See more...
Hi Have you done this on all SHC members? Configure each search head cluster member as a search head on the indexer cluster. Use the CLI splunk edit cluster-config command. For example: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/SHCandindexercluster One correction for those default ports. There is no default ports (or alt least earlier haven't been) for IDX replication or SHC replication. There are some commonly used ports, those are not default, you must always define those manually in CLI, conf files or in GUI! r. Ismo 
@yuanliu @gcusello it worked thanks
is there any other way of handling json content for using rex command which would be much easier. although my request is not completely in a json format. You must understand why @PickleRick and ... See more...
is there any other way of handling json content for using rex command which would be much easier. although my request is not completely in a json format. You must understand why @PickleRick and I keep telling you not to try using rex to handle structured data like JSON: rex is the wrong tool because syntax is not bound by format in JSON.  The same semantics can be expressed by a million variants of format while conforming to the same syntax. {"ka":"va","kb":"vb"} is exactly the same as {"kb":"vb","ka":"va"}.  Any rex you develop will always be instable.  By insisting on using regex, i.e., treating structured data as pure text, you are just reinforcing some bad habit that will inhibit your abilities in the future.
just voted, thanks a lot  
I have a similar requirement but only if i could send my GLUE DQ results direcrtly to splunk to showcase a dashboard
| streamstats list(Count) as Count list(errorid) as errorid by provider errorname global=f window=4 | where mvcount(Count) = 3 By setting the window to 4, only the 3rd one will have 3 values in the ... See more...
| streamstats list(Count) as Count list(errorid) as errorid by provider errorname global=f window=4 | where mvcount(Count) = 3 By setting the window to 4, only the 3rd one will have 3 values in the list
A more fundamental problem is that by insisting  on using regex for this log, you are treating structured JSON log eilog.EILOG as text string.  It is NOT.   It is much more robust to use Splunk's bui... See more...
A more fundamental problem is that by insisting  on using regex for this log, you are treating structured JSON log eilog.EILOG as text string.  It is NOT.   It is much more robust to use Splunk's built-in, QA tested capabilities to handle structured data.  Have you tried my suggestion | rex "eilog.EILog:\s*(?<eilog>{.+})" | spath input=eilog | spath input=jsonRecord and not getting all data fields in this JSON?   As I illustrated previously, this should give you Task Name Cash Apps PAPI along with dozens of other key-value pairs.