Hello community, I have an issue with one forwarder, was working and suddenly stopped sending data to the Indexers. The Splunk services at the UF are running but the data is not sent to the Indexers. The internal logs are not sent either. But if I run a restart at the UF the logs are send but almost immediately are stopped again. I have checked the logs but I cannot find a logical reason for this to happen. I have changed the [inputproc] max_fd = <integer>  From 100 to 8192 then I restarted the splunk service. I have checked the ulimits and currently is in 6400.   If I stop and start the service and I check the logs I cannot see any clue about can be happening, these are the logs that appears before the UF stops sending data: 03-30-2023 05:07:26.260 +0200 INFO TailReader [20263 MainTailingThread] - Setting maxFDs to 8192 03-30-2023 05:07:31.013 +0200 INFO ProxyConfig [20259 TcpOutEloop] - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled. 03-30-2023 05:07:31.013 +0200 INFO ProxyConfig [20259 TcpOutEloop] - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled. 03-30-2023 05:07:31.013 +0200 INFO ProxyConfig [20259 TcpOutEloop] - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled. 03-30-2023 05:07:31.013 +0200 INFO ProxyConfig [20259 TcpOutEloop] - Failed to initialize the proxy_rules setting from server.conf for splunkd. Please provide a valid set of proxy_rules in case HTTP proxying needs to be enabled. 03-30-2023 05:07:31.013 +0200 INFO ProxyConfig [20259 TcpOutEloop] - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled. 03-30-2023 05:07:31.674 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.675 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.675 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.675 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.685 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.685 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.685 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.685 +0200 WARN TcpOutputProc [20259 TcpOutEloop] - 'sslCertPath' deprecated; use 'clientCert' instead 03-30-2023 05:07:31.695 +0200 INFO AutoLoadBalancedConnectionStrategy [20259 TcpOutEloop] - Will resolve indexer names at 330.000 second interval. 03-30-2023 05:07:36.671 +0200 INFO TailReader [20266 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log' 03-30-2023 05:07:37.774 +0200 INFO DC:DeploymentClient [20219 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected 03-30-2023 05:07:49.774 +0200 INFO DC:DeploymentClient [20219 PhonehomeThread] - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected   After certain time I see the message "The TCP output processor has paused the data flow. Forwarding to host_dest..." But I assume this is because the Splunkd is not able of send data.   Do you have any idea about what can be going on?   Thanks in advance ... View more

Hello hello, We have the Splunk db connect app working in our environment, but suddenly stops working And I can see this log: 2022-11-21 23:22:39.050 -0500 [dw-203047 - PUT /api/inputs/server_average_latency] ERROR c.s.d.m.repository.DefaultConfigurationRepository - action=failed_to_get_the_conf reason=HTTP 401 -- call not properly authenticated com.splunk.HttpException: HTTP 401 -- call not properly authenticated at com.splunk.HttpException.create(HttpException.java:84) at com.splunk.DBXService.sendImpl(DBXService.java:131) at com.splunk.DBXService.send(DBXService.java:43) at com.splunk.HttpService.get(HttpService.java:154) at com.splunk.Entity.refresh(Entity.java:381) at com.splunk.Entity.refresh(Entity.java:24) at com.splunk.Resource.validate(Resource.java:186) at com.splunk.Entity.validate(Entity.java:462) at com.splunk.Entity.getContent(Entity.java:157) at com.splunk.Entity.size(Entity.java:416) at java.util.HashMap.putMapEntries(HashMap.java:501) at java.util.HashMap.<init>(HashMap.java:490) at com.splunk.dbx.model.repository.JsonMapperEntityResolver.apply(JsonMapperEntityResolver.java:34) at com.splunk.dbx.model.repository.JsonMapperEntityResolver.apply(JsonMapperEntityResolver.java:18) at com.splunk.dbx.model.repository.DefaultConfigurationRepository.get(DefaultConfigurationRepository.java:92) at com.splunk.dbx.server.dbinput.task.DbInputTaskLoader.load(DbInputTaskLoader.java:63) at com.splunk.dbx.server.api.service.conf.impl.InputServiceImpl.update(InputServiceImpl.java:221) at com.splunk.dbx.server.api.service.conf.impl.InputServiceImpl.update(InputServiceImpl.java:38) at com.splunk.dbx.server.api.resource.InputResource.updateInput(InputResource.java:81) at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43... CONTINUES   I am not going to add the whole log because is huge.   We have a cluster and our dbconnect is install at the search heads and all the inputs are configured at the Heavy Forwarder.   Is you have any idea what I can check to see what is the issue, please let me know.   The environment is over linux.   Thanks in advance. Best Regards. ... View more

Hello everyone, I have noticed that some users in our Splunk environment are always using base searches and Post-process searches, because they was told that was a good practice to do that. But there are some cases I have noticed that the use of the base search is not speeding up the dashboard instead spent more time. For example there is a dashboard that uses a base search and they use something like this: Base search index=example sourcetype=testing | fields *   And then at the subsearch I can see that when Splunk uses that best search is doing something weird adding the | fields *  at search for example: Example: index=example sourcetype=testing | fields * | eval Date=strftime(_time, "%m/%d/%Y") | dedup s1, s1 | fields * | search something=tosearch | fields * | eval _time = strptime(Date,"%m/%d/%Y") When the post-search is <query> | eval Date=strftime(_time, "%m/%d/%Y") | dedup s1, s1 | search something=tosearch | eval _time = strptime(Date,"%m/%d/%Y") </query>   So I would like to understand why Splunk does it. And also I would like to know if there are some scenarios where use the base search is not recommended.   Thanks in advantage. Best Regards, ... View more

Hello community, I hope you are doing well. I have a customer that wants to have a Version Control system with Splunk. And the option that I see is Git to be able of do all the Backup and Restore process. I tried with this app https://splunkbase.splunk.com/app/4355#/overview But this app doesn't work as I expected, because is not able of do the commit automatically and is just for save the Knowledge Objects in another location.   I would like to know if there is any app that helps doing the commit process to the remote repository or this must be done using an script, I would like to avoid the customer do it manually. Thanks for the inputs.   Best Regards. ... View more

Hello community, since a couple of months ago we are having an issue into Splunk and is so weird... The issue is that we are working as usual and suddenly we lost the session and when we refresh the UI is shown like the image below: So as you can see the menu at the right top is broken and also the icon is broken. Also if you try to click one of the available option, don't work. We have a cluster search head and a Load Balancer.   If you have any idea I'll really appreciate it.   Thanks in advance. Version 8.2.2 ... View more

Hello everyone, I am trying to ingest data into Splunk and the data is into some .tgz files, but within those files are a lot of different folders and levels of directories, the thing is that I want to read just one type of file that is into those directories and is not an absolute path is a relative the path can change and can be into any directory. So the inputs .conf was set up with something like this: [monitor:///dir1/dir2/Spk/Test/*.tgz] whitelist=my.log   But this is not working because of this: When you configure wildcards in a file input path, Splunk Enterprise creates an implicit allow list for that stanza. The longest wildcard-free path becomes the monitor stanza, and Splunk Enterprise translates the wildcards into regular expressions.  https://docs.splunk.com/Documentation/Splunk/latest/Data/Specifyinputpathswithwildcards?_gl=1*srk1nm*_ga*MjE0MDA2MDA2MS4xNjI4ODcyNDg2*_gid*MTUyMjkxNTIzLjE2NTY5Njg1NDE.&_ga=2.178037989.152291523.1656968541-2140060061.1628872486   So I am looking the way to filter those logs using whitelisting, should I use regular expressions to filter the logs?   Thank you in advance. ... View more

Hello everyone,   We found that the VM has more RAM allocated currently we have 12GB of RAM for each Search Head, but that can be increase to 48GB of RAM. I have been reading and increase the capacity of the Search Heads can affect the indexer nodes: An increase in search tier capacity corresponds to increased search load on the indexing tier, requiring scaling of the indexer nodes. Scaling either tier can be done vertically by increasing per-instance hardware resources, or horizontally by increasing the total node count. And that make sense, currently the environment has more searches than indexers and I think increase the capacity of the SH can overwhelm the  indexers. Current environment: 3 SH (cluster) and 2 Indexers(cluster). I would appreciate any recommendation to do this as good as possible and be able to use the memory allocated.   Kind Regards. ... View more