Hi @bishida , Thanks for the inputs. I checked the documentation and have updated the diagram to show what should be happening under the hood. The setups on the left are part of the organisation's infrastructure, and on the right are exposed to the internet. Is this feasible to implement?   Thanks, Pravin ... View more

Hi @livehybrid , Currently, I am using the intermediary Splunk for sending data to Splunk Cloud (again on the internet) via HEC. Now, in addition to this existing setup, I want to use this intermediary Splunk to send data to Splunk APM as well, using another HEC token, but the data has to be converted to protobuf format first before sending to APM. Thanks, Pravin   ... View more

Thanks @PickleRick  I understand that the SHC is not configured properly, which is causing the issue with UI, but does it affect the Splunk search performance as well? The redirection between different SHs is not happening, which is my concern now, but how does this cause Slplunk slowness? Thanks, Pravin ... View more

Hi @livehybrid ,   Thanks for your response. It was really useful. When I have a glitchy UI, the error is mostly - Failed to load resource - the server responded with a status of 400 (Bad Request) (OR)  net::ERR_CONNECTION_RESET Does this load balancer stickiness also affect how searches are managed and loaded in Splunk?  Do they also slow down the search speed when LB caching is present?   Regards, Pravin ... View more

When we set up a cluster, the SH, CM and the indexers stay connected over the management port 8089 and will keep sending _internal logs no matter what, but the forwarders use the inputs port 9997 to send data to the indexers. In our case, we only flip the port to XXXX or 9998, depending on the type of forwarding setup used. We have controlled data ingestion and always stay within limits, but sometimes unexpected testing causes a high input flow, and thus, we have to take measures to make sure we don't breach the license. ... View more

Hi @isoutamo , Thanks for the answer. Could you please provide more clarity on this part? So my proposal is that you just switch receiving port of indexers to some valid port which are allowed only from SH side by FW. Then your SH (and LM in SH) can continue send their internals into indexers and everything should works. And same time all UFs with static indexer information cannot send events as the receiving port has changed. If you have any real dat inputs on SH then you should set up HF and move those inputs there. Are there multiple receiving ports for an indexer? And, if so, how can I do that? Thanks, Pravin ... View more

Hi @isoutamo , You may be aware that Splunk has its panel that records license warnings and breaches, but once the number of warnings/breaches (I assume it was 5) exceeds the 30-day limit, Splunk would cut the data intake, and the panels become unusable. To make sure that data isn't completely cut off, we at our company made an app that keeps track whenever we hit the mark of 3 breaches in a 30-day rolling period. So, upon hitting the mark, the port flip comes into action, and it flips the default receiving port from 9997 to XXXX. Some random alphabet because the indexer discovery will determine the new port as well, once the indexer is restarted. This strategy was initially implemented as a port switch from 9997 to 9998, and the inputs.conf was configured in the usual static way, where I mention the names of the <server>:<port>  format, but later reformatted to suit the indexer discovery technique. What was strange about this technique was that we never had network issues in the search head during the classic forwarding technique, but noticed the same in the indexer discovery technique.  Also, to make sure the problem exists only after the indexer discovery, I simulated the same in a test environment and noticed the worse network usage when the indexers are not reachable, but still the Search head was usable. The only difference between the two environments is that production has a lot of incoming data to the indexers, and the SH also acts as the license master to a lot of other sources where whereas the test environment doesn't do the same. The data flow begins again as we switch the ports back to 9997 after midnight, once the new day license period starts and the SH is back to its normal state. ... View more

Hi @PrewinThomas , Thanks for your response. We used to have a classic way of connecting the forwarder to peers, and we had the same port flip technique, but didn't have the SH slowness. This is making me doubt if I am missing something in the configuration of the indexer discovery.  Also, to get back on the same issue, I always tried to check if the SH responds, but on the flipside, I never checked the indexers UI. So should it be the case where all the UI should fail due to cluster instability, and not just the SH? But when this was the case, I tried the SH restart, but of no use. To address license overages at source, reducing the ingestion or increasing the license is not possible at this stage because overages are a rare one-off scenario, but Splunk's license enforcement is something which I can do. Is there a way I can cut off the data when I am approaching a license breach? One more important thing to notice is how the Splunk license works. Splunk logs license through the _internal index, and meter gets data based on license.log, but if the license.log file is being indexed late with a time delay, still the license gets updated for the _time of the data, and not the indextime. Any thoughts on this process @livehybrid @PrewinThomas  Thanks, Pravin ... View more

Hi @livehybrid , The SH sends its internal logs to the indexers as well, but we used to have a classic way of connecting the forwarder earlier, but didn't have the SH slowness. This is making me doubt if I am missing something in the configuration of the indexer discovery. To get back to your comment, when you say SH starts queuing data, is there an intermediate queue in the SH, or maybe any forwarder to store data when it's unable to connect to the indexers (data layer). Thanks, Pravin ... View more

Hi @livehybrid , To answer your questions in order, I can get events, and it works when I run the first line. I am logged in as an admin, and I created the index, so I have permission to read and write to the index. Data is available, but not written to the new index. info : The limit has been reached for log messages in info.csv. 65149 messages have not been written to info.csv. Refer to search.log for these messages or limits.conf to configure this limit. warn : No results to summary index. Once I complete the execution, I get the above error message in the job inspector. Thanks, Pravin ... View more

Hi @gcusello , We have enough space in the servers, it's not an issue with the disk. Thanks, Pravin ... View more

I didn't copy all the files and directories mentioned by you and also didn't put the old CM in maintenance mode but did change the URL and FDQN for all the instances. Probably the problem arose from the fact that the production instance has a lot of moving data and not going into maintenance mode caused the problems. Even, the test site had moving data but did not have a lot of data like production. What was surprising is that there were no logs that showcase any exact reason of error. However, I used the techniques mentioned by you and was able to migrate CM to new hardware. Thanks to you @isoutamo      ... View more

Hi @isoutamo ,   I followed a similar strategy and I'll list it below: Set up new CM, and turn on indexer clustering. Turn off the instance and copy  'etc/master-apps' and 'etc/manager-apps' Setup server.conf in the new instance with relevant changes ( pass4symkey in string format) Start the new CM Change the URL of the indexer cluster one by one. Finally, change the URL in the search head after the indexers are migrated. This had worked in the test environment but when it was time for the production setup, the indexers failed to connect and would keep stopping after changing to new CM.   Regards, Pravin   ... View more

I use Splunk Enterprise 9.0.4 and I tried adding _meta field which didn't work. I also tried adding INGEST_EVAL to transforms and tried sending the data, still no luck identifying the source. ... View more

Hi @hieuba6868 ,   I am sending open telemetry data to heavy forwarder and the HF forwards the data to indexers. When I look at the  field 'splunk_server' I can see only the name of indexers. If I look at the data I can see the name of the otel source. In my current scenario I want to know which is the HF sending the data.   Regards, Pravin ... View more