About justynap_ldz

verbal_666 · ‎11-28-2023

Whis is as you have 2 UF on same machine. Maybe you should only increase the limits.conf, [thruput] maxKBps = <integer> * The maximum speed, in kilobytes per second, that incoming data is processed through the thruput processor in the ingestion pipeline. * To control the CPU load while indexing, use this setting to throttle the number of events this indexer processes to the rate (in kilobytes per second) that you specify. * NOTE: * There is no guarantee that the thruput processor will always process less than the number of kilobytes per second that you specify with this setting. The status of earlier processing queues in the pipeline can cause temporary bursts of network activity that exceed what is configured in the setting. * The setting does not limit the amount of data that is written to the network from the tcpoutput processor, such as what happens when a universal forwarder sends data to an indexer. * The thruput processor applies the 'maxKBps' setting for each ingestion pipeline. If you configure multiple ingestion pipelines, the processor multiplies the 'maxKBps' value by the number of ingestion pipelines that you have configured. * For more information about multiple ingestion pipelines, see the 'parallelIngestionPipelines' setting in the server.conf.spec file. * Default (Splunk Enterprise): 0 (unlimited) * Default (Splunk Universal Forwarder): 256 Since by deault it send at 256Kb/s. I set it to 2048 for many UFs which send much data. You could also try a 0 to disable thruput control.

emallinger · ‎08-18-2023

HEllo, Same here on splunk version 9.1.0.2. Plus, displays : Plus, I've errors on debug console : Some js files seems to be blocked because of the X-Content-Type Options that are still installation defaults. Does anyone have any info on this topic ? Thanks ! Ema

justynap_ldz · ‎08-01-2023

Hey, does anyone know any best practice or clever way of removing orphaned Knowledge Objects in a Search Head cluster when it is already too late for reassignment? For each orphaned object we are doing manual job like checking if AD accounts still exist, emailing the users and asking if they still need Splunk etc. For non-existing accounts, we delete /opt/splunk/etc/users<user_id> catalogue from each SH separately (there are 4 SHs in our cluster), but we are looking for more clever solution Unfortunately, there is no option in our case to be informed by the users that they are going to leave the company in order to react in advance and avoid orphaned KOs at all... Greetings, Justyna

isoutamo · ‎07-21-2023

Hi one way to try to find where it's stored it use find like this find $SPLUNK_HOME $ -path $SPLUNK_HOME/var/log -o -path $SPLUNK_HOME/var/run/splunk/dispatch -o -path $SPLUNK_HOME/var/lib/splunk/\*/\*db\* -o -path $SPLUNK_HOME/var/lib/splunk/fishbucket $ -prune -o -type f $ -ctime -1m -o -mtime -1m $ -ls Modify -1m what ever time you need to look, when you thing that token has created. It's told how many minutes backward it is looking when file have created or modified. r. Ismo

justynap_ldz · ‎07-21-2023

Hi @isoutamo, It worked, thank you for your prompt reply! Much appreciated BR, Justyna

woodcock · ‎04-16-2023

Here are some things that hopefully you can change/disable that can get in the way: FIPS selinux firewall (firewalld) missing route dns

kaeleyt · ‎04-14-2023

I had a similar thing happen. Mine was closed and I was told to post on Splunk Ideas. Here's the post in case anyone wants to upvote it: https://ideas.splunk.com/ideas/EID-I-854 Seems based on one reply that is was supposedly "under investigation" but that was 2 years ago and still seems to be an issue in 9.0.4. I'm the team member responsible for managing this type of thing for all of the teams who use our enterprise wide Splunk environment but do not have the access/permissions to use a lot of the solutions people are suggesting on this thread so it's still a bit frustrating that we have objects owned by users who left the company 4 years ago.

justynap_ldz · ‎03-02-2023

Hello All, We keep getting some errors from Splunk Add-on for Java Management Extensions 5.3.0: splunkd.log 1. ERROR ExecProcessor [20951 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" INFO: Loading mapping descriptors from jar:file:/opt/splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml 2. ERROR ExecProcessor [20951 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" Mar 01, 2023 11:59:59 PM org.exolab.castor.mapping.Mapping loadMapping jmx.log 3. 2023-03-02 06:37:18,168 - com.splunk.modinput.ModularInput -1968729474 [Thread-11] INFO [] - Failed connection with 'service:jmx:rmi:///jndi/rmi://server1.domain.com:port/jmxrmi', trying to collect data with a short URL: 'service:jmx:rmi://server1.domain.com:port/jndi/jmxrmi' . 2023-03-02 06:37:18,168 - com.splunk.modinput.ModularInput -1968729474 [Thread-11] ERROR [] - Exception@checkConnector, e= java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[?:1.8.0_51] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[?:1.8.0_51] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229) ~[?:1.8.0_51] at com.splunk.jmx.ServerTask.connect(Unknown Source) ~[jmxmodinput.jar:?] at com.splunk.jmx.ServerTask.checkConnector(Unknown Source) ~[jmxmodinput.jar:?] at com.splunk.jmx.Scheduler.run(Unknown Source) ~[jmxmodinput.jar:?] Caused by: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:662) ~[?:1.8.0_51] at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[?:1.8.0_51] at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:350) ~[?:1.8.0_51] at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_51] at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957) ~[?:1.8.0_51] at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924) ~[?:1.8.0_51] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_51] ... 5 more 2023-03-02 06:37:18,168 - com.splunk.modinput.ModularInput -1968729474 [Thread-11] WARN [] - Server misconfiguration already notified once for jmx://_Splunk_TA_jmx_:server1 stanza with message: Failed to connect with the JMX server. Review the configuration of the jmx://_Splunk_TA_jmx_:server1 stanza, and try again. 2023-03-02 06:37:18,174 - com.splunk.modinput.ModularInput -1968729480 [Thread-11] INFO [] - 1 servers found in stanza jmx://_Splunk_TA_jmx_:server1 Add-on has been configured on a Heavy Forwarder Our jmx_servers.conf is: [hostname] description = <description> destinationapp = Splunk_TA_jmx host = <FQDN> jmxport = <port> protocol = rmi stubSource = jndi disabled = 0 lookupPath = /jmxrmi Does anyone have any idea on how to fix it? Greetings, Justyna

PaulPlum · ‎02-03-2023

try [mail://username@thisisourmail.com]

saurabh_ha · ‎01-27-2023

Hello, I faced the same issue, following are the troubleshooting steps has been followed. As ERROR=401 that unauthrized request is made from deployer to splunk SHC Go to the /splunk-home/etc/system/local/server.conf of all SH cluster member and deployer and change the following, keep the value the same. [shclustering] pass4SymmKey=<clear_text_string> Restart splunkd. /opt/splunk/bin/splunk apply shcluster-bundle --answer-yes -target https://xx.xx.xxx.xx:8089 -auth admin:xxxmxmxm hope @erw550 this will help to you.

PickleRick · ‎01-27-2023

Try this. https://conf.splunk.com/files/2019/slides/FN1402.pdf and this https://conf.splunk.com/files/2019/summit/FN1402.mp4

justynap_ldz · ‎01-26-2023

Hi @JECN , we have the same issue? Did you solve your case?

isoutamo · ‎01-20-2023

Hi how many HFs you have vs indexers? I have seen a rule of thumb that you should have (min) 7 hf pipeline per one IDX pipeline to get this working smoothly. r. Ismo

guilmxm · ‎12-13-2022

Hi @justynap_ldz Have you checked the logs? https://ta-ms-teams-alert-action.readthedocs.io/en/latest/configuration.html#

richgalloway · ‎12-05-2022

The ignoreOlderThan setting has no bearing on data retention. It only controls what input files Splunk reads. The delete command does not remove data from indexes. It merely hides the events. Data is deleted by bucket and doesn't happen until the newest event in a bucket is at least 6 months old. The dbinspect command can show the newest and oldest event dates for each bucket. Many sites avoid this problem by setting a maximum lifetime of 24 hours for a hot bucket. ( maxHotSpanSecs = 86400 in indexes.conf).

justynap_ldz · ‎12-05-2022

Hello All, we have just found some pretty old catalogues and delta files in /opt/splunk/var/run/searchpeers on our indexers (3 indexers in the cluster, Splunk v. 8.2.9.). We have never deleted anything from that location as it always seemed to be a bit risky for us. At the same time the catalogues were modified in 2017 and seem to contain some unneeded and outdated stuff from already decommissioned servers. Is that safe to delete old catalogues in /opt/splunk/var/run/searchpeers? If so, should splunkd be stopped prior to deletion or should splunkd rolling restart be initiated after deletion as this is indexer cluster? Any hints would be much appreciated! BR, Justyna

justynap_ldz · ‎11-23-2022

Hi @mpavlas , we have similar issue after upgrading to 8.2.9. Have you already fixed the issue or contacted Splunk Support by any chance? Greetings, Justyna

justynap_ldz · ‎10-11-2022

The solution was to request new cert in company's CA, generate new private key and new certificate chains

justynap_ldz · ‎09-23-2022

Hi @kamlesh_vaghela, thanks for replying! We have not configured any inputs on SearchHeads and Indexers, but your comment just reminded me about https://docs.splunk.com/Documentation/AddOns/released/Overview/Distributedinstall We have deleted default inputs.conf and inputs.conf.spec from Splunk_TA_jmx on SHs and the errors are gone! Thank you 🙂

kkong_splunk · ‎08-05-2022

(Full disclaimer: I'm the product manager for Splunk Mobile) Splunk Mobile is a free app for Android or iOS that allows you to send push notifications to your phone based on the alerts in Enterprise. We actually had a user that we featured in our breakout session this year (PLA1488A) at .conf22 who used Mobile alerts to monitor his indexers, and was able to catch them going down on a weekend because of Splunk Mobile. You can configure alerts to be sent to a specific role, so if you already have a Splunk role setup for the folks on your team, you can send alerts to that role to receive them on mobile. Documentation on getting started with Splunk Mobile is here: https://docs.splunk.com/Documentation/Alerts/2.31.1/Alerts/GetStarted Documentation on alerts specifically is here: https://docs.splunk.com/Documentation/Alerts/2.31.1/Alerts/SendAlerts An overview of what Splunk Mobile looks like is here: https://www.youtube.com/watch?v=FOHwU00IVUE&ab_channel=Splunk

justynap_ldz · ‎07-12-2022

Thank you @richgalloway, I have just contacted Splunk Support

gcusello · ‎07-08-2022

Hi @justynap_ldz, I fell into this mistake some time ago, which is why I went almost without fail! good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated 😉

TheExpert · ‎03-25-2022

Hi @justynap_ldz, no I wasn't able to solve it with Splunk. I never changed something in the .conf files you mentioned. But I had to stop sending the syslogs of the VMware ESXi hosts to Splunk because the free amount about 500 MB per day was overloaded by the VMware log data. I also use Splunk for the logs of my Sophos UTM to have a better tool for troubleshooting firewall and proxy issues. So there's not enough free space for the VMware syslogs. And i found an alternative way by using VMware PowerCLI to get the SMART data from the ESXi hosts. With a PowerShell script I can read all SMART data and send a warning mail when there are issues. Í even can read data that isn't shown in the syslog of the ESXi hosts. Kind Regards

PickleRick · ‎03-25-2022

Ahh, my mistake. I misinterpreted the file path in inputs.conf. I thought you were mounting remote storage from ESXi by some means (NFS? sshfs?) and pulling log files from there. So you're using some kind of syslog daemon on the HF machine and then the forwarder pulls data from the logs that syslog daemons writes to? Most probably your syslog solution strips some headers or reorganizes the data in some way before writing it to disk. There are many possible approaches to this situation 1. Correct the transform to catch proper field (might not be compatible with future add-on upgrades and you might have to adjust the resulting transform by hand after such upgrades) 2. Correct the syslog daemon configuration so it writes the events literarily as it was received from network to the file - relatively foolproof solution. 3. Get rid of the intermediate logfile layer - reorganize your syslog receiving process to receive events from network and send them directly to HEC input on your HFs/indexers by means of sc4s or rsyslog - most flexible solution but requires some work.

MaroofaTanweer · ‎01-07-2022

hi, I had same problem . I changed two things of server.conf file of all indexers : from, master_uri = https://splunk-master.splunk.svc.cluster.local:8089 mode = slave to, manager_uri = https://splunk-master.splunk.svc.cluster.local:8089 mode = peer and restart the splunk. these changes worked for me.

Posts	44
Solutions	2
Karma Given	30
Karma Received	0
Member Since	‎08-21-2020

Online Status	Offline
Date Last Visited	‎09-15-2023 08:30 AM

What is Best practice for removing orphaned KOs fr...

How to delete authorization token for non-existent...

Splunk Add-on for Java Management Extensions 5.3.0...

Cannot publish Splunk alert in MS Teams with TA-ms...

Indexer cluster: "Oldest Data Age" extremely high ...

Is that safe to delete old catalogues in /opt/splu...

How to resolve Indexer ERROR TcpInputConfig - SSL ...

Splunk Add-on for Java Management Extensions ExecP...

Splunk crashes after error popped up: ENGINE: Bus ...

Splunk serverclass is uninstalling a deployment ap...

Re: Why are we receiving this ingestion latency er...

Re: Failed to load source for MC KPI Indicator vis...

What is Best practice for removing orphaned KOs fr...

Re: Where does token gets stored when created by s...

Re: How to delete authorization token for non-exis...

Re: ERROR TcpOutputFd - Read error. Connection res...

Re: Bug: Unable to Reassign Knowledge Objects with...

Splunk Add-on for Java Management Extensions 5.3.0...

Re: How to configure TA_mailclient with a differen...

Re: Why can I not push config from deployer to sea...

Re: Uneven indexing rate by instance

Re: Add-on for Java Management Extensions errors

Re: Parsing queue and Aggregation queue sizes on h...

Re: Cannot publish Splunk alert in MS Teams with T...

Re: Indexer cluster: "Oldest Data Age" extremely h...

Is that safe to delete old catalogues in /opt/splu...

Re: After upgrade we are receiving this error UFMA...

Re: How to resolve Indexer ERROR TcpInputConfig - ...

Re: Splunk Add-on for Java Management Extensions E...

Re: How to get Splunk alerting for Admins during o...

Re: Splunk crashes after error popped up: ENGINE: ...

Re: Splunk serverclass is uninstalling a deploymen...

Re: Getting syslog events from VMware ESXi: I don'...

Re: Incorrect sourcetypes for ESXi syslog

Re: Failed to replicate Streaming bucket