About performancemoni

performancemoni · ‎09-20-2022

Hello everyone, I am trying to create an add-on REST API with Splunk to download a .csv file from a confluence page and push the information to Splunk. I would like to have a table with the content of the .csv file, but so far I was just able to get one entry with 130 lines in the index and sourcetype defined by the add-on. It would help if I could split at the end of every line.. I've tried to define the sourcetype in porps.conf with line breaker and some other regex params. and at this point I don't know how to move on. Data sample: 1 UserID,Profile,Role1,Role2,Licenses,Programs,Domain,FieldX 2 user139,Profile description,Role 1,Role 2,"lic1,lic2,lic3,lic4,lic5,lic6",protram,domain,extra_field . . 130 user139,Profile description,Role 1,Role 2,"lic1,lic2,lic3,lic4,lic5,lic6",protram,domain,extra_field I've tried to reproduce the content as best as I could, included spaces. The content between the quotes "", all belong to the header "licenses".

performancemoni · ‎11-04-2020

Thank you for your answer ! I agree that UFs->IDX is usually the optimal solution, I should have mentioned that we have constraints that make the HF necessary. We don't want to manage N network flows between the forwarders and the Indexer, and the data should not reach the Indexer without being fully anonymized first. About the indexing on the Heavy Forwarder, this is not really long term, we would just like to enable indexing for specific data sources temporarily to test things locally, and then we would disable indexing and open forwarding again for these data sources like normal. The goal is not to make this HF an Indexer. It is true that our current setup isn't optimal at all, but we would not be able to get additonal Splunk machines fast enough to answer the need... So that's why I was wondering if a "quick and dirty" solution like the one in my first message could work right now.

performancemoni · ‎11-03-2020

Hello, I have read the documentation on routing and filtering events (https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Routeandfilterdatad), but can you tell me if I'm going in the right direction ? Here is my scenario: I have 1 Heavy Forwarder that receives events from multiple Universal Forwarders, the HF then forwards the events to an Indexer. UFs -> HF -> IDX There is a new data source that we want to collect from one of the Universal Forwarders, but we don't want this data to be sent to the Indexer yet. We would like to index it on the Heavy Forwarder first because we need to test out some data anonymization on the events before sending them to the Indexer. All other data collected by the Universal Forwarders should still be sent to the Indexer as usual. Our HF is configured with a tcpout defaultGroup pointing to the Indexer. What I understand is that I have to configure this on my Univeral Forwarder for the input i want to index locally: [monitor:///mydata/source1.log] _INDEX_AND_FORWARD_ROUTING=local ... If I understand correctly: these events will be indexed locally on the HF, but they will also be forwarded to the Indexer because of the defaultGroup. Would it be a good solution to use props/transforms on the HF to change the _TCP_ROUTING of these events (to give them an incorrect output group like "NoForward" for instance) ? Or is there a another/simpler solution ? Thank you !

performancemoni · ‎08-28-2020

We managed to fix the issue by changing the ulimits in /etc/security/limits.conf: * hard core 0 * hard maxlogins 10 * soft nofile 65535 * hard nofile 65535 * soft nproc 20480 * hard nproc 20480 * soft fsize unlimited * hard fsize unlimited Then we stopped splunk, exited our SSH session, and after reconnecting via SSH and restarting Splunk it works like a charm, no more issues. So the problem was that our ulimits were not properly configured.

performancemoni · ‎08-25-2020

Splunk 7.3.3 was installed on the machine, we had to uninstall it and re-install it (same version). And yes we made sure to: stop splunk, uninstall it with rpm and remove everything from $SPLUNK_HOME afterwards, so there was no file left...

performancemoni · ‎08-24-2020

Hello, We are having an issue, we have uninstalled a previous Splunk instance on our linux server and now we are trying to re-install Splunk (version 7.3.3). After starting Splunk for the first time with "splunk start" there doesn't seem to be any error at first, we manage to go to the login web page but when we login we receive a 500 server error. And when we check the status of Splunk with "splunk status" we get the following message: Error encountered, failed to start pid_check.sh to validate splunkd.pid. errno=11 Failed to determine if splunkd 5292 was running. Can't run "btool web list settings --no-log": Resource temporarily unavailable Do you know how to solve the problem ? Or how to investigate further ? Here are some logs from splunkd.log: 08-24-2020 14:57:23.353 +0200 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 08-24-2020 15:00:00.003 +0200 INFO ExecProcessor - setting reschedule_ms=3599997, for command=python /data/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py 08-24-2020 15:08:17.884 +0200 WARN Thread - webui: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 62 threads active ((same logs on repeat...)) 08-24-2020 15:10:18.355 +0200 INFO ProcessTracker - Start process: type=SplunkOptimize idx=_internal procId=199 latency_sec=118.000 08-24-2020 15:10:18.357 +0200 INFO ProcessTracker - Start process: type=SplunkOptimize idx=_audit procId=200 latency_sec=116.000 08-24-2020 15:10:18.357 +0200 WARN ProcessTracker - executable=splunk-optimize failed to start reason='': Resource temporarily unavailable 08-24-2020 15:10:19.355 +0200 INFO ProcessTracker - Start process: type=SplunkOptimize idx=_introspection procId=201 latency_sec=116.999 08-24-2020 15:17:00.654 +0200 ERROR SearchProcessRunner - preforked search=0/3 on process=0/3 caught exception. completed_searches=0, process_started_ago=0.029, search_started_ago=0.028, search_ended_ago=0.000, total_usage_time=0.028 08-24-2020 15:17:00.654 +0200 ERROR SearchProcessRunner - preforked process=0/3 died on exception: Main Thread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 4 threads active 08-24-2020 15:23:44.592 +0200 ERROR SearchProcessRunner - Error reading from preforked process=0/5: Connection reset by peer 08-24-2020 15:23:44.607 +0200 ERROR SearchProcessRunner - preforked search=0/4 on process=0/4 caught exception. completed_searches=0, process_started_ago=0.029, search_started_ago=0.027, search_ended_ago=0.000, total_usage_time=0.027 08-24-2020 15:23:44.607 +0200 ERROR SearchProcessRunner - preforked process=0/4 died on exception: Main Thread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 4 threads active It seems like there is problem with the threads maybe ? We have never encountered that before. Thank you vey much !

performancemoni · ‎08-19-2020

Hello, We are having some issues finalizing the installation of our Splunk environment. We have 2 Linux servers: 1 Search Head and 1 Indexer as search peer. We had just finished to set up the search peer in "Distributed search", so we tried to run a search "index=_internal sourcetype=splunkd" on the last 60 minutes but it only returned logs from the Indexer. But then we realized that TailReader-0 on the Search Head was in error "The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.". and the related messages such as: 08-19-2020 12:4650.607 +0200 WARN TailReader - Could not send data to output queue (parsingQueue), retrying... This is weird because we configured the outputs.conf on the Search Head to send data to the Indexerand we configured inputs.conf of the Indexer to receive data so we are not sure what's wrong. outputs.conf on the SH: [tcpout] defaultGroup=indexer [tcpout:indexer] server=indexer_hostname:9997 inputs.conf on the IDX: [splunktcp://9997] Both servers have been restarted, I guess the queues are full because the Search Head can't send the data, but why ? The port 9997 is open and the connection from SH to IDX is fine. Also we don't have any forwarder or data input configured so it should not be because of a sudden burst of incoming data. We restarted the Search Head after this, and now we are not able to run searches anymore, all searches return in error, the job inspector says "This search has encountered a fatal error and has been marked as zombied". Could it be a performance issue ? Our servers have only 4CPU and 12GB of RAM. Do we need more CPU to solve these issues ? Thank you very much !

performancemoni · ‎07-01-2020

Hello, We have remote linux servers with universal forwarders already installed by root, we are not admin of these servers and we don't even have a user for us to connect on these servers either. We would like to be able to access the servers as basic users, for instance with a "splunk" user, manage the forwarder configuration and use the splunk command lines (splunk start/restart/btool etc...) with this user, without requiring admin rights. How can we achieve that ? I guess we first need the "splunk" user to be created on the servers with a password so that we can at least access the machines. Then do we just need to change the boot-start configuration and the owner of the forwarder file system ? /opt/splunk/splunkforwarder/bin/splunk enable boot-start -user splunk chown -R splunk /opt/splunk/splunkforwarder And now we should be able to manipulate the forwarder as we want without admin rights, right ? We wish we had a spare linux server to test this on but sadly we don't have any means to do this... Thank you for your help !

performancemoni · ‎06-18-2020

Hello everyone, We have configured some automatic field extractions using regular expressions on some logs that can get really big. These field extractions are very important, if they fail we are missing critical information in our daily monitoring. At some point the field extraction didn't work and we realized that was because of the regex depth limit, when we ran it manually with rex and we got Streamed search execute failed because: Error in 'rex' command: regex="<the_regex>" has exceeded the configured depth_limit, consider raising the value in limits.conf. We fixed it by optimizing the regex and now it's working fine. But we cannot be sure that the issue has been absolutely fixed, it could potentially happen again in the future. We would like to configure a Splunk alert that warns us when this type of error occurs. Does Splunk log anything about this type of error ? We could not find anything in _internal or anywhere else, or maybe we didn't look correctly ? Thank you for your help !

performancemoni · ‎05-12-2020

Are you sure about this ? Is this behavior documented somewhere ? I just tested the following: Configured a tcpout group that refers to a non existing indexer [tcpout:dummy_output_group] server=dummy_indexer:9997 Configured an input sending the data to a group of existing indexer (indexer) and to the dummy_output_group [monitor:///splunk/etc/apps/test_app_data_forwarding/test_file.txt] index = main sourcetype = test_data_forwarding _TCP_ROUTING = indexer, dummy_output_group I refreshed the configuration of the Splunk server, it took into account the new dummy output group. And the group indexer did index correctly all the events from the test file (even beyond the max queue size of the dummy output group that started to drop events). Doesn't this test represent a case where one output group is unavailable ? Yet the data is collected by the other group. Am I missing something ?

performancemoni · ‎05-12-2020

Your comment is interesting, it might not answer my specific question but maybe it raises another issue that we didn't anticipated. So wait, is this true even if the group indexer1 has acknowledgement enabled and group indexer2 has the default setting (acknowledgment disabled) ? And you mean that if both groups are down, data forwarding will only restart when both groups are up (at least one indexer available in each group) ? I would have thought that if the group indexer2 doesn't care about acknowledgement, data will still be forwarded to it anyway, even if the the group indexer1 is unavailable.

performancemoni · ‎05-11-2020

Hello, We have a question regarding a specific use case of data forwarding, we would like to know if there is a risk with the situation. Let's say we have two Splunk platforms, one has a set of indexers (indexer1) that stores data for a specific use case, the other platform has also its indexer (indexer2) for another use case. At some point, these 2 platforms will have to collect data from the same machines, not necessarily the same data, but it could be. So there is a forwarder on a remote machine, let's say we have to collect the same file and forward it to both platforms. We are using the data cloning technique, with for example: In inputs.conf [monitor://file_path] index = … sourcetype = ... _TCP_ROUTING=indexer1,indexer2 Or with outputs.conf [tcpout] defaultGroup=indexer1,indexer2 Or by using props/transforms to alter the routing of the events on an intermediate Heavy Forwarder, whatever (please, do tell if there is a significant difference with any of these methods for this situation). Now the question is: In general, is there a risk that one of the two platforms will stop receiving events if the other is down, depending on the configuration of the indexers/forwarders ? We have heard of the concept of indexer acknowledgment, we are not sure if it can have any impact on this situation. For example, if the group indexer1 is configured with acknowledgement enabled, is there any risk that the group indexer2 won't receive data when indexer1 is not acknowledging the reception ? This topic is a little bit confusing for us, we have heard claims that the data forwarding could be blocked if another platform needs to receive the data and one of the indexer is down, but it doesn't seem right. We just want to clarify, with the set up described above, if there would be any issue. Thank you very much for your help Update: We learned about the parameters in outputs.conf to consider when configuring the behavior of the queues: - dropEventsOnQueueFull = - dropClonedEventsOnQueueFull = - blockOnCloning = It is indeed possible to block the data collect of a splunk instance (in a data cloning configuration) when not paying attention to these parameters. In the little test we did, the default value of dropClonedEventsOnQueueFull made it so that the data collect didn't block. However we have to watch out for dropEventsOnQueueFull as well, which can cause data forwarding issues when a splunk instance is unavailable (with default value) => But it also depends on whether you accept the loss of data or not in your deployment. Very interesting parameters to know about.

performancemoni · ‎03-20-2020

Hello, We have a weird warning on the integrity of the file etc/users/users.ini, if we look at the file, it contains: [contains-uppercase] user@domain> =user@domain_.5ec8143af5fe7888fddd625f69d704b3 It might have occured when we created a user and made a mistake with their username, we used their email address as username but we forgot to remove the ">" at the end. But anyway, now we have this stanza with this entry and a warning on file integrity. We deleted the user and recreated it with the correct username "user@domain". The file is still the same... We checked on other Splunk instances where we don't have the integrity problem and the file users.ini is always empty. So we tried to remove the stanza from the file to make the file empty but when we restart splunk it tells us that the file is missing a stanza... In the end we re-added the content of the file, just in case, to not have an error on restart. This file doesn't seem to have an impact on Splunk (?), but how can we remove or fix this integrity warning ? Thank you for your help

performancemoni · ‎12-02-2019

Hello, We would like to use Enterprise SSO to authenticate the users on one of our Splunk platforms. On another platform we had used the PingFederate solution so we know how to configure this type of SSO, but we were wondering if Enterprise SSO was compatible with Splunk and if the configuration is different. We had a look at the web documentation (https://docs.splunk.com/Documentation/Splunk/8.0.0/Security/HowSAMLSSOworks) but I don't know if anything refers/applies to Enterprise SSO, can you help ? (The target Splunk platform will be in a version >7.2.1) Thank you very much

Posts	21
Solutions	1
Karma Given	1
Karma Received	2
Member Since	‎06-17-2019

Online Status	Offline
Date Last Visited	‎09-20-2022 11:39 AM

Splunk Add-On builder for import of .csv file form...

DB Connect v3 MS-SQL Server: driver is not configu...

Selective indexing of events from UF on Heavy Forw...

Splunk server error and failed to start pid_check....

Why are the queues full and searches in error on S...

Changing the owner of the splunk universal forward...

Can we be alerted when a field extraction fails be...

Any data forwarding issue using data cloning and d...

File etc/users/users.ini presenting integrity chec...

Is Splunk compatible with Enterprise SSO ?