About kaurinko

kaurinko

Hi, I just upgraded splunk to 9.0.0 and realized the log ~/var/log/splunk/splunkd.log started to get populated with messages like 06-14-2022 16:41:00.924 + 0300 ERROR script [ 2047436 SchedulerThread ] - Script execution failed for external search command ' runshellscript '. 06-14-2022 16:41:00.906 + 0300 WARN SearchScheduler [ 2047436 SchedulerThread ] - addRequiredFields: SearchProcessorException is ignored , sid=AlertActionsRequredFields_1655214060.1451 , error=Error in ' script ' : Script execution failed for external search command ' runshellscript '. The above comes to the logs regardless of whether the alert has been fired or not, and we rely quite heavily on running external scripts to make external systems aware of problems. I thought, now all our script bindings to alerts are broken and we must do a rollback. However, I tested and the scripts were executed nicely. My question is, what has changed here, if anything? I would like to get rid of those messages cluttering the logs in vain. An the other things is, if something else really has changes, what should I do to make splunk happy about the scripts in alerts? I am looking for something else than "Please write a Python script to do the job." Any clues?

kaurinko · ‎06-24-2021

Hi, I just realized a problem that had surfaced with the installation of Splunk v. 8.2.0. I have a number of alerts executing external scripts in ~splunk/bin/scripts and this system has worked fine fine for years. Yesterday I realized the scripts were being executed, but at the end of the chain the Perl-scripts tries to execute a system-command, which simply fails with return code 134. Naturally I could execute the scripts interactively as splunk user without trouble. Only executing them from splunkd would fail. After hours of headbanging I had to work around the system commands, and write the commands to a queue and separately handle the queue with an external system. Now it works, but I find it disturbing, that all of a sudden something that has worked fine starts to fail without a clear reason. Time history associates the problems to the installation of Splunk v. 8.2.0. I won't bother filing a bug report, because it would be impossible for me to show beyond reasonable doubt that it is the Splunk 8.2.0 that actually is broken. This is just one of those issues of mixing Python, bash, Perl and se-linux to name a few well known candidates to blame. It is always someone else's fault. Anyone else experiencing similar issues? It would be delightful to avoid fighting this problem again at some other point. Br, Petri

kaurinko · ‎05-18-2021

Hi @potnuru , Based on the log there most probably already is a process listening to 127.0.0.1:9998. Try issuing the following (as root to get the process name): netstat -lntp | grep 9998 The error message Caused by: java.net.BindException: Address already in use means some other process is already listenining to the address:port you are trying to bind to. Best regards, Petri

kaurinko · ‎11-30-2020

@dlpco Try downgrading the UF to version 8.0.1. It solved my problems. I also noticed, that all the logs were not transported to the indexer during the error messages. With UF v. 8.0.1 normal operation was resumed. Br, Petri

kaurinko · ‎11-25-2020

Hi, I noticed by accident I also had this problem, and it all started when I upgraded the Universal Forwaders to version 8.1.0. Now I downgraded all UFs back to version 8.0.1 and the problems disappeared. The Indexer version was originally when the problems started to appear 8.1.0, and right now it is 8.1.0.1. For me this was not only an annoyance. I first found the problem, when I realized some logs were not transferred to the indexer. Some trouble shooting took me to the UF version issue. In my opinion this is clearly a bug, and it should be fixed. Best regards, Petri

kaurinko · ‎11-03-2020

Hi, I bumped into this thread while looking for something similar. Thanks to your groundwork I came to a solution as follows: my search | stats count AS Usage by USERID | stats dc(USERID) AS Users by Usage What I would like to do is to get some sort of summary where Usage would be grouped by e.g. 0-10, 11-100,101- where the limits would of course be something depending on the problem at hand. Any ideas how that might be accomplished?

kaurinko · ‎10-21-2020

Hi, I just upgraded our Splunk server to 8.1.0 and after a while realized some of our good old searches utilized in a couple of dashboards had stopped working. After a lot of digging around, it seems like the problem has been narrowed down to tags not working as before in 8.0.6. The code below will serve as an example. index="myindex" summaryline | stats count(eval(STATUS_CODE=10 OR STATUS_CODE=42)) AS "No-go" count(eval(STATUS_CODE=50)) AS OK by tag::CUSTOMER_CODE The ides is, that summaryline is just a keyword to match the appropriate lines of data. The lines will populate the fields CUSTOMER_CODE and STATUS_CODE, and tag::CUSTOMER_CODE is defined based on the value of CUSTOMER_CODE under Settings > Tags. Running the above in Verbose mode works, even though previously I didn't need to include CUSTOMER_CODE in the fields list. Running in Fast mode will produce nothing for the same data but results will be shown, if statistics are calculated "by CUSTOMER_CODE". For this reason, the dashboard using the above construct stopped producing the data we wanted. Looking at the problem, there is an obvious workaround to use lookup, and the example search looks like this index="myindex" summaryline | lookup CustomerLookup.csv CUSTOMER_CODE OUTPUT CUSTOMER AS Customer | stats count(eval(STATUS_CODE=10 OR STATUS_CODE=42)) AS "No-go" count(eval(STATUS_CODE=50)) AS OK by Customer CustomerLookup.csv will then hold columns CUSTOMER_CODE, CUSTOMER and possibly others separated by commas. I tested, and I could not output the CUSTOMER column as tag::CUSTOMER, or no results were produced. Anyone else with this kind of experiences and is there a way to make tags work again like they used to?

kaurinko · ‎10-15-2020

Hi, Thanks for asking. Your problem looks interesting. Unfortunately I have never worked with anything like that, but on a rainy day I might experiment with some ideas. Right now I can't contribute, sorry.

kaurinko · ‎09-24-2020

Hi, Thanks for the suggestion. In the actual subsearch I am trying to use the result is passed with a $Break, not a bare Break. That returns in my test "Yes" and the match should return true and the IsBreak should be set to 1. Looking at your suggestion, I have a hard time identifying how would it actually change the value of Break and ever return anything else than IsBreak=0. Best regards, Petri

kaurinko · ‎09-24-2020

Hi, What I am trying to do, is to determine from a lookup table whether we have a maintenance window active in order to effectively disable a number of alerts. Excluding the log lines from the searches is not an option, because the alerts will interpret that as an error situation since the successfull cases would be missing. I already have a lookup table containing the start and end times for the maintenance windows. The following produces promising results: | inputlookup maintenancetimes.csv | convert timeformat="%Y/%m/%d %H:%M:%S %p" mktime(MaintStart) mktime(MaintEnd) | eval Break=if( now() > MaintStart AND now() < MaintEnd, "Yes", "") | sort -Break | return 500 Break The result is (Break="Yes") OR (Break="") Which I interpret as presence of both active and inactive maintenance windows. However, when I am trying to use the data from a subsearch, it isn't doing what I want. | makeresults count=2 annotate=true | eval IsBreak=if(match([ | inputlookup maintenancetimes.csv | convert timeformat="%Y/%m/%d %H:%M:%S %p" mktime(MaintStart) mktime(MaintEnd) | eval Break=if( now() > MaintStart AND now() < MaintEnd, "Yes", "") | sort -Break | return 500 $Break ],"Yes"),1,0) | table IsBreak _time The results show 0 as the value of IsBreak, and I can't figure out why. The intention is of course to utilize this as a part of a more complicated search/alert. What am I doing wrong? Best regards, Petri

kaurinko · ‎07-06-2020

Hi @potnuru , The configurations in etc/system/local are as follows: inputs.conf: [default] host = splunk.tupislab.net [splunktcp:9997] [splunktcp-ssl:9998] disabled=0 [SSL] serverCert = /opt/splunk/etc/auth/My-CA/SplunkIndexerCerts.txt requireClientCert = true sslVersions = tls1.2 and server.conf: [general] serverName = splunk.mydomain.net pass4SymmKey = *** [sslConfig] sslPassword = ***** sslRootCAPath = /opt/splunk/etc/auth/My-CA/CACertificates.txt serverCert = /opt/splunk/etc/auth/My-CA/SplunkIndexerCerts.txt [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial quota = MAX slaves = * stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free quota = MAX slaves = * stack_id = free [license] master_uri = https://license.tupislab.net:8089/ I hope this helps you. Best regards, Petri

kaurinko · ‎06-04-2020

Dear Sirs, I am using lookup to enrich my event data on the fly, and it seems to work fine. However, every invocation of the function lookup produces a warning to the splunkd.log as follows: 06-04-2020 18:39:27.136 +0300 WARN CsvDataProvider - Unable to find filename property for lookup=splunk-installation-info.csv will attempt to use implicit filename. The search producing the above is | rest splunk_server=lic /services/licenser/slaves | lookup splunk-installation-info.csv splunk_uuid AS title OUTPUT BU It works, but I am puzzled by the warning. I would like to get rid of it, so how do I tell Splunk the explicit filename? Best regards, Petri

kaurinko · ‎04-30-2020

I can confirm, that this only happens for source based field extractions. Ones with sourcetype-based searches are not affected.

kaurinko · ‎04-30-2020

I have the same problem with 8.0.1. Would be interested to know if there is a solution.

kaurinko · ‎03-09-2020

I'll think about that. Thanks for the conversation.

kaurinko · ‎03-09-2020

I guess you got mixed somewhere writing your comment, or I can't follow the logic "CRL signing by browser" doesn't make any sense as CRLs are signed by CAs. Anyhow, my guess is Splunk isn't interested in CRLs as I haven't found anything about them in the documentation. The other thing I can't find mentioned is validity period checking, so it looks like the certificates are treated as never expiring certificates. Despite all that, I still think setting up certificates for the connections is worth the trouble. And maybe some day there will be better validation.

kaurinko · ‎03-09-2020

I think it is the responsibility of any relying party to proper validation. That would include checking that the certificate is valid, signed by the issuer, not revoked and the issuer certificate is also valid. That last step adds recursion to the process, but most chains are not that long. I would say that the browser should be interested in validating the server certificate, but if a client certificate is requested and presented, it would be the server's duty to validate it.

kaurinko · ‎03-09-2020

Hi, That nailed it, thanks! I had to add the Root-CA certificate of the splunkweb-cert to the file pointed at by sslRootCAPath and obviously all the other trusted Root-CAs. The documentation for this part is confusing. With certificates the keyword I am looking for is trust. The documentation is not clear about how to define the trusted CAs (trust anchors) for different purposes. Actually the wording lets one believe that there can only be one trusted CA, which would be very strange. Now that I know how things work, I can understand it, but it should have been the other way around. Do you know anything about doing proper certificate validation with CRLs or an ocsp in Splunk? /Petri

kaurinko · ‎03-09-2020

Hi! I have worked for a while to make Splunk use TLS and PKI as much as possible. At present the system contains of version 8.0.1 components only. I have managed to get Splunk Indexer to require client certificate from the UFs, and it seems to work. The Splunk serverCert is in a file containing the certificate, private key, issuer certificate and the root-CA certificate (the issuer of the issuer certificate). The trusted root-CA -certificates are in a separate text file. For this connection requiring requireClientCert = true works fine. With the web-UI things are not going as elegantly. My web.conf looks like this: [settings] enableSplunkWebSSL = 1 privKeyPath = etc/auth/splunkweb/splunk.pki.key serverCert = etc/auth/splunkweb/splunk.pki.txt requireClientCert = false sslVersions = tls1.2 loginBackgroundImageOption = none login_content = This is a <b>Test installation</b>. With the above, things are just fine. Changing requireClientCert to true breaks everything and the webGUI is not started. The splunkd.log gets populated with lines like: 03-09-2020 12:45:04.223 +0200 ERROR X509Verify - X509 certificate (CN=Root CA,O=X,C=Y) failed validation; error=19, reason="self signed certificate in certificate chain" I get exactly the same error message, if I connect using openSSL to the indexer port, where UFs connect: openssl s_client -connect splunk:9998 -state -prexit * Certificate chain 0 s:/C=Y/O=X/CN=Test Splunk Indexer i:/C=Y/O=X/CN=TestCA-1 1 s:/C=Y/O=X/CN=TestCA-1 i:/C=Y/O=X/CN=Root CA 2 s:/C=Y/O=X/CN=Root CA i:/C=Y/O=X/CN=Root CA * SSL-Session: Protocol : TLSv1.2 * Verify return code: 19 (self signed certificate in certificate chain) My point here is that the connection between UF and indexer still works fine. The question is: Is requireClientCertificate simply not supported in the web-GUI, or is there something in the documentation I have not understood correctly? If it is possible to require a certificate from the client (i.e. a web browser), is there a way to define the trusted CA-certificates and should any intermediate CA certifictes be included as well? Another thing I have been wondering about is certificate validation and CRLs. Is there a way to make Splunk actually validate the certificates it is presented? Best regards, Petri

kaurinko · ‎12-24-2019

It seems like I had a stupid error in my configuration. I added [splunktcp:9997] [splunktcp-ssl:9998] disabled=0 before the [SSL] stanza and left out the sslRootCAPath configuration entry. Now I get two listeners and I get a proper SSL/TLS handshake from port 9998.

Posts	33
Solutions	2
Karma Given	0
Karma Received	5
Member Since	‎03-26-2015

Online Status	Offline
Date Last Visited	2 weeks ago

Splunk started complaining "Script execution faile...

Problems executing scripts after an alert has been...

Splunk 8.1.0 tags are unreliable

How to pass result from subsearch to main level

CsvDataProvider - Unable to find filename property...

SSL/TLS with requireClientCert in web.conf fails

How to set up SSL/TLS for the Splunk indexer

How to exclude multiple time ranges from multiple ...

How can I combine several transactions into one se...

How to limit the number of result lines from stats

Splunk started complaining "Script execution faile...

Problems executing scripts after an alert has been...

Re: How to set up SSL/TLS for the Splunk indexer

Re: Universal Forwarder - Repeating message TcpOut...

Re: Universal Forwarder - Repeating message TcpOut...

Re: How to generate a count of occurrences of dist...

Splunk 8.1.0 tags are unreliable

Re: Splunk Dashboard to accept Input Data from the...

Re: How to pass result from subsearch to main leve...

How to pass result from subsearch to main level

Re: How to set up SSL/TLS for the Splunk indexer

CsvDataProvider - Unable to find filename property...

Re: Possible bug with changing permission on sourc...

Re: Possible bug with changing permission on sourc...

Re: SSL/TLS with requireClientCert in web.conf fai...

Re: SSL/TLS with requireClientCert in web.conf fai...

Re: SSL/TLS with requireClientCert in web.conf fai...

Re: SSL/TLS with requireClientCert in web.conf fai...

SSL/TLS with requireClientCert in web.conf fails

Re: How to set up SSL/TLS for the Splunk indexer