About vzabawski

vzabawski · ‎09-15-2022

"Too little too late", but hope it might lead someone to the solution. When I was setting up some backup client to work with Openstack Swift, there was a parameter s3.path.style.access: true Without this parameter, connection to Swift didn't work properly. Maybe it's the same case, since googling this error message lead me to cases when paths in S3 were malformed.

vzabawski · ‎09-14-2022

GUI doesn't support deleting users created with SAML. I can edit internal users. I guess it's an expected, yet inconvenient behavior from Splunk's side.

vzabawski · ‎09-01-2022

I don't know what the exact problem was, but the root cause of it was a cloned search app. User's alert was created in a separate app which was cloned from Splunk 7.0.0 search app, so it contained default with configs from that version of Splunk. After I've upgraded Splunk to 8.2, this legacy configuration started causing troubles because Splunk 7 and 8 are very different. Solution: I've deleted default dir from the app. Another step was to copy $SPLUNK_HOME/etc/apps/search/default/data/ui/nav/default.xml to $SPLUNK_HOME/etc/apps/<yourapp>/local/data/ui/nav/default.xml. That allowed me to preserve tabs from the search app.

vzabawski · ‎08-10-2022

Unfortunately, no. There are no errors or any other records close to the log records from the splunkd.log Closest records are 2022-08-10 13:02:34,782 and 2022-08-10 14:01:08,141. They don't seem to be related.

vzabawski · ‎08-10-2022

I have an alert configured in Splunk which should send the email when the alert is triggered. The alert is being added to the list of triggered alerts, but the email is not being sent. In logs I see the following errors: 08-09-2022 13:30:14.510 +0200 ERROR ScriptRunner [26046 AlertNotifierWorker-0] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/mycommunity/bin/se ndemail.py "results_link=https://mysplunk.example.com/app/mycommunity/@go?sid=sc heduler__d051859__mycommunity__RMD540c2da3bac08625c_at_1660044600_69194" "ssnam e=MYBLOG PROD Broken Requests" "graceful=True" "trigger_time=1660044614" result s_file="/opt/splunk/var/run/splunk/dispatch/scheduler__d051859__mycommunity__RM D540c2da3bac08625c_at_1660044600_69194/results.csv.gz" "is_stream_malert=False"' : File "/opt/splunk/etc/apps/mycommunity/bin/sendemail.py", line 111 08-09-2022 13:30:14.510 +0200 ERROR ScriptRunner [26046 AlertNotifierWorker-0] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/mycommunity/bin/se ndemail.py "results_link=https://mysplunk.example.com/app/mycommunity/@go?sid=sc heduler__d051859__mycommunity__RMD540c2da3bac08625c_at_1660044600_69194" "ssnam e=MYBLOG PROD Broken Requests" "graceful=True" "trigger_time=1660044614" result s_file="/opt/splunk/var/run/splunk/dispatch/scheduler__d051859__mycommunity__RM D540c2da3bac08625c_at_1660044600_69194/results.csv.gz" "is_stream_malert=False"' : except Exception, e: 08-09-2022 13:30:14.510 +0200 ERROR ScriptRunner [26046 AlertNotifierWorker-0] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/mycommunity/bin/sendemail.py "results_link=https://mysplunk.example.com/app/mycommunity/@go?sid=scheduler__d051859__mycommunity__RMD540c2da3bac08625c_at_1660044600_69194" "ssname=MYBLOG PROD Broken Requests" "graceful=True" "trigger_time=1660044614" results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__d051859__mycommunity__RMD540c2da3bac08625c_at_1660044600_69194/results.csv.gz" "is_stream_malert=False"': ^ 08-09-2022 13:30:14.510 +0200 ERROR ScriptRunner [26046 AlertNotifierWorker-0] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/mycommunity/bin/sendemail.py "results_link=https://mysplunk.example.com/app/mycommunity/@go?sid=scheduler__d051859__mycommunity__RMD540c2da3bac08625c_at_1660044600_69194" "ssname=MYBLOG PROD Broken Requests" "graceful=True" "trigger_time=1660044614" results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__d051859__mycommunity__RMD540c2da3bac08625c_at_1660044600_69194/results.csv.gz" "is_stream_malert=False"': SyntaxError: invalid syntax 08-09-2022 13:30:14.512 +0200 ERROR script [26046 AlertNotifierWorker-0] - sid:scheduler__d051859__mycommunity__RMD540c2da3bac08625c_at_1660044600_69194 External search command 'sendemail' returned error code 1. . The error message is very ambiguous, so I don't know what causes this error. It seems like a bug to me. Any ideas? Environment: Splunk 8.2.7 OS: SLES 15 SP3

vzabawski · ‎06-30-2022

Token authentication mechanism kind of works in parallel with SAML, so it requires SAML Attribute Query support in order to retrieve the information about group membership. Without AQR, this can be done with a script which extends Splunk auth and retrieves the information about group membership on its own, without AQR. You have 3 possible options: 1. Use identity provider which supports Attribute Query (AQR) 2. Use Azure or Okta since Splunk has auth extensions for them out of the box 3. Create your own authentication extension. If I'm not mistaken, Splunk cloud doesn't support auth extensions, so option 3 might be not applicable to your case.

vzabawski · ‎06-30-2022

Internal users co-exist with your authentication mechanism without any issues. Have been using internal users with LDAP and SAML. You just need to add en-US/account/login?loginType=Splunk to your Splunk url in order to log in with the internal user.

vzabawski · ‎06-30-2022

Authentication tokens are supported with SAML, internal and LDAP authentication mechanisms. However, for SAML, your identity provider needs to support AQR (Attribute Query) or have a custom authentication extension. Splunk provides custom authentication extension out of the box for Okta and Azure. Source: https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Security/Setupauthenticationwithtokens#Supported_authentication_schemes

vzabawski · ‎05-24-2022

Splunk 8.2.5, but I'm getting this alert starting from Splunk 8.1.6 (if I'm not mistaken).

vzabawski · ‎05-24-2022

Yes, I've upgraded from 7.x to 8.2.x and after that I've started receiving those notifications. I've measured iops and seems like everything is fine: fiotest: (groupid=0, jobs=1): err= 0: pid=29: Tue May 24 13:40:16 2022 read: IOPS=1796, BW=7186KiB/s (7358kB/s)(6141MiB/875157msec) bw ( KiB/s): min= 5584, max= 9976, per=100.00%, avg=7192.71, stdev=565.38, samples=1748 iops : min= 1396, max= 2494, avg=1798.08, stdev=141.33, samples=1748 write: IOPS=599, BW=2400KiB/s (2457kB/s)(2051MiB/875157msec); 0 zone resets bw ( KiB/s): min= 1888, max= 2885, per=100.00%, avg=2401.51, stdev=73.89, samples=1748 iops : min= 472, max= 721, avg=600.34, stdev=18.47, samples=1748 cpu : usr=0.89%, sys=2.59%, ctx=533118, majf=0, minf=6 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64

vzabawski · ‎05-24-2022

I'm using Azure Premium SSD which should have max IOPS of 20,000, according to the documentation, but I'll run a test to see some real-life results.

vzabawski · ‎04-01-2022

This issue is caused by wrong home directory set for "splunk" user. I've compared the output of echo $HOME on healthy and this server I'm having troubles with and on "healthy" server HOME is set to /opt/splunk. On affected server HOME is set to /opt/splunk/splunk, so that's what was causing problems.

vzabawski · ‎03-30-2022

Good idea, thanks! SPLUNK_HOME looks fine, so I guess it's not the case.

vzabawski · ‎03-30-2022

Hello. I'm trying to view Splunk configuration, but getting a very odd error: splunk@test1:/> /opt/splunk/bin/splunk show config authentication Splunk username: admin Password: Can't create directory "/opt/splunk/splunk/.splunk": No such file or directory Changing dir to /opt/splunk/bin/ and running command from it also doen't help. Doesn't matter if I run the command under splunk user (this user owns files in /opt/splunk) or with sudo. This Splunk instance was upgraded from 7.0.0 to 8.1.5 and then to 8.2.5, so maybe it affected somehow. How can I fix this?

vzabawski · ‎11-08-2021

To whoever might find this interesting. I've recently encountered with such issue after installing Dynatrace OneAgent chart in the same k8s cluster with Splunk. In my case I wasn't able to just delete the liboneagentproc.so file, so I had to uninstall Dynatrace chart and then delete /opt/oneagent directory in pod where Splunk runs. Link to upgrade notes: https://docs.splunk.com/Documentation/Splunk/8.2.1/ReleaseNotes/Knownissues

vzabawski · ‎09-15-2021

Thanks for your reply. What should I do if it isn't an indexer cluster? Those indexers are standalone, so there's no replication going on, and no indexer master is present. Anyway, I think restarting indexers might be a good idea, so I'll try it.

vzabawski · ‎09-14-2021

Hello everyone. I'm getting Forced bundle replication failed. Reverting to old behavior - using most recent bundles on all on a search head, and I'm not sure how to fix this. I excluded heavy files from the bundle, also restarted the search head, but nothing changes. Where should I dig? I wasn't able to find this error message in Splunk documentation and on the internet. The closest topic on Splunk answers was related to search head clustering, but since I wasn't setting up SH clustering, I guess it's not applicable. Additional info. Before the issue occurred, I've noticed that disk usage on indexers went to 100%. I solved it by deleting data from /opt/splunk/var/run/searchpeers (except the latest files). My environment: - 4 indexer VMs. - 2 search head VMs (not clustered, just testing Splunk 7 and Splunk 8 in parallel). 4 indexers are connected as distributed search peers to each of those search heads. - No deployment server in use. Sometimes network connection is not good between indexers and search head, so maybe it contributes somehow. Any suggestions and ideas appreciated.

vzabawski · ‎07-07-2021

I don't have much experience with running Splunk 8 on SLES 15 for a long time, but it starts and works there, so I assume there should be no problems. Splunk is a pretty independent tool, so all you need to worry about is kernel version, I guess. https://docs.splunk.com/Documentation/Splunk/8.2.1/Installation/Systemrequirements

vzabawski · ‎10-14-2020

Worked for me. Also, you can reload only savedsearches.conf, without restarting Splunk with this: https://spliunk:8080/en-US/debug/refresh?entity=admin/savedsearch

vzabawski · ‎09-07-2020

I have same problem with Splunk 7.0. Let's assume there's a role "my_role". In my case that role had inherited role "power" and that was the problem. After switching from "power" to "user", the restriction worked.

vzabawski · ‎03-27-2020

I have Splunk 7 with SSO enabled. Is there any way to extract certain value remoteUser header? For example, I want to extract from header with content "CN=cnvalue,O=MYORG,C=US" the value of CN, so Splunk wiil use only "cnvalue" as remote user. I know this can be done by Apache, but is there any way to do it on Splunk's side?

Posts	21
Solutions	3
Karma Given	17
Karma Received	7
Member Since	‎02-26-2020

Online Status	Offline
Date Last Visited	‎04-18-2023 05:49 AM

Sendmail throws SyntaxError when sending email as ...

Why the error "Can't create directory" when runnin...

Forced bundle replication failed. Reverting to old...

Parsing remoteUser header value

Re: SmartStore OpenStack Swift configuration isn't...

Re: How to remove users from Splunk registered by ...

Re: Sendmail throws SyntaxError when sending email...

Re: Sendmail throws SyntaxError when sending email...

Sendmail throws SyntaxError when sending email as ...

Re: Creating Tokens in Splunk via GUI

Re: Creating Tokens in Splunk via GUI

Re: Creating Tokens in Splunk via GUI

Re: Health status of IOWait is read!!

Re: Health status of IOWait is read!!

Re: Health status of IOWait is read!!

Re: Why the error "Can't create directory" when ru...

Re: "Can't create directory" error when running "s...

Why the error "Can't create directory" when runnin...

Re: Terminated with signal 4 (core dumped) when up...

Re: Forced bundle replication failed. Reverting to...

Forced bundle replication failed. Reverting to old...

Re: Splunk Enterprise SLES compatibility

Re: Error while deleting an alert

Re: Restrict searches for a role using search term...

Parsing remoteUser header value