About edoardo_vicendo

edoardo_vicendo · ‎03-30-2022

Same here, the UF was working fine on a Windows server than suddenly started to not ingest the file anymore. To see the error I had to put in debug mode the universal forwarder. The strange thing is that other files are still monitored and forwarded correctly. I have even added followSymlink=false and CHECK_METHOD=modtime without any luck #inputs.conf [monitor://D:\path\to_*\file-*_*.csv] disabled=0 index=myindex sourcetype=mysourcetype crcSalt = <SOURCE> followSymlink=false #props.conf [source::D:\path\to_*\file-*_*.csv] CHECK_METHOD=modtime

edoardo_vicendo · ‎12-30-2021

Hello, We have IBM VIOS servers running AIX and we need to monitor them, mainly in term of Security. Is there anyone having experience on that? Did you installed a Splunk Universal Forwarder or are you sending data out via syslog? Thanks a lot, Edoardo

edoardo_vicendo · ‎12-16-2021

Unfortunately old stile, less/tail etc... Not the best way but currently the only solution as HTTP output does not allow to split the output, it is all or nothing

edoardo_vicendo · ‎12-16-2021

Same issue, did you were able to solve it? 12-16-2021 16:23:59.872 +0100 ERROR S2SOverHttpOutputProcessor [1631141 parsing] - HTTP 502 Bad Gateway

edoardo_vicendo · ‎12-16-2021

@richgalloway :Thanks for your feedback, initially I though about this solution but wanted to have something "more robust" that does not depend from manual instruct which are the inputs to exclude. By the way you are right, the transforms.conf does not apply on the UF, I'll try with your suggestion.

edoardo_vicendo · ‎12-16-2021

Because this Splunk UF will receive specific data and have to forward only them out in HTTP

edoardo_vicendo · ‎12-15-2021

Hello, Due to a specific requirement we have to install a Splunk Universal Forwarder acting as "intermediate forwarder". Basically it will receive data via TCP (to leverage persistent queue), and it has to forward them in output in HTTP. Forwarding data in HTTP is possible since Splunk Universal Forwarder 8.x: https://docs.splunk.com/Documentation/Forwarder/8.2.3.1/Forwarder/Configureforwardingwithoutputs.conf#Configure_the_universal_forwarder_to_send_data_over_HTTP Here the set-up: # inputs.conf [tcp://9997] persistentQueueSize=1000MB connection_host=none disabled=false # outputs.conf #Example from Splunk [httpout] httpEventCollectorToken = eb514d08-d2bd-4e50-a10b-f71ed9922ea0 uri = https://10.222.22.122:8088 What we also want to achieve is to forward only data received via TCP, and to do not forward the Splunk UF internal logs. I didn't found a sort of _HTTP_ROUTING setting (like for example _TCP_ROUTING) to be put in inputs.conf Therefore listing all the Splunk UF inputs with that command: /opt/splunkforwarder/bin/splunk btool inputs list --debug I was thinking about this configuration: #props.conf [source::/opt/splunkforwarder/...] force_local_processing = true TRANSFORMS-null = setnull #transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue Do you think it is going to work? Maybe another option could be tag TCP inputs host based on DNS or IP, and then move to nullQueue all the logs produced by the Splunk UF: #inputs [tcp://9997] persistentQueueSize=1000MB connection_host=dns disabled=false #props.conf [host::mysplunkUFhostname] force_local_processing = true TRANSFORMS-null = setnull #transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue Do you see any other possible configuration? Thanks a lot, Edoardo

edoardo_vicendo · ‎12-03-2021

Hi All, I had this error at it took a while to understand and fix it. Here my environment: Splunk 8.0.5 Splunk DB Connect 3.6.0 Java /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-0.el6_10.x86_64/jre/bin/java Red Hat Enterprise Linux Server release 6.10 (Santiago) Target DB is PostgreSQL We have several query all properly running, just one was giving the error. The query is the following: index=myindex sourcetype=mysourcetype etc… | dbxoutput output=my_stanza “my_stanza” refers to one present on db_outputs.conf The error in Splunk Search Head was: rx.exceptions.OnErrorNotImplementedException at rx.internal.util.InternalObservableUtils$ErrorNotImplementedAction.call(InternalObservableUtils.java:386) at rx.internal.util.InternalObservableUtils$ErrorNotImplementedAction.call(InternalObservableUtils.java:383) at rx.internal.util.ActionSubscriber.onError(ActionSubscriber.java:44) at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:153) at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:115) at rx.exceptions.Exceptions.throwOrReport(Exceptions.java:212) at rx.observers.SafeSubscriber.onNext(SafeSubscriber.java:139) at rx.internal.operators.OperatorBufferWithSize$BufferExact.onCompleted(OperatorBufferWithSize.java:128) at rx.internal.operators.OnSubscribeMap$MapSubscriber.onCompleted(OnSubscribeMap.java:97) at rx.internal.operators.OperatorPublish$PublishSubscriber.checkTerminated(OperatorPublish.java:423) at rx.internal.operators.OperatorPublish$PublishSubscriber.dispatch(OperatorPublish.java:505) at rx.internal.operators.OperatorPublish$PublishSubscriber.onCompleted(OperatorPublish.java:305) at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.slowPath(OnSubscribeFromIterable.java:134) at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:89) at rx.Subscriber.setProducer(Subscriber.java:211) at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:63) at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:34) at rx.Observable.unsafeSubscribe(Observable.java:10327) at rx.internal.operators.OperatorPublish.connect(OperatorPublish.java:214) at rx.observables.ConnectableObservable.connect(ConnectableObservable.java:52) at com.splunk.dbx.command.DbxOutputCommand.process(DbxOutputCommand.java:161) at com.splunk.search.command.StreamingCommand.process(StreamingCommand.java:58) at com.splunk.search.command.ChunkedCommandDriver.execute(ChunkedCommandDriver.java:109) at com.splunk.search.command.AbstractSearchCommand.run(AbstractSearchCommand.java:50) at com.splunk.search.command.StreamingCommand.run(StreamingCommand.java:16) at com.splunk.dbx.command.DbxOutputCommand.main(DbxOutputCommand.java:100) Caused by: java.lang.NullPointerException at java.math.BigDecimal.<init>(BigDecimal.java:809) at com.splunk.dbx.service.output.OutputServiceImpl.setParameterAsObject(OutputServiceImpl.java:288) at com.splunk.dbx.service.output.OutputServiceImpl.setParameter(OutputServiceImpl.java:270) at com.splunk.dbx.service.output.OutputServiceImpl.processInsertion(OutputServiceImpl.java:216) at com.splunk.dbx.service.output.OutputServiceImpl.output(OutputServiceImpl.java:76) at rx.internal.util.ActionSubscriber.onNext(ActionSubscriber.java:39) at rx.observers.SafeSubscriber.onNext(SafeSubscriber.java:134) ... 19 more Looking at search.log from job inspector: 12-03-2021 17:26:18.187 INFO DispatchExecutor - END OPEN: Processor=noop 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: Exception in thread "main" java.lang.IllegalStateException: I/O operation on closed writer 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.search.command.AbstractWriteHandler.checkValidity(AbstractWriteHandler.java:100) 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.search.command.AbstractWriteHandler.flush(AbstractWriteHandler.java:228) 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.search.command.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:69) 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.search.command.AbstractWriteHandler.close(AbstractWriteHandler.java:233) 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.search.command.ChunkedCommandDriver.execute(ChunkedCommandDriver.java:120) 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.search.command.AbstractSearchCommand.run(AbstractSearchCommand.java:50) 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.search.command.StreamingCommand.run(StreamingCommand.java:16) 12-03-2021 17:26:18.188 ERROR ChunkedExternProcessor - stderr: at com.splunk.dbx.command.DbxOutputCommand.main(DbxOutputCommand.java:100) I solved in this way (adding fillnull): index=myindex sourcetype=mysourcetype etc… | fillnull value=0.00 mbytes_in | fillnull value=0.00 mbytes_out | dbxoutput output=my_stanza There were 2 records in the extraction having "mbytes_in" and "mbytes_out" fields without any value. I am sure before upgrading to Splunk DB Connect 3.6.0 it was working properly. The target DB is a PostgreSQL and the table is defined as below, as you can see "mbytes_in" and "mbytes_out" can accept NULL values (and I can see several records in the PostgreSQL DB populated in the past with "mbytes_in" and "mbytes_out" having NULL values) Here the table definition in PostgreSQL: CREATE TABLE myschema.mytable ( field01 integer NOT NULL, field02 character varying(6) NOT NULL, field03 character varying(6), field04 character varying(15) NOT NULL, field05 timestamp(6) with time zone NOT NULL, mbytes_in numeric(12, 2), mbytes_out numeric(12, 2), field06 character varying(15) NOT NULL, field07 character varying(50), field08 character varying(50), field09 character varying(50) NOT NULL, field10 character varying(255) NOT NULL, field11 character varying(15) NOT NULL, field12 character varying(255), field13 date NOT NULL, field14 character varying(255) NOT NULL, CONSTRAINT my_pkey PRIMARY KEY (field01) ) WITH ( OIDS = FALSE ) TABLESPACE mytablespace; ALTER TABLE myschema.mytable OWNER to myuser; GRANT ALL ON TABLE myschema.mytable TO myuser; The log error that pointed me to a solution was the following: at com.splunk.dbx.command.DbxOutputCommand.main(DbxOutputCommand.java:100) Caused by: java.lang.NullPointerException at java.math.BigDecimal.<init>(BigDecimal.java:809) By the way no valuable logs were present in Splunk _internal index, usually when some SPL query fail to insert into our PostgreSQL DB I find valuable information like SQL codes and SQL errors. This time it was not present. I hope this post will help someone having the same issue. Best Regards, Edoardo

edoardo_vicendo · ‎12-03-2021

After a system patch JRE path was modified and DB Connect not working anymore with this error in _internal index: 12-03-2021 13:40:51.626 +0100 ERROR ModularInputs - <stderr> Introspecting scheme=dbxquery: /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh: line 33: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java: No such file or directory 12-03-2021 13:40:51.687 +0100 ERROR ModularInputs - Unable to initialize modular input "dbxquery" defined in the app "splunk_app_db_connect": Introspecting scheme=dbxquery: script running failed (exited with code 126).. 12-03-2021 13:40:52.092 +0100 ERROR ModularInputs - <stderr> Introspecting scheme=server: /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh: line 33: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java: No such file or directory 12-03-2021 13:40:52.092 +0100 ERROR ModularInputs - Unable to initialize modular input "server" defined in the app "splunk_app_db_connect": Introspecting scheme=server: script running failed (exited with code 126).. To fix I solved in this way Before: splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/local > cat dbx_settings.conf [java] javaHome = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/linux_x86_64/bin > cat customized.java.path /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java After: splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/local > cat dbx_settings.conf [java] javaHome = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-0.el6_10.x86_64/jre splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/linux_x86_64/bin > cat customized.java.path /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-0.el6_10.x86_64/jre/bin/java + Restart splunk

edoardo_vicendo · ‎10-22-2021

Thank you, I had exactly the same issue. D uring the upgrade, with Cluster Master in maintenance mode, the affected Indexer had an outage at storage level and then it was unable to join back the cluster. I solved with proposed steps, just wanted to add that not all the buckets have to be renamed, just the ones that are replicated (for instance in our environment metrics and other specific Splunk indexes are not)

edoardo_vicendo · ‎10-22-2021

I had the same issue in our test environment upgrading from 8.0.5 to 8.2.2.1 One of the 3 Indexer in the cluster was having this message in splunkd.log: 10-22-2021 12:42:49.859 +0200 WARN CMSlave [2763 indexerPipe] - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json manager=xxxxxxx:yyyy rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=xxx.xxx.xxx.xxx mgmtport=yyyy (reason: This peer re-added a clustered bucket as standalone and is rejected from joining the cluster). [ event=addPeer status=retrying AddPeerRequest: { _indexVec="iiiiii:event" batch_serialno=10 batch_size=14 guid=AAAAAAAA-1111-1111-AAAA-AAAAAAAAAAAA server_name=aaaaaaa status=Up } Batch 10/14 ]. The effect was being stuck in "batchadding" in the Monitoring Console. The main reason was that during the upgrade, with Cluster Master in maintenance mode, the affected Indexer had an outage due to an issue at storage level, therefore I am not completely sure with the proposed solution that suggest to do not put the Cluster in maintenance mode during the upgrade... To solve it I followed the steps proposed here: https://community.splunk.com/t5/Getting-Data-In/Indexer-fails-to-join-back-cluster-due-to-standalone-buckets/m-p/437846 Be aware not all the buckets have to be renamed, just the ones that are replicated (for instance in our environment metrics and other specific Splunk indexes are not)

edoardo_vicendo · ‎10-21-2021

Coming back again on this. I read different suggestions about enabling Hyper-threading on the Search Heads, but I am wondering about a configuration present in limits.conf: Maximum # of Concurrent Searches per SH Instance: – (max_searches_per_cpu x Logical # of CPUs) + base_max_searches max_searches_per_cpu = <integer> * The maximum number of concurrent historical searches for each CPU. The system-wide limit of historical searches is computed as: max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches * NOTE: The maximum number of real-time searches is computed as: max_rt_searches = max_rt_search_multiplier x max_hist_searches * Default: 1 Therefore if you do not enable Hyper-threading but you increase max_searches_per_cpu to 2 you are more or less obtaining the same result?

edoardo_vicendo · ‎10-20-2021

Hello, In our environment we have Splunk HF with 2 parallel Ingestion Pipelines. https://docs.splunk.com/Documentation/Splunk/8.2.2/Capacity/Parallelization#Index_parallelization One of the aim of those Splunk HF is to offload the Splunk Indexer on parsing Pipeline, Merging Pipeline and Typing Pipeline. Due to that the data coming from Splunk HF are already "processed" and our Indexer are mostly processing them only in the Index Pipeline. https://wiki.splunk.com/Community:HowIndexingWorks On the Indexers we only have 1 Ingestion Pipeline, the CPU Cores used for indexing are typically 4-6. Does our Indexers are taking advantage using pretty much all the 4-6 CPU Cores for the Index Pipeline only OR they are "wasted" on the other mostly idle pipelines? Thanks a lot, Edoardo

edoardo_vicendo · ‎10-20-2021

I confirm you Splunk_TA_nix has been deprecated (I remember having received an email from Splunk about this). Anyway it doesn't mean it will stop working 🙂 About your issue you probably don't receive any log because most of the inputs are disabled by default. What I would suggest you to do is to copy the ../default/inputs.conf in .../local/inputs.conf and then modify it to enable the scripted input OR the monitor stanza related to linux_secure Best Regards, Edoardo

edoardo_vicendo · ‎10-20-2021

Did you ended up with a solution?

edoardo_vicendo · ‎10-15-2021

Hello, In the Monitoring Console Summary Dashboard, under Deployment Metrics, there is an "Avg. Search Latency" indicator. I searched in the official documentation but I didn't found an extensive explanation. What does this metric shows? Thanks a lot, Edoardo

edoardo_vicendo · ‎10-13-2021

Hello, We are going to have several OpenShift 4 clusters running CoreOS, therefore without having the possibility to install a standard version of the Splunk Universal Forwarder. Do you think is possible, installing a containerized version of the Splunk Universal Forwarder on each OpenShift node, to let it connect to a Splunk Deployment server (not containerized) in order to control them centrally? Thanks a lot, Edoardo

edoardo_vicendo · ‎10-05-2021

We solved the issue with this configuration in indexes.conf [default] maxRunningProcessGroups = 12 processTrackerServiceInterval = 0 What we observed is that splunk-optimize process runs for very few seconds, therefore is probably more important to set processTrackerServiceInterval=0 so that every second can be spawn a new process, instead of waiting 15 seconds as default. With maxRunningProcessGroups=12 then there is even more "room" for splunk-optimize processes.

edoardo_vicendo · ‎09-28-2021

Hello, I need your thought on this topic. With suggested modification I guess there is chance to completely fill the process queue ( maxRunningProcessGroups) with splunk-optimize processes (maxConcurrentOptimizes) because maxConcurrentOptimizes=25 > maxRunningProcessGroups=12 https://docs.splunk.com/Documentation/Splunk/8.2.2/Admin/Indexesconf By default maxConcurrentOptimizes=6 < maxRunningProcessGroups=8 maxConcurrentOptimizes = <nonnegative integer> * The number of concurrent optimize processes that can run against a hot bucket. * This number should be increased if: * There are always many small tsidx files in the hot bucket. * After rolling, there are many tsidx files in warm or cold buckets. * You must restart splunkd after changing this setting. Reloading the configuration does not suffice. * The highest legal value is 4294967295. * Default: 6 maxRunningProcessGroups = <positive integer> * splunkd runs helper child processes like "splunk-optimize", "recover-metadata", etc. This setting limits how many child processes can run at any given time. * This maximum applies to all of splunkd, not per index. If you have N indexes, there will be at most 'maxRunningProcessGroups' child processes, not N * 'maxRunningProcessGroups' processes. * Must maintain maxRunningProcessGroupsLowPriority < maxRunningProcessGroups * This is an advanced setting; do NOT set unless instructed by Splunk Support. * Highest legal value is 4294967295. * Default: 8 On the other hand, in this post they just left maxConcurrentOptimizes to default increasing maxRunningProcessGroups only: https://community.splunk.com/t5/Getting-Data-In/The-index-process-has-paused-data-flow-Too-many-tsidx-files/m-p/524522 Therefore maxConcurrentOptimizes=6 < maxRunningProcessGroups=12 This setting seems to be better, but if the aim of this change is to increase the parallel processes of splunk-optimize (to avoid the warning message "The index processor has paused data flow. Too many tsidx files in idx= ..."), shouldn't be better increasing maxConcurrentOptimizes keeping it always lower than maxRunningProcessGroups ? Thanks a lot, Edoardo

edoardo_vicendo · ‎09-22-2021

I think it depends on how you manage your environment. We applied this modification only on a subset of client machines (having UF installed) because the side effect is that you are not able anymore to do "su - splunk". On a UF it could be acceptable, mainly because you deploy the apps with the Splunk Deployment Server, and because to restart the UF if needed you can leverage on init or systemd. On the other hand, if you are on a Splunk server (having Splunk Enterprise installed), it is usually necessary to become splunk user to modify configuration files, run CLI commands etc... Of course you can also manage it adding requested commands in sudoers file, but it could take time to define them all, and using wildcards on sudoers file it's not the best choice in term of security. This obviously applies unless you are not root on that machine 🙂

Posts	103
Solutions	6
Karma Given	60
Karma Received	14
Member Since	‎09-19-2016

Online Status	Offline
Date Last Visited	‎03-30-2022 01:06 PM

How do you monitor IBM VIOS ?

Configure Splunk Universal Forwarder with TCP inpu...

Splunk DB Connect error DbxOutputCommand.main(DbxO...

CPU Cores assigned to Index Pipeline

Monitoring Console Summary Dashboard, what Search ...

Containerized Splunk Universal Forwarder on OpenSh...

Why are we encountering an issue after a data migr...

Splunk Enterprise Security enable Hyper-threading

Setting useACK in outputs.conf in a Distributed En...

Match records based on a WHERE condition defined i...

Re: Skipping itemPath, does not match path....

How do you monitor IBM VIOS ?

Re: Configure Splunk Universal Forwarder with TCP ...

Re: Configure Universal forwarder 8.1.0 to send da...

Re: Configure Splunk Universal Forwarder with TCP ...

Re: Configure Splunk Universal Forwarder with TCP ...

Configure Splunk Universal Forwarder with TCP inpu...

Splunk DB Connect error DbxOutputCommand.main(DbxO...

Re: Splunk DB Connect: How to resolve "Can not com...

Re: Indexer fails to join back cluster due to stan...

Re: 8.0.9 indexers all stuck "batchadding" after u...

Re: Splunk Enterprise Security enable Hyper-thread...

CPU Cores assigned to Index Pipeline

Re: Search for sudo (linux_secure does not exist?)

Re: Full queues and low IOWait???!

Monitoring Console Summary Dashboard, what Search ...

Containerized Splunk Universal Forwarder on OpenSh...

Re: Search peer has the following message: idx=_in...

Re: Search peer has the following message: idx=_in...

Re: Splunk needs a shell?