Hello, In a distributed environment with Universal Forwarder, Heavy Forwarder and Indexers, like this one: UF --> HF --> IDX How do you set useACK=true in outputs.conf ? Is it needed to be enabled both on Universal Forwarder and Heavy Forwarder? We currently have it enabled only on the Heavy Forwarder. Thanks a lot, Edoardo ... View more

Hi, I have seen this new feature in Splunk 8.1.1 https://docs.splunk.com/Documentation/Splunk/8.1.1/ReleaseNotes/MeetSplunk Enhanced TSIDX compression Enhanced TSIDX compression for improved performance and up to 40% reduced storage.   I was just wondering if this feature is related to the tsidxWritingLevel as mentioned here: https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/Reducetsidxdiskusage#The_tsidx_writing_level Splunk Enterprise Version The tsidxWritingLevel supported The supported tsidx level for searching 7.2.x 1, 2 1, 2 7.3.x 1, 2, 3 1, 2, 3 8.0.x 1, 2, 3 1, 2, 3 8.1.x 1, 2, 3, 4 1, 2, 3, 4   that can be set-up in indexes.conf as follow: tsidxWritingLevel = [1|2|3|4] * Enables various performance and space-saving improvements for tsidx files. * For deployments that do not have multi-site index clustering enabled, set this to the highest value possible for all your indexes. * For deployments that have multi-site index clustering, only set this to the highest level possible AFTER all your indexers in the cluster have been upgraded to the latest code level. * Do not configure indexers with different values for 'tsidxWritingLevel' as downlevel indexers cannot read tsidx files created from uplevel peers. * The higher settings take advantage of newer tsidx file formats for metrics and log events that decrease storage cost and increase performance * Default: 1   Moreover, is there anyone having some (good) feedback after have enabled it? Thanks a lot, Edoardo ... View more

Hello, We are in a multi-site indexer cluster environment, and we are going to upgrade our infrastructure from 3 Indexer to 6 Indexer. Basically we will add 6 new indexers, and decommission the 3 old indexers. Do you know which is the speed in MBytes/s that will be reached once the data will start to be copied from the 3 old Indexers to the 6 new indexers? We did some test in our development environment, where we have simulated a similar scenario, and it seems it started copying very fast with 170 MBytes/s peak (we have checked it with an nmon session at the source Indexer machine). In order to start the copying process we had: added the new Indexers to the Master Node switched the Splunk HFs to forward data to the new Indexers run, one by one on each old Indexer, the command splunk offline --enforce-counts Before starting the test we have read the following documentation: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Takeapeeroffline#Estimate_the_cluster_recovery_time_when_a_peer_gets_decommissioned In the official documentation it is mentioned: 10GB (rawdata and/or index files) from one peer to another across a LAN takes about 5-10 minutes therefore copying speed should go from 136 Mbit/s to 272 Mbit/s. If the speed we have observed is correct (more than 1Gbit/s), do you know if there is any way to limit the output bandwidth? Thanks a lot, Edoardo ... View more

We have to forward some data from a Splunk Heavy Forwarder to a third party syslog server. This is possible as indicated here: https://docs.splunk.com/Documentation/Splunk/8.0.6/Forwarding/Forwarddatatothird-partysystemsd   The challenge is to select only some files from a particular host and forward only the logs that contain a particular string. Here is what we were able to achieve (basically 2 rules out of 3, so some files that contain a particular string), I don’t know if it is possible to add in some way a reference also for the host. Do you know if it is feasible?   outputs.conf [syslog:syslog_target] type = udp server = 111.222.333.444:514 props.conf [source::/path/of/myfile/*filename.log] TRANSFORMS-syslog_forward = syslog_forward_rule transforms.conf [syslog_forward_rule] REGEX = www\.mywebsite\.com DEST_KEY = _SYSLOG_ROUTING FORMAT = syslog_target   Thanks a lot, Edoardo ... View more

Hi All, As indicated here (https://community.splunk.com/t5/Getting-Data-In/Why-am-I-unable-to-monitor-SPLUNK-HOME-var-log-splunk-audit-log/m-p/506185#M86203), I have been able to get the audit.log from our Universal Forwarders with audittrail sourcetype. Unfortunately sometimes those events read from $SPLUNK_HOME/var/log/splunk/audit.log are merged in a unique event (even if each event is in a new line and starts with a timestamp). In our deployment we have Universal Forwarders sending data to Heavy Forwarders that then send them to Indexers: UF --> HF --> IDX What I tried to do is to deploy a props.conf on the HF to indicate the following:   [audittrail] SHOULD_LINEMERGE = false SEDCMD = s/\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}\.\d{3}.* INFO AuditLogger - //g   But even the SEDCMD is not applied. And I can see with the following command that the configuration are properly read in the HF:   splunk btool props list --debug   Due to that I tried adding this props.conf directly on the UF and it is working (but it is not a good solution for us because we don't want to force the local processing on the UF).   [audittrail] SHOULD_LINEMERGE = false SEDCMD = s/\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}\.\d{3}.* INFO AuditLogger - //g force_local_processing = true   I believe the issue is related to the fact that the audit logs from UF are sent to HF indexQueue instead of parsingQueue.  I tried also to add the audit.conf file both in UF and HF as follow without any luck:   [default] queueing=false   Reading further on Splunk documentation (https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Auditconf) :   queueing = <boolean> * Whether or not audit events are sent to the indexQueue. * If set to "true", audit events are sent to the indexQueue. * If set to "false", you must add an inputs.conf stanza to tail the audit log for the events reach your index. * Default: true   My questions are: Do you know what does it means "If set to "false", you must add an inputs.conf stanza to tail the audit log for the events reach your index." Do you have any idea on how to apply on the HF the props.conf to the audit events coming from the UF without having to deploy it directly on UF with  force_local_processing=true Thanks a lot, Edoardo   ... View more