About satyenshah

satyenshah

Of course there's no simple setting, but the question is if there's a combination of settings (e.g. timeouts, retries, retry intervals, iptables) which can mimic that absent setting. For example, I've tried simply blocking port 8080 traffic from distant site3 to site2, to see if a recently-rebuilt peer in site2 would be able to fixup exclusively from site1, by exhausting retries for replicating a bucket from site3 until finally giving up, then retrying from site1, and then succeeding. That apparently throws errors that jam up the fixup tasks. But, with clever settings, it might work.

satyenshah

Is there a combination of settings that can optimize replication in a multisite cluster with 3+ sites where the links between sites are not identical? Suppose you have three sites (site1, site2, site3). Suppose site1 and site2 are connected by a fast link, and that site3 is far away with a slow link. Suppose a bucket exists on site2 and site3 but not site1. I there a combination of settings or hacks that can ensure site1 gets the bucket from nearby site2 and not from distant site3?

satyenshah · ‎03-26-2025

I didn't realize that table forces localop. So, optimizing further would be: index=main sourcetype="access_combined_wcookie" action=purchase status=200 file="success.do" | fields - _* | fields JSESSIONID, action, status | stats count by JSESSIONID, action, status | rename JSESSIONID as UserSessions to discard _raw and other internal fields.

satyenshah · ‎02-27-2025

I believe this answer is not quite correct. The optimized query is: index=main sourcetype="access_combined_wcookie" action=purchase status=200 file="success.do" | table JSESSIONID, action, status | stats count by JSESSIONID, action, status | rename JSESSIONID as UserSessions In a clustered Splunk environment, lines 1-2 execute in parallel on your indexers, the minimized data is then passed to the searchhead, and the searchhead executes line 3, and then line 4 only operates on 1 row of data. I try to always do a TABLE early in the qeury especially before doing an expensive DEDUP, STATS, or BIN. That reduces the dataset on all your indexers, discarding unneeded fields, before it's merged on your searchead. Instead of TABLE you could alternately do two FIELDS commands, one to include the necessary fields and another to remove _raw. Computationally I don't know whether Splunk is more efficient handling event data from FIELDS or handling transformed data from TABLE, but TABLE makes the query simpler.

satyenshah · ‎01-10-2025

Based on the docs, I can't tell if there's a functional difference between this: [clustering] multisite = true [clustermanager:prod] multisite = true [clustermanager:dev] multisite = false and this: [clustering] [clustermanager:prod] multisite = true [clustermanager:dev] in server.conf on searchheads.

satyenshah · ‎07-14-2023

The only reason I can think of for your _raw data shrinking but not impacting your license usage is if your Splunk license is workload-model not ingest-model.

satyenshah · ‎07-12-2023

In your case, change log{} for fortinet_fortigate to reference rewrite{}, so modify your config like this: rewrite rewrite_fortigate { <subst commands go here> }; log { source(s_net); filter(f_fortinet_fortigate); rewrite(rewrite_fortigate); destination(d_fortinet_fortigate); };

satyenshah · ‎07-03-2023

I haven't looked at this in a while, but I think the fortidgate TA looks at the DEVID which I discard. Workaround is FIELDALIASing the DEVID to map to HOST, or you could even just manually recreate DEVID in PROPS since in most environments you you can probably fit all your Fortigates into one CASE() statement.

satyenshah · ‎07-03-2023

Yes you can also do the rewrites using SEDCMD in props.conf or REGEX/FORMAT in transforms.conf. It's more efficient doing it ahead of the index pipeline. But doing it Typing shouldn't be too expensive. There's also a second set of transforms worth applying to fortigate traffic logs to CIM-normalize the fields (e.g. srcip to src_ip) and more significantly to rewrite "action=" to "vendor_action=" or "fgt_action=", which causes a lot of confusion when users search for blocked/denied traffic because the CIM and fortigate conflict with each other. I'll share here.

satyenshah · ‎03-01-2023

The guidance from Intel is to disable hyperthreading on searchheads for best performance. On searchheads, single-thread performance is more helpful than core count. On indexers, hyperthreading helps indexing but hurts searching. So you can enable/disable it to optimize for one or the other.

satyenshah · ‎01-28-2022

The "searchable" rolling restart feature frequently fails leaving indexers stuck in a "Reassigning primaries" state. It's so prone to failure, and the impact of failure is so severe that you're better off never doing it.

satyenshah · ‎09-03-2021

To manually merge buckets from multiple legacy indexers onto one new indexer, I used these commands which work on RHEL7/8: 1. on the legacy indexers, in indexes.conf, set maxVolumeDataSizeMB=400 for the warm volume to force all buckets to roll to cold 2. on the legacy indexers, in /etc/sudoers add johndoe ALL=NOPASSWD:/usr/bin/rsync to enable passwordless sudo with rsync 3. on the new indexer, run this command to rysnc cold buckets from each legacy indexer to a subfolder on the new indexer: sudo rsync --delete --compress-level=0 -aPe ssh --rsync-path="sudo rsync" [email protected]:/splunk_cold /splunk_cold/idx8 in the above case, 192.168.1.108 is the address for legacy indexer#8 4. on the new indexer, run this command to renumber the buckets from each indexer for i in /splunk_cold/idx8/*/colddb ; do echo $i ; cd $i ; ls ; for f in ` ls -rtd db_* `; do jj=` echo $f | cut -d "_" -f 4 `; kk=$(($jj + 8000)) ; ff=` echo $f | sed -e "s/_$jj\$/_$kk/" ` ; mv $f $ff ; done ; done in the above case, each bucket from indexer#8 with 3-digit id=xxx is changed to id=8xxx 5. on the new indexer, run this command to merge each subfolder to the parent folder cd /splunk_cold/idx8 ; 596 find -type d -exec mkdir -vp "/splunk_cold"/{} \; -or -exec mv -nv {} "/splunk_cold"/{} \;

satyenshah · ‎08-13-2020

I don't see that happening in my environment. Am running TA version 1.6.1. I haven't checked if it works with 1.6.2.

satyenshah · ‎03-26-2020

It's inconvenient that a Splunk deployment-server cannot reload an 'app'. It only reloads a 'serverclass'. Here's a shortcut. The following bash script takes the name of a deployment-app as a parameter. It will reload serverclasses that use the deployment-app. Copy the script to your Deployment Server to make it easier to push out app changes to clients from the CLI. #!/bin/bash if [ $# -lt 1 ]; then echo 1>&2 "$0: not enough arguments" exit 2 elif [ $# -gt 1 ]; then echo 1>&2 "$0: too many arguments" exit 2 elif (! (grep -E -q "^\[serverClass:[^:]*:app:$1" /opt/splunk/etc/system/local/serverclass.conf )); then echo 1>&2 "$0: No serverclasses found for $1" exit 0 fi grep -E "^\[serverClass:[^:]*:app:$1" /opt/splunk/etc/system/local/serverclass.conf | cut -d':' -f2 | sed -E 's/^(.*)$/ -class \1/g' | tr -d '\n' | xargs echo sudo -H -u splunkserviceaccount /opt/splunk/bin/splunk reload deploy-server -auth admin:changeme read -p "Press [Enter] to execute above command..." grep -E "^\[serverClass:[^:]*:app:$1" /opt/splunk/etc/system/local/serverclass.conf | cut -d':' -f2 | sed -E 's/^(.*)$/ -class \1/g' | tr -d '\n' | xargs sudo -H -u splunkserviceaccount /opt/splunk/bin/splunk reload deploy-server -auth admin:changeme

satyenshah · ‎12-11-2019

One frustration with the Linux TA (v6.0.2) is that the output contains so much whitespace. Ingesting that data as a metric spares your license, but you still lose on storage and speed. For example, I enabled vmstat.sh, bandwidth.sh, df.sh, and cpu.sh on one host. Those four inputs generate 64,552 bytes / hour of raw events. If I dedup the whitespace (replace \s+ with \s) then that shrinks to 36,465 bytes, which is 44% reduction. In other words, metrics from the Linux TA are 51% whitespace. Anybody try to fix this? The whitespace originates from the awk/printf commands in the bash scripts. The commands format the output into pretty printed tables, which doesn't make sense for machine data.

satyenshah · ‎12-02-2019

The app hosted at Splunk can be deployed to all instances, and is clean. Ourselves we're updating everything to 7.3.3.

satyenshah · ‎12-02-2019

I see! Will fix that.

satyenshah · ‎11-27-2019

The comment is present in datetime.xml in old versions of Splunk going back to Splunk v4. An engineer must have written it in 2012, when epoch time was due to roll over to 1.5e10 in 2014, warning that the datetime.xml file will again need to be updated around 2017 to still work in 2020.

satyenshah · ‎11-27-2019

Thanks! I can't think of an edge-case when the logic would go awry, because the same if-then condition needs to return two different results in a row to get to the reboot step. I can't imagine any way that could happen in an endless loop. The logic only injects '/opt/splunkforwarder' if $SPLUNK_HOME is somehow empty. I did that mostly for testing.

satyenshah · ‎11-27-2019

Is the Apache Add-on falling out of favor? It hasn't been updated since 1.0.0 release It doesn't officially support 7.3 or 8.0 (though unofficially it works fine) Splunk can natively parse apache logs using the access_combined and apache_error pretrained sourcetypes If you need more thorough CIM mapping than the pretrained sourcetypes offer, then there's an unofficial 'Add-on for Access Combined' available on splunkbase that does the job with fewer field extractions than the Apache add-on, and without the need for another sourcetype

satyenshah · ‎11-27-2019

I fatfingered cp -rp by habit. Changed to cp -f. Thanks for the heads up!

satyenshah · ‎11-25-2019

Yes, version 7.2.9.1 forwarders are compatible with 7.2.4 indexers: https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwardersandindexers We have a mix of UF versions ranging from 7.0 to 8.0.0, alongside indexers that have been are frequently updated, and have never run into an indexer-compatibility issue. At the same time, you can get away without doing anything to the UFs, since the timestamp extraction / cooking of data happens downstream of the UF.

satyenshah · ‎11-11-2019

Edit 2019-11-28: Splunk has released a better fix-it app than the one below. Edit 2019-11-25: I didn't notice the more immediate January 1, 2020 bug. Splunk has also released an updated datetime.xml in 7.3.3. Updated the app to copy over the old datetime.xml file with the newer datetime.xml file, instead of doing an inline string-replace. There's a looming problem with the way Splunk parses Unix/epoch time that will fail once epoch rolls over to 1600000000 on September 13, 2020. The specific problem is this regex, present in $SPLUNK_HOME/etc/datetime.xml in all Splunk installs: <define name="_utcepoch" extract="utcepoch, subsecond">  <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text> </define> which won't match epoch times past 1599999999. This bug exists in all currently released versions of Splunk, including 8.0.0. Upcoming versions of Splunk ought to resolve the problem (SPL-179221). For instances that can't/won't get updated in time, this Splunk app can be deployed as a workaround. The app executes bash/Powershell at Splunk startup to check for the above regex and add a '6' if needed. It may not be the best fix, but it does kick-the-can on the problem for another 3 years, at least until epoch time reaches 1.7e10.

satyenshah · ‎11-07-2019

Below is one improvement (which saves a significant amount of money/resources) and one bug fix (which improves ingest) relevant for everyone who supports Fortinet. Event Trimming In our environment, Fortigate firewall logs accounted for about 20% of our Splunk license usage. This is a typical Fortigate traffic event: time=17:28:50 devname="my3700d" devid="FGT37D4615801234" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="mirror-dmz" eventtime=1570055330 srcip=192.0.2.35 srcname="venus.example.local" srcport=41394 srcintf="internal.123" srcintfrole="undefined" dstip=18.71.0.20 dstport=389 dstintf="external.466" dstintfrole="undefined" poluuid="b64cd2aa-8b22-51e9-36fd-43ef903c833b" sessionid=3049101971 proto=6 action="accept" policyid=188 policytype="policy" service="LDAP" dstcountry="United States" srccountry="Reserved" trandisp="noop" duration=275 sentbyte=386 rcvdbyte=60 sentpkt=6 rcvdpkt=1 appcat="unscanned" sentdelta=386 rcvddelta=60 this event is over 650 bytes long. Since Splunk licensing is based on raw event size, stripping down that event before it gets ingested by Splunk can have a dramatic effect on license usage. Fortinet does not have any settings for making its events less chatty. So, we used syslog-ng's rewrite functionality to trim the events like this: time=17:28:50 devname="my3700d" devid="FGT37D4615801234" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="mirror-dmz" eventtime=1570055330 srcip=192.0.2.35 srcname="venus.example.local" srcport=41394 srcintf="internal.123" srcintfrole="undefined" dstip=18.71.0.20 dstport=389 dstintf="external.466" dstintfrole="undefined" poluuid="b64cd2aa-8b22-51e9-36fd-43ef903c833b" sessionid=3049101971 proto=6 action="accept" policyid=188 policytype="policy" service="LDAP" dstcountry="United States" srccountry="Reserved" trandisp="noop" duration=275 sentbyte=386 rcvdbyte=60 sentpkt=6 rcvdpkt=1 appcat="unscanned" sentdelta=386 rcvddelta=60 which is under 400 bytes. Many events (like drops and blocks) shrink even further. It's up to you to decide which fields to keep and which to discard. We removed fields that were not of interest to the firewall admins. In addition to the fields shown, we could have safely deleted the human-readable 'time' field (see item #2 below). We could also have totally discarded the src/dst name fields which are pulled from reverse DNS, but in our case we elected to just trim the FQDN down to the short name. I'm probably also going to rename the 'action' field to 'vendor-action' or something else to avoid collision and confusion with the CIM-mapped 'action'. In syslog-ng, the configuration looks like this: # TCP/601 - RFC5424, octet counting source s_aggregation_tcp_601 { syslog( ip(0.0.0.0) transport(tcp) port(601) ); }; rewrite rewrite_fortigate { #subst ( '^time=\d\d:\d\d:\d\d(\.\d{3,6})? ', "", value("MESSAGE") ); subst ( ' devid="[A-Z0-9]{16}" ', " ", value("MESSAGE") ); subst ( ' logid="[0-9]{10}" ', " ", value("MESSAGE") ); subst ( ' srcname="([\w-]+)\.example\.local" ', ' srcname="$1" ', value("MESSAGE") ); subst ( ' srcintfrole="[a-z]+" ', " ", value("MESSAGE") ); subst ( ' dstname="([\w-]+)\.example\.local" ', ' dstname="$1" ', value("MESSAGE") ); subst ( ' dstintfrole="[a-z]+" ', " ", value("MESSAGE") ); subst ( ' poluuid="[a-z0-9-]{36}" ', " ", value("MESSAGE") ); subst ( ' sessionid=[0-9]+ ', " ", value("MESSAGE") ); subst ( ' policytype="policy" ', " ", value("MESSAGE") ); subst ( ' srccountry="[^"]+" ', " ", value("MESSAGE") ); subst ( ' dstcountry="[^"]+" ', " ", value("MESSAGE") ); subst ( ' service="[^"]+" ', " ", value("MESSAGE") ); subst ( ' appcat="unscanned"', "", value("MESSAGE") ); subst ( ' crscore=[0-9]+ craction=[0-9]+ crlevel="[a-z]+"', "", value("MESSAGE") ); }; filter f_firewall { netmask(192.0.2.1/32) ;}; template t_msg_only { template("${MESSAGE}\n"); template_escape(no); }; destination d_firewall { file("/app/$LOGHOST/log/$R_YEAR-$R_MONTH-$R_DAY/firewall/$SOURCEIP/$R_HOUR-$LEVEL-$FACILITY.log" suppress(0) template(t_msg_only)); }; log { source(s_aggregation_tcp_601); filter(f_firewall); rewrite(rewrite_fortigate); destination(d_firewall); flags(final); }; Note that since last year Fortinet versions v6.0+ support modern RFC5424/RFC6514 syslog. So, you can now do syslog over TCP instead of over UDP. However, be careful that in syslog-ng you must use the syslog() driver instead of the network() driver. The network() driver is for legacy BSD syslog, which is what most devices today still use. Because I discard the 'devid' field, I needed to make one modification to the Fortinet TA deployed to the Heavy Forwarder on the syslog-ng server. I added this file to the TA: # $SPLUNK_HOME/etc/apps/Splunk_TA_fortinet_fortigate/local/transforms.conf [force_sourcetype_fgt_traffic] DEST_KEY = MetaData:Sourcetype REGEX = \stype=\"?traffic\"?\s FORMAT = sourcetype::fgt_traffic [force_sourcetype_fgt_utm] DEST_KEY = MetaData:Sourcetype REGEX = \stype=\"?utm\"?\s FORMAT = sourcetype::fgt_utm [force_sourcetype_fgt_event] DEST_KEY = MetaData:Sourcetype REGEX = \stype=\"?event\"?\s FORMAT = sourcetype::fgt_event By using syslog-ng to trim our logs, we cut our daily firewall log ingest from 180GB to about 70GB. At today's pricing, the value of that 110GB savings in Splunk license is maybe $60,000 per year. Trimming events also made them more readable (when using the 'Search and Reporting' app), faster to query, and consume less disk. For any deep debugging that requires those trimmed fields, the firewall admins still have access to those events in Fortianalyzer. Timestamp Extraction There is a bug in current versions (up to 1.6.1) of the Splunk add-on for Fortinet that breaks timestamp extraction. The bug causes Splunk timestamp extraction to guess the timestamp, which sometimes fails. When it fails, Splunk will intermittently backdate recent events with the right time but a random date from the past ~10 years. Most events do get properly assigned today's date - only a few events get backdated. So perhaps the biggest ramification of the bug is that it gives Splunk buckets a longer date range. Your firewall log buckets will say they contain years of data instead of a few days or weeks of data, because a few events have the wrong date. The fix is adding this file to the TA, to explicitly extract epoch time from events: # $SPLUNK_HOME/etc/apps/Splunk_TA_fortinet_fortigate/local/props.conf [fgt_log] TIME_PREFIX = \seventtime= TIME_FORMAT = %s MAX_TIMESTAMP_LOOKAHEAD = 28 This bug has been reported to Fortinet and should be fixed in upcoming versions of the TA.

satyenshah · ‎03-04-2019

The Splunk Add-on for Microsoft IIS still requires the administrator to manually create an inputs.conf file on each IIS host, customized with the paths of all Log directories. For large, diverse environments, this can be time consuming. The following app can be deployed to your Windows servers, alongside the IIS add-on. On first-run it will automatically populate its inputs.conf with monitor stanzas. Each monitor will correspond to one IIS website's log folder. ├── bin │ ├── generate_inputs.cmd │ ├── generate_inputs.ps1 │ └── restart_splunk.cmd ├── default │ └── app.conf └── local ├── app.conf └── inputs.conf bin/generate_inputs.cmd REM store path of this bin folder set "BINPATH=%~dp0" REM set "SPLUNK_HOME=C:\Program Files\SplunkUniversalForwarder" Powershell -ExecutionPolicy ByPass -File "%BINPATH%generate_inputs.ps1" -splunkHome "%SPLUNK_HOME%" bin/generate_inputs.ps1 # generate inputs.conf for IIS Import-Module WebAdministration $INPUTSCONF = "$PSScriptRoot\..\local\inputs.conf" # only if IIS websites exist and inputs.conf contains no monitors If ((Get-Website) -And (-Not (Select-String -Pattern "^\[monitor" -Path "$INPUTSCONF" -quiet))) { foreach($WebSite in $(get-website)) { $logFile="$($Website.logFile.directory)\W3SVC$($website.id)".replace("%SystemDrive%",$env:SystemDrive) Add-Content -Path "$INPUTSCONF" -Value "" Add-Content -Path "$INPUTSCONF" -Value "# $($website.name)" Add-Content -Path "$INPUTSCONF" -Value "[monitor://$logfile]" Add-Content -Path "$INPUTSCONF" -Value "disabled = false" Add-Content -Path "$INPUTSCONF" -Value "ignoreOlderThan = 14d" Add-Content -Path "$INPUTSCONF" -Value "sourcetype = ms:iis:auto" Add-Content -Path "$INPUTSCONF" -Value "index = default" } # restart splunk if inputs.conf now contains monitors if (Select-String -Pattern "^\[monitor" -Path "$INPUTSCONF" -quiet) { Invoke-WmiMethod -Class Win32_Process -Name Create -ArgumentList "$PSScriptRoot\restart_splunk.cmd" } } bin/restart_splunk.cmd @echo off timeout /t 5 rem "C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" restart sc stop SplunkForwarder sc start SplunkForwarder exit local/inputs.conf [script://.\bin\generate_inputs.cmd] disabled = False index = _internal interval = -1 # autogenerated monitors will be written below this line Limitations: the script will not change inputs.conf once it has been written out. It would need to be undeployed/redeployed to catch up to any IIS configuration changes. It also will not resolve environment variables in the paths other than systemdrive (which would be unusual, but could happen). It's also possible for a Windows foldername to contain a bracket "]" character which might crash the [monitor://] stanza. I haven't tested, and don't know if there's a method for escaping it.

Posts	35
Solutions	3
Karma Given	15
Karma Received	33
Member Since	‎03-15-2018

Online Status	Offline
Date Last Visited	Saturday

optimizing Splunk multisite indexer replication tr...

Script to reload serverclasses that use a specifie...

Splunk Add-on for Apache vs pretrained sourcetypes

Splunk Y2K-type bug arriving on September 13, 2020

Trimming firewall logs using syslog-ng to conserve...

App for generating inputs.conf for IIS

Lousy eventtypes in Unix add-on: only Unix uses *....

Good practice to remove serverName from server.con...

Major boot-start change with 7.2.3

How do you detect a Universal Forwarder (UF) vs En...

Re: optimizing Splunk multisite indexer replicatio...

optimizing Splunk multisite indexer replication tr...

Re: Best practice dedup: should I use it as early ...

Re: Best practice dedup: should I use it as early ...

Re: Multisite and Single-Site Environment

Re: Trimming firewall logs using syslog-ng to cons...

Re: Trimming firewall logs using syslog-ng to cons...

Re: Trimming firewall logs using syslog-ng to cons...

Re: Trimming firewall logs using syslog-ng to cons...

Re: Splunk Enterprise Security enable Hyper-thread...

Re: Rolling restart hung on "Reassigning primaries...

Re: How can I find all duplicate bucket id's that ...

Re: Trimming firewall logs using syslog-ng to cons...

Script to reload serverclasses that use a specifie...

Re: Splunk Metrics via Splunk Add on for *nix

Re: Splunk Y2K-type bug arriving on September 13, ...

Re: Splunk Y2K-type bug arriving on September 13, ...

Re: Splunk Y2K-type bug arriving on September 13, ...

Re: Splunk Y2K-type bug arriving on September 13, ...

Splunk Add-on for Apache vs pretrained sourcetypes

Re: UF and Platform version level compatibility wi...

Re: UF and Platform version level compatibility wi...

Splunk Y2K-type bug arriving on September 13, 2020

Trimming firewall logs using syslog-ng to conserve...

App for generating inputs.conf for IIS

Join the Conversation