We've SH Cluster environment and are seeing the following error ; "Gave up waiting for the captain to establish a common bundle version across all search peers; using most recent bundles on all peers instead" After some re-search and looking through answers site, this could be due to inconsistent distsearch.conf on some of the search heads in the cluster ; so I updated and removed all the values to servers key in distsearch.conf on all the search heads in the cluster and restarted splunk; but immediately following restart the changes made are overridden and restored to old distsearch.conf file. We're not deploying this file with these changes using deployer. Following was done (multiple times) on each search head in the cluster (IPs hashed for security purposes) - cat /opt/splunk/etc/system/local/distsearch.conf [distributedSearch] servers = https://10.xxx.36.000:8089,https://10.xxx.46.00:8089,https://10.xxx.46.00:8089,https://10.xxx.46.00:8089,https://eo1vmsk011.lema:8089 Changed distsearch.conf to [distributedSearch] servers = Restarted splunk Checked the distsearch.conf file to find contents restored We even tried to delete the distsearch.conf file across all the search heads in the cluster , followed by restarting all the members, but the distsearch.conf file gets recreated. output of btool command on distsearch from one of the affected search heads in the cluster. I have checked for any monitoring/CM tool, but we don't have any to manage splunk process. [spnksvc@ep3vmnspk199 bin]$ ./splunk cmd btool distsearch list --debug /opt/splunk/etc/system/default/distsearch.conf [bundleEnforcerBlacklist] /opt/splunk/etc/system/default/distsearch.conf [bundleEnforcerWhitelist] /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf [distributedSearch] /opt/splunk/etc/system/default/distsearch.conf authTokenConnectionTimeout = 5 /opt/splunk/etc/system/default/distsearch.conf authTokenReceiveTimeout = 10 /opt/splunk/etc/system/default/distsearch.conf authTokenSendTimeout = 10 /opt/splunk/etc/system/default/distsearch.conf bestEffortSearch = false /opt/splunk/etc/system/default/distsearch.conf connectionTimeout = 10 /opt/splunk/etc/system/default/distsearch.conf defaultUriScheme = https /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf disabled = 0 /opt/splunk/etc/system/default/distsearch.conf receiveTimeout = 600 /opt/splunk/etc/system/default/distsearch.conf sendTimeout = 30 /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf serverTimeout = 900 /opt/splunk/etc/system/local/distsearch.conf servers = https://10.xxx.36.000:8089,https://10.xxx.46.00:8089,https://10.xxx.46.00:8089,https://10.xxx.46.00:8089,https://eo1vmsk011.lema:8089 /opt/splunk/etc/system/default/distsearch.conf shareBundles = true /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf statusTimeout = 900 /opt/splunk/etc/system/default/distsearch.conf useSHPBundleReplication = true /opt/splunk/etc/apps/Splunk_TA_windows/default/distsearch.conf [replicationBlacklist] /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/distsearch.conf MSAD_lookups = .../splunk_app_windows_infrastructure/lookups/(tHostInfo|tSessions).csv$ /opt/splunk/etc/system/default/distsearch.conf conf = (system|(apps/))/(default|local)/server.conf /opt/splunk/etc/system/default/distsearch.conf framework = apps/framework/... /opt/splunk/etc/system/default/distsearch.conf lookupindexfiles = (system|apps/|users(/reserved)?//)/lookups/.(tmp$|index($|/...)) /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf noBinDir = (.../bin/) /opt/splunk/etc/apps/Splunk_TA_windows/default/distsearch.conf nontsyslogmappings = ...ntsyslog_mappings.csv /opt/splunk/etc/system/default/distsearch.conf sampleapp = apps/sample_app/... /opt/splunk/etc/system/default/distsearch.conf user_specific_meta = users(/_reserved)?///metadata/local.meta /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf [replicationSettings] /opt/splunk/etc/system/default/distsearch.conf allowDeltaUpload = true /opt/splunk/etc/system/default/distsearch.conf allowSkipEncoding = true /opt/splunk/etc/system/default/distsearch.conf allowStreamUpload = auto /opt/splunk/etc/system/default/distsearch.conf concerningReplicatedFileSize = 500 /opt/splunk/etc/system/default/distsearch.conf connectionTimeout = 60 /opt/splunk/etc/system/default/distsearch.conf excludeReplicatedLookupSize = 0 /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf maxBundleSize = 14438892420 /opt/splunk/etc/system/default/distsearch.conf maxMemoryBundleSize = 10 /opt/splunk/etc/apps/splunk_dist_conf/default/distsearch.conf replicationThreads = 8 /opt/splunk/etc/system/default/distsearch.conf sanitizeMetaFiles = true /opt/splunk/etc/system/default/distsearch.conf sendRcvTimeout = 60 /opt/splunk/etc/system/default/distsearch.conf [replicationSettings:refineConf] /opt/splunk/etc/system/default/distsearch.conf replicate.app = true /opt/splunk/etc/system/default/distsearch.conf replicate.authorize = true /opt/splunk/etc/system/default/distsearch.conf replicate.collections = true /opt/splunk/etc/system/default/distsearch.conf replicate.commands = true /opt/splunk/etc/system/default/distsearch.conf replicate.eventtypes = true /opt/splunk/etc/system/default/distsearch.conf replicate.fields = true /opt/splunk/etc/system/default/distsearch.conf replicate.literals = true /opt/splunk/etc/system/default/distsearch.conf replicate.lookups = true /opt/splunk/etc/system/default/distsearch.conf replicate.multikv = true /opt/splunk/etc/system/default/distsearch.conf replicate.props = true /opt/splunk/etc/system/default/distsearch.conf replicate.segmenters = true /opt/splunk/etc/system/default/distsearch.conf replicate.tags = true /opt/splunk/etc/system/default/distsearch.conf replicate.transactiontypes = true /opt/splunk/etc/system/default/distsearch.conf replicate.transforms = true /opt/splunk/etc/system/default/distsearch.conf [replicationWhitelist] /opt/splunk/etc/system/default/distsearch.conf kvstore = kvstore/... /opt/splunk/etc/system/default/distsearch.conf other = (system|(apps/(?!pdfserver))|users(/_reserved)?//)/(bin|lookups)/... /opt/splunk/etc/system/default/distsearch.conf refine.conf = (system|(apps/)|users(/_reserved)?//)/(default|local)/.conf /opt/splunk/etc/system/default/distsearch.conf refine.metadata = (system|(apps/)|users(/_reserved)?//)/metadata/.meta /opt/splunk/etc/system/default/distsearch.conf searchscripts = searchscripts/... /opt/splunk/etc/system/default/distsearch.conf [tokenExchKeys] /opt/splunk/etc/system/default/distsearch.conf certDir = $SPLUNK_HOME/etc/auth/distServerKeys /opt/splunk/etc/system/default/distsearch.conf genKeyScript = $SPLUNK_HOME/bin/splunk, createssl, audit-keys /opt/splunk/etc/system/default/distsearch.conf privateKey = private.pem /opt/splunk/etc/system/default/distsearch.conf publicKey = trusted.pem ... View more

Our application logs events to the Windows application events with custom SourceNames. Need help to extract the fields using the props and transforms. I am able to extract the fields search time using the rex command, but the same doesn't work in props.conf . tried extracting one field - REGEX working in search - rex field=Message "^[^\]\n]*\]\s+(?P\w+)" props.conf [ebs_prod_api] REPORT-ebs_type = ebs_type_extract transforms.conf [ebs_type_extract] SOURCE_KEY = Message REGEX = ^[^\]\n]*\]\s+(?P\w+) Need to extract the following fields - Log_type - VERBOSE , ThreadId - {117} CorrelationId, Http status Code, Duration, Execution, Session ID . Sample data - 07/03/2019 11:33:38 AM LogName=Application SourceName=exs_nmon EventCode=0 EventType=4 Type=Information ComputerName=ex1nmkilo TaskCategory=%1 OpCode=Info RecordNumber=19214030 Keywords=Classic Message=[07/03/19 11:33:38.9356] **VERBOSE** **{117}** (NewFolder): (**CorrelationId**: IO-78904a2a-f22b-43bc-b39c-1188e9530622) Read Data for file version "Pipeline folder 2030\{1d38cda6-13f7-4353-bd46-8bde2659af97}" from \\nmsxlfs\jkil11638\nuimo1927\encdata\files\Pipeline folder 2030\{1d38cda6-13f7-4353-bd46-8bde2659af97}\Versions\00135_file.mr 07/03/2019 01:26:54 PM LogName=Application SourceName=exs_nmon EventCode=0 EventType=4 Type=Information ComputerName=ex1nmkilo TaskCategory=%1 OpCode=Info RecordNumber=32205553 Keywords=Classic Message=[07/03/19 13:26:54.8825] RESTAPI {121} (GET-/v1/folder/{fileid}/metadata): Instance ID: xcert123987; User ID: one_lo; **CorrelationId**: io19db84-f68b-4328-93eb-963cf77f0feb; **Execution**: Complete; **Duration**: 260 ms; **Http status Code**: 200; **Session ID**: ui90k2b4-b714-4caa-ab26-7d0ee7df3681; Node: enc169094num; **PODVersion**: 13.1; fileid: 29006663-92a5-4ff9-ad3b-64f387004cf7 Note - Not all events contain all the fields. Thanks Naveed ... View more

I need help in masking data in the payload emitted in the log. The application writes logs to Windows Event logs - Message=[2019-01-29 07:00:24,706] {1302} INFO SomeHelper::SendToDestination - {"RefId":"78c2511d-5aa6-4b92-9a50-3622d62ac1a2","Lender":{"Id":"4be018e4-de81-4142-adf0-cfa3dd923c7c","FirstName":"SomeFirstName","LastName":"SomeLastName","Quailification":false,"ID":"4be018e4-de81-4142-adf0-cfa3dd923c7c"} I have tried the following SEDCMD in props.conf , but it masks all the fields after the "LastName" in payload in the EventLog. SEDCMD-ananlname = s/("LastName":".*"/"LastName":"*****"/ ... View more

We have 4 tasks that run on different schedules and log an event in the application logs when the job starts. The task is to alert if the job doesn't run on a prescribed schedule. Can this be done with a single search command ? JobA 0 02 * * * JobB 0 2/4 * * * JobC 0 03 * * * JobD . 0 0/2 * * * Event produced in Log - [2018-08-29 06:00:00,912] {Worker #04283bd8} INFO JobA::Execute - ********************** Job Name : AgedApplicationNotificationJob - started at : 8/29/2018 6:00:00 AM ********************** Thanks ... View more