Hello, everybody! I have Splunk Enterprise 7.3.2 infrastructure with Splunk UF's deployed particularly to our corporate on-premises Microsoft Exchange Server 2016 servers. We have 40 Exchange servers with really huge mail flow and we want to collect all the Message Tracking log into Splunk for future investigations. We do not have a license for Splunk App for Microsoft Exchange / Splunk Add-on for Microsoft Exchange, so I wrote a simple custom app for UF with inputs.conf / props.conf to reach my goal. Microsoft Exchange Server Message Tracking log files are CSV-files with first 4 commented # lines, then 5th commented filed headers line preceded with #Fields: , next data lines. Detailed format description can be find here https://docs.microsoft.com/en-us/exchange/mail-flow/transport-logs/message-tracking?view=exchserver-2019 I deployed the following inputs.conf to my UF's: [monitor://C:\Queue\TransportLogs\MessageTracking] disabled = 1 time_before_close = 30 sourcetype = my_exchange_logs_message_tracking ignoreOlderThan = 1d crcSalt = <SOURCE> whitelist = \.log$|\.LOG$ # blacklist = I deployed the following props.conf to my UF's: [my_exchange_logs_message_tracking] disabled = false SHOULD_LINEMERGE = false MAX_TIMESTAMP_LOOKAHEAD = 25 INDEXED_EXTRACTIONS = CSV FIELD_HEADER_REGEX = ^#Fields:\s*(.*) # HEADER_FIELD_LINE_NUMBER = 5 PREAMBLE_REGEX = ^#.* HEADER_FIELD_DELIMITER = , FIELD_DELIMITER = , FIELD_QUOTE = " TRUNCATE = 0 Everything seems to be working fine, but sometimes I see my INDEXED_EXTRACTIONS do not work for some source log file lines. I see two problems while looking to the events with SH: For some source lines I do not see any fields according to headers extracted and indexed For some other source lines I see fields extracted and indexed not to header line field names but to set of EXTRA_FIELD_## fields. Moreover, if one row fails configured extractions with described effect - all the future lines till the EOF. I carefully checked the "problem" lines and actually found no problems to these! I also put a test but the same configured inputs.conf / props.conf to another server, brought the "problem" source files here - and these got indexed with no problems. With that written I expect my UF's on production Exchange Servers sometimes maybe have some problems, that prevent processing configured field extractions. I took a look into _internal but did not mention anything suspicious according to field extractions. Does maybe anybody have any ideas what should I check next to catch and fix the problem? ... View more

Hello! I have a distributed deployment of Splunk Enterprise. All my UFs send raw events to two HFs, these send cooked data to three-node IDX cluster. My search interface is three-node SH cluster. I plan to use a few ingest-time eval fields, first of all I tested how it works, placing: props.conf to HFs: props.conf ------------------------------------------------------------ [airwatch_iis_flogs] TRANSFORMS = ingest-eval-rule-size_bytes, ingest-eval-rule-orig_host, ingest-eval-rule-orig_time transforms.conf to HFs: transforms.conf ------------------------------------------------------------ [ingest-eval-rule-size_bytes] INGEST_EVAL = size_bytes=len(_raw) [ingest-eval-rule-orig_host] INGEST_EVAL = orig_host=upper(host) [ingest-eval-rule-orig_time] INGEST_EVAL = orig_time=_time fields.conf to SHs: fields.conf ------------------------------------------------------------ [size_bytes] INDEXED = True [orig_host] INDEXED = True [orig_time] INDEXED = True I put props.conf/transforms.conf to HFs (not to IDXs) as these servers process all the raw events and cook the data for indexers. This configuration works like a charm: querying index=* sourcetype=airwatch_iis_flogs I get the events having expected indexed fields size_bytes (calculated), orig_host (uppercase) and orig_host (match _time). Now, field of interest to me is _indextime. I want to index the latency: transforms.conf ------------------------------------------------------------ [ingest-eval-rule-latency_sec] INGEST_EVAL = latency_sec=_indextime-_time I also put the relevant changes to props.conf and fields.conf, but unfortunately this configuration doesn't work. Is it maybe because _indextime is empty while cooking events on HFs, and actually filled up while writing events to disk on indexers (not sure, where IndexQueue lives - on IDXs or HFs)? What should I do to use this _indextime field in ingest-time eval - maybe put: outputs.conf ------------------------------------------------------------ sendCookedData = false to my HFs, move all props.conf/transforms.conf to IDXs? I feel myself, there are more drawbacks then benefts from this decision. Are there any more limitations using _indextime ingest-time eval? ... View more

Hello guys! I now work on ITSI service models and health. I want my service models to be lightweight and elegant, but support drill-down to the exact problematic component and most importantly reflect the real service state from user perspective. I have done many experiments, but still cannot implement some ITSI services and KPIs satisfying my requirements. For example, I want to build a 3-node cluster service, containing host1 , host2 and host3 . I have a KPI query to get each host UP and DOWN state. I want to achieve the following: 1 My cluster service KPI value does not decrease at all until at least two of three nodes go to DOWN state. The service ServiceHealthScore decrease if two of three nodes go to DOWN state: If only any one node goes to DOWN state, cluster service ServiceHealthScore KPI should remain Normal (Green color) with value 100. The broken host itself ServiceHealthScore should be High (Orange color). If two nodes go to DOWN state, cluster service ServiceHealthScore KPI should be Low (Yellow color). The broken hosts - High (Orange color). If all three nodes go to DOWN state, cluster service ServiceHealthScore KPI should be High (Orange color). All three broken hosts - High (Orange color). 2 Visualize the state of each host in cluster in ITSI Service Analyzer. 3 Have a possibility to alert state change of each individual host and whole cluster by ITSI Notable Event Aggregation Policies. Below is what I managed to achieve. Test 1: A very simple service model, one KPI counting number of alive nodes. I know, how many hosts in cluster are UP, I can create events on service state change. Neither I cannot show individual hosts state (poor drill-down), nor alert host state change. Administrator must find affected nodes beyond the Service Analyzer. This is unacceptable. Service health is calculated as I want it: If one host DOWN - service has Normal severity, correct: If two hosts DOWN - service has Low severity, correct: If all three hosts DOWN - service has High severity, correct: Test 2: Same simple service model, but KPI has split by entity setting. I know how many hosts in cluster are UP, I know which hosts are UP and DOWN. Unfortunately, ServiceHealthScore does not work as it should for the cluster: If one host DOWN - service has High severity, wrong! (despite aggregate KPI is Normal, correct): If two hosts DOWN - service has High severity, wrong! (despite aggregate KPI is Low, correct): If all three hosts DOWN - service has High severity, correct: Test 3: This is what I like the most in Service Analyzer. UP/DOWN KPI are linked to individual services representing hosts, cluster state is calculated automatically. I could not find any way to force ServiceHealthScore KPI to behave like 'cluster' functionality: If one host DOWN - service has High severity, wrong: If two hosts DOWN - service has High severity, wrong: If all three hosts DOWN - service has High severity, correct: Actually, I want to implement my KPIs with Test 3 service model, which I personally like the most. Can anyone help me deal with ServiceHealthScore? Alternatively, maybe any workaround? ... View more

Hello, everybody! I now work on ITSI service models, I want my services to be created automatically from search, based on pre-created templates and support entity filtering to simplify KPI in template. I want my service models support deep drill-down to exact problem components, I decided to make every service a separate small ITSI service, base building blocks for huge business IT services. I created the sample service models manually and I love how it looks and works. To test service autodiscovery I have three entities named okd-node001 , okd-node002 and okd-node003 : I put the following scheduled search into /opt/splunk/etc/shcluster/apps/itsi/local/inputs.conf: [itsi_csv_import://okd-node test 01] log_level = INFO disabled = False backfill_enabled = 0 entity_title_field = DependentEntities import_from_search = true index_earliest = -15m index_latest = now interval = */15 * * * * search_string = | inputlookup itsi_entities | rename title as HostName | search HostName="okd-node*" | eval ServiceTitle = HostName." test 01", DependentEntities = HostName | fields ServiceTitle, DependentEntities service_enabled = 1 service_security_group = default_itsi_security_group service_title_field = ServiceTitle update_type = upsert and got the expected results: Three services named okd-node001 test 01 , okd-node002 test 01 and okd-node003 test 01 . Each service is filtered on appropriate entity. After that, I created a test service template named okd-node-template : and the followind service discovery search: [itsi_csv_import://okd-node test 02] log_level = INFO disabled = False backfill_enabled = 0 entity_title_field = DependentEntities import_from_search = true index_earliest = -15m index_latest = now interval = */15 * * * * search_string = | inputlookup itsi_entities | rename title as HostName | search HostName="okd-node*" | eval ServiceTitle = HostName." test 02", DependentEntities = HostName, ServiceTemplate = "okd-node-template" | fields ServiceTitle, DependentEntities, ServiceTemplate service_enabled = 1 service_security_group = default_itsi_security_group service_template_field = ServiceTemplate service_title_field = ServiceTitle update_type = upsert I got the following results: Three services named okd-node001 test 02 , okd-node002 test 02 and okd-node003 test 02 , all linked to okd-node-template service template. But none of the new services has entity filtering rule! So my KPIs work for all the entities in each service. I wonder, where where am i wrong with my second query? How should I fix this to enable both linkage to service template and entity filtering rule? ... View more