Hello, everybody! Does anybody can help with such an easy problem as counting events in summary index? I have a summary index populated with something like SS:   | tstats prestats=true summariesonly=false min(CPU.CPU_Performance.cpu_load_percent), avg(CPU.CPU_Performance.cpu_load_percent), max(CPU.CPU_Performance.cpu_load_percent) from datamodel=MODEL where nodename=CPU.CPU_Performance by host, CPU.CPU_Performance.cpu_instance | sistats min(CPU.CPU_Performance.cpu_load_percent), avg(CPU.CPU_Performance.cpu_load_percent), max(CPU.CPU_Performance.cpu_load_percent) by host, CPU.CPU_Performance.cpu_instance | addinfo | eval _time=info_min_time, host=upper(host) | fields - info_sid, info_search_time, info_min_time, info_max_time | collect index=my_summary   My SS is scheduled to run once an hour, so I every hour get 1 event for each orig_host in summary index. Now I want to check, if all the required events are here in summary index. I expect to get count=24 events for each orig_host in summary index for each day. When I try the search:   index=my_summary | stats count by orig_host   I get all the  psrsvd _ ct_ values summarized giving me not what I expected. How should I change my search to count events in summary index? ... View more

Hello, everybody! I have Splunk Enterprise 7.3.2 infrastructure with Splunk UF's deployed particularly to our corporate on-premises Microsoft Exchange Server 2016 servers. We have 40 Exchange servers with really huge mail flow and we want to collect all the Message Tracking log into Splunk for future investigations. We do not have a license for Splunk App for Microsoft Exchange / Splunk Add-on for Microsoft Exchange, so I wrote a simple custom app for UF with inputs.conf / props.conf to reach my goal. Microsoft Exchange Server Message Tracking log files are CSV-files with first 4 commented # lines, then 5th commented filed headers line preceded with #Fields: , next data lines. Detailed format description can be find here https://docs.microsoft.com/en-us/exchange/mail-flow/transport-logs/message-tracking?view=exchserver-2019 I deployed the following inputs.conf to my UF's: [monitor://C:\Queue\TransportLogs\MessageTracking] disabled = 1 time_before_close = 30 sourcetype = my_exchange_logs_message_tracking ignoreOlderThan = 1d crcSalt = <SOURCE> whitelist = \.log$|\.LOG$ # blacklist = I deployed the following props.conf to my UF's: [my_exchange_logs_message_tracking] disabled = false SHOULD_LINEMERGE = false MAX_TIMESTAMP_LOOKAHEAD = 25 INDEXED_EXTRACTIONS = CSV FIELD_HEADER_REGEX = ^#Fields:\s*(.*) # HEADER_FIELD_LINE_NUMBER = 5 PREAMBLE_REGEX = ^#.* HEADER_FIELD_DELIMITER = , FIELD_DELIMITER = , FIELD_QUOTE = " TRUNCATE = 0 Everything seems to be working fine, but sometimes I see my INDEXED_EXTRACTIONS do not work for some source log file lines. I see two problems while looking to the events with SH: For some source lines I do not see any fields according to headers extracted and indexed For some other source lines I see fields extracted and indexed not to header line field names but to set of EXTRA_FIELD_## fields. Moreover, if one row fails configured extractions with described effect - all the future lines till the EOF. I carefully checked the "problem" lines and actually found no problems to these! I also put a test but the same configured inputs.conf / props.conf to another server, brought the "problem" source files here - and these got indexed with no problems. With that written I expect my UF's on production Exchange Servers sometimes maybe have some problems, that prevent processing configured field extractions. I took a look into _internal but did not mention anything suspicious according to field extractions. Does maybe anybody have any ideas what should I check next to catch and fix the problem? ... View more

Hello! I have a distributed deployment of Splunk Enterprise. All my UFs send raw events to two HFs, these send cooked data to three-node IDX cluster. My search interface is three-node SH cluster. I plan to use a few ingest-time eval fields, first of all I tested how it works, placing: props.conf to HFs: props.conf ------------------------------------------------------------ [airwatch_iis_flogs] TRANSFORMS = ingest-eval-rule-size_bytes, ingest-eval-rule-orig_host, ingest-eval-rule-orig_time transforms.conf to HFs: transforms.conf ------------------------------------------------------------ [ingest-eval-rule-size_bytes] INGEST_EVAL = size_bytes=len(_raw) [ingest-eval-rule-orig_host] INGEST_EVAL = orig_host=upper(host) [ingest-eval-rule-orig_time] INGEST_EVAL = orig_time=_time fields.conf to SHs: fields.conf ------------------------------------------------------------ [size_bytes] INDEXED = True [orig_host] INDEXED = True [orig_time] INDEXED = True I put props.conf/transforms.conf to HFs (not to IDXs) as these servers process all the raw events and cook the data for indexers. This configuration works like a charm: querying index=* sourcetype=airwatch_iis_flogs I get the events having expected indexed fields size_bytes (calculated), orig_host (uppercase) and orig_host (match _time). Now, field of interest to me is _indextime. I want to index the latency: transforms.conf ------------------------------------------------------------ [ingest-eval-rule-latency_sec] INGEST_EVAL = latency_sec=_indextime-_time I also put the relevant changes to props.conf and fields.conf, but unfortunately this configuration doesn't work. Is it maybe because _indextime is empty while cooking events on HFs, and actually filled up while writing events to disk on indexers (not sure, where IndexQueue lives - on IDXs or HFs)? What should I do to use this _indextime field in ingest-time eval - maybe put: outputs.conf ------------------------------------------------------------ sendCookedData = false to my HFs, move all props.conf/transforms.conf to IDXs? I feel myself, there are more drawbacks then benefts from this decision. Are there any more limitations using _indextime ingest-time eval? ... View more