Have a nice day, everyone! For continuous event truncation tracking, I have a simple alert that notifies me about truncation: index="_internal" sourcetype="splunkd" log_level="WARN" "Truncating" | rex field=_raw "limit\s+of\s+(?<Limit>\d+)\s" | rex field=_raw "length\s+>=\s+(?<MessageLength>\d+)\s+" | rex field=_raw "data_host=\"(?<Host>.*?)\",\s" | rex field=_raw "data_sourcetype=\"(?<Sourcetype>.*?)\"" | stats count by Sourcetype, Host, Limit, MessageLength | top limit=5 Sourcetype, MessageLength | table Sourcetype, MessageLength After my last update, I have started to notice that the 'splunk_python' sourcetype hits the default TRUNCATE value a couple of times per hour. Overall, the events look fine, so I decided to tune the TRUNCATE value on the indexer cluster layer (manager-apps\_cluster\local\props.conf): [splunk_python] SHOULD_LINEMERGE = false TRUNCATE = 20000 Unfortunately, it also didn't help, and I'm still getting alerts with the default message length (10000). Then I set TRUNCATE to 0 and found no difference. What am I doing wrong? ... View more

Hello everyone! I'm trying to create a table view of IIS logs. The main issue I've encountered is some very long URL fields. In similar situations elsewhere, I've seen interactive "URL wrapping" — like clicking or hovering to reveal the full link. But Splunk's table view doesn't seem to offer anything like that. How can I handle this? ... View more

For some reason, I needed to share some data from an index with a different set of permissions. After a bit of research, I found that the CLONE_SOURCETYPE option could help me with this stuff. I created the required settings in props.conf and transforms.conf, and then pushed them to the IDXC layer. At first glance, everything seemed fine, but then I discovered that CLONE_SOURCETYPE clones all events from the original sourcetype and redirects only a few to the new one. Is that the intended behavior, or did I make serious mistakes in the configuration? I expected to see only the events matching the REGEX in the original index.   props.conf [vsi_file_esxi-syslog] LINE_BREAKER = (\n) MAX_TIMESTAMP_LOOKAHEAD = 24 SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE_DATE = false BREAK_ONLY_BEFORE = \d{1,3} TIME_PREFIX = ^<\d{1,3}> TRANSFORMS-remove_trash = vsi_file_esxi-syslog_rt0, vsi_file_esxi-syslog_ke0 TRANSFORMS-route_events = general_file_esxi-syslog_re0 transforms.conf [general_file_esxi-syslog_re0] CLONE_SOURCETYPE = general_re_esxi-syslog REGEX = FIREWALL-PKTLOG: DEST_KEY = _MetaData:Index FORMAT = general WRITE_META = true     ... View more

Hello everyone! I came across a strange behavior. I was building a dashboard and noticed that some results look unexpected. The results are presented at the top of the screenshot. On the last row, you can see that ProvDuration is 0. Also, StartTime and EndTime are equal. Moreover, other fields are also equal, and it's illogical due to the search specifics. As you can see, StartTime and EndTime represent the min and max values of the _time field.   index="hrz" (sourcetype="hrz_file_log" AND "*is provisioning") OR (sourcetype="hrz_file_syslog" AND EventType="AGENT_STARTUP") | rex field=_raw "VM\s+(?<MachineName>.*)$" | table _time, PoolId, MachineName, _raw | transaction MachineName startswith="Pool" endswith="startup" maxevents=2 keeporphans=false | search (PoolId="*") (MachineName="*") | search duration<=700 | stats min(duration) AS DurationMin, avg(duration) AS DurationAvg, max(duration) AS DurationMax, min(_time) AS StartTime, max(_time) AS EndTime BY PoolId | eval DurationMin = round(DurationMin, 2) | eval DurationAvg = round(DurationAvg, 2) | eval DurationMax = round(DurationMax, 2) | eval ProvDuration = round((EndTime - StartTime), 2) | eval StartTime = strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3Q") | eval EndTime = strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3Q") | table PoolId, DurationMin, DurationAvg, DurationMax, ProvDuration, StartTime EndTime   I decided to dig deeper and try to analyze the search more carefully. After I moved to the search through the dashboard, I found that the search results look different. The last row looks as it should be. You can see these results at the bottom of the screenshot. What could be wrong with my search, and what am I missing? ... View more

Hello to everyone! I'm not sure how to correctly name this thing, but I will carefully try to explain what I want to achieve. In our infrastructure we have plenty of Windows Server instances with Universal Forwarder installed. All servers are divided into groups according to the particular application that the servers host. For example, Splunk servers have group 'spl,' remote desktop session servers have group 'rdsh,' etc. Each server has an environment variable with this group value. By design, the access policy to logs was built on these groups. One group - one index. Because of it, each UF input stanza has the option "index = group.". According to this idea, introspection logs of UF agents are related to the SPL (or Splunk) group\index. And here the nuisance started. Sometimes UF agents report about errors that demand some things on the running hosts, for example, restarting the agent manually. I see these errors because I have access to the 'spl' index, but I don't have access to all Windows machines and I have to notify the machine owner about it manually. So, the question is how to create a sort of tag or field on the UF that can help me separate all Splunk UF logs by these groups? Maybe I can use our environment variable to achieve it? I only need to access this field during search time to create various alerts that will notify machine owners instead of me. ... View more

Hello to everyone! I want to build a dashboard with which I can access information from config files of indexer cluster I know that the typical scenario to access config files is using REST endpoints "/services/configs/conf-*" But as I understood, these endpoints show only configuration files stored under /system/local/*.conf Is it a way to access config files stored under /manager-apps/local ? ... View more