I happen to have several long running searches having the same error message. Have you found any clue regarding these messages ? ... View more

@PaveIP, interesting information. Events I'm ingesting are indeed potentially multiline (95% of them are single line) and I break events based on the date. Here's the relevant props.conf stanza. [db.command.trace] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\.\d+\s #LINE_BREAKER_LOOKBEHIND = 4096 MAX_TIMESTAMP_LOOKAHEAD = 40 TIME_PREFIX = ^ #Wed Mar 25 04:05:00.978 2020 TIME_FORMAT = %a %b %e %H:%M:%S.%Q %Y I get errors from indexers regarding date/time extraction although when I search for events, I don't see any problem. I have not configured event breaker on the UF but I believe I should, I'll give it a try. But we are also ingesting many other source types, including multiline ones and we've never experienced such problem before so I was wondering how it works under normal situations. Thanks for the test but I don't completely understand how it's done. Is your event multiline ? Yes, a slow inputs will generally not work well with multiline. Splunk will send an event as soon as it reaches "\n"+EOF , so you need to play with the following parameter in inputs.conf: multiline_event_extra_waittime = <boolean> * By default, the file monitor sends an event delimiter when: * It reaches EOF of a file it monitors and * The last character it reads is a newline. * In some cases, it takes time for all lines of a multiple-line event to arrive. * Set to "true" to delay sending an event delimiter until the time that the file monitor closes the file, as defined by the 'time_before_close' setting, to allow all event lines to arrive. * Default: false ... View more

Hello PaveIP, thanks for your answer. I already knew about this parameter but it's disabled by default. [ By the way, note this documentation is partly wrong: the default value must be a capturing group. I reported it to splunk which acknowledged it as a documentation mistake and the true value will be mentioned in the next documentation release : "([\r\n]+)". ] . If you read props.conf : EVENT_BREAKER_ENABLE = <boolean> * Whether or not a universal forwarder (UF) uses the 'ChunkedLBProcessor' data processor to improve distribution of events to receiving indexers for a given source type. * When set to true, a UF splits incoming data with a light-weight chunked line breaking processor ('ChunkedLBProcessor') so that data is distributed fairly evenly amongst multiple indexers. * When set to false, a UF uses standard load-balancing methods to send events to indexers. * Use this setting on a UF to indicate that data should be split on event boundaries across indexers, especially for large files. * This setting is only valid on universal forwarder instances. * Default: false So by default, the UF doesn't break events on file's EOL but "a UF uses standard load-balancing methods to send events to indexers.". The standard load-balancing method is based on autoLBVolume and autoLBFrequency (see outputs.conf) and chunks of data (outputs.conf : "Non-parsing forwarders, such as universal forwarders, send blocks, which can be up to 64KB."). So like I mention, by default, one indexer may receive a chunk of data and another indexer will receive the next chunk. My question is "how do they get reassembled ?". ... View more

In addition, I'm wondering how this all works with an indexer cluster where chunks are spreaded over multiple indexers (if you have UFs connecting to indexers in a round robin). ... View more

I was wondering the same and came up more or less with the same conclusion as Masa BUT I have more questions. Let's take a concrete example: chunk1 : [data1]\n 2017-01-03 12:00:00 [data2]\n 2017-01-03 12:0 chunk2 : 0:01 [data3]\n ... So when splunk processes chunk1, the regex will match only once: after [data1]. So data1 will be correctly identified as being part of an event (but not yet data2 and the preceding timestamp). When processing chunk2, splunk will prefix the chunk with LOOKBEHIND bytes of chunk1. The regex will then match if LOOKBEHIND is greater or equal than length(data2) + 2*length(timestamp). However, I find it all quite clumsy. First, splunk should know that ALL the leftover of chunk1 should be processed (why limit it to LOOK_BEHIND ?). Then, the default of 100 bytes appear quite small: it will only work for events whose total length + timestamp length is below 100 bytes. Am I missing something ? ... View more

@mmodestinosplunk, I've compared your search with the one in the DMC for the total license. This one seems to take into account you can have multiple license masters (splunkserver in the search). It also uses a join which is a bit more explicit than a "|search []" in my opinion. So I've rewritten partly your search and the result is below. However, in both cases, you consider a license group can only contain 1 single stack (by renaming stackids to stackid), is it really the case ? Here's my search : | rest splunk_server=local /services/licenser/pools | rename title AS pool | join type=outer splunk_server stack_id [ rest splunk_server=local /services/licenser/groups | eval stack_id=stack_ids | fields stack_id splunk_server is_active] | search is_active=1 | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "% used"=round(used_bytes/quota*100,2) | fields pool "% used" | where '% used' >= 0 ... View more

Please define "soon"... Still the same issue today 🙂 ... View more

I downvoted this post because i upvoted it but it's actually a bad answer ... View more

I think he's talking about encryption and compression on disk as he is mentioning Pure Flash Array. By default, data is compressed but not encrypted. There used to be a parameter to disable encryption but it's now obsolete (see indexes.conf documentation ) compressRawdata = true|false * This parameter is ignored. The splunkd process always compresses raw data. ... View more

The youtube channel is empty, otherwise there would have been a workaround as you can subscribe to YouTube channels via rss... I won't have a look at splunk blogs any longer until this is fixed, sorry for you. ... View more

I think custom viz apps are inherating certain properties from more general vizualisation elements such as . I'm not an expert but anyway, the following worked for me to disable the "timeline" default drilldown : Under the element of your source XML, just add a dummy drilldown which sets a token, for instance: <viz type=...> ... <drilldown> <set token="sourcetype">1</set> </drilldown> </viz> ... View more

Try the gantt visualization app ... View more

Hello, I have not yet found time to create this chart (low priority here) but I found this viz app which could reveal quite useful to create such a dashboard: https://splunkbase.splunk.com/app/1741/ ... View more