We have a distributed splunk (8.x) environment on-prem, with CM and 3 peers, 2 SH, 1 deployment server, and many clients. On on of my Windows 10 clients, I have a csv file  that gets new data appended to it every 5 minutes via a separate python script that runs locally. I have verified the appending is working as intended. On the UF, I have setup an App with a monitor input  to watch the file. A custom props.conf is located on the indexer peers. I am experiencing some unexpected behavior and am struggling to find a solution.  The first odd thing I have noticed is that when data is appended  to the csv file, either via the python script or manually adding, I sometimes will get an event ingested but sometimes it will not. If I manually add lines 4 or 5 times over a 15 minute period, I might get 1 event. Sometimes I wont get any events at all, but if I get one event, its never more than 1.  The second weird thing noticed is that the event is always the top line of the csv file. Never the line that I added manually or via python to the bottom of the CSV file. The file is over 2500 lines. I have verified that the lines are actually appended to the bottom, and persist.  I suspect that there might be an issue with the timestamp or possibly the LINE_BREAK but I cannot say definitively that is the issue. (Maybe the LINE_BREAK regex is not actually there?) I can take the csv file, and add it without issue using the "Add Data" process in the SplunkWeb. It breaks the events exactly how I would expect it should (not just the top line) with correct timestamps, and looks perfect. Using the Add Data method, I copied the props from the WebUI and tried to add it to my Apps custom props.conf that is pushed to the peers. I am still left with the same weird behavior as I experienced with my custom props.conf file. I am reaching a point where googling is becoming too specific to get me a solid lead on my next steps in the troubleshoot. Does anyone know what might be causing either of these issues? Here is a snippet of the csv file (the top 3 lines):   profile_val,start_time,scheduler_tree,bot_name,tree_name,query_val,kill_option,stopped_on,remarks,status_val,ts Fit,11/03/2022 19:34:00.277,lb_prod_scheduler,spotbot,,%20redacted%20auto,false,11/03/2022 19:40:00.107,BOT did not pinged for more than 300 seconds,failed,2022-11-04 00:40:05 Fit,11/03/2022 19:49:00.143,lb_prod_scheduler,spotbot,,%20redacted%20auto,false,11/03/2022 19:55:00.091,BOT did not pinged for more than 300 seconds,failed,2022-11-04 00:55:05     Here is my custom props.conf that I initially tried:   [bot:logs] description = BOT Application Status Logs category = Custom disabled = false CHARSET=UTF-8 KV_MODE = none LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_FORMAT = %m/%d/%Y%t%H:%M:%S.%f TIME_PREFIX = Fit, INDEXED_EXTRACTIONS=CSV FIELD_DELIMITER=, FIELD_NAMES=profile_val,start_time,scheduler_tree,bot_name,tree_name,query_val,kill_option,stopped_on,remarks,status_val,ts     Here is the "Add Data" props.conf that works in the WebUI but not on the indexers:   [bot:logs] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=UTF-8 EXTRACT-Ex_account=^(?:[^:\n]*:){4}(?P<Ex_account>\d+) INDEXED_EXTRACTIONS=csv KV_MODE=none category=Structured description=Comma-separated value format. Set header and other settings in "Delimited Settings" disabled=false pulldown_type=true     ... View more

I am trying to understand what would cause a variance in the volume used in our quota vs the log size it is ingesting. I have been failing to find an explanation, and am wondering if anybody else has figured our the reason for the variance.  As an example, I am sending syslog from a wifi AP to our syslog server that is running a UF. On the syslog server,  on Oct 27 I am getting 484 lines of logs and those logs have 91,462 bytes in the log file.  In Splunk if I search for these events using this search:     index=network sourcetype=wifi | eval eventSize=len(_raw) | stats sum(eventSize) count by sourcetype     I also get 484 events but using len(_raw) function, I get length of characters of 90,978. I would assume these numbers should be pretty close and they are.  Now when I look in the _internal index for the *license_usage.log  metrics for the wifi sourcetype, using this query:     index=_internal sourcetype=splunkd source=*license_usage.log type=Usage st=wifi | stats sum(b) as bytes by st     I get a different sum of 103,794 bytes.  I am trying to determine how this could be or makes sense. This isn't a large difference (roughly 12%), but its spread out over every index and sourcetype and combined equals a large part of our license quota  that I cannot explain for.  Another example is our firewall logs, which is one of our largest indexes. For Oct 27, len(_raw) = 62,899,298,079 chars in length.  *license_usage.log = 81,139,209,296  This is a difference of roughly 22% over much larger percentage of our quota. There are other large indexes with similar variance.  I only have one production cluster, and I don't have a great way to verify this would be the same result on someone else's cluster. Do other people have this issue? I am trying to find a logical reason for why this would be the case. Things that I have tried to track down: 1) Using the License Usage dashboards in the Monitoring Console of our LM, on License Usage - Today I see numbers that align with the metrics using the len(_raw) metric. Using the Historical License Usage dashboard - If I switch to "NoSplit" parameter, I get numbers that also align with the len(_raw) metrics. If I change the Historical License Usage parameter to "SplitByIndex" I get numbers that align with the *license_usage.log metrics.  2) I have a support case opened to try to understand this difference. My SE told me that the "NoSplit" parameter (which is using the Type=RolloverSummary attribute in its base search) is the correct metric to measure license usage. My Support tech has told me that this is false, and that the "SplitByIndex" metric (using type=Usage) is the true count. Based on my manual measurements of the logs on the syslog server, I have to agree with the SE, but do not have any way to prove my LM is reporting incorrectly. 3) I have looked for duplicates using a variety of searches. Most show a couple of events, but we are talking less then 10, and its typically isolated on log sources with verbose or debugging output like DNS. 4) I have looked for misconfigured inputs.conf or outputs.conf files. This did yield some results. I found one SH that had multiple outputs.conf files that were cloning some of the data inputs originating from that SH (a WIN!!), but in regards to the syslog wifi and firewall sourcetypes, this doesn't seem to be the case. 5) I have reviewed the character encoding that is described here:  https://docs.splunk.com/Documentation/Splunk/latest/Data/Configurecharactersetencoding Every props.conf that I can find is set to CHARSET=UTF-8, which as I understand means Splunk is encoding all ingested logs using UTF-8. I thought this might be the culprit as higher order characters can take up multiple bytes in UTF-8. I do not think this is the case, as the syslog logs for wifi are only using the lower ASCII characters of UTF-8 from 0-127, which I believe are only taking up 1 byte per character. I am making this claim based on https://www.rfc-editor.org/rfc/rfc5424 and just visually looking at the logs, there is not that much to them. This also aligns with my manual measurement of the characters in the shell and using len(_raw).  Am I missing something here (or not understanding the underlaying process)? Is there other root causes I am overlooking? Does my cluster have some sort of issue, maybe with the configuration or architecture? Do others have this similar extra volume that is not easily explained, and this is normal behavior? Anything helps, thanks. ... View more

I have suspicious that my outputs.conf configuration files are causing some unwanted data cloning in my forwarders. I am trying to make sense of some weird behavior I am observing, I am hoping someone can fact-check my assumptions for validity, or tell me what if I am not understanding this issue correctly.  I have a UF on a syslog server. On the UF is a variety of apps, only a few of which possess a outputs.conf file.  If I search for outputs.conf files, these are the 4 that I find:     ./apps/SplunkUniversalForwarder/default/outputs.conf ./apps/comp_all_forwarder_outputs/local/outputs.conf ./apps/comp_all_outputs/local/outputs.conf ./system/default/outputs.conf     Based on the conf file hierarchy rules, I would expect that the two with ./local/outputs.conf would take priority over the other two with ./default/outputs.conf. Taking a look at each file, one is specifying indexer peers by FQDN, and the other is specifying the peers as IP addresses. Since both files have the same priority, and they are not the same conf file, would this create a scenario where Splunk sends data to the indexer tier twice (once for each outputs.conf file) cloning the data into the same indexing tier? /opt/splunkforwarder/etc/apps/comp_all_outputs/local/outputs.conf     [tcpout] defaultGroup = primary_indexers [tcpout:primary_indexers] server = spkidx01.comp.com:9997, spkidx02.comp.com:9997, spkidx03.comp.com:9997 autoLB = true       /opt/splunkforwarder/etc/apps/comp_all_forwarder_outputs/local/outputs.conf     [tcpout] defaultGroup = primary_indexers [tcpout:primary_indexers] server = 10.15.4.229:9997, 10.15.5.85:9997, 10.15.4.250:9997     The IP Addresses listed resolve to the FQDNs in the previous outputs.conf file. I would expect Splunk or maybe the OS would call these two separate outputs.conf files   TIA!     ... View more

Hello - I am trying to troubleshoot an issue and have not had much success in determining a root cause. I was wondering if anybody else has ever seen this issue. I inherited this cluster about a year ago, which is a distributed cluster hosted mostly in our own cloud environment with a few parts located on-prem. The indexer tier consists of a CM with 3 peer nodes. The search tier is two SH (not setup as a cluster), one being the main SH used by the company, and the other SH is running ES and is also he LM for the deployment. We have a license quota of 300Gb/Day, and until very recently I believed we were averaging around 280-320 Gb/day during the workweek.   In the past year, we have been told on multiple occasions that we are exceeding our license quota, and it has been a continuous effort to monitor and reduce our sourcetypes when possible to manage our daily ingestion volume. This has been something that we have spent a lot of time trying to maintain. Doing some log analysis on our sourcetypes, I have discovered what I believe is a duplicate scenario somewhere between the ingest point and the indexer tier. In a rough example to show my case, I have been trying to take measurements of our ingest to better understand what is happening. Using the method recommended in the Splunk Forums, I started out measuring our ingest using this base search.      index=_internal sourcetype=splunkd source=*license_usage.log type=Usage earliest=-1d@d | stats sum(b) as bytes by idx | eval gb=round(bytes/1024/1024/1024,3)     This sum came out to be between 280-300 Gb /day. I then tried to measure the bytes in the logs using the len() aggregate function, in a search like this:     index=* earliest=-1d@d | eval EventSize=len(_raw) | stats sum(EventSize) by index       This search sums out to around 150 Gb /day.  From my understanding, this is completely the opposite of what I expected. I did not expect the numbers to be exactly the same, but being roughly double does not make sense to me. If I am ingesting bytes, and Splunk is counting as (bytes)*2, this is a big issue for us. Please let me know if I am incorrect in this assumption.  Another case example. To reproduce this phenomena, I created a scenario that ingests a small amount of data to hopefully make measuring the difference easier to do.  I created a script that makes an API call to another solution in our network, to pull down information about its clients. Since this list is mostly static, I figured it would be a nice and small data source that I can work with. Running the script locally in a bash shell returns 606 json events. When redirected to a local file, these 606 events equal 396,647 bytes in size.  Next I put this script in an app on my searchhead. I created a sourcetype specifically for the json data that the API call will be returning. I created a Script modular input to execute the API call every 60 seconds.  I enabled the modular input, let it execute once, then disabled it. Looking for the logs in search, Splunk is calling it 1210 events and checking the len(_raw) shows 792,034 bytes. This seems to be a very large issue for us, and it seems to be effecting every sourcetype that we have. I did a similar test on our DNS logs, and our firewall logs, which are our two largest sourcetypes. A close examination of the local log file, shows logs that are near exactly half the size in bytes as the same _raw log examined in Splunk. Has anyone every seen a issue like this before? I have a Case opened with Splunk but so far it has not yielded any actionable results and we are going on 2 weeks now of back and forth. Any ideas or insights would be greatly appreciated. TY   ... View more