Hi team, I have been experiencing issues with log ingestion in a Windows Server and I was hoping to get some advice. The files are generated in a mainframe and transmitted onto a local share in a windows server via TIBCO Jobs. The files are generated in 9 windows throughout the day -  3 files at a time varying in size from a few Mb to up to 3Gb. The solution has worked fine in lower environments, likely, because of looser file/folder restrictions, but in PROD, only one or two files per window get ingested. The logs in indicate that Splunk can't open or read the files:   The running theory is that the process that is writing the files to the disk is locking them so Splunk can't read them.  I'm currently reviewing the permission sets for the TIBCO Service Account and the Local System Account (Splunk UF runs as this account) in the lower environments to try and spot any differences that could be causing the issue - based on the information in the post below: https://community.splunk.com/t5/All-Apps-and-Add-ons/windows-file-locking/m-p/14126 in addition to that, I was exploring the possibility of user the "monitornohandle" stanza as it seems to fit the use case I am dealing with - monitor single files that don't get updated frequently. But I haven't been able to determine, based on documentation, if I can use wildcards in the filename - for reference, this is the documentation I'm referring to: https://docs.splunk.com/Documentation/SplunkCloud/9.2.2406/Data/Monitorfilesanddirectorieswithinputs.conf#MonitorNoHandle.2C_single_Windows_file I'd appreciate if I could get any insights from the community either regarding permission or the use of the "monitornohandle" input stanza. Thanks in advance, ... View more

Hi community, I have observed an issue with the ingestion of the first line in a log file that, at first glance, seemed to have been truncated. Here's a screenshot for reference: My apologies for the poor job at blurring the data, but the first event should look like the second event, with a whole lot of data after the highlighted field. The field DistPoint itself should have a value of "DEPSY.IM2" and, it got, apparently, truncated at such a weird point. All other subsequent lines in the log were successfully ingested. There were 3 log files landing on the ingestion point in quick succession - seconds apart, so I am not sure if this could have been the issue. I was about to update the truncate value for the sourcetype, but all lines in the logs are 3551 bytes, by default. Any ideas as to what could the problem have been? Thank you. ... View more

Hi Splunk Community, I need to build an alert that will be triggered if a specific signature is not present in the logs for a period of time. The message shows up in the logs every 3 or 4 seconds in BAU conditions, but there are some instances of longer intervals going up to 4 minutes. What I had in mind was a query that ran over a 15-time timeframe using 5-minute buckets - to ensure that I would catch the negative trend and not only the one offs. I have made it this far in the query:   index=its-em-pbus3-app "Checking the receive queue for a message of size" | bin _time span=5m aligntime=@m | eval day_of_week = strftime(_time,"%A") | where NOT (day_of_week="Saturday" OR day_of_week="Sunday") | eval date_hour = strftime(_time, "%H") | where (date_hour > 7 AND date_hour < 19) | stats count by _time   **I only need the results for Monday to Friday between the hours of 7AM and 7PM. The query returns the count by _time, which is great, but if the signature is not present I don't get any hits, obviously. So I can count the number of occurrences within the 5-minute buckets, but I can't assess the intervals or determine the absence using count. I thought of, perhaps, manipulating timestamps so I could calculate the difference between current time and the last timestamp of the event, but I am not exactly sure how to compare a timestamp to "now". I would appreciate if I could get some advice on either how to count "nulls" or how to cross-reference the timestamps of the signature against current time. Thank you all in advance. ... View more

Hi Splunk Community, I need to create an alert that only gets triggered if two conditions are met. As a matter of fact, the conditions are layered: Search results are >3 in a 5-minute interval. Condition 1 is true 3 times over a 15-minute interval. I thought I would create 3 sub-searches within the search and output the result in a "counter" and I would, then, run a search to identify if the "counter" values are >3: index=foo mal_code="foo" source="foo.log" | search "{\\\"status\\\":{\\\"serverStatusCode\\\":\\\"500\\\"" earliest=-5m@m latest=now | stats count as event_count1 | search "{\\\"status\\\":{\\\"serverStatusCode\\\":\\\"500\\\"" earliest=-10m@m latest=-5m@m | stats count as event_count2 | search "{\\\"status\\\":{\\\"serverStatusCode\\\":\\\"500\\\"" earliest=-15m@m latest=-10m@m | stats count as event_count3 | search event_count*>0 | stats count as result I am not sure my time modifiers are working correctly, but I am not getting the results I expected. I would appreciate if I could get some advice on how to go about this. ... View more