What comes after the last filter line? Have you checked whether the cases where it is not going as expected are related to a specific host (and does that occur in your list of filters)? ... View more

Hi thanks for your inputs updated as below UF CHARSET = UTF-8 KV_MODE = none Indexer [json_odelia] DATETIME_CONFIG = CURRENT SEDCMD-cut_footer = s/]\,\n\s*\"total\":.$/g SEDCMD-cut_header = s/^{\n\s\"matches\":\s[/g category = Structured disabled = false HEADER_FIELD_LINE_NUMBER = 3 SHOULD_LINEMERGE = 0 TRUNCATE = 0 INDEXED_EXTRACTIONS = json still not working ... View more

Yeah, so that messes things up. Splunk keeps track of the last line it has read and expects new logs to be added after that. In your case, that last line is the line with </Events> . When a new event is added, it is added on that same line and the </Events> is shifted down. So splunk detects that the last line it had already read is now changed and that triggers re-reading the entire file. I don't see any settings in inputs.conf that can change that behavior, so I guess looking at the data source to see if it can log in a different way is your best bet. I've upgraded my comment to an answer, so you can accept it. ... View more

Thanks, this worked like magic 🙂 ... View more

By configuring your esxi hosts to send their syslog to a separate syslog server. See: https://docs.splunk.com/Documentation/AddOns/released/VMW/ESXihosts#Configure_ESXi_hosts_to_send_data How exactly you setup that syslog server and send the logs from that syslog server to Splunk Cloud is up to you. This can be done in various ways. The classic way of doing it is have the syslog server run rsyslog/syslog-ng which receives the syslog from esxi and write it to files. Then put a UF on the syslog server to read those files and send them to Splunk Cloud. More modern approaches involve solutions that enable you to receive syslog data and send it to HEC. E.g. Splunk Connect for Syslog, or using rsyslog/syslog-ng's built in http output modules (requires recent versions of rsyslog / syslog-ng), or using Ryan Faircloth's omsplunkhec. ... View more

Assuming your regex is correct. The way to change the index at index time (with props/transforms) is : on the first splunk instance that will parse the data. It means usually on the indexers. But if you have intermediary heavy forwarder, it will have to be on the first one of the forwarding chain. And if your data is parsed by the monitoring forwarder (look for INDEXED_EXTRACTIONS setting in props), then the parsing is happening on the first forwarder (could be the UF) ... View more

Any configuration pushed from the deployment server will be in specific apps under C:\Program Files\SplunkUniversalForwarder\etc\apps\ on your windows universal forwarders. So have a look there, what apps your team has configured to push to the UF and what is inside those apps. You might also want to ask around in the team you joined how they set it up, hopefully they also have some documentation on the setup. After all: we can only explain how things generically work and how things are set up according to best practice. But only your teammates can tell you how it is really done in your environment 🙂 ... View more

So, you want to ingest all those lines as individual events into splunk, but you want to use the overall timestamp from the header of that structured file? Not sure that is possible really. You might get better results by ingesting this whole thing as a single event, with the timestamp from the header, and do further manipulation of the event as part of your searches. The multikv command will be very useful for that, see https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Multikv Either that or change the way you produce this log to produce a log that contains something more compatible with Splunk (ie. line by line events, including a timestamp, rather than such structured log files). Or do some preprocessing to transform these files into something that is easy to ingest into splunk. ... View more

Don't think the join approach will work like that, but I think your first approach should work with a few adjustments. One thing you might want to do is is add the following in the subsearch: | fields cardid | format earliest=-5m index=iis_openapi /internal/loyalty/v1/ cs_uri_stem="registrations" cs_method = "POST" cardid="*" NOT [ search earliest=-5m index=log-cdx-test source=kubernetes sourcetype=_json "PROCESSOR_E2E_TRACKING" "cardRegistered" "cardId" cardNumber="" | rename cardNumber as cardid | fields cardid | format] | table cardid Do note that there are limits to the number of results subsearches can handle. Not sure what number of cardids you would expect to be involved in this. If that is low (since you only look over past 5min) I think this approach should work. An alternative approach (not using any subsearches and hence not restricted by subsearch restrictions) could be something along these lines: (earliest=-5m index=iis_openapi /internal/loyalty/v1/ cs_uri_stem="registrations" cs_method = "POST" cardid="*") OR (earliest=-5m index=log-cdx-test source=kubernetes sourcetype=_json "PROCESSOR_E2E_TRACKING" "cardRegistered" "cardId" cardNumber="") | rename cardNumber as cardid | stats values(index) by cardid | where index!="log-cdx-test" ... View more

Hi @essibong1, sorry, probably I wasn't so clear in my previous answer! this is a macro definition that I hinted in my answer to your previous question ( https://answers.splunk.com/answers/789359/can-any-one-provide-me-with-a-good-search-to-monit.html#answer-789363 😞 I used a macro to have in only one point the working time definition so I can modify it in only one point instead in each search of my app. Anyway, if you don't want to use a macro, you have to create your search starting from the filter for the access events (e.g. in windows EventCode=4624) and then filter for the hours and minutes. The other needed information is if you want only to filter for time or you want also to filter for holydays: if you want to filter only for time, you can run something like this (working time 8.00 - 17.00): index=your_index EventCode=4624 (date_hour<8 OR date_hour>18) | ... If instead you want also holydays (using the lookup), you have to create a lookup (called e.g. SIEMCAL.csv) containing two information: day in a format you like (e.g. dd/mm/yyyy), type (0 for working time, 1 for half working time, 2 for holydays). So for windows systems you can run something like this: index=your_index EventCode=4624 | eval day=strftime(_time,"%d/%m/%Y") | lookup SIEMCAL.csv day OUTPUT type | search type=2 OR (Tipo=1 (date_hour>14 OR (date_hour<8)) OR (type=0 (date_hour>18 OR date_hour<8))) | ... Ciao. Giuseppe ... View more

If you're just interested in yesterday's license usage and prefer to get it from the files, rather than by querying splunk. Look at the lines in those files that contain type=RolloverSummary . Those lines occur right after midnight and contain the bytes (b) ingested in the past day. If it is a single instance with a single license stack, there should be 1 event per day like that. And that may also explain why you were getting weird results, as the license_usage.log contains (at least) 3 different types of events: - Usage: every minute one event per index/source/sourcetype/host combo with bytes (b) ingested in past minute - RolloverSummary: daily summary as described above - SlaveWarnSummary: something else, not containing license usage info So if you sum the b field, without taking into account those different types of logs, you will be summing 'live' usage logs together with the daily summary. ... View more

I think the issue is with your initial conversion. You say you are using %m-%d-%Y %H:%M:%S %p UTC to interpret your original timestamps, that is incorrect. Your timestamps don't have 24h clock, they have 12h clock + AM/PM. When you use %H I bet the %p (AM/PM) gets ignored. Use %m-%d-%Y %I:%M:%S %p UTC Also: it would help if you would post the actual search code that you are using, so we can see more accurately what you are doing. ... View more

Those fields are indeed fine, if they are there and you don't run into timezone issues. Regarding the working time filter: I think you missed my point. By doing AND date_minute < 45 you'll never get any events back that happen in the last 15 minutes of each hour. With your filter, you get events from 0:00 - 0:44, 1:00 - 1:44, 2:00 - 2:44, 3:00 - 3:44, 4:00 - 4:44, 5:00 - 5:44, 6:00 - 6:44 and then 21:00 - 23:59. ... View more

Yes, assume case sensitivity ... As for your use case, i'd recommend to not even allow the option to write some conf files as you mentioned ... View more

"Total entries = 14995" props.conf LINE_BREAKER in single line printed JSON doc I hope this can be done well. ... View more

Hi @ CONSORP, the problem is that SplunkAppForBlueCoatProxySG works using summary indexes (because usually there are too many events to directly display in dashboards and dashboards are too slow). This means that you cannot limit to extract the new field but you have to do some other jobs: extract the new field as usual from the logs (e.g. using regex); modify the scheduled search that updates summary index adding the new field; modify all searches in dashboards panels. Ciao. Giuseppe ... View more

Correct, they both take place in the same phase. Not sure how it exactly works under the hood, but basically when the event enters the phase, splunk looks at host, source and sourcetype and based on that determines what props and transforms to apply during typing phase and then applies all those in a certain order. Those props and transforms may change host, source and sourcetype, but this does not trigger Splunk to re-evaluate what props and transforms to apply. Changing those 3 fields is basically only useful for search time. The only exception is when you do a CLONE_SOURCETYPE transform, there the cloned events are injected back in to the start of the pipeline and SED etc. is applied as if it had come in originally with the new sourcetype. But I guess you're not interested in cloning anything. ... View more

screenshot 1 - is the output of some scheduled search which is categorised as sourcetype stash, and it has latency in _indextime. Following is the search I used, which triggered output from screenshot2 | eval timediff=(_indextime-_time) | eval Hoursoff=round(timediff/3600) | search NOT Hoursoff=0 | rename _indextime as Indextime | eval Indextime=strftime(Indextime, “%d%m%y %H:%M:%S”) | eval Time=strftime(_time, “%d%m%y %H:%M:%S”) | dedup sourcetype | table Time, Indextime, Hoursoff, date_zone, host, sourcetype, source ... View more

Not with a simple single search command or so, no. The most basic way would be to just use that stats, look at the results and then think of some threshold (e.g. any counts ❤️ are suspicious). But you can also do some more elaborate statistics on it to calculate a threshold per user (maybe some admins are more dynamic in what IP they use than others). Splunk's Machine Learning Toolkit could also be used, but that might be a bit overkill. ... View more

Hi Frank that definitely works for me. I was working according to the examples in the link for subsearches, but as I am not looking for the events with the most counts, but for the calculated bandwidth, I never got it to work. Thanks for your quick and efficient answer! ... View more

I suggest brushing up on your basics regarding the use of stats etc. You're doing impossible things. You can't sum ip addresses like sum(Source_Network_Address) , you can't put as Total at the end of a stats command after the by clause. ... View more

Well, the average user typically doesn't look at the properties of the source type I think. Adding this kind of extra information to event fields would indeed be useful, that I fully agree with, but that is something slightly different as I mentioned in my answer. ... View more

Can you show a sample of your data showing the MANAGER_NAME and AMONAME fields and the result of the case statement as it is put into MANAGER_NAME1? ... View more

Ah, good to know! And glad you got it resolved! ... View more