Nice summary of the options. I'm much more familiar with rsyslog than with syslog-ng, so not suprised you found a few gaps / improvements on my suggested solution. Glad to see I helped you look in the right direction 🙂 ... View more

I think best practice is for network equipment like that to be set to UTC, regardless of the location it is in. That way a team monitoring/managing a global collection of equipment does not have to worry about time zone differences etc. when assessing logs and such. If you want to keep the devices set to various local time zones, then I think with what you have figured out around syslog-ng should give you the tools to at least enable splunk to interpret the (extra) timestamps correctly, so that the end user in splunk sees correctly interpreted time stamps that can be correlated across devices. Still, only the extra timestamps can be correctly interpreted and there may be cases where there is a delay between the event happening and syslog-ng receiving it. Which could hinder accurate correlation / sequencing. Alternatively if the amount of devices is fairly small and not too dynamic, you could also specify TZ setting in props.conf on hostname basis. That way you don't need the extra timestamp and splunk will use the more accurate original timestamp from the event itself. ... View more

Regarding your first comment: "it would be the same". The same to what? Not sure I follow you here. Regarding the second part of your comment: Yes, if you can configure your devices from different regions to log to different ports, you can either use syslog-ng logic to do the time zone interpretation based on that. Or: have syslog-ng simply write to regional folders and configure TZ setting in splunk's props.conf based on source field (which is the log file path). Then again, if you need to touch all your network equipment to make it log to a time zone specific port, you may just as well change the time zone they use (unless you have some specific reason to keep network equipment running in its local timezone) 😉 ... View more

The best way to solve this would be to configure all your network equipment (or at least those that do not log time zone info) to run / log in a common time zone (e.g. UTC). Alternatively, you could configure Splunk with per-host TZ setting in props.conf, but with many devices that might become hard to manage. If you really want to go down the route of adding an additional timestamp based on when syslog-ng processed the event, you can do that by defining your own template for how the events get written to disk. For example: template t_extra_timestamp { template("${R_ISODATE} ${RAWMSG}\n"); }; destination d_juniper { file( "/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-juniper.log" template(t_extra_timestamp) ); }; Note the R_ variant of the ISODATE macro, this uses the syslog-ng receipt time, rather than the timestamp from the message. There are other timestamp macros available as well, if you'd prefer a different timestamp format. The downside of this approach of course is that you can no longer rely on the indexed _time field in splunk for any searches that rely on when the event exactly happened (e.g. for Flow Tracing as you mentioned). Which likely means you also can no longer use accelerated data models for that kind of work, as those will be based on _time. And if you want to correlate events from different devices that are not in the same timezone, using search time extracted timestamps from the event will also not work (unless you come up with some tricky evals to adjust for the time zone offset as observed against the syslog-ng receipt time). ... View more

Thanks for the quick response Chris. We'll give it a go in a test environment first anyway, but good to hear that it should work and thanks for the specific pointers. ... View more

What comes after the last filter line? Have you checked whether the cases where it is not going as expected are related to a specific host (and does that occur in your list of filters)? ... View more

A $ sign is only needed for those date/time variables, not for message properties like %fromhost-ip%. ... View more

That sounds really strange, especially because you also filter for that same property. So if it was missing (which it normally never is for data that came from a UDP or TCP input) then it shouldn't even trigger the action? Is the config you shared here the full config, or did you perhaps simplify it to a point where the cause of the issue is no longer visible in your example? ... View more

Indeed and if %HOSTNAME% is also showing mixed results, then take a look at the messages to check if they consistently contain the hostname or not (probably not). Maybe see if the data source can be configured as to how it formats the messages, to ensure it always includes its hostname. ... View more

Indeed. Key thing here is adding the | mvexpand data . Otherwise you are working with a single row, with multivalued fields, which results in the outcome as per the screenshot. ... View more

What if you remove the INDEXED_EXTRACTIONS = json from the UF's config (and enable kvmode again, or move the indexed extractions to the indexer)? The UF will try to do the json extractions, without any of the custom line breaking and header stripping. And once the indexed extractions have been done, the downstream splunk enterprise instance will no longer apply linebreaking stuff if I'm not mistaken. Indexing JSON files that contain multiple events in one json structure is anyway a pain in the proverbial butt. You might also want to look at setting sensible EVENT_BREAKER settings on your UF, to at least make sure events arrive in one piece at the indexers. Or consider using a heavy forwarder for this, so that indexed extractions and linebreaking and such happen at the same place. But all in all, I think changing how this data gets logged, or do some pre-processing on the json file to transform it into individual events, might be the best (but not necessarily the easiest) thing to do here. ... View more

A virtual machine running a recent linux distro (or at least has a recent version of syslog-ng installed) with sufficient resources to handle your data volume. If you're worried about data loss, or want extra performance and better data balancing towards your splunk environment, you can also setup multiple of these boxes with a load balancer in front. As mentioned in my comment above, it is usually recommended to have the syslog daemon write to file and install a UF on the same box to read from those files. Alternatively you could look at solutions that allow the syslog server to send straight to HEC: - Splunk Connect for Syslog: https://splunkbase.splunk.com/app/4740/ - its much simpler predecessor omsplunkhec: https://bitbucket.org/rfaircloth-splunk/rsyslog-omsplunk/src/445676ad128d8ca5de3b573c55450ecc13b3dd88/omsplunkhec.py - your syslog daemon's native http destination: https://www.rfaircloth.com/2019/04/22/to-hec-with-syslog-all-grown-up/ ... View more

Why recommend using a HF, when a UF would serve perfectly fine for network inputs? Also, there are several reasons why using network inputs is not considered best practice for ingesting syslog data. Using a syslog daemon that writes to files is significantly more robust against data loss (e.g. due to splunk blocking its input queues / during splunk restart); especially with UDP. It also allows making use of syslog's built in features to write logs from different devices to separate files / folders, potentially making host assignment in splunk easier and more efficient. It also makes troubleshooting easier. Also more modern approaches are appearing the last few years, with several solutions for having a syslog daemon forward logs directly to a HEC endpoint. Cutting out the need of a forwarder completely resulting in a more performant, easier to load balance flow. ... View more

Well yes, using splunktcp between HF and Indexer would indeed be much better. Not sure why you would ever want to use syslog for a splunk-to-splunk connection. ... View more

Can you show what multiple consequtive syslog messages looks like when there are multiline events involved? Also: what do you mean by "I think I realy need to use intermediate forwarder to proper send data to my indexer."? How is that relevant to understanding what happens in syslog forwarding? ... View more

So, you are using a UF to collect windows logs locally on a server and forward them using normal splunktcp to a HF and from there you want to forward them as syslog? How are you determining what the syslog events look like? Are you checking that on the receiving server, or using tcpdump or so on the HF to see what it is actually sending? My experience is that Splunk will nicely send 1 event as 1 syslog message (as in: only prepend the syslog header 1x per event, not for each line). The problem is that syslog is not really designed for multiline logs, so likely the receiving syslog server will split it line by line. So you'll likely need to look at tuning the receiving side to handle these multiline logs better. But whether your syslog server is capable of doing that is quite a question. One alternative would be to configure a clone sourcetype TRANSFORMS on the HF (assuming you also want this data to flow to splunk without modification), and on the cloned sourcetype apply a SEDCMD that strips out the newlines (or replaces them with some specific character) and then send that data to syslog output. ... View more

Normally you'd find that in the documentation of the product that is producing those logs. If not there, than your guess is as good as mine. You could post a few sample events here, see if someone recognizes it. ... View more

Is this bit in the same file: 2020-03-10T11:20:38.710+1100: 687207.416: [Event2, 0.0204509 secs] and that should go into a second event? Meaning: you don't actually want to ingest the whole file? Try this in props.conf for the relevant sourcetype on your indexers: TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z MAX_TIMESTAMP_LOOKAHEAD = 28 TRUNCATE = 0 SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)\d+-\d+-\d+T ... View more

Yeah, so that messes things up. Splunk keeps track of the last line it has read and expects new logs to be added after that. In your case, that last line is the line with </Events> . When a new event is added, it is added on that same line and the </Events> is shifted down. So splunk detects that the last line it had already read is now changed and that triggers re-reading the entire file. I don't see any settings in inputs.conf that can change that behavior, so I guess looking at the data source to see if it can log in a different way is your best bet. I've upgraded my comment to an answer, so you can accept it. ... View more

Can you please edit your question and put the content of the last line between ` characters? I guess the file actually ends with some xml tag, but due to how this splunk answers forum works, anything like <bla> disappears when it is not posted in code tags. If the last line contains a closing XML tag, which shifts down, that is why splunk fails on that. Not sure if there is any way to fix that. Really weird logging format to be honest, can this not be configured differently? What kind of logs are we actually talking about? ... View more

When a new event is appended, are older events removed from the file as well? Or does the file keep growing? ... View more

There are some under Data Source Category "Webserver Access Logs". Some, maybe even most, of those rely on using the CIM Web data model. ... View more

The 'normal' SEP TA (https://splunkbase.splunk.com/app/2772/) is for use with other ingestion methods and will not work with the log format you get from symantec's syslog output. Have a look at the specific SEP TA for use with syslog input: https://splunkbase.splunk.com/app/3121/ Make sure kiwi writes the log messages in a format that matches what the TA expects. Downside of Kiwi is that, as far as I know, it does not support writing the original full raw syslog message, but hopefully it has an output format that is suitable. If not: you may have to customize the TA a bit to make it work properly (or consider moving away from kiwi, but that is a separate discussion). Also: what exact symantec app are you using for the dashboards you mention? ... View more