All Apps and Add-ons

Syslog Message from different Time Zone, Timestamp Syslog-ng

vessev
Path Finder

Hello fellow splunker,

i have a problem with syslog messages from a juniper SRX firewall in a different timezone. I will give a lot of details below "like always" but at first let me give you an short overview of the situation.
The raw Syslog Message from a Firewall in Tokyo when send to my syslog-ng (intermediate forwarder) in germany looks like this:

<14>May  8 11:44:44 juniperfw01 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 10.8.5.63/16744->10.0.22.222/30013 0x0 None 10.8.5.63/16744->10.0.22.222/30013 0x0 N/A N/A N/A N/A 6 KAPFW-Tmp-KP01 FDMZ WAN 76060 N/A(N/A) reth0.0 UNKNOWN UNKNOWN UNKNOWN N/A N/A

You will see there is a timezone information missing per default. Juniper..

Syslog-ng now manipulates the message header (Host, Timestamp) depending on the config. After that the syslog message gets written to the disk where a universal forwarder collects the logfile and sends it to my splunk indexer.

The Message will appear in Splunk like this (different flow but its the header which is important here):

May 11 10:29:00 juniperfw01.contoso.com junos-alg: RT_ALG_WRN_CFG_NEED: MSRPC ALG detected packet from 10.46.0.11/53461 which need extra policy config with UUID:12345678-1234-abcd-ef00-01234567cffb or 'junos-ts-apc-any' to let it pass-through on ASL session

What i really whant is what the syslog collecter integrated in splunk is doing per default (Adding a second header):

May  8 11:44:40 juniperfw01.contoso.com May  8 18:44:40 juniperfw01 RT_FLOW: RT_FLOW_SESSION_DENY: session denied 10.46.30.21/54085->61.205.120.130/123 junos-ntp 17(0) DEFAULT-DENY-LOG(global) FRONT WAN UNKNOWN UNKNOWN N/A(N/A) reth3.0 UNKNOWN policy deny

A Splunk Syslog collector has a few options which are very nice:
1. If an event timestamp does not include a recognizable timezone, the Splunk platform uses the time zone of the host that indexes the event.
2. For example, if an event with field timestamp="2016-01-26 21:58:38.107" is indexed by a Splunk platform instance in GMT-8, it will subtract 8 hours from the timestamp and set the _time of event to: 2016-01-26T13:58:38.000-08:00

3. The Splunk platform normally looks at the text of the event for the timestamp, and by default will select the leftmost recognizable timestamp.

I'm sure there are multiple ways to accomplish this. But my solution needs a few necessary things:
1. I need the original timestamp included (like done by splunk) - this is necessary because the index timestamp and the event time are slightly different. This would make a Flow Tracing to be impossibile.
2. The traffic must be collected from the intermediate forwarder first.

I would love to accomplish this by modify the syslog-ng config which would add another header but i did not find anything about that.

Details about my config:

        +-----------------+         +-------------------------------+         +-----------------------+
        |Syslog Originator|---->----|intermediate forwarder udp:5517|---->----|Splunk Indexer tcp:9997|
        +-----------------+         +-------------------------------+         +-----------------------+

Intermediate Forwarder:
syslog-ng conf:

options {
    log_fifo_size(1024);
    time_reopen (10);
    chain_hostnames(no);
    keep_hostname(yes);
    use_dns(yes);
    use_fqdn(yes);
    create_dirs (yes);
    group(syslog-ng);
    dir_perm(0750);
    perm(0640);
};


source s_udp5517 {
    udp(
        port(5517)
        keep-hostname(no)
        use-dns(yes)
    );
};

destination d_juniper { file( "/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-juniper.log"  ); };

# Log
log { source(s_udp5517); destination(d_juniper); };

And the splunk inputs.conf from my universal forwarder running on this intermediate forwarder:

# Juniper
[monitor:///data/syslog/forwarder/u5517/*/*juniper.log]
sourcetype = juniper
index = juniper
disabled = 0
host_segment = 5
1 Solution

vessev
Path Finder

Hello FrankVI,

i tested your suggestions and found some of my possibile solutions.

1.RawMessage
Your idea with the ${RAWMSG}\n is brilliant but you've forgot to mentioned annother setting to make this work flags(store-raw-message) in the source:

source s_udp5517 {
    udp(
        port(5517)
        keep-hostname(no)
        use-dns(yes)
        flags(store-raw-message) 
    );
};


template t_extra_timestamp { template("${R_ISODATE} $HOST ${RAWMSG}\n"); };

destination d_u5517 { file("/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-juniper.log" template(t_extra_timestamp) ); };

log { source(s_udp5517); destination(d_u5517); };

2.No Parse
Since i only needed a second Timestamp and the Hostname, it was easier to accomplish that by using the no parse function of syslog-ng -> with that syslog-ng does not parse the message at all but simply add a hostname and a timestamp. (i didn't now this exist before i found it after your suggestions)

source s_udp5517 {
    udp(
        port(5517)
        keep-hostname(no)
        use-dns(yes)
        flags(no-parse)
    );
};

destination d_u5517 { file("/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-test.log"  ); };

log { source(s_udp5517); destination(d_u5517); };

3.TimeManipulation
The Juniper Firewalls have no configuration to setup a timezone in syslog messages. That is quite sad but i can't configure it.
What i can do is to write a new config on a new port. Before writing the messages to the destination i can modify the timezone offset.

source s_udp5519 {
    udp(
        port(5519)
        keep-hostname(no)
        use-dns(yes)
    );
};

destination d_juniperu5519 { file("/data/syslog/forwarder/u5519/$HOST/$YEAR$MONTH$DAY-juniper.log" time-zone("-07:00") ); };

log { source(s_udp5519); destination(d_juniperu5519); };

There are also existing other possibilities to manipulate the timezone but i didn't got it to work. But i left the rest of the people who reading this a few notes on that too. As far as i understood it there are 2 global options regarding the time-zone. You can setup recv-time-zone and the send-time-zone as a global option. If you want to manipulate this you need to do the following:
You can add a time-zone("-07:00") into a destination which would override the send-time-zone() global option. Think of it like changing the difference not set it up to a specific time zone. This is the difference from JST Japan->Germany CEST. But i'm not sure if this would break in times of summer/winter time changes. You also can put a specific timezone "Europe/Berlin" into that setting but i've not tested that. The second setting would be to manipulate the recv-time-zone() global option. Therefore you have to setup the source setting, like s_udp5517 in my example. This only works if the sending event does not have a timezone in the TIMESTAMP. You set it up like in the destination simply use time-zone("-07:00") or the specific time zone. This is what the documentation wrote about that but the manipulation in the source did not worked for me. The timestamp did not change at all ...

At last i want to thank you FrankVI for your help - i was searching very long for those solutions and was about to give up and simply send everything to splunk directly. I will mark my response at answer but will give you the amount of points that you would normally get since you came up with those ideas.

*Update*: Manipulation of Time Zones can be done better
3.TimeManipulation

source s_udp5519 {
    udp(
        port(5519)
        keep-hostname(no)
        use-dns(yes)
        time-zone("Japan")
        #ZoneInfo /usr/shared/zoneinfo/*
    );
};

destination d_u5519 { file("/data/syslog/forwarder/u5519/$HOST/$YEAR$MONTH$DAY-juniper.log" time-zone("Europe/Berlin") ); };

log { source(s_udp5519); destination(d_u5519); };

Syslog-ng will always write the correct difference. I'm quite sure that with this config the summer/winter time is also covered.

Syslog-ng normally use the syslog originator timezone to modify the timestamp - this is the default. If there is none then you can define it. This will always change the timezone even if the originator has a time zone written in a syslog event.

BR vess

View solution in original post

vessev
Path Finder

Hello FrankVI,

i tested your suggestions and found some of my possibile solutions.

1.RawMessage
Your idea with the ${RAWMSG}\n is brilliant but you've forgot to mentioned annother setting to make this work flags(store-raw-message) in the source:

source s_udp5517 {
    udp(
        port(5517)
        keep-hostname(no)
        use-dns(yes)
        flags(store-raw-message) 
    );
};


template t_extra_timestamp { template("${R_ISODATE} $HOST ${RAWMSG}\n"); };

destination d_u5517 { file("/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-juniper.log" template(t_extra_timestamp) ); };

log { source(s_udp5517); destination(d_u5517); };

2.No Parse
Since i only needed a second Timestamp and the Hostname, it was easier to accomplish that by using the no parse function of syslog-ng -> with that syslog-ng does not parse the message at all but simply add a hostname and a timestamp. (i didn't now this exist before i found it after your suggestions)

source s_udp5517 {
    udp(
        port(5517)
        keep-hostname(no)
        use-dns(yes)
        flags(no-parse)
    );
};

destination d_u5517 { file("/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-test.log"  ); };

log { source(s_udp5517); destination(d_u5517); };

3.TimeManipulation
The Juniper Firewalls have no configuration to setup a timezone in syslog messages. That is quite sad but i can't configure it.
What i can do is to write a new config on a new port. Before writing the messages to the destination i can modify the timezone offset.

source s_udp5519 {
    udp(
        port(5519)
        keep-hostname(no)
        use-dns(yes)
    );
};

destination d_juniperu5519 { file("/data/syslog/forwarder/u5519/$HOST/$YEAR$MONTH$DAY-juniper.log" time-zone("-07:00") ); };

log { source(s_udp5519); destination(d_juniperu5519); };

There are also existing other possibilities to manipulate the timezone but i didn't got it to work. But i left the rest of the people who reading this a few notes on that too. As far as i understood it there are 2 global options regarding the time-zone. You can setup recv-time-zone and the send-time-zone as a global option. If you want to manipulate this you need to do the following:
You can add a time-zone("-07:00") into a destination which would override the send-time-zone() global option. Think of it like changing the difference not set it up to a specific time zone. This is the difference from JST Japan->Germany CEST. But i'm not sure if this would break in times of summer/winter time changes. You also can put a specific timezone "Europe/Berlin" into that setting but i've not tested that. The second setting would be to manipulate the recv-time-zone() global option. Therefore you have to setup the source setting, like s_udp5517 in my example. This only works if the sending event does not have a timezone in the TIMESTAMP. You set it up like in the destination simply use time-zone("-07:00") or the specific time zone. This is what the documentation wrote about that but the manipulation in the source did not worked for me. The timestamp did not change at all ...

At last i want to thank you FrankVI for your help - i was searching very long for those solutions and was about to give up and simply send everything to splunk directly. I will mark my response at answer but will give you the amount of points that you would normally get since you came up with those ideas.

*Update*: Manipulation of Time Zones can be done better
3.TimeManipulation

source s_udp5519 {
    udp(
        port(5519)
        keep-hostname(no)
        use-dns(yes)
        time-zone("Japan")
        #ZoneInfo /usr/shared/zoneinfo/*
    );
};

destination d_u5519 { file("/data/syslog/forwarder/u5519/$HOST/$YEAR$MONTH$DAY-juniper.log" time-zone("Europe/Berlin") ); };

log { source(s_udp5519); destination(d_u5519); };

Syslog-ng will always write the correct difference. I'm quite sure that with this config the summer/winter time is also covered.

Syslog-ng normally use the syslog originator timezone to modify the timestamp - this is the default. If there is none then you can define it. This will always change the timezone even if the originator has a time zone written in a syslog event.

BR vess

FrankVl
Ultra Champion

Nice summary of the options. I'm much more familiar with rsyslog than with syslog-ng, so not suprised you found a few gaps / improvements on my suggested solution. Glad to see I helped you look in the right direction 🙂

0 Karma

vessev
Path Finder

Thanks 🙂 if you find it sufficient you can upvote my question that also the community can identify this as helpful.
BR vess

0 Karma

FrankVl
Ultra Champion

The best way to solve this would be to configure all your network equipment (or at least those that do not log time zone info) to run / log in a common time zone (e.g. UTC).

Alternatively, you could configure Splunk with per-host TZ setting in props.conf, but with many devices that might become hard to manage.

If you really want to go down the route of adding an additional timestamp based on when syslog-ng processed the event, you can do that by defining your own template for how the events get written to disk. For example:

template t_extra_timestamp { template("${R_ISODATE} ${RAWMSG}\n"); };
destination d_juniper { file( "/data/syslog/forwarder/u5517/$HOST/$YEAR$MONTH$DAY-juniper.log" template(t_extra_timestamp) ); };

Note the R_ variant of the ISODATE macro, this uses the syslog-ng receipt time, rather than the timestamp from the message.
There are other timestamp macros available as well, if you'd prefer a different timestamp format.

The downside of this approach of course is that you can no longer rely on the indexed _time field in splunk for any searches that rely on when the event exactly happened (e.g. for Flow Tracing as you mentioned). Which likely means you also can no longer use accelerated data models for that kind of work, as those will be based on _time.

And if you want to correlate events from different devices that are not in the same timezone, using search time extracted timestamps from the event will also not work (unless you come up with some tricky evals to adjust for the time zone offset as observed against the syslog-ng receipt time).

vessev
Path Finder

Hi FrankVl,

thank you for your suggestion i will try that asap.
There is just something i want to mention. If i would send the syslog directly to splunk for example as input udp:5517, splunk at the time of collection could not identify the timezone and would add the indexing time as syslog TIMESTAMP.
-> This is as you've mentioned not very nice - but it would be the same?

Your suggestion made me also think about a different apporach:
I could create a syslog-ng config (separate port f.e. 5518) for a specific time zone like japan. That means i could collect juniper firewalls from specific time zones with a config that has the syslog-ng option send-time-zone or recv-time-zone.

FrankVl
Ultra Champion

Regarding your first comment: "it would be the same". The same to what? Not sure I follow you here.

Regarding the second part of your comment: Yes, if you can configure your devices from different regions to log to different ports, you can either use syslog-ng logic to do the time zone interpretation based on that. Or: have syslog-ng simply write to regional folders and configure TZ setting in splunk's props.conf based on source field (which is the log file path).
Then again, if you need to touch all your network equipment to make it log to a time zone specific port, you may just as well change the time zone they use (unless you have some specific reason to keep network equipment running in its local timezone) 😉

vessev
Path Finder

I meant that adding a second timestamp with syslog-ng at the time an event got collected should be the same as sending the logs directly to splunk. Splunk cannot identify a timezone and adds automatically a second timestamp. This time stamp is the time an event got indexed and also referees to the _time like the timestamp from syslog-ng would do.
Therefore it would make no difference sending logs without timezone directly to splunk or sending the logs to a syslog-ng which we setup to add a second timestamp.

As I've mentioned in my answer the juniper srx firewalls are not sending timezone information.
They are located in a branch office in Tokyo. Our HQ and our log monitoring is in Germany. Since we have only 10 devices in Tokyo its not that big of a deal. We also have equipment in Detroit and Romania. They all have different time zones. But, all those devices communicating with each other at the same time -> therefore i should change all timestamps to the most relevant zone. The local zone where most of the equipment is located don't you think?

Best regards, vess

0 Karma

FrankVl
Ultra Champion

I think best practice is for network equipment like that to be set to UTC, regardless of the location it is in. That way a team monitoring/managing a global collection of equipment does not have to worry about time zone differences etc. when assessing logs and such.

If you want to keep the devices set to various local time zones, then I think with what you have figured out around syslog-ng should give you the tools to at least enable splunk to interpret the (extra) timestamps correctly, so that the end user in splunk sees correctly interpreted time stamps that can be correlated across devices.

Still, only the extra timestamps can be correctly interpreted and there may be cases where there is a delay between the event happening and syslog-ng receiving it. Which could hinder accurate correlation / sequencing.

Alternatively if the amount of devices is fairly small and not too dynamic, you could also specify TZ setting in props.conf on hostname basis. That way you don't need the extra timestamp and splunk will use the more accurate original timestamp from the event itself.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...