topic Re: Very long log events coming over syslog 514/udp are cut in Getting Data In

Very long log events coming over syslog 514/udp are cut- How would I resolve this?

jrodriguezap — Mon, 17 Oct 2022 17:34:55 GMT

Hello
Someone will have happened that the logs come with a length of 1000 characters at most, and these are indexed incompletely?
How could I do to be stored completely all characters?
The source I use is syslog 514/udp.

I would appreciate your support.
greetings

Re: Very long log events coming over syslog 514/udp are cut

pradeepkumarg — Fri, 10 Oct 2014 20:12:39 GMT

set below property in props.conf

TRUNCATE =
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

More details here
http://docs.splunk.com/Documentation/Splunk/6.1.4/Admin/Propsconf

Re: Very long log events coming over syslog 514/udp are cut

jrodriguezap — Fri, 10 Oct 2014 20:43:59 GMT

Hi gpradeepkumarreddy
Thanks for your answer, I have understood that 1 character = 1 byte approx.
However, the lines are approximately 1500 characters, yet are cut, switch to 0 and continue slashing. Have something to do it syslog 514/udp?

Re: Very long log events coming over syslog 514/udp are cut

jrodman — Sat, 11 Oct 2014 05:44:29 GMT

syslog-udp cannot transport 1500 character events, beause that exceeds ethernet MTU, and syslog-over-udp is a single packet.
The data is likely being truncated before splunk receives it.

Consider a more reliable transport, such as syslog over tcp or splunk forwarders.

Re: Very long log events coming over syslog 514/udp are cut

jrodriguezap — Sat, 11 Oct 2014 05:54:42 GMT

Hi jrodman
I tried also with syslog via TCP / 514, and also truncated, that's normal?
I am what I am doing from ironport wsa

Re: Very long log events coming over syslog 514/udp are cut

sgailey_splunk — Sat, 11 Oct 2014 08:56:16 GMT

Syslog-ng can transport much longer events; up to 8K I believe but the syslog shipped with most linux and UNIX distributions can't, even when using a tcp transport.

Re: Very long log events coming over syslog 514/udp are cut

jrodman — Sat, 11 Oct 2014 09:00:19 GMT

Yeah i think the typical syslog follows the recommended max length specification. Oops, I forgot.

Re: Very long log events coming over syslog 514/udp are cut

jrodman — Sat, 11 Oct 2014 09:01:32 GMT

I think this really is a question for Cisco/Ironport. How can the device be configured to produce its data in a complete way?

Re: Very long log events coming over syslog 514/udp are cut

jrodriguezap — Sat, 11 Oct 2014 23:03:45 GMT

that's the detail. The same happens to me with an F5 ASM

Re: Very long log events coming over syslog 514/udp are cut

jrodman — Sat, 11 Oct 2014 23:17:06 GMT

For syslog-udp there is a maximum possible size that the messages can be in the way they are transmitted over the network. Splunk cannot help here.
For syslog-tcp, splunk does not even know that it is accepting syslog when accepting the data. There is nothing in splunk that truncates lines other than the TRUNCATE setting described by MuS, which defaults to 10KB.

Re: Very long log events coming over syslog 514/udp are cut

nitsud — Mon, 17 Oct 2022 17:10:20 GMT

You can only choose syslog for text-based logs.

The Syslog Push method sends log messages to a remote syslog server on port 514. This method conforms to RFC 3164.

Maximum message size is configurable on the WSA

You can increase the maximum message size on a log subscription in the WSA. It defaults to 1024.

When you choose this method, you must enter the following information:

Syslog server hostname
Protocol to use for transmission, either UDP or TCP
Maximum message size
Valid values for UDP are 1024 to 9216.
Valid values for TCP are 1024 to 65535.
Maximum message size depends on the syslog server configuration.
Facility to use with the log