When collecting logs through syslog, why does it comes in json format?

[guerrillalds]

EPP: {"syslog_type":"AGENT_EVENT", "syslog_data":{"log_string_args":null,"computer_name":"F0-P-N0017","login_id":"POO10","client_time":{"$date ":"2022-10-01T04:31:03.752Z"},"log_string_id":"EPPAGENT_LOGID_11002","user_name":"ppart","data_uuid":"35a5e74d-c4ad-4f25-ba9f-830c6-WINDOWS", "유형":"11000"}}

필드 추출이 잘 안되는데 어떻게 해야 하나요?

[yuanliu]

I assume that you meant index-time extraction because Splunk can handle JSON at search time; JSON field extraction at search time is automatic. (Or you can use spath in search window.)

But first, you want to decide whether it is absolutely necessary to extract custom fields at index time. From Create custom fields at index time:

Unless absolutely necessary, do not add custom fields to the set of default fields that Splunk software automatically extracts and indexes at index time... Adding to this list of fields decreases performance, as each indexed field increases the size of the searchable index.

If you really need to do it at index time, ask your system admin to change to plain text can be one option. Depending on which field(s) you want to extract, using regex on JSON can be attempted, but it is not very robust.

[PickleRick]

Unfortunately, automatic json extraction works only if the whole event is a properly formatted json (unless something changed in 9.0 but I don't recall anything like that).

So if you have an event delivered as, for example: "<PRIO>Date host whatever {json data}" - which is quite typical for syslog delivered logs - it will _not_ get automatically extracted unless you cut the whole header and index only the json portion.

[yuanliu · Accepted solution]

@PickleRickis correct; the correct form of question should be "... but it does not come in JSON form." I am not familiar with syslog in (quasi-)JSON form. If your system admin cannot configure collector in truly conformant JSON form, you should use forwarder to mangle the input before forwarding.

The first string "EPP:" outside of the {} block violates JSON, even though the {} block is perfectly conformant. I assume that EPP is just one level of logs. The question then is: Does the source issue other levels of logs? If not, you can simply ask the forwarder to strip "EDD:" from the raw event. If you must preserve log level, you can ask the forwarder to add appropriate wraps to these events, e.g., the sample you posted should be modified to conformant form

{"EPP": {"syslog_type":"AGENT_EVENT", "syslog_data":{"log_string_args":null,"computer_name":"F0-P-N0017","login_id":"POO10","client_time":{"$date ":"2022-10-01T04:31:03.752Z"},"log_string_id":"EPPAGENT_LOGID_11002","user_name":"ppart","data_uuid":"35a5e74d-c4ad-4f25-ba9f-830c6-WINDOWS", "유형":"11000"}}}

[PickleRick]

There are some solutions which push json events via syslog. One of them I use myself on my laptop 🙂 - NXLog exporting windows EventLog events packed in json. Luckily I don't need to ingest those into Splunk but I only save them to disk for archiving.

But on the other hand - there are solutions which send properly formated json structures via "syslog transport" - my example here is rsyslog's internal stats generated by impstats module.