Connect winlogbeat log format to Splunk_TA_Windows

[gcusello]

Hi at all,

I have to ingest logs from securelog and I'm able to take and parse linux logs, but I have an issue when parsing windows logs:

how can I connect winlogbeat format to a Splunk_TA_Windows to correctly parse events?

in winlogbeat events format is different from the normal windows logs so te Splunk_TA_windows doesn't reach to parse logs.

Is there a connector or must I manually parse winlogbeat logs to reduce them to Splunk_TA_windows logs?

Thank you for your help.

Ciao.

Giuseppe

[tscroggins]

Hi @gcusello,

How are you indexing winlogbeat events today? You can configure winlogbeat to write to one or more files and index the files with Splunk as you would any other file. In e.g. winlogbeat.yml, add or modify:

winlogbeat.event_logs:
  - name: Security
    include_xml: true

output.file:
  path: "C:/Temp"
  filename: winlogbeat
  rotate_every_kb: 10240
  number_of_files: 7
  # don't do this
  permissions: 0666
  rotate_on_startup: false
  codec.format:
    string: '%{[event.original]}'

In the example configuration, winlogbeat will create:

C:\Temp\winlogbeat-yyyymmdd.ndjson
C:\Temp\winlogbeat-yyyymmdd-1.ndjson
C:\Temp\winlogbeat-yyyymmdd-2.ndjson
C:\Temp\winlogbeat-yyyymmdd-3.ndjson
C:\Temp\winlogbeat-yyyymmdd-4.ndjson
C:\Temp\winlogbeat-yyyymmdd-5.ndjson
C:\Temp\winlogbeat-yyyymmdd-6.ndjson

You may need to increase the size and number of files (the "buffer") to accommodate the event write rate relative to the Splunk read rate.

Each line will be an XML formatted Windows event with the same content generated by renderXml = true in inputs.conf with the addition of a rendered <Message> element. Break on each line and extract the timestamp from <TimeCreated SystemTime='YYYY-MM-DDTHH:MM:SS.FFFFFFFZ'/>. Remove the <Message> element from _raw for a pure XmlWinEventLog event.

That said, if you have Splunk installed to read files, you can read event logs directly as well. It's an interesting winlogbeat feature nonetheless.

If you're receiving winlogbeat events in some other way, you may need to set include_xml: true (as shown above) in winlogbeat.yml or the active configuration file, replace _raw with json_extract(_raw, "event.original") from the JSON event, and remove the <Message> element.

Using output.logstash to send events directly to a raw tcp input (or properly configured tcp-ssl input) may be possible, but I don't see a way to disable acknowledgements in the Logstash Lumberjack protocol. Splunk has no way to acknowledge events at the application layer. Has anyone written a Lumberjack protocol modular input? I don't see one in Splunkbase, and the protocol isn't very complex.

[gcusello]

Hi @tscroggins ,

I have a data flow from logstash containing linux and windows logs.

I', able ro reconvert linux logs to standard readable by the Splunk_TA-Linux Add-On, but, the Windows logs, because they were took using WinLogBeat, have a different format than standard so I'm not able to connect to the Splunk_TA_Windows.

Now the only solution seems that I have to wride a custom add-on to manually match all the fields between the WinLogBeat and the Splunk_TA_Windows formats.

Have you a different experience of approach?

I cannot modify the WinLogBeat extraction, e.g. writing other files.

Thank you for you help.

Ciao.

Giuseppe

[tscroggins]

Hi @gcusello,

If you're receiving Elastic Common Schema (ECS) events in JSON format, i.e. Logstash tcp output plugin to Splunk raw tcp input , then a combination of search-time automatic field extractions, field aliases, etc. may be preferred if your site distinguishes between the Splunk administrator and Splunk knowledge manager functions; otherwise, I would transform the events using TRANSFORMS or RULESET.

For example, given a source ECS event in _raw:

{
  "@timestamp": "2023-11-26T14:40:57.209Z",
  "@metadata": {
    "beat": "winlogbeat",
    "type": "_doc",
    "version": "8.11.1"
  },
  "event": {
    "action": "Sensitive Privilege Use",
    "created": "2023-11-26T16:52:21.698Z",
    "code": "4673",
    "kind": "event",
    "provider": "Microsoft-Windows-Security-Auditing",
    "outcome": "failure"
  },
  "log": {
    "level": "information"
  },
  "message": "A privileged service was called.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-21-**********-**********-**********-1234\n\tAccount Name:\t\tuser\n\tAccount Domain:\t\tCONTOSO\n\tLogon ID:\t\t0x1960F1B\n\nService:\n\tServer:\tSecurity\n\tService Name:\t-\n\nProcess:\n\tProcess ID:\t0x37d0\n\tProcess Name:\tC:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe\n\nService Request Information:\n\tPrivileges:\t\tSeProfileSingleProcessPrivilege",
  "host": {
    "os": {
      "kernel": "10.0.22621.2715 (WinBuild.160101.0800)",
      "build": "22621.2715",
      "type": "windows",
      "platform": "windows",
      "version": "10.0",
      "family": "windows",
      "name": "Windows 11 Pro"
    },
    "id": "6d403ce9-3f79-4551-b651-4d3eb6c53bc9",
    "ip": [
      "192.0.2.1"
    ],
    "mac": [
      "ff-ff-ff-ff-ff-ff",
    ],
    "name": "my-pc",
    "hostname": "my-pc",
    "architecture": "x86_64"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "version": "8.11.1",
    "ephemeral_id": "080792d1-f86c-4a5d-9c46-265212f944f7",
    "id": "287205c6-b0d0-46f5-875a-1bcc6e013cf2",
    "name": "my-pc",
    "type": "winlogbeat"
  },
  "winlog": {
    "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
    "channel": "Security",
    "opcode": "Info",
    "process": {
      "pid": 4,
      "thread": {
        "id": 940
      }
    },
    "provider_name": "Microsoft-Windows-Security-Auditing",
    "event_data": {
      "Service": "-",
      "ProcessId": "0x37d0",
      "SubjectUserSid": "S-1-5-21-**********-**********-**********-1234",
      "SubjectDomainName": "MY-PC",
      "ObjectServer": "Security",
      "PrivilegeList": "SeProfileSingleProcessPrivilege",
      "ProcessName": "C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe",
      "SubjectUserName": "user",
      "SubjectLogonId": "0x1960f1b"
    },
    "keywords": [
      "Audit Failure"
    ],
    "computer_name": "my-pc",
    "record_id": 383388,
    "api": "wineventlog",
    "event_id": "4673",
    "task": "Sensitive Privilege Use"
  }
}

we can use INGEST_EVAL to reformat _raw as WinEventLog:

# transforms.conf
[ecs-to-wineventlog]
INGEST_EVAL = _raw:=strftime(strptime(json_extract(_raw, "event.created"), "%Y-%m-%dT%H:%M:%S.%N%Z"), "%m/%d/%Y %I:%M:%S %p").urldecode("%0a")."LogName=".json_extract(_raw, "winlog.channel").urldecode("%0a")."EventCode=".json_extract(_raw, "winlog.event_id").urldecode("%0a")."ComputerName=".json_extract(_raw, "winlog.computer_name").urldecode("%0a")."RecordNumber=".json_extract(_raw, "winlog.record_id").urldecode("%0a")."Keywords=".json_extract(_raw, "winlog.keywords{}").urldecode("%0a")."TaskCategory=".json_extract(_raw, "winlog.task").urldecode("%0a")."OpCode=".json_extract(_raw, "winlog.opcode").urldecode("%0a")."Message=".json_extract(_raw, "message")

We can use the same INGEST_EVAL or a separate one (before _raw is transformed) to extract _time from event.created, set sourcetype, etc.

Note that my sample event does not have direct translations of EventType, SourceName, and Type. The output would be similar to a WinEventLog input with the related suppress_* settings set to true. ECS does have event.outcome and log.level fields; however, they've been normalized for Elastic. The ECS event.provider field is the internal Windows event log provider/source name, e.g. Microsoft-Windows-Security-Auditing, not the rendered name, e.g. Microsoft Windows security auditing.

Translating to XmlWinEvnetLog is similar, but without access to the mvmap() function in INGEST_EVAL, construction of the <EventData><Data Name="Foo">Bar</Data></EventData> array wouldn't be dynamic.

A search-time translation of EventData might look like this:

| eval _raw="<EventData>".mvjoin(mvmap(split(replace(json_extract(_raw, "winlog.event_data"), "\",\"", "\"}".urldecode("%08")."{\""), urldecode("%08")), "<Data Name=\"".mvjoin(json_array_to_mv(json_keys(_raw)), "")."\">".replace(replace(replace(replace(replace(json_extract(_raw, mvjoin(json_array_to_mv(json_keys(_raw)), "")), "&", "&amp;"), "\"", "&quot;"), "'", "&apos;"), "<", "&lt;"), ">", "&gt;")."</Data>"), "")."</EventData>"

--but it isn't very maintainable. It also doesn't help with your problem; it's just fun. 😜

(Edit: The community editor may have stripped some of the characters from my examples. Apologies if they don't work as shown! This edit will also remove syntax highlighting--old community bug!)

Some XML elements are not present in every event, so we would need to wrap segments in coalesce() to make them optional during extraction, e.g. coalesce(json_extract(_raw, "winlog.some_missing_field"), ""), or just inject an empty element, e.g. <Security />. The Windows Event schema is documented at <https://learn.microsoft.com/en-us/windows/win32/wes/eventschema-schema>.

[gcusello]

Hi @tscroggins m

this is the solution I tought, that I'd like to avoid: I'll try to persuade the customer to use the Universal Forwarder, otherwise I will create a translator from WinLogBeat to Splunk_TA_Windows, as I did for Linux.

Thank you.

Ciao.

Giuseppe

[PickleRick]

I'm afraid I don't have good news for you here.

The methods for getting events from windows in order of decreasing preference:

1. Directly pulling data from local eventlog with UF - obvious case

2. Use native Windows Event Forwarding to get events to another host and ingest with UF from there - WEF can be tricky in some cases to set up (especialy in a domainless situation), forwarded events can hit EventLog input performance limits (can happen on a standalone host as well but it's more probable on a WEF collector) but generally works quite well and is often the only available reasonable options (domain admins don't like 3rd party tools on domain controllers)

3. Pull data from another host with WMI input - that's way more tricky to set up, works only in AD environment, needs UF run with a domain Service Account, is worse performancewise and harder to debug.

4. And at the very end you have the worst possible option of pulling the data with 3rd party tools. They are in strange formats, contain some additional data specific to that 3rd party solution but may not forward all data included in original events. So far the only solutions I saw that suggested it could be able to push the full eventlog XML via syslog was the paid version of NXLog (the free one when configured to push XML sends an XML of NXLog event with eventlog entry in plaintext form as one of the fields).

There are two problems you face with 3rd party forwarders. One is the necessity of parsing the completely foreign format. And another thing is that you need to make it either CIM compliant so you can do without TA_windows which is a huge undertaking or do a "mapping" to TA_windows supplied sourcetype which seems equally absurd amount of work.

So I'm afraid your knee-deep in manure here and I'd advise you to try hard to get those events in any of those first 3 ways. Even WMI is better than external tools.