Getting Data In

Use RegEx to set sourcetype on UF where events are in a JSON format

donaldwayne1975
Path Finder

Scenario:

  1. two different source types being sent to UF (snort and firewall) from the same IP/source.
  2. format of data is JSON (see below example)

    {
    "client": {"ip": "...",...
    ...
    "type": "snort"
    }

    {
    "fw": {"iptables": "...",...
    "client": {"ip": "...",...
    ...
    "type": "firewall"
    }{
    "client": {"ip": "...",...
    ...
    "type": "snort"
    }

    {
    "fw": {"iptables": "...",...
    "client": {"ip": "...",...
    ...
    "type": "firewall"
    }

I've tried using regexes like below to separate them.

props.conf

[udp:514]
TRANSFORMS-set_sourcetype = snort,firewall

transforms.conf

[snort]
REGEX = ^(.*?type\"\:)(?P<type>("([^}]|"")*"))
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$type

[firewall]
REGEX = ^(.*?type\"\:)(?P<type>("([^}]|"")*"))
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$type

Looking for the best method to assign the sourcetypes. I do not have control over the data structure or how it is sent to the UF.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

A UF doesn't parse the data, so this functionality must be done on the indexer, or an HF. Your title for the question specifically states doing it on a UF, so the answer is, you can't. But you can do it on the indexers or an HF.

0 Karma

to4kawa
Ultra Champion

Why do you want to separate sourcetypes?
JSON is extracted fields appropriately .

If you search , you can use type=snort or type=firewall .

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...