Getting Data In

How to add a static field using a lookup file for a partial match in the Universal forwarder?

varunanand
New Member

I am new to splunk and trying to add a static field (action) using a lookup file. It needs to be a partial match with the log entry.
I would prefer doing it in the forwarder because the indexer is common many projects.

lookups/lookup-file.csv

raw,action
*BoExceptions*,exclude
*No existing PackageTrade is found*,include
*deadLetter | 145 | ExchangeExchange[ExchangePattern:InOnly, BodyType:String]*,exclude

transforms.conf

[default]
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)
[lookup-app-log]
filename=lookup-file.csv

I tried the following two approaches.
props.conf

[default]
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log OUTPUT action
[source::.../server-2-*.log]
sourcetype=luxor-gemfire-server
REPORT-action=lookup-app-log
0 Karma
1 Solution

lguinn2
Legend

Here is what you need:

props.conf
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log raw as _raw OUTPUT action

transforms.conf
[lookup-app-log]
filename=lookup-file.csv
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)

Note that the name of the field in the events is _raw not raw. That is why you have to specify them both in LOOKUP setting. Also, this lookup may be pretty inefficient.

If you do not have a large number of entries in the lookup table, you might consider using eventtypes instead. Or a combination of eventtypes and tags.

View solution in original post

lguinn2
Legend

Here is what you need:

props.conf
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log raw as _raw OUTPUT action

transforms.conf
[lookup-app-log]
filename=lookup-file.csv
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)

Note that the name of the field in the events is _raw not raw. That is why you have to specify them both in LOOKUP setting. Also, this lookup may be pretty inefficient.

If you do not have a large number of entries in the lookup table, you might consider using eventtypes instead. Or a combination of eventtypes and tags.

varunanand
New Member

Thanks Iguinn. Solved my issue.

0 Karma

lguinn2
Legend

You can't do a lookup on a forwarder. Lookups happen only at search time - forwarders work at input time only.

Here are some references that may help:
Splunk docs: Index time vs. Search time

Splunk docs: Configuration parameters and the data pipeline

Splunk wiki: Where do I configure my Splunk settings?

varunanand
New Member

I configured it on the indexer and it still doesn't work. Is the conf right? Also added:

fields.conf

[action]
INDEXED_VALUE=false
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...