Getting Data In

How to add a static field using a lookup file for a partial match in the Universal forwarder?

varunanand
New Member

I am new to splunk and trying to add a static field (action) using a lookup file. It needs to be a partial match with the log entry.
I would prefer doing it in the forwarder because the indexer is common many projects.

lookups/lookup-file.csv

raw,action
*BoExceptions*,exclude
*No existing PackageTrade is found*,include
*deadLetter | 145 | ExchangeExchange[ExchangePattern:InOnly, BodyType:String]*,exclude

transforms.conf

[default]
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)
[lookup-app-log]
filename=lookup-file.csv

I tried the following two approaches.
props.conf

[default]
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log OUTPUT action
[source::.../server-2-*.log]
sourcetype=luxor-gemfire-server
REPORT-action=lookup-app-log
0 Karma
1 Solution

lguinn2
Legend

Here is what you need:

props.conf
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log raw as _raw OUTPUT action

transforms.conf
[lookup-app-log]
filename=lookup-file.csv
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)

Note that the name of the field in the events is _raw not raw. That is why you have to specify them both in LOOKUP setting. Also, this lookup may be pretty inefficient.

If you do not have a large number of entries in the lookup table, you might consider using eventtypes instead. Or a combination of eventtypes and tags.

View solution in original post

lguinn2
Legend

Here is what you need:

props.conf
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log raw as _raw OUTPUT action

transforms.conf
[lookup-app-log]
filename=lookup-file.csv
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)

Note that the name of the field in the events is _raw not raw. That is why you have to specify them both in LOOKUP setting. Also, this lookup may be pretty inefficient.

If you do not have a large number of entries in the lookup table, you might consider using eventtypes instead. Or a combination of eventtypes and tags.

varunanand
New Member

Thanks Iguinn. Solved my issue.

0 Karma

lguinn2
Legend

You can't do a lookup on a forwarder. Lookups happen only at search time - forwarders work at input time only.

Here are some references that may help:
Splunk docs: Index time vs. Search time

Splunk docs: Configuration parameters and the data pipeline

Splunk wiki: Where do I configure my Splunk settings?

varunanand
New Member

I configured it on the indexer and it still doesn't work. Is the conf right? Also added:

fields.conf

[action]
INDEXED_VALUE=false
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...