Getting Data In

How to add a static field using a lookup file for a partial match in the Universal forwarder?

varunanand
New Member

I am new to splunk and trying to add a static field (action) using a lookup file. It needs to be a partial match with the log entry.
I would prefer doing it in the forwarder because the indexer is common many projects.

lookups/lookup-file.csv

raw,action
*BoExceptions*,exclude
*No existing PackageTrade is found*,include
*deadLetter | 145 | ExchangeExchange[ExchangePattern:InOnly, BodyType:String]*,exclude

transforms.conf

[default]
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)
[lookup-app-log]
filename=lookup-file.csv

I tried the following two approaches.
props.conf

[default]
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log OUTPUT action
[source::.../server-2-*.log]
sourcetype=luxor-gemfire-server
REPORT-action=lookup-app-log
0 Karma
1 Solution

lguinn2
Legend

Here is what you need:

props.conf
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log raw as _raw OUTPUT action

transforms.conf
[lookup-app-log]
filename=lookup-file.csv
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)

Note that the name of the field in the events is _raw not raw. That is why you have to specify them both in LOOKUP setting. Also, this lookup may be pretty inefficient.

If you do not have a large number of entries in the lookup table, you might consider using eventtypes instead. Or a combination of eventtypes and tags.

View solution in original post

lguinn2
Legend

Here is what you need:

props.conf
[source::.../server-1-*.log]
sourcetype=luxor-server
LOOKUP-action=lookup-app-log raw as _raw OUTPUT action

transforms.conf
[lookup-app-log]
filename=lookup-file.csv
max_matches=1
min_matches=1
default_match=exclude
case_sensitive_match=false
match_type=WILDCARD(raw)

Note that the name of the field in the events is _raw not raw. That is why you have to specify them both in LOOKUP setting. Also, this lookup may be pretty inefficient.

If you do not have a large number of entries in the lookup table, you might consider using eventtypes instead. Or a combination of eventtypes and tags.

varunanand
New Member

Thanks Iguinn. Solved my issue.

0 Karma

lguinn2
Legend

You can't do a lookup on a forwarder. Lookups happen only at search time - forwarders work at input time only.

Here are some references that may help:
Splunk docs: Index time vs. Search time

Splunk docs: Configuration parameters and the data pipeline

Splunk wiki: Where do I configure my Splunk settings?

varunanand
New Member

I configured it on the indexer and it still doesn't work. Is the conf right? Also added:

fields.conf

[action]
INDEXED_VALUE=false
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...