Getting Data In

ingest_eval lookup not working

Niro
Explorer

I have the following transforms.conf file:

[pan_src_user]
INGEST_EVAL=src_user_idx=json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")

and props.conf file:

[pan:traffic]
TRANSFORMS-pan_user = pan_src_user

user_ip_mapping.csv file sample:

src_ipsrc_user
10.1.1.1someuser

 

However it's not working - not sure what I'm doing wrong? The src_user_idx field is not showing up in any of the logs

Labels (3)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Did you put the definition in the proper place?

2. Do you have your lookup defined on that component?

3. Most importantly - why would you use json functions when pan:traffic format does not have anything to do with json? (unless you have some completely non-standard configuration we know nothing about).

0 Karma

Niro
Explorer

3. Most importantly - why would you use json functions when pan:traffic format does not have anything to do with json? (unless you have some completely non-standard configuration we know nothing about).

 

I have no idea, I was following the some ingest_eval examples and was confused why it was always using json functions, kind of assumed it was just some way splunk was interpreting it in the backend. How else would I go about accomplishing this at ingest?

 

For the first two questions - it's in system/local and the lookups are globally available 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. As a general rule of thumb - avoid using system/local. It might not have anything to do with this particular case or even not matter much in your environment in general but it's a good practice to split your confiugration into apps and maintain it as apps. system/local is the directory with the highest priority (except for the clustered indexers) and you can get into some undesired situations if you put your settings into system/local and can't later overwrite them with apps. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles

2. json functions are meant for working with json data. I suppose the examples were meant for json events.

3. I didn't notice it before having concentrated on that json part but your INGEST_EVAL, however you write it, has no chance of working if you base it on search-time extracted fields. Remember that most Splunk extractions are search-time. So in order to use part of the event for your lookup, you need to either find it by means of - for example - substr(), l/rtrim() or replace() or extract it as indexed field in order to be able to use it as argument for INGEST_EVAL (you can later assign null-value to it so it doesn't get indexed in the end).

isoutamo
SplunkTrust
SplunkTrust

Hi

one more. Have you test it on GUI with 

<your base search>
|eval src_user_idx = json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")
| table src*

In that way you can validate your INGEST_EVAL expression.  And as @PickleRick said there are some other action to do after you have successfully validate it.

r. Ismo 

0 Karma

Niro
Explorer

This actually DOES work - why isn't it working using the transforms/props files?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please explain what you mean by "not working".  What is the input, what are the expected results, and what are the actual results?

---
If this reply helps you, Karma would be appreciated.
0 Karma

Niro
Explorer

I mean it's not adding the src_user_idx field to the logs - the log files contain a `src_ip` field, so I expect to get a src_user_idx field to get populated, using this search:

 

index=pan_logs sourcetype=pan:traffic earliest=-1m
| fields src_ip,src_user*

 

I get src_ip but no src_user_idx. I did confirm the src_ip values are in fact in the lookup table

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @Niro,

If your issue isn't resolved, it might happen because of sourcetype overwrite on pan logs. 

pan:traffic is overridden sourcetype please try putting the transforms setting to your original sourcetpe.

It should be pan:log or pan_log according to your input setting.

[pan:log]
TRANSFORMS-pan_user = pan_src_user
If this reply helps you an upvote and "Accept as Solution" is appreciated.

Esky73
Builder

Hi did you get this working?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...