Getting Data In

ingest_eval lookup not working

Niro
Explorer

I have the following transforms.conf file:

[pan_src_user]
INGEST_EVAL=src_user_idx=json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")

and props.conf file:

[pan:traffic]
TRANSFORMS-pan_user = pan_src_user

user_ip_mapping.csv file sample:

src_ipsrc_user
10.1.1.1someuser

 

However it's not working - not sure what I'm doing wrong? The src_user_idx field is not showing up in any of the logs

Labels (3)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. Did you put the definition in the proper place?

2. Do you have your lookup defined on that component?

3. Most importantly - why would you use json functions when pan:traffic format does not have anything to do with json? (unless you have some completely non-standard configuration we know nothing about).

0 Karma

Niro
Explorer

3. Most importantly - why would you use json functions when pan:traffic format does not have anything to do with json? (unless you have some completely non-standard configuration we know nothing about).

 

I have no idea, I was following the some ingest_eval examples and was confused why it was always using json functions, kind of assumed it was just some way splunk was interpreting it in the backend. How else would I go about accomplishing this at ingest?

 

For the first two questions - it's in system/local and the lookups are globally available 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. As a general rule of thumb - avoid using system/local. It might not have anything to do with this particular case or even not matter much in your environment in general but it's a good practice to split your confiugration into apps and maintain it as apps. system/local is the directory with the highest priority (except for the clustered indexers) and you can get into some undesired situations if you put your settings into system/local and can't later overwrite them with apps. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles

2. json functions are meant for working with json data. I suppose the examples were meant for json events.

3. I didn't notice it before having concentrated on that json part but your INGEST_EVAL, however you write it, has no chance of working if you base it on search-time extracted fields. Remember that most Splunk extractions are search-time. So in order to use part of the event for your lookup, you need to either find it by means of - for example - substr(), l/rtrim() or replace() or extract it as indexed field in order to be able to use it as argument for INGEST_EVAL (you can later assign null-value to it so it doesn't get indexed in the end).

isoutamo
SplunkTrust
SplunkTrust

Hi

one more. Have you test it on GUI with 

<your base search>
|eval src_user_idx = json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")
| table src*

In that way you can validate your INGEST_EVAL expression.  And as @PickleRick said there are some other action to do after you have successfully validate it.

r. Ismo 

0 Karma

Niro
Explorer

This actually DOES work - why isn't it working using the transforms/props files?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please explain what you mean by "not working".  What is the input, what are the expected results, and what are the actual results?

---
If this reply helps you, Karma would be appreciated.
0 Karma

Niro
Explorer

I mean it's not adding the src_user_idx field to the logs - the log files contain a `src_ip` field, so I expect to get a src_user_idx field to get populated, using this search:

 

index=pan_logs sourcetype=pan:traffic earliest=-1m
| fields src_ip,src_user*

 

I get src_ip but no src_user_idx. I did confirm the src_ip values are in fact in the lookup table

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @Niro,

If your issue isn't resolved, it might happen because of sourcetype overwrite on pan logs. 

pan:traffic is overridden sourcetype please try putting the transforms setting to your original sourcetpe.

It should be pan:log or pan_log according to your input setting.

[pan:log]
TRANSFORMS-pan_user = pan_src_user
If this reply helps you an upvote and "Accept as Solution" is appreciated.

Esky73
Builder

Hi did you get this working?

0 Karma
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...