topic Re: ingest_eval lookup not working in Getting Data In

ingest_eval lookup not working

Niro — Thu, 04 Jan 2024 05:45:26 GMT

I have the following transforms.conf file:

[pan_src_user]
INGEST_EVAL=src_user_idx=json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")

and props.conf file:

[pan:traffic]
TRANSFORMS-pan_user = pan_src_user

user_ip_mapping.csv file sample:

src_ip	src_user
10.1.1.1	someuser

However it's not working - not sure what I'm doing wrong? The src_user_idx field is not showing up in any of the logs

Re: ingest_eval lookup not working

richgalloway — Thu, 04 Jan 2024 14:14:37 GMT

Please explain what you mean by "not working". What is the input, what are the expected results, and what are the actual results?

Re: ingest_eval lookup not working

Niro — Fri, 05 Jan 2024 04:35:15 GMT

I mean it's not adding the src_user_idx field to the logs - the log files contain a `src_ip` field, so I expect to get a src_user_idx field to get populated, using this search:

index=pan_logs sourcetype=pan:traffic earliest=-1m | fields src_ip,src_user*

I get src_ip but no src_user_idx. I did confirm the src_ip values are in fact in the lookup table

Re: ingest_eval lookup not working

PickleRick — Fri, 05 Jan 2024 09:06:03 GMT

1. Did you put the definition in the proper place?

2. Do you have your lookup defined on that component?

3. Most importantly - why would you use json functions when pan:traffic format does not have anything to do with json? (unless you have some completely non-standard configuration we know nothing about).

Re: ingest_eval lookup not working

isoutamo — Fri, 05 Jan 2024 09:26:07 GMT

one more. Have you test it on GUI with

<your base search> |eval src_user_idx = json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user") | table src*

In that way you can validate your INGEST_EVAL expression. And as @PickleRick said there are some other action to do after you have successfully validate it.

r. Ismo

Re: ingest_eval lookup not working

Niro — Fri, 05 Jan 2024 14:33:56 GMT

I have no idea, I was following the some ingest_eval examples and was confused why it was always using json functions, kind of assumed it was just some way splunk was interpreting it in the backend. How else would I go about accomplishing this at ingest?

For the first two questions - it's in system/local and the lookups are globally available

Re: ingest_eval lookup not working

Niro — Fri, 05 Jan 2024 14:35:43 GMT

This actually DOES work - why isn't it working using the transforms/props files?

Re: ingest_eval lookup not working

PickleRick — Fri, 05 Jan 2024 14:51:15 GMT

1. As a general rule of thumb - avoid using system/local. It might not have anything to do with this particular case or even not matter much in your environment in general but it's a good practice to split your confiugration into apps and maintain it as apps. system/local is the directory with the highest priority (except for the clustered indexers) and you can get into some undesired situations if you put your settings into system/local and can't later overwrite them with apps. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles

2. json functions are meant for working with json data. I suppose the examples were meant for json events.

3. I didn't notice it before having concentrated on that json part but your INGEST_EVAL, however you write it, has no chance of working if you base it on search-time extracted fields. Remember that most Splunk extractions are search-time. So in order to use part of the event for your lookup, you need to either find it by means of - for example - substr(), l/rtrim() or replace() or extract it as indexed field in order to be able to use it as argument for INGEST_EVAL (you can later assign null-value to it so it doesn't get indexed in the end).

Re: ingest_eval lookup not working

Esky73 — Wed, 14 Feb 2024 06:10:06 GMT

Hi did you get this working?

Re: ingest_eval lookup not working

scelikok — Wed, 14 Feb 2024 06:33:58 GMT

Hi @Niro,

If your issue isn't resolved, it might happen because of sourcetype overwrite on pan logs.

pan:traffic is overridden sourcetype please try putting the transforms setting to your original sourcetpe.

It should be pan:log or pan_log according to your input setting.

[pan:log] TRANSFORMS-pan_user = pan_src_user

Re: ingest_eval lookup not working

victor1004k — Mon, 26 May 2025 08:52:38 GMT

[Solution]

@Niro You can get the desired result by modifying transforms.conf as follows:

1. /opt/splunk/etc/apps/myapp/local/transforms.conf

[pan_src_user] INGEST_EVAL = src_ip=replace(_raw, ".*src_ip=([0-9.]+).*","\1"), src_user_idx=json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")

Result: