ingest_eval lookup not working

[Niro]

I have the following transforms.conf file:

[pan_src_user]
INGEST_EVAL=src_user_idx=json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")

and props.conf file:

[pan:traffic]
TRANSFORMS-pan_user = pan_src_user

user_ip_mapping.csv file sample:

src_ip	src_user
10.1.1.1	someuser

However it's not working - not sure what I'm doing wrong? The src_user_idx field is not showing up in any of the logs

[victor1004k]

[Solution]

@Niro  You can get the desired result by modifying transforms.conf as follows:

1. /opt/splunk/etc/apps/myapp/local/transforms.conf

[pan_src_user]
INGEST_EVAL = src_ip=replace(_raw, ".*src_ip=([0-9.]+).*","\1"), src_user_idx=json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")

Result:

[PickleRick]

1. Did you put the definition in the proper place?

2. Do you have your lookup defined on that component?

3. Most importantly - why would you use json functions when pan:traffic format does not have anything to do with json? (unless you have some completely non-standard configuration we know nothing about).

[Niro]

3. Most importantly - why would you use json functions when pan:traffic format does not have anything to do with json? (unless you have some completely non-standard configuration we know nothing about).

I have no idea, I was following the some ingest_eval examples and was confused why it was always using json functions, kind of assumed it was just some way splunk was interpreting it in the backend. How else would I go about accomplishing this at ingest?

For the first two questions - it's in system/local and the lookups are globally available 

[PickleRick]

1. As a general rule of thumb - avoid using system/local. It might not have anything to do with this particular case or even not matter much in your environment in general but it's a good practice to split your confiugration into apps and maintain it as apps. system/local is the directory with the highest priority (except for the clustered indexers) and you can get into some undesired situations if you put your settings into system/local and can't later overwrite them with apps. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles

2. json functions are meant for working with json data. I suppose the examples were meant for json events.

3. I didn't notice it before having concentrated on that json part but your INGEST_EVAL, however you write it, has no chance of working if you base it on search-time extracted fields. Remember that most Splunk extractions are search-time. So in order to use part of the event for your lookup, you need to either find it by means of - for example - substr(), l/rtrim() or replace() or extract it as indexed field in order to be able to use it as argument for INGEST_EVAL (you can later assign null-value to it so it doesn't get indexed in the end).

[isoutamo]

Hi

one more. Have you test it on GUI with 

<your base search>
|eval src_user_idx = json_extract(lookup("user_ip_mapping.csv",json_object("src_ip", src_ip),json_array(src_user_idx)),"src_user")
| table src*

In that way you can validate your INGEST_EVAL expression.  And as @PickleRick said there are some other action to do after you have successfully validate it.

r. Ismo 

[Niro]

This actually DOES work - why isn't it working using the transforms/props files?

[richgalloway]

Please explain what you mean by "not working".  What is the input, what are the expected results, and what are the actual results?

[Niro]

I mean it's not adding the src_user_idx field to the logs - the log files contain a `src_ip` field, so I expect to get a src_user_idx field to get populated, using this search:

index=pan_logs sourcetype=pan:traffic earliest=-1m
| fields src_ip,src_user*

I get src_ip but no src_user_idx. I did confirm the src_ip values are in fact in the lookup table

[scelikok]

Hi @Niro,

If your issue isn't resolved, it might happen because of sourcetype overwrite on pan logs. 

pan:traffic is overridden sourcetype please try putting the transforms setting to your original sourcetpe.

It should be pan:log or pan_log according to your input setting.

[pan:log]
TRANSFORMS-pan_user = pan_src_user

[Esky73]

Hi did you get this working?