ingest_eval works on AIO instance but different results when applied at the HF tier

[Skins]

I have syslog events being written to a HF locally via syslog-ng - these events are then consumed via file reader and the IP address in the log name is extracted as host.

I now want to run an ingest_eval on the ip address and use a lookup to change the host

If i run the cmd from search i get the required result:

index=... | eval host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

this replaces host with "host_value"

I have this working on an AIO instance with the following config below:

Now adding to HF tier : /opt/splunk/etc/apps/myapp/lookups/lookup.csv
lookup has global access and export = system
host,host_value
1.2.3.4, myhostname

props.conf:
[mysourcetype]
TRANSFORMS-host_override = host_override

transforms.conf:
[host_override]
INGEST_EVAL =host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

When applied on the HF (restarted)  i see some of the hostnames are changed to "localhost" the others remain unchanged (but this is due to the config not working OR the data coming from another HF with the test config not applied

Any ideas - thx

[victor1004k]

@Skins , @moja Hello, Bellow is the solution for your question.

1.  /opt/log/syslog-ng-sample.log

May 13 15:09:09 1.2.3.4 sim: logging for test

2. /opt/splunk/etc/apps/myapp/lookups/lookup.csv

host,host_value
1.2.3.4,myhostname

 3. /opt/splunk/etc/apps/myapp/local/props.conf

[mysourcetype]
TRANSFORMS-host_override = host_override

4. /opt/splunk/etc/apps/myapp/local/transforms.conf

[host_override]
INGEST_EVAL = host=replace(_raw, "^\w+\s+\d+\s+\d+:\d+:\d+\s+([^ ]+)\s+.*", "\1"), hostname=host,host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

5. Result

[moja]

@Skins hi) I faced the same limitation - the inability to use Ingest-time lookup on hw, did you manage to solve this issue?

[victor1004k]

Hello, Bellow is the solution for your question.

1. /opt/log/syslog-ng-sample.log

May 13 15:09:09 1.2.3.4 sim: logging for test

2. /opt/splunk/etc/apps/myapp/lookups/lookup.csv

host,host_value
1.2.3.4,myhostname

 3. /opt/splunk/etc/apps/myapp/local/props.conf

[mysourcetype]
TRANSFORMS-host_override = host_override

4. /opt/splunk/etc/apps/myapp/local/transforms.conf

[host_override]
INGEST_EVAL = host=replace(_raw, "^\w+\s+\d+\s+\d+:\d+:\d+\s+([^ ]+)\s+.*", "\1"), hostname=host,  host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

5. Result