topic Re: ingest_eval works on AIO instance but different results when applied at the HF tier in Getting Data In

ingest_eval works on AIO instance but different results when applied at the HF tier

Skins — Sun, 18 Feb 2024 23:35:57 GMT

I have syslog events being written to a HF locally via syslog-ng - these events are then consumed via file reader and the IP address in the log name is extracted as host.

I now want to run an ingest_eval on the ip address and use a lookup to change the host

If i run the cmd from search i get the required result:

index=... | eval host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

this replaces host with "host_value"

I have this working on an AIO instance with the following config below:

Now adding to HF tier : /opt/splunk/etc/apps/myapp/lookups/lookup.csv
lookup has global access and export = system
host,host_value
1.2.3.4, myhostname

props.conf:
[mysourcetype]
TRANSFORMS-host_override = host_override

transforms.conf:
[host_override]
INGEST_EVAL =host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

When applied on the HF (restarted) i see some of the hostnames are changed to "localhost" the others remain unchanged (but this is due to the config not working OR the data coming from another HF with the test config not applied

Any ideas - thx

Re: ingest_eval works on AIO instance but different results when applied at the HF tier

moja — Mon, 12 Aug 2024 14:34:47 GMT

@Skins hi) I faced the same limitation - the inability to use Ingest-time lookup on hw, did you manage to solve this issue?

Re: ingest_eval works on AIO instance but different results when applied at the HF tier

victor1004k — Mon, 26 May 2025 08:47:33 GMT

@Skins , @moja Hello, Bellow is the solution for your question.

1. /opt/log/syslog-ng-sample.log

May 13 15:09:09 1.2.3.4 sim: logging for test

2. /opt/splunk/etc/apps/myapp/lookups/lookup.csv

host,host_value 1.2.3.4,myhostname

3. /opt/splunk/etc/apps/myapp/local/props.conf

[mysourcetype] TRANSFORMS-host_override = host_override

4. /opt/splunk/etc/apps/myapp/local/transforms.conf

[host_override] INGEST_EVAL = host=replace(_raw, "^\w+\s+\d+\s+\d+:\d+:\d+\s+([^ ]+)\s+.*", "\1"), hostname=host,host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

5. Result

Re: ingest_eval works on AIO instance but different results when applied at the HF tier

victor1004k — Mon, 19 May 2025 06:51:44 GMT

Hello, Bellow is the solution for your question.

1. /opt/log/syslog-ng-sample.log

May 13 15:09:09 1.2.3.4 sim: logging for test

2. /opt/splunk/etc/apps/myapp/lookups/lookup.csv

host,host_value 1.2.3.4,myhostname

3. /opt/splunk/etc/apps/myapp/local/props.conf

[mysourcetype] TRANSFORMS-host_override = host_override

4. /opt/splunk/etc/apps/myapp/local/transforms.conf

[host_override] INGEST_EVAL = host=replace(_raw, "^\w+\s+\d+\s+\d+:\d+:\d+\s+([^ ]+)\s+.*", "\1"), hostname=host, host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value")

5. Result