CSV file ingest, exclude header

[NullZero]

I'm ingesting logs from DNS (Next DNS via API) and struggling to exclude the header. I have seen @woodcock resolve some other examples and I can't quite see where I'm going wrong. The common mistake is not doing this on the UF.

Sample data: (comes in via a curl command and writes out to a file)

timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name
2023-09-01T09:09:21.561936+00:00,beam.scs.splunk.com,AAAA,false,DNS-over-HTTPS,213.31.58.70,,,,splunk.com,8D512,"NUC10i5",,,,nextdns-cli
2023-09-01T09:09:09.154592+00:00,time.cloudflare.com,A,true,DNS-over-HTTPS,213.31.58.70,,,,cloudflare.com,14D3C,"NUC10i5",,,,nextdns-cli

UF (On syslog server) v8.1.0

props.conf
[nextdns:dns]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1
HEADER_FIELD_DELIMITER =,
FIELD_NAMES = timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name
TIMESTAMP_FIELDS = timestamp

inputs.conf
[monitor:///opt/remote-logs/nextdns/nextdns.log]
index = nextdns
sourcetype = nextdns:dns
initCrcLength = 375

Indexer (SVA S1) v9.1.0

Disabled the options, I will apply Great8 once I have this fixed. All the work needs to happen on the UF.

[nextdns:dns]
#INDEXED_EXTRACTIONS = CSV
#HEADER_FIELD_LINE_NUMBER = 1
#HEADER_FIELD_DELIMITER =,
#FIELD_NAMES = timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name
#TIMESTAMP_FIELDS = timestamp

 Challenge:

• I'm still getting the header field ingest
• I have deleted the indexed data, regenerated updated log, reingested and still issues. Obviously I have restarted splunk on each instance after respective changes.

[NullZero]

Thanks @gcusello . I saw other options but I didn't think them necessary, appreciate the assistance and good to have solved it.

[gcusello · Accepted solution]

Hi @NullZero,

as you can see at https://docs.splunk.com/Documentation/ITSI/4.17.0/Configure/props.conf#props.conf.example you should try add to your props.conf PREAMBLE_REGEX:

[nextdns:dns]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER = 1
HEADER_FIELD_DELIMITER =,
FIELD_NAMES = timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name
TIMESTAMP_FIELDS = timestamp
PREAMBLE_REGEX = ^timestamp,domain,query_type,

 Ciao.

Giuseppe