I have tested the EVAL statement as provided in the transforms.conf at seaech time and it is working fine. But the new fields that i want to add from the csv file is not getting appended to the logs that are getting ingested on a match dst_ip field of log with the dst_ip field of csv. From the documentation i came to know that i have to configure fields.conf also. I have configured the same with INDEXED=true for the new field that i want to append to the logs. But still the logs are not appended with the new fields.  i followed the link https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Configureindex-timefieldextraction#Define_additional_indexed_fields . this shows to append new fields to the logs based on extraction from the actual log. What i actually require is that i want the logs to be appended with fields from my csv file. Can you please guide us in configuring the props.conf and transforms.conf properly such that the logs are enriched with fields from the csv file for match. thanks and regards ... View more

The following are the configurations that we have made in props and transforms conf files props.conf [source::logserver] TRANSFORMS-enrich = malicious_ip transforms.conf [malicious_ip] filename=malicious_ip.csv match_type=WILDCARD(dst_ip) INGEST_EVAL = json_data=lookup("malicious_ip.csv", json_object("dst_ip" , dst_ip), json_array("creationdate", "confidence", "tags")) INGEST_EVAL = creationdate=json_extract(json_data, "creationdate") INGEST_EVAL = tags=json_extract(json_data, "tags") .csv file is located at /opt/splunk/etc/system/local/lookups/malicious_ip.csv and it is accessible through search and reporting app. still the logs are not getting enriched at ingest time. Kindly provide correct conf for props and transforms and any additionaly observation / recommendation. thanks and regards   ... View more

@bowesmana  let me clarify you the exact issue. We are ingesting logs from syslogserver in real time manner (meaning that as and when the logs are getting generated at the device, immediately the splunk forwarder is forwarding it to splunk for indexing. Now we are having threat intel in the form of .csv file containing multiple headers viz date, ip, valid_from, valid_until etc. we have ingested this csv file in lookup and it is accessible through searh and reporting. Our architecture is having one master/search-head and two indexers. We have configured deployment server on master and indexers (clients) and in sync with deployment server successfully. The deployment app has been created and is getting deployed on the clients also. The deployment app is aimed at enriching the logs with the threat intel in csv file. However, this enrichmet has to be done before the logs are getting indexed and any match of ip in the log event with the ip in csv should generate additional field "Add_field" which should also get indexed alongwith syslog logs. we have configured props.conf and transforms.conf in the deployment app, however exact configuration is not being achieved.  regarding your specific query about real time: when we say real time, it means that logs are getting enriched at the time of indexing and additional contenxtual information present in the threat intel is also getting indexed in additional fields. the query run on the logs therefore does not need any lookup to be incorporated in search query. the match of threat intel done today should stay in the logs in case the csv file is updated tomorrow.  looking forward for suitable solution / configurations to be done in props.conf and transforms.conf for index time enrichment (real time enrichment) and not search time enrichment. thanks and regards ... View more