Splunk deployment for enrichment based on csv file

[prateek123]

Hi

     I have deployed Splunk enterprise and my logs are getting ingested into the indexer. Now i have created an app for enriching the logs with additional fields from a csv file. I have deployed the app by making configuration changes in props.conf and transforms.conf and i am able to view search time enrichment. But my requirement is real time enrichment as my csv file would change every 2 days. Can anyone provide a sample configuration for props.conf and transforms.conf for real time enrichment of logs with fields from csv based on match with one of the fields of the logs.

Regards

[richgalloway]

You can enrich your data at index time using INGEST_EVAL and your CSV file.  See https://docs.splunk.com/Documentation/Splunk/9.4.0/Data/IngestLookups

[isoutamo]

Yes one can use lookup, but it needs INGEST_EVAL, not the “normal” lookup definition which are working only in search time.

[prateek123]

The following are the configurations that we have made in props and transforms conf files

props.conf

[source::logserver]

TRANSFORMS-enrich = malicious_ip

transforms.conf

[malicious_ip]

filename=malicious_ip.csv

match_type=WILDCARD(dst_ip)

INGEST_EVAL = json_data=lookup("malicious_ip.csv", json_object("dst_ip" , dst_ip), json_array("creationdate", "confidence", "tags"))

INGEST_EVAL = creationdate=json_extract(json_data, "creationdate")

INGEST_EVAL = tags=json_extract(json_data, "tags")

.csv file is located at /opt/splunk/etc/system/local/lookups/malicious_ip.csv and it is accessible through search and reporting app.

still the logs are not getting enriched at ingest time. Kindly provide correct conf for props and transforms and any additionaly observation / recommendation.

thanks and regards

[isoutamo]

You must have own transform stanzas for those INGEST_EVAL definitions. Or at least one which have those all in one line.
Lookup tables are used only in search time, not in index time.
You should test your INGEST_EVAL settings in search time and after it works in one eval xx=yy, zz=xyz then you can copy this into your transforms.conf.
See more https://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf#transforms.conf.example

[prateek123]

I have tested the EVAL statement as provided in the transforms.conf at seaech time and it is working fine. But the new fields that i want to add from the csv file is not getting appended to the logs that are getting ingested on a match dst_ip field of log with the dst_ip field of csv. From the documentation i came to know that i have to configure fields.conf also. I have configured the same with INDEXED=true for the new field that i want to append to the logs. But still the logs are not appended with the new fields. 

i followed the link https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Configureindex-timefieldextraction#Define_ad... . this shows to append new fields to the logs based on extraction from the actual log. What i actually require is that i want the logs to be appended with fields from my csv file.

Can you please guide us in configuring the props.conf and transforms.conf properly such that the logs are enriched with fields from the csv file for match.

thanks and regards

[richgalloway]

Lookup tables *can* be used at index-time as explicitly stated in the docs page linked in my reply.

[prateek123]

@bowesmana  let me clarify you the exact issue. We are ingesting logs from syslogserver in real time manner (meaning that as and when the logs are getting generated at the device, immediately the splunk forwarder is forwarding it to splunk for indexing. Now we are having threat intel in the form of .csv file containing multiple headers viz date, ip, valid_from, valid_until etc. we have ingested this csv file in lookup and it is accessible through searh and reporting. Our architecture is having one master/search-head and two indexers. We have configured deployment server on master and indexers (clients) and in sync with deployment server successfully. The deployment app has been created and is getting deployed on the clients also. The deployment app is aimed at enriching the logs with the threat intel in csv file. However, this enrichmet has to be done before the logs are getting indexed and any match of ip in the log event with the ip in csv should generate additional field "Add_field" which should also get indexed alongwith syslog logs. we have configured props.conf and transforms.conf in the deployment app, however exact configuration is not being achieved. 

regarding your specific query about real time: when we say real time, it means that logs are getting enriched at the time of indexing and additional contenxtual information present in the threat intel is also getting indexed in additional fields. the query run on the logs therefore does not need any lookup to be incorporated in search query. the match of threat intel done today should stay in the logs in case the csv file is updated tomorrow. 

looking forward for suitable solution / configurations to be done in props.conf and transforms.conf for index time enrichment (real time enrichment) and not search time enrichment.

thanks and regards

[bowesmana]

Thanks for clarifying. I understand you want to mark your ingested data at the time of ingest, so that is forever constant regardless of any changes made to the lookup.

As @richgalloway has said, it should be possible - I am unsure of the sequence of INGEST_EVAL statemensts where there are more than one. Have you tried putting the json_extract AND the lookup in a single statement, as in Rich's linked example to see if that works, at least for one of the fields.

[bowesmana]

Can you clarify what you did to get the "search time enrichment". Did you create an automatic lookup or are you using a lookup to enrich the data in your search SPL or are you doing something else?

If you change your lookup, then the lookup results will change, so I am not sure what you mean by "real time enrichment".

The principle of a CSV lookup is to give you data from the lookup file based on a field or fields in an event. That principle would give you "search time" AND "real time" enrichment, as they would be one and the same thing.