Getting Data In

HTTP Event Collector and CSV files

Explorer

Hello,

I would like to know if it was possible to send a CSV to the HEC, and to take into consideration the names of the columns in the index. Here is my CSV file:

url,vuln,fix,severity,package,imageTag
https://security-tracker.debian.org/tracker/CVE-2018-1302,CVE-2018-1302,None,Unknown,apache2-2.4.25-...
https://security-tracker.debian.org/tracker/CVE-2007-3303,CVE-2007-3303,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2003-1580,CVE-2003-1580,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2003-1581,CVE-2003-1581,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2008-0455,CVE-2008-0455,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2001-1534,CVE-2001-1534,None,Negligible,apache2-2.4....

I would like the url,vuln,fix,severity,... headers to be recognized as a column in splunk, to get a table.

When I manually import my CSV file the fields are well recognized and everything works as I want. But with HEC, the headers line is simply added to the event list.

To send my CSV to HEC, I use a python script and the PyHEC modue (github jonromero pyHEC) with this piece of code I send line by line the content of my CSV:

with open("vuln.csv") as fp:

for line in fp:
print hec.send(line)

I also tried without a loop, but the whole CSV content is stored in a single event.

I also tried field extraction, but I send 2 different CSVs in the same index, with different fields.

I specify that this works with CSV files sent manually to splunk.

Do you have an idea to make splunk recognize CSV headers when sending via HTTP Event Collector?

Sincerely.

Motivator

What a great idea. I've been testing having users upload to a Heavy Forwarder, and then monitoring the app folder they upload to ingest the csv as a log instead of a lookup, however your approach seems much more pragmatic.

0 Karma

New Member

@skhedim Hey, I am facing the same problem. Did you find a solution for it?

0 Karma

SplunkTrust
SplunkTrust

Use a dict reader in python to read it in as a list of dicts.

Then send as the event dict.
https://github.com/georgestarcher/Splunk-Class-httpevent