HTTP Event Collector and CSV files

[skhedim]

Hello,

I would like to know if it was possible to send a CSV to the HEC, and to take into consideration the names of the columns in the index. Here is my CSV file:

url,vuln,fix,severity,package,imageTag
https://security-tracker.debian.org/tracker/CVE-2018-1302,CVE-2018-1302,None,Unknown,apache2-2.4.25-...
https://security-tracker.debian.org/tracker/CVE-2007-3303,CVE-2007-3303,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2003-1580,CVE-2003-1580,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2003-1581,CVE-2003-1581,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2008-0455,CVE-2008-0455,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2001-1534,CVE-2001-1534,None,Negligible,apache2-2.4....

I would like the url,vuln,fix,severity,... headers to be recognized as a column in splunk, to get a table.

When I manually import my CSV file the fields are well recognized and everything works as I want. But with HEC, the headers line is simply added to the event list.

To send my CSV to HEC, I use a python script and the PyHEC modue (github jonromero pyHEC) with this piece of code I send line by line the content of my CSV:

with open("vuln.csv") as fp:

for line in fp:
print hec.send(line)

I also tried without a loop, but the whole CSV content is stored in a single event.

I also tried field extraction, but I send 2 different CSVs in the same index, with different fields.

I specify that this works with CSV files sent manually to splunk.

Do you have an idea to make splunk recognize CSV headers when sending via HTTP Event Collector?

Sincerely.

[bandit]

What a great idea. I've been testing having users upload to a Heavy Forwarder, and then monitoring the app folder they upload to ingest the csv as a log instead of a lookup, however your approach seems much more pragmatic.

[cbhattad1]

@skhedim Hey, I am facing the same problem. Did you find a solution for it?

[starcher]

Use a dict reader in python to read it in as a list of dicts.

Then send as the event dict.
https://github.com/georgestarcher/Splunk-Class-httpevent