topic Index csv files with grouped fields? in Getting Data In

Index csv files with grouped fields?

kjell_ml — Thu, 13 Aug 2020 11:59:43 GMT

I'm using Universal forwarder and trying to consume a complex csv file. Usually this works OK by configuring props.conf correctly on the forwarder. However, this CSV file is quite complex, with many Grouped Field. "{ }" to be used for encapsulating outer most list and "[]" for internal lists.
Every internal list within {} or [] will be comma separated.

Is this possible to achieve? I mean to get the naming of the header fields correct?

Since the header fields will change depending on which groups or lists have data.

I have a good documentation of the csv file format, but haven't found any ways to make props.conf handle these grouped fields and lists.....

Re: Index csv files with grouped fields?

richgalloway — Thu, 13 Aug 2020 13:21:51 GMT

Example data would be helpful, but I suspect the file is more complex than Splunk can handle. Splunk expects all CSV rows to have the same set of fields.
Consider creating a scripted input to ingest that file.

Re: Index csv files with grouped fields?

kjell_ml — Thu, 13 Aug 2020 13:36:36 GMT

Here's an example, I have anonymized the data and broken it up on separate lines where the internal groups and lists occur: (Inside the [] brackets there can be data or not, depending on type of transactions. This is what messes up the header_field naming)

0-0-9-1,64,xxx.xxx.x.x,28fa2342-5b1b-4605-b178-f2ec2b0b5327,20200813151737.417061,ff-14f-3fffffff-00152,,,
{REGISTER,,3600,0,
[sip:+4700000000@xxx.xxxxxx.xxxxxx.xxx.xxxxxxxxxxx.xxx],
[sip:+4700000000@xxx.xxxxx.xxxxxx.xxx.xxxxxxxxxxx.xxx,tel:+4700000000],
sip:+4700000000@xxx.xxxxxx.xxxxxx.xxx.xxxxxxxxxxx.xxx,
[],
20200813151737.417083,417,20200813151742.479075,479,,0,-1,xxxx-x-xxxxx-xxx;xxxxx-xxxx-xx-xxxx=xxxxxxxxxxxxxxxx,0,xxxxxxx.xxx,,0,
[],
,
[
[255.671.3043243843-1293344657.140,Ioi1,12345,]
]},
{[0,0,,0,0,0,0,xxxx-x-xxxxx-xxx,0],
},
{1,0,,0,0,0,false},
,,,,,,,,,,,,,
{[3,2,20200813151737.419758,REGISTER,0,xx.xx.x.xx,xxx.xxx.x.xx],
[3,2,20200813151742.434086,REGISTER,0,xxx.xxx.xx.xxx,xxx.xxx.xx.xxx],
[3,2,20200813151742.467122,200,0,xxx.xxx.xx.xxx,xxx.xxx.xx.xxx],
[3,2,20200813151742.478905,200,0,xxx.xxx.xx.xxx,xx.xx.x.xx]}\x00