Using text files as input as a single set of data

[Linze99]

Hi, I got production logs as txt files containing many Fields that are always in the format $_XXX: YYY with XXX being the Field name and YYY being the field value. All fields belong together as one set of production data for one device. The complete file has a timestamp ($_Date: ...) somewhere in the text. 

 I now want that whole file parsed like a csv file but with only one row of data. So that the XXXs are my value names and the YYYs are the actual values like on this picture from a csv: 

Whatever I try, Splunk always wants to handle my Values as separate events instead of one large single event.

Is there a simple way to achieve this?

[richgalloway]

It would help if you said what you've tried so far and how those attempts failed to meet expectations.

A file that is not in CSV format cannot be parsed by Splunk as a CSV.  Use a Python script (as a scripted input) or a third-party utility like Cribl to re-format the data into CSV.

It should be a straightforward process to ingest the file in it's existing format as it's in K=V format with the '=' replaced by ':'.  Have you tried using a transform like this?

[mytransform]
REGEX = ([^:]+):(\S+)
FORMAT = $1::$2

[Linze99]

Hi, thanks for the fast reply, I just started working with splunk so I'm not very experienced. Its additionally harder to start of with an unusual task like this.

I read about transforms but I dont know where to put those lines of code so thats why I haven tried anything so far. I dont necessarily want to parse it as a csv that was just an idea. I cant format the text files before importing them to splunk because depending on the manufacturer or product they change completely. The only thing that is always the same is the definition of Values like $_XXX: YYY. For example my file looks like that:

$_A: 15.3
$_B: 18.4
Random Text that isnt a value!
$_Date: 2021-10-08 15-23-12
$_C: False
$_D: True
Random numbers 12314234234234234

and so on.

I want Splunk to import this file and recognize only the $_ Values and leave the rest. 

Thats why I tried to use REGEX and Split the Vlaues into separate events. This worked fine but the events had no relation to each other after that, besides the filename they came from. But the filename should be completely irellevant as its just a number given by the Manufacturer of the Device which could repeat in the future. I need the data to be bundled together. So that one event is one Production log from one device containing all said Fields.

I really want to try your answer but as I said I dont know where to put the code. I know that there is a transform.conf file but I dont know how to change it/where it is located. 

[richgalloway]

You're jumping into the deep end of the pool.  

Let's step back a little.  Before we try to parse the events, let's get the event onboard properly.  This means combining lines that belong together.  What can Splunk look for to determine when one event ends and the next begins?  For example, if every event starts with "$_A" then we'd tell Splunk to break events immediately before that string.

[Linze99]

Yes I already achieved this so far, but one whole txt file all belongs to one single log of one tested device, which should be one single vent right?

So that this event contains all the fields/values

[richgalloway]

Yes, one file can be a single event, if that makes sense in your world.  Once you have that, just add the transform I gave you earlier to your transforms.conf file then reference that transform in props.conf like this:

[mysourcetype]
TRANSFORMS-parse = mytransform

If you share the current props.conf settings for that sourcetype, we can advise of any other changes that should be made.

[Linze99]

So I tried your Solution.

both my transforms and props are in .etc/system/local.

I changed them according to your answers.

When I add a new textfile to splunk it gets shown in Search & Reporting but there are no additional fields extracted. I can extract the fields manually but that would take very long with that many fields. In Field extractions menu I added a new extraction with my source type and mytransform and so on. But how can I apply this extraction?

[richgalloway]

Check the permissions on the field extractions you created to make sure they're accessible by other users and from other apps.

[Linze99]

Ok, I want to try that, but I'm using splunk enterprise on a server. I cant find the files on the splunk user interface. How can I access the .conf files?

Edit: Found them, testing it now