We measure 50 values every 5 seconds during each hour long experiment. We do these experiments many times under different conditions (different ambient temp, SW build, HW type, etc. A dozen or so different metadata parameters). Each run saves measurements in a csv file I want to import into splunk but I'd like some advice on where to put the metadata about the experiment. The SW build number, for example, applies to the whole experiment (it doesn't change per row) and I want to be able to search on it.
Initially I thought I would just add a new column in the csv for each metadata value. That works fine, but the values are all the same in the column, so that seems wasteful. Then I thought of encoding the metadata into a simple string and using that for the source value, but then At some point I have to parse. Then I thought about a separate lookup table with some foreign key in the csv. That seems too database-y. (Not that there's anything wrong with that. Some of my best friends are DBAs)
If my goal is filter on the metadata values and analyze the measurements for just those experiments (e.g. "For all experiments run at 30C with SW build 1234, plot values for measurement x"), where/how should I store the metadata?
Thanks for any advice
... View more