I’m trying to use the Splunk_TA_linux app (3412) with an old system (CentOS 5 vintage) as the target. Getting collectd to send its observations to Splunk is problematic (the collectd version is too old and I’m limited as to what I can change on the target), so I’ve been forced to set up collectd to merely dump its data locally in csv format, and I intend to have the Universal Forwarder monitor the data dump directory. The problem is in converting the event formats into what Splunk_TA_linux expects, namely event types such as linux_collectd_cpu , linux_collectd_memory , and so forth. I think I need to define a bunch of new sourcetypes, which will manipulate the events to transform them into the various event types expected. The forwarder is limited to INDEXED_EXTRACTIONS , but that should be enough.
Collectd has been configured to monitor several system metrics, and uses its csv plugin for output. The csv files go in the /var/collectd/csv folder. Collectd then creates a single subfolder, named using <hostname> (in this case, sv3vm5b.etv.lab ). There are then a bunch of subfolders for the various metrics: cpu-0, cpu-1, cpu-2, cpu-3, df, disk-vda, disk-vda1, disk-vda2, interface, irq, load, memory, processes, processes-all, swap, tcp-conns-22-local, tcp-conns-111-local, tcp-conns-698-local, tcp-conns-2207-local, tcp-conns-2208-local, tcp-conns-8089-local, uptime . The cpu-* folders are tracking several cpu metrics ( idle, interrupt, nice, softirq, steal, system, user, wait ). The first metric (CPU idle time) generates daily files, e.g. cpu-idle-2017-12-12, cpu-idle-2017-12-13 , etc. This pattern is the same for each metric. The contents of cpu-idle-<date> are:
epoch,value
1513025715,491259
1513025725,492242
...
Again, this pattern is the same for the other files: a header line listing the fields (although the names are pretty generic), then regular measurements consisting of a Unix timestamp followed by one to three integer or floating-point values. What the collectd input plugins measure is documented on the collectd wiki.
In collectd JSON or Graphite mode, the Splunk source type is linux:collectd:http:json or linux:collectd:graphite , the event type is linux_collectd_cpu and the data model is "ITSI OS Model Performance.CPU". Splunk_TA_linux's eventtypes.conf
ties linux_collectd_cpu to the two source types, so this gives rise to a first question: Will Splunk_TA_linux's eventtypes.conf need tweaking?
Assuming I set the forwarder to monitoring /var/collectd/csv/*/cpu-*/cpu-idle-* (can I specify paths using jokers like that?), I could then set the source type for those daily files as a custom type. The process would be repeated for the various other collectd files and folders, resulting in a slew of custom source types.
source type: collectd_csv_cpu_idle
dest app: Search & Reporting (should this be Splunk_TA_linux ?)
category: Metrics
indexed extractions: csv
timestamp: auto (this will recognise a Unix timestamp, right?)
field delimiter: comma
quote character: double quote (unused)
File preamble: ^epoch,value$
Field names: custom
…and that’s where I’m stumped. This expects a comma-separated list of field names. Is the first one _time or is that assumed? The “ITSI OS Model Performance.CPU” documentation has no fields for the jiffy counts ( cpu-idle, -interrupt, -nice, -softirq, -steal, -system, -user, -wait are reporting the number of jiffies spent in each of the possible CPU states, respectively idle, IRQ, nice, softIRQ, steal, system, user, wait-IO ) but does have cpu_time and cpu_user_percent fields. Isn’t there supposed to be a correspondence? Is Splunk_TA_linux further transforming the collectd inputs to fit them to the data models, so that I need more than just INDEXED_EXTRACTIONS ? And what about those fields that can only be extracted from the source paths, like the host ( sv3vm5b.etv.lab ) and number of CPUs, for instance?
... View more