I'm using Subsystem Device Drivers (SDD) on an AIX system to monitor SAN LUNs. When I run "datapath query devstats" command, I get output that looks like this:
Device #: 35 ============= Total Read Total Write Active Read Active Write Maximum I/O: 301295802 7081834 0 0 40 SECTOR: 967435533 1003883755 0 0 11424 Transfer Size: <= 512 <= 4k <= 16K <= 64K > 64K 13566 32412 301058990 4143978 3128690
(The spacing isn't coming out correctly on this form, but you get the idea).
It lists X number of the above depending on how many LUNs I have assigned. In this case, there will be 36 entries (0-35, #35 being the one I just pasted).
What I need is to capture the device IDs as well as "Total Read" and "Total Write" fields for both I/O and SECTOR for each device.
This command would be running every hour, its output sent directly to Splunk.
The trick here will be telling Splunk to split the output up per Device, and then extracting the needed fields across multiple lines.
Is this possible? If so, how do I tell Splunk to break the output up into chunks divided in a certain spot?
I'd modify the linebreak to include the whole event.
In addition, I was able to extract to fields using something like this:
|file /tmp/test.txt | rex field=_raw "Device #:\s+(?P<device_id>[\d]+)" | rex field=_raw "I\/O:\s+(?P<total_read_io>[\d]+)\s+(?P<total_write_io>[\d]+)\s+" | rex field=_raw "SECTOR:\s+(?P<sector_write_io>[\d]+)\s+(?P<sector_read_io>[\d]+)\s+"
You can change how events are broken up (on input, so you may need to reindex data until it's right) using the BREAK_ONLY_BEFORE parameter in props.conf. That's probably the easiest way to configure it, something like
BREAK_ONLY_BEFORE = ^Device #: \d+