Combining searches and the data gets scrambled. Ideas?

[talbot7]

I have two different sets of data coming in Splunk:

Dec  1 08:43:07 a4-hpc2-2.llnl.gov logger: dom0stat42 : timestamp=08:43:02 pool=General2 hardware=a4-hpc2-2 dom0_one_min_load=0.50 dom0_free_memory=114 dom0_total_memory=1024 dom0_used_memory=910 xen_free_memory=12507 xen_total_memory=49149 xen_used_memory=36642 dom0_total_vmos_mb=3355444 dom0_used_vmos_mb=2684594 dom0_free_vmos_mb=670850 stolen_cpu_ticks= steal_time=0.20

Dec  1 08:42:12 a4-hpc2-2.llnl.gov logger: gvmstats timestamp=08:42:02 pool=General2 hardware=a4-hpc2-2 gvm=splatint0007 memory=2048 vcpu=1 cpu_seconds=244751.0 vnc_console=a4-hpc2-2:5906

I am trying to search through them and pull out some key information:

index=unix gvmstats OR dom0stat42 hardware=a4-hpc2-2 
| eval xen_free_memory_GB=round(xen_free_memory/1024,2) 
| stats values(xen_free_memory_GB), values(gvm), values(vcpu), values(memory), values(vnc_console) by pool hardware

All the data gets combined, but scrambled:

    pool hardware  values(xen_free_memory_GB) values(gvm) values(vcpu) values(memory) values(vnc_console)
    General2    a4-hpc2-2   12.21   Domain-0      1   1024    a4-hpc2-2:5900
                                    dbdev0003     2   2048    a4-hpc2-2:5901
                                    oidev0001     4   4096    a4-hpc2-2:5902
                                    oidvqa0001    8   8192    a4-hpc2-2:5903
`                              `    savidev0006               a4-hpc2-2:5904
                                    saviqa0010                a4-hpc2-2:5905
                                    secwsint0003              a4-hpc2-2:5906
                                    splatint0007              a4-hpc2-2:5907

The pool, hardware, and "xen_free_memory_GB" are correct. Every thing past that is all screwed up. Domain-0 has 1024 (correct, but only by luck), 12vcpu, and no vnc_console.

Any ideas of how to fix this?

[talbot7]

Table gives me all the data, and formats it correctly. But I get ALL the data, not just the "last" values. using last() does not seem to work. Ideas?

[lguinn2]

Last as in "most recent"? Remember that Splunk naturally sorts that data in reverse chronological order, so the first results are the most recent.

[lguinn2]

Your stats command is saying "for each combination of pool and hardware, show me ALL the values for the other fields."

Stats is used to summarize. I think you may prefer the table command for this case:

...
| table pool hardware xen_free_memory_GB  gvm vcpu memory) vnc_console

Or maybe you can describe the output that you want in more detail...

If you just want to see the most recent event, do this

...
| table pool hardware xen_free_memory_GB  gvm vcpu memory) vnc_console
| head 1

Or, perhaps you want this

...
| table pool hardware xen_free_memory_GB  gvm vcpu memory) vnc_console
| tail 1