Security
Highlighted

Splunk anomalies

Path Finder

I have found the following output after runing the following command. What does the out means ?

host="xxx" | anomalies

USER               PID   PSR   pctCPU       CPUTIME  pctMEM     RSZ_KB     VSZ_KB   TTY      S       ELAPSED  COMMAND             ARGS
root                 1     0      0.0      00:00:00     0.0        684      10324   ?        S    8-21:12:55  init                [3]
root                 2     0      0.0      00:00:01     0.0          0          0   ?        S    8-21:12:55  [migration/0]       <noArgs>
root                 3     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [ksoftirqd/0]       <noArgs>
root                 4     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [watchdog/0]        <noArgs>
root                 5     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [events/0]          <noArgs>
root                 6     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [khelper]           <noArgs>
root                 7     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [kthread]           <noArgs>
root                 9     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [xenwatch]          <noArgs>
root                10     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [xenbus]            <noArgs>
root                17     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [migration/1]       <noArgs>
root                18     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [ksoftirqd/1]       <noArgs>
root                19     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [watchdog/1]        <noArgs>
root                20     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [events/1]          <noArgs>
root                21     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [migration/2]       <noArgs>
root                22     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [ksoftirqd/2]       <noArgs>
root                23     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [watchdog/2]        <noArgs>
root                24     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [events/2]          <noArgs>
root                25     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [migration/3]       <noArgs>
root                26     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [ksoftirqd/3]       <noArgs>
root                27     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [watchdog/3]        <noArgs>
root                28     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [events/3]          <noArgs>
root                33     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [kblockd/0]         <noArgs>
root                34     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [kblockd/1]         <noArgs>
root                35     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [kblockd/2]         <noArgs>
root                36     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [kblockd/3]         <noArgs>
root                37     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [cqueue/0]          <noArgs>
root                38     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [cqueue/1]          <noArgs>
root                39     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [cqueue/2]          <noArgs>
root                40     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [cqueue/3]          <noArgs>
root                44     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [khubd]             <noArgs>
root                46     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [kseriod]           <noArgs>
root               126     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [pdflush]           <noArgs>
root               127     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [pdflush]           <noArgs>
root               128     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [kswapd0]           <noArgs>
root               129     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [aio/0]             <noArgs>
root               130     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [aio/1]             <noArgs>
root               131     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [aio/2]             <noArgs>
root               132     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:55  [aio/3]             <noArgs>
root               262     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:52  [kpsmoused]         <noArgs>
root               316     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:51  [ksnapd]            <noArgs>
root               319     0      0.0      00:00:01     0.0          0          0   ?        S    8-21:12:51  [kjournald]         <noArgs>
root               348     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:50  [kauditd]           <noArgs>
root               377     2      0.0      00:00:00     0.0       1804      13524   ?        S    8-21:12:49  udevd               -d
root               843     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:44  [kmpathd/0]         <noArgs>
root               844     1      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:44  [kmpathd/1]         <noArgs>
root               845     2      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:44  [kmpathd/2]         <noArgs>
root               846     3      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:44  [kmpathd/3]         <noArgs>
root               872     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:43  [kjournald]         <noArgs>
root               874     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:43  [kjournald]         <noArgs>
root               876     0      0.0      00:00:00     0.0          0          0   ?        S    8-21:12:43  [kjournald]         <noArgs>
root              1340     0      0.0      00:00:00     0.3      14200      24000   ?        S    8-21:12:39  restorecond         <noArgs>
root              1350     3      0.0      00:00:00     0.0        604       5888   ?        S    8-21:12:39  syslogd             -m_0
root              1354     3      0.0      00:00:00     0.0        424       3780   ?        S    8-21:12:39  klogd               -x
root              1368     0      0.0      00:00:01     0.0        556      14528   ?        S    8-21:12:39  mcstransd           <noArgs>
dbus              1379     0      0.0      00:00:00     0.0        892      31488   ?        S    8-21:12:39  dbus-daemon         --system
root              1431     2      0.0      00:00:04     0.0       1256      65936   ?        S    8-21:12:39  bash                /usr/sbin/xe-daemon_-p_/var/run/xe-daemon.pid
root              1510     0      0.0      00:00:01     0.1       6616     148276   ?        S    8-21:12:38  snmpd               -Lsd_-Lf_/dev/null_-p_/var/run/snmpd.pid_-a
root              1525     0      0.0      00:00:00     0.0       1216      60524   ?        S    8-21:12:37  sshd                <noArgs>
root              1535     0      0.0      00:00:00     0.0       1212      74812   ?        S    8-21:12:37  crond               <noArgs>
root              1546     1      0.0      00:00:00     0.0        312      58900   ?        S    8-21:12:37  rhnsd               --interval_240
68                1556     0      0.0      00:00:00     0.0       3276      30260   ?        S    8-21:12:37  hald                <noArgs>
root              1557     1      0.0      00:00:00     0.0       1008      21648   ?        S    8-21:12:37  hald-runner         <noArgs>
root              1569     0      0.0      00:00:00     0.0        532       3780   ?        S    8-21:12:36  agetty              xvc0_9600_vt100-nav
root             12893     3      0.0      00:00:00     0.0        428       3764   ?        S         00:25  sleep               60
s-splunk         12900     2      0.0      00:00:00     0.0       1136      63844   ?        S         00:00  sh                  /opt/splunk/etc/apps/unix/bin/ps.sh
s-splunk         12913     0      0.0      00:00:00     0.0        932      65600   ?        R         00:00  ps                  -wweo_uname,pid,psr,pcpu,cputime,pmem,rsz,vsz,tty,s,etime,args
s-splunk         12914     3      0.0      00:00:00     0.0        540      58904   ?        S         00:00  tee                 /dev/null
s-splunk         12915     2      0.0      00:00:00     0.0       1056      63912   ?        S         00:00  awk                 {NR_==_1_&&_$0_=_header}_{sub("^_",_"",_$1);_if_(NF>12)_{args=$13;_for_(j=14;_j<=NF;_j++)_args_=_args_"_"_$j}_else_args="<noArgs>";_sub("^[^\134[:_-]*/",_"",_$12)}_(NR>1)_{if_($4<0_||_$4>100)_$4=0;_if_($6<0_||_$6>100)_$6=0}_{if_(NR_==_1)_{print_$0}_else_{printf_"%-14.14s_%6s_%4s_%6s_%12s_%6s_%8s_%8s_%-7.7s_%1.1s_%12s_%-18.18s_%s\n",_$1,_$2,_$3,_$4,_$5,_$6,_$7,_$8,_$9,_$10,_$11,_$12,_args}}_header=USER_PID_PSR_pctCPU_CPUTIME_pctMEM_RSZ_KB_VSZ_KB_TTY_S_ELAPSED_COMMAND_ARGS
s-splunk         18321     3      0.1      00:00:48     1.2      51700     179848   ?        S      12:17:48  splunkd             -p_8089_restart
s-splunk         18322     3      0.0      00:00:02     0.1       7112      47060   ?        S      12:17:48  splunkd             -p_8089_restart
s-splunk         18388     2      0.0      00:00:00     0.6      28560     671264   ?        S      12:17:38  python              -O_/opt/splunk/lib/python2.6/site-packages/splunk/appserver/mrsparkle/root.py_restart
Tags (2)
0 Karma
Highlighted

Re: Splunk anomalies

Splunk Employee
Splunk Employee

The anomalies command, http://www.splunk.com/base/Documentation/latest/SearchReference/Anomalies acts as a filter as well as a labeller. It tries to figure out, for a data stream, which events are unusual. It will pass through more unusual events, and filter out less unusual ones (configurable), and it will apply fields to events indicating how unusual it felt they were. If you used it for a particular thing, such as a web server log, it might help you identify trouble points, changes, and other things of interest.

In this case, it seems you are pointing it at all data from a particular host. Since this is probably a heterogeneous set of data, I think anomolies will be very hard pressed to guess which is unexpected.

The event you actually are seeing appears to be the listeing of processes running on that system, probably as produced by 'ps', probably as produced by the unix app. Compared to most data you receive, this event is very large, and differently structured, so it would be unsurprising if anomalies found it to be relatively unusual.