Splunk Search
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

How to Parse EMR Log to generate table output?

edwinmae
Path Finder
‎03-24-2022 06:47 AM

I have a log events (each about 260 lines) related to our AWS EMR Cluster 'performance' metrics. It seems it's just a collection of output from certain Linux commands.

edwinmae_0-1648129144646.png

**

If I want to parse e.g. like free -m, to generate some table output / timechart out of those, how would I start to parse these (assuming it's possible) ?

edwinmae_1-1648129230417.png

Extract New fields, using Regular Expression didn't seem to work ...

Labels (4)
0 Karma
Reply

somesoni2
Revered Legend
‎03-24-2022 09:14 AM

Could you provide a sample entry/log (whole log, mask anything sensitive) in text format?

0 Karma
Reply

edwinmae
Path Finder
‎03-24-2022 11:26 PM 
3/25/22
6:07:02.000 AM	
Date: Fri, 25 Mar 2022 06:07:02 GMT
x-amz-bucket-region: us-east-1
x-amz-access-point-alias: false
Content-Type: application/xml
Server: AmazonS3
# Now traceroute it
traceroute -T --sport=17241 -p 443 -w 3 -n -m 10 elasticmapreduce.s3.amazonaws.com
traceroute to elasticmapreduce.s3.amazonaws.com (52.217.108.28), 10 hops max, 60 byte packets
 1  10.119.0.247  0.110 ms  0.078 ms  0.100 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * 241.0.10.12  0.854 ms
 6  241.0.10.15  0.799 ms 241.0.9.199  0.842 ms 240.1.100.16  0.823 ms
 7  240.1.100.19  0.742 ms 240.1.100.24  0.714 ms 240.1.100.20  0.818 ms
 8  242.3.185.1  16.959 ms  26.163 ms 242.3.183.129  1.169 ms
 9  100.95.3.19  1.328 ms 100.95.19.31  1.364 ms 100.95.3.17  1.308 ms
10  100.91.176.205  66.646 ms 100.91.176.217  66.800 ms 100.91.177.131  65.994 ms
# listing of last logged in users
last -w -n 25
reboot   system boot  4.14.241-184.433.amzn2.x86_64 Fri Mar 25 05:15 - 06:07  (00:51)    
reboot   system boot  4.14.241-184.433.amzn2.x86_64 Fri Aug  6 20:41 - 20:51  (00:09)    
wtmp begins Fri Aug  6 20:41:09 2021
# whats io usage look like
iostat -x 1 5
Linux 4.14.241-184.433.amzn2.x86_64 (ip-10-20-30-40) 	03/25/22 	_x86_64_	(32 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.27    0.00    1.28    2.72    0.00   83.73
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme1n1           0.00     5.11    0.24   68.54     5.32 14455.52   420.51     0.49    8.54    0.18    8.57   1.42   9.77
nvme2n1           0.00     0.28    0.14   61.21     2.22 15169.38   494.59     0.51   10.06    0.18   10.08   1.69  10.34
nvme3n1           0.00     0.40    0.14   56.51     2.22 13949.54   492.62     0.46    9.86    0.25    9.88   1.65   9.36
nvme4n1           0.00     0.40    0.14   58.78     2.22 14544.40   493.83     0.50   10.09    0.19   10.12   1.69   9.95
nvme0n1           0.02     6.74    7.85    5.38   276.56   897.95   177.55     0.04    4.61    2.02    8.41   0.83   1.10
dm-0              0.00     0.00    0.06    0.53     1.04    64.44   221.22     0.01    9.97    0.19   11.07   0.77   0.05
dm-1              0.00     0.00    0.06   24.40     1.07 14390.98  1176.70    53.67 2193.89    0.25 2199.03   4.09  10.00
dm-2              0.00     0.00    0.06   10.22     1.04 15169.34  2951.65    50.23 4886.42   35.81 4915.42  10.09  10.37
dm-3              0.00     0.00    0.06    9.96     1.04 13949.50  2783.36    36.04 3595.08    0.25 3617.01   9.37   9.40
dm-4              0.00     0.00    0.06   10.00     1.04 14544.36  2891.29    42.12 4185.86    0.29 4211.43   9.93   9.99
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.06    0.00    0.00   99.94
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme1n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme2n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme3n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme4n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme0n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.31    0.00    0.19    0.06    0.00   99.44
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme1n1           0.00     0.00    0.00    1.00     0.00     4.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme2n1           0.00     0.00    0.00   34.00     0.00  8388.00   493.41     0.06    2.24    0.00    2.24   0.59   2.00
nvme3n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme4n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme0n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    1.00     0.00     4.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    6.00     0.00  8388.00  2796.00     0.06    9.33    0.00    9.33   3.33   2.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.09    0.06    0.00   99.84
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme1n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme2n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme3n1           0.00   246.00    0.00   68.00     0.00 12784.00   376.00     0.09    1.76    0.00    1.76   0.47   3.20
nvme4n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme0n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00  275.00     0.00 12784.00    92.97     4.86   17.69    0.00   17.69   0.12   3.20
dm-4              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.64    0.00    0.22    0.00    0.00   93.14
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme1n1           0.00     0.00    1.00    0.00    16.00     0.00    32.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme2n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme3n1           0.00     0.00    1.00    0.00    16.00     0.00    32.00     0.00    0.00    0.00    0.00   0.00   0.00
nvme4n1           0.00     0.00    1.00    0.00    16.00     0.00    32.00     0.00    4.00    4.00    0.00   0.00   0.00
nvme0n1           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    1.00    0.00    16.00     0.00    32.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    1.00    0.00    16.00     0.00    32.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-4              0.00     0.00    1.00    0.00    16.00     0.00    32.00     0.00    4.00    4.00    0.00   4.00   0.40
# whats memory usage look like
free -m
              total        used        free      shared  buff/cache   available
Mem:         255139       27090      187176           4       40872      225853
Swap:             0           0           0
# trend memory
vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 191668400   8632 41844724    0    0     9  1845   51  221 12  1 84  3  0
15  0      0 187757920   8632 41845156    0    0     0     0 196146 187847 35 18 47  0  0
15  0      0 186603456   8632 41845280    0    0     0     0 23810 17182 45  3 52  0  0
15  0      0 186288448   8632 41845312    0    0     0   288 5776 3159 47  1 52  0  0
19  0      0 186075552   8632 41845632    0    0     0     0 119725 126526 32  7 61  0  0
# amount of disk free
df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               125G     0  125G   0% /dev
tmpfs                  125G     0  125G   0% /dev/shm
tmpfs                  125G  768K  125G   1% /run
tmpfs                  125G     0  125G   0% /sys/fs/cgroup
/dev/nvme0n1p1          10G  5.5G  4.6G  55% /
/dev/mapper/nvme1n1p1  5.0G  213M  4.8G   5% /emr
/dev/mapper/nvme1n1p2  123G  9.4G  114G   8% /mnt
/dev/mapper/nvme2n1    128G   11G  118G   8% /mnt1
/dev/mapper/nvme3n1    128G  8.1G  120G   7% /mnt2
/dev/mapper/nvme4n1    128G  8.7G  120G   7% /mnt3
tmpfs                   25G     0   25G   0% /run/user/990
tmpfs                   25G     0   25G   0% /run/user/991
tmpfs                   25G     0   25G   0% /run/user/0
set +o verbose
Top 10 folders in /emr in MB:
172	/emr/instance-controller/lib/bootstrap-actions/2
3	/emr/instance-state
1	/emr/setup-devices
1	/emr/logpusher/run
1	/emr/logpusher/log
1	/emr/logpusher/db
1	/emr/instance-controller/run
1	/emr/instance-controller/log/system-actions/3
1	/emr/instance-controller/log/system-actions/2
1	/emr/instance-controller/log/system-actions/1
Top 20 folders including subdirectories in /emr in MB:
176	/emr
173	/emr/instance-controller
172	/emr/instance-controller/lib
172	/emr/instance-controller/lib/bootstrap-actions
172	/emr/instance-controller/lib/bootstrap-actions/2
3	/emr/instance-state
1	/emr/apppusher
1	/emr/apppusher/log
1	/emr/apppusher/run
1	/emr/instance-controller/db
1	/emr/instance-controller/lib/bootstrap-actions/1
1	/emr/instance-controller/lib/info
1	/emr/instance-controller/lib/sslKeys
1	/emr/instance-controller/log
1	/emr/instance-controller/log/bootstrap-actions
1	/emr/instance-controller/log/bootstrap-actions/1
1	/emr/instance-controller/log/bootstrap-actions/2
1	/emr/instance-controller/log/system-actions
1	/emr/instance-controller/log/system-actions/1
1	/emr/instance-controller/log/system-actions/2
# dump network statistics
netstat -s -e
Ip:
    27677847 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    27677843 incoming packets delivered
    18440811 requests sent out
    3 outgoing packets dropped
    31 dropped because of missing route
Icmp:
    73 ICMP messages received
    55 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 12
        timeout in transit: 55
        echo replies: 6
    6 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        echo request: 6
IcmpMsg:
        InType0: 6
        InType3: 12
        InType11: 55
        OutType8: 6
Tcp:
    6584 active connections openings
    4777 passive connection openings
    56 failed connection attempts
    48 connection resets received
    150 connections established
    27676466 segments received
    48951171 segments send out
    6558 segments retransmited
    0 bad segments received.
    7645 resets sent
Udp:
    1307 packets received
    0 packets to unknown port received.
    0 packet receive errors
    10621 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    12160 packets pruned from receive queue because of socket buffer overrun
    5265 TCP sockets finished time wait in fast timer
    18967 delayed acks sent
    335 delayed acks further delayed because of locked socket
    Quick ack mode was activated 474 times
    17105542 packet headers predicted
    4513674 acknowledgments not containing data payload received
    4324063 predicted acknowledgments
    123 times recovered from packet loss by selective acknowledgements
    Detected reordering 33 times using time stamp
    1 congestion windows partially recovered using Hoe heuristic
    TCPLostRetransmit: 269
    6392 fast retransmits
    1 other TCP timeouts
    TCPLossProbes: 250
    11732 packets collapsed in receive queue due to low socket buffer
    478 DSACKs sent for old packets
    19 DSACKs sent for out of order packets
    198 DSACKs received
    358 connections reset due to unexpected data
    1137 connections reset due to early user close
    TCPDSACKIgnoredNoUndo: 187
    TCPSackShifted: 19344
    TCPSackMerged: 9843
    TCPSackShiftFallback: 2633
    TCPRcvCoalesce: 10144666
    TCPOFOQueue: 404026
    TCPOFOMerge: 18
    TCPChallengeACK: 5
    TCPAutoCorking: 3105716
    TCPFromZeroWindowAdv: 12071
    TCPToZeroWindowAdv: 12071
    TCPWantZeroWindowAdv: 806608
    TCPSynRetrans: 1
    TCPOrigDataSent: 36414854
    TCPHystartTrainDetect: 205
    TCPHystartTrainCwnd: 8723
    TCPACKSkippedSeq: 28
    TCPACKSkippedChallenge: 150
    TCPWinProbe: 2
IpExt:
    InOctets: 170168321507
0 Karma
Reply
Get Updates on the Splunk Community!

New This Month in Splunk Observability Cloud - Metrics Usage Analytics, Enhanced K8s ...

The latest enhancements across the Splunk Observability portfolio deliver greater flexibility, better data and ...

Alerting Best Practices: How to Create Good Detectors

At their best, detectors and the alerts they trigger notify teams when applications aren’t performing as ...

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...