Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Splunk Search
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- How to Parse EMR Log to generate table output?
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How to Parse EMR Log to generate table output?
edwinmae
Path Finder
03-24-2022
06:47 AM
I have a log events (each about 260 lines) related to our AWS EMR Cluster 'performance' metrics. It seems it's just a collection of output from certain Linux commands.
**
If I want to parse e.g. like free -m, to generate some table output / timechart out of those, how would I start to parse these (assuming it's possible) ?
Extract New fields, using Regular Expression didn't seem to work ...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
somesoni2
Revered Legend
03-24-2022
09:14 AM
Could you provide a sample entry/log (whole log, mask anything sensitive) in text format?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
edwinmae
Path Finder
03-24-2022
11:26 PM
3/25/22
6:07:02.000 AM
Date: Fri, 25 Mar 2022 06:07:02 GMT
x-amz-bucket-region: us-east-1
x-amz-access-point-alias: false
Content-Type: application/xml
Server: AmazonS3
# Now traceroute it
traceroute -T --sport=17241 -p 443 -w 3 -n -m 10 elasticmapreduce.s3.amazonaws.com
traceroute to elasticmapreduce.s3.amazonaws.com (52.217.108.28), 10 hops max, 60 byte packets
1 10.119.0.247 0.110 ms 0.078 ms 0.100 ms
2 * * *
3 * * *
4 * * *
5 * * 241.0.10.12 0.854 ms
6 241.0.10.15 0.799 ms 241.0.9.199 0.842 ms 240.1.100.16 0.823 ms
7 240.1.100.19 0.742 ms 240.1.100.24 0.714 ms 240.1.100.20 0.818 ms
8 242.3.185.1 16.959 ms 26.163 ms 242.3.183.129 1.169 ms
9 100.95.3.19 1.328 ms 100.95.19.31 1.364 ms 100.95.3.17 1.308 ms
10 100.91.176.205 66.646 ms 100.91.176.217 66.800 ms 100.91.177.131 65.994 ms
# listing of last logged in users
last -w -n 25
reboot system boot 4.14.241-184.433.amzn2.x86_64 Fri Mar 25 05:15 - 06:07 (00:51)
reboot system boot 4.14.241-184.433.amzn2.x86_64 Fri Aug 6 20:41 - 20:51 (00:09)
wtmp begins Fri Aug 6 20:41:09 2021
# whats io usage look like
iostat -x 1 5
Linux 4.14.241-184.433.amzn2.x86_64 (ip-10-20-30-40) 03/25/22 _x86_64_ (32 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
12.27 0.00 1.28 2.72 0.00 83.73
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme1n1 0.00 5.11 0.24 68.54 5.32 14455.52 420.51 0.49 8.54 0.18 8.57 1.42 9.77
nvme2n1 0.00 0.28 0.14 61.21 2.22 15169.38 494.59 0.51 10.06 0.18 10.08 1.69 10.34
nvme3n1 0.00 0.40 0.14 56.51 2.22 13949.54 492.62 0.46 9.86 0.25 9.88 1.65 9.36
nvme4n1 0.00 0.40 0.14 58.78 2.22 14544.40 493.83 0.50 10.09 0.19 10.12 1.69 9.95
nvme0n1 0.02 6.74 7.85 5.38 276.56 897.95 177.55 0.04 4.61 2.02 8.41 0.83 1.10
dm-0 0.00 0.00 0.06 0.53 1.04 64.44 221.22 0.01 9.97 0.19 11.07 0.77 0.05
dm-1 0.00 0.00 0.06 24.40 1.07 14390.98 1176.70 53.67 2193.89 0.25 2199.03 4.09 10.00
dm-2 0.00 0.00 0.06 10.22 1.04 15169.34 2951.65 50.23 4886.42 35.81 4915.42 10.09 10.37
dm-3 0.00 0.00 0.06 9.96 1.04 13949.50 2783.36 36.04 3595.08 0.25 3617.01 9.37 9.40
dm-4 0.00 0.00 0.06 10.00 1.04 14544.36 2891.29 42.12 4185.86 0.29 4211.43 9.93 9.99
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.06 0.00 0.00 99.94
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme2n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme3n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme4n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.31 0.00 0.19 0.06 0.00 99.44
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme1n1 0.00 0.00 0.00 1.00 0.00 4.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme2n1 0.00 0.00 0.00 34.00 0.00 8388.00 493.41 0.06 2.24 0.00 2.24 0.59 2.00
nvme3n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme4n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 1.00 0.00 4.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 6.00 0.00 8388.00 2796.00 0.06 9.33 0.00 9.33 3.33 2.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.09 0.06 0.00 99.84
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme2n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme3n1 0.00 246.00 0.00 68.00 0.00 12784.00 376.00 0.09 1.76 0.00 1.76 0.47 3.20
nvme4n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 275.00 0.00 12784.00 92.97 4.86 17.69 0.00 17.69 0.12 3.20
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
6.64 0.00 0.22 0.00 0.00 93.14
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme1n1 0.00 0.00 1.00 0.00 16.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme2n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme3n1 0.00 0.00 1.00 0.00 16.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme4n1 0.00 0.00 1.00 0.00 16.00 0.00 32.00 0.00 4.00 4.00 0.00 0.00 0.00
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 1.00 0.00 16.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 1.00 0.00 16.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 1.00 0.00 16.00 0.00 32.00 0.00 4.00 4.00 0.00 4.00 0.40
# whats memory usage look like
free -m
total used free shared buff/cache available
Mem: 255139 27090 187176 4 40872 225853
Swap: 0 0 0
# trend memory
vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 191668400 8632 41844724 0 0 9 1845 51 221 12 1 84 3 0
15 0 0 187757920 8632 41845156 0 0 0 0 196146 187847 35 18 47 0 0
15 0 0 186603456 8632 41845280 0 0 0 0 23810 17182 45 3 52 0 0
15 0 0 186288448 8632 41845312 0 0 0 288 5776 3159 47 1 52 0 0
19 0 0 186075552 8632 41845632 0 0 0 0 119725 126526 32 7 61 0 0
# amount of disk free
df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 125G 0 125G 0% /dev
tmpfs 125G 0 125G 0% /dev/shm
tmpfs 125G 768K 125G 1% /run
tmpfs 125G 0 125G 0% /sys/fs/cgroup
/dev/nvme0n1p1 10G 5.5G 4.6G 55% /
/dev/mapper/nvme1n1p1 5.0G 213M 4.8G 5% /emr
/dev/mapper/nvme1n1p2 123G 9.4G 114G 8% /mnt
/dev/mapper/nvme2n1 128G 11G 118G 8% /mnt1
/dev/mapper/nvme3n1 128G 8.1G 120G 7% /mnt2
/dev/mapper/nvme4n1 128G 8.7G 120G 7% /mnt3
tmpfs 25G 0 25G 0% /run/user/990
tmpfs 25G 0 25G 0% /run/user/991
tmpfs 25G 0 25G 0% /run/user/0
set +o verbose
Top 10 folders in /emr in MB:
172 /emr/instance-controller/lib/bootstrap-actions/2
3 /emr/instance-state
1 /emr/setup-devices
1 /emr/logpusher/run
1 /emr/logpusher/log
1 /emr/logpusher/db
1 /emr/instance-controller/run
1 /emr/instance-controller/log/system-actions/3
1 /emr/instance-controller/log/system-actions/2
1 /emr/instance-controller/log/system-actions/1
Top 20 folders including subdirectories in /emr in MB:
176 /emr
173 /emr/instance-controller
172 /emr/instance-controller/lib
172 /emr/instance-controller/lib/bootstrap-actions
172 /emr/instance-controller/lib/bootstrap-actions/2
3 /emr/instance-state
1 /emr/apppusher
1 /emr/apppusher/log
1 /emr/apppusher/run
1 /emr/instance-controller/db
1 /emr/instance-controller/lib/bootstrap-actions/1
1 /emr/instance-controller/lib/info
1 /emr/instance-controller/lib/sslKeys
1 /emr/instance-controller/log
1 /emr/instance-controller/log/bootstrap-actions
1 /emr/instance-controller/log/bootstrap-actions/1
1 /emr/instance-controller/log/bootstrap-actions/2
1 /emr/instance-controller/log/system-actions
1 /emr/instance-controller/log/system-actions/1
1 /emr/instance-controller/log/system-actions/2
# dump network statistics
netstat -s -e
Ip:
27677847 total packets received
3 with invalid addresses
0 forwarded
0 incoming packets discarded
27677843 incoming packets delivered
18440811 requests sent out
3 outgoing packets dropped
31 dropped because of missing route
Icmp:
73 ICMP messages received
55 input ICMP message failed.
ICMP input histogram:
destination unreachable: 12
timeout in transit: 55
echo replies: 6
6 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo request: 6
IcmpMsg:
InType0: 6
InType3: 12
InType11: 55
OutType8: 6
Tcp:
6584 active connections openings
4777 passive connection openings
56 failed connection attempts
48 connection resets received
150 connections established
27676466 segments received
48951171 segments send out
6558 segments retransmited
0 bad segments received.
7645 resets sent
Udp:
1307 packets received
0 packets to unknown port received.
0 packet receive errors
10621 packets sent
0 receive buffer errors
0 send buffer errors
UdpLite:
TcpExt:
12160 packets pruned from receive queue because of socket buffer overrun
5265 TCP sockets finished time wait in fast timer
18967 delayed acks sent
335 delayed acks further delayed because of locked socket
Quick ack mode was activated 474 times
17105542 packet headers predicted
4513674 acknowledgments not containing data payload received
4324063 predicted acknowledgments
123 times recovered from packet loss by selective acknowledgements
Detected reordering 33 times using time stamp
1 congestion windows partially recovered using Hoe heuristic
TCPLostRetransmit: 269
6392 fast retransmits
1 other TCP timeouts
TCPLossProbes: 250
11732 packets collapsed in receive queue due to low socket buffer
478 DSACKs sent for old packets
19 DSACKs sent for out of order packets
198 DSACKs received
358 connections reset due to unexpected data
1137 connections reset due to early user close
TCPDSACKIgnoredNoUndo: 187
TCPSackShifted: 19344
TCPSackMerged: 9843
TCPSackShiftFallback: 2633
TCPRcvCoalesce: 10144666
TCPOFOQueue: 404026
TCPOFOMerge: 18
TCPChallengeACK: 5
TCPAutoCorking: 3105716
TCPFromZeroWindowAdv: 12071
TCPToZeroWindowAdv: 12071
TCPWantZeroWindowAdv: 806608
TCPSynRetrans: 1
TCPOrigDataSent: 36414854
TCPHystartTrainDetect: 205
TCPHystartTrainCwnd: 8723
TCPACKSkippedSeq: 28
TCPACKSkippedChallenge: 150
TCPWinProbe: 2
IpExt:
InOctets: 170168321507
Get Updates on the Splunk Community!
.conf24 | Registration Open!
Hello, hello! I come bearing good news: Registration for .conf24 is now open!
conf is Splunk’s rad annual ...
ICYMI - Check out the latest releases of Splunk Edge Processor
Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.
HEC Receiver authorization ...
Introducing the 2024 SplunkTrust!
Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!
The ...
