Splunk Hadoop Logs Parsing

[don12]

Hello Guys,

Am having with hadoop logs that is not properly parsed when I use the sourcetype:linux_secure or access_combined. I have gone through the splunk documentation and hadoop docs to see if there is a way for me to parse the logs properly but not seeing anything of help.

I would be really glad if someone can point me in the right direction.

Sample Logs below:

... 10 more
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) ~[hadoop-common-3.0.0-cdh6.3.0.jar:?]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
2021-04-20 12:34:21,906 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-56]: Error occurred during processing of message.
... 10 more
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) ~[hadoop-common-3.0.0-cdh6.3.0.jar:?]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_181]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-2.1.1-cdh6.3.0.jar:2.1.1-cdh6.3.0]
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
2021-04-20 12:34:21,905 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-56]: Error occurred during processing of message.
2021-04-20 12:34:21,887 INFO org.apache.hadoop.hive.ql.session.SessionState: [a11894f3-4ce4-478a-a11a-8fa624429f33 HiveServer2-Handler-Pool: Thread-6527170]: Resetting thread name to HiveServer2-Handler-Pool: Thread-6527170
2021-04-20 12:34:21,887 INFO org.apache.hadoop.hive.conf.HiveConf: [a11894f3-4ce4-478a-a11a-8fa624429f33 HiveServer2-Handler-Pool: Thread-6527170]: Using the default value passed in for log id: a11894f3-4ce4-478a-a11a-8fa624429f33
2021-04-20 12:34:21,884 INFO org.apache.hadoop.hive.ql.session.SessionState: [HiveServer2-Handler-Pool: Thread-6527170]: Updating thread name to a11894f3-4ce4-478a-a11a-8fa624429f33 HiveServer2-Handler-Pool: Thread-6527170
2021-04-20 12:34:21,884 INFO org.apache.hadoop.hive.conf.HiveConf: [HiveServer2-Handler-Pool: Thread-6527170]: Using the default value passed in for log id: a11894f3-4ce4-478a-a11a-8fa624429f33
... 10 more

[venkatasri]

Hi @don12 

Following config would work for above logs format, copy this to props.conf file to either HF/indexer layer under /opt/splunk/etc/apps/<app_name>/local (or)  /opt/splunk/etc/system/local followed by a restart. 

[hadoop_logs]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
TIME_PREFIX=^
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3Q

----------------------------------------------------

An upvote would be appreciated if it helps!

[don12]

Hello,

I tried pushing the props.conf changes to all the indexes but I am not seeing any changes. I still see something like this below:

... 10 more

• host = goodies
• index = blala_linux
• source = /var/log/hive/hadoop-cmf-hive-HIVESERVER2-*.out
• sourcetype = linux_secure

Thank you so much for your help.