Getting Data In

Splunkd tainted with universal forwarder 7.1.2 on Linux kernel 4.9

sakti
Engager

My Splunk Universal forwarder crashes with following logs as soon as started . I don't see same crash on a different kernel,

Splunk universal forwarder version: 7.1.2-a0c72a66db66.i386
splunkd.log has,

02-01-2019 10:46:23.792 -0800 ERROR ProcessRunner - Error from ProcessRunner helper process: ERROR - Failed opening "": No such file or directory
02-01-2019 10:46:23.792 -0800 ERROR ProcessRunner - Error from ProcessRunner helper process: terminate called after throwing an instance of 'EventLoopException'
02-01-2019 10:46:23.792 -0800 ERROR ProcessRunner - Error from ProcessRunner helper process: what(): Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory
02-01-2019 10:46:23.959 -0800 FATAL ProcessRunner - Unexpected EOF from process runner child!
02-01-2019 10:46:23.959 -0800 ERROR ProcessRunner - helper process seems to have died (child killed by signal 6: Aborted)!

dmesg has,

Feb 1 10:13:37 co169 kernel: [65023.192026] CPU: 0 PID: 13240 Comm: splunkd Tainted: P O 4.9.108.Ar-10738448.4213F #1
Feb 1 10:13:37 co169 kernel: [65023.192029] task: ffff880128fb8bc0 task.stack: ffffc90005518000
Feb 1 10:13:37 co169 kernel: [65023.192031] RIP: 0023:[<00000000f77d6c09>] [<00000000f77d6c09>] 0xf77d6c09
Feb 1 10:13:37 co169 kernel: [65023.192038] RSP: 002b:00000000ffb93760 EFLAGS: 00200206
Feb 1 10:13:37 co169 kernel: [65023.192040] RAX: 0000000000000000 RBX: 00000000000033b8 RCX: 00000000000033b8
Feb 1 10:13:37 co169 kernel: [65023.192041] RDX: 0000000000000006 RSI: 00000000ffb93828 RDI: 00000000f70f8000
Feb 1 10:13:37 co169 kernel: [65023.192043] RBP: 00000000ffb93778 R08: 0000000000000000 R09: 0000000000000000
Feb 1 10:13:37 co169 kernel: [65023.192044] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Feb 1 10:13:37 co169 kernel: [65023.192046] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Feb 1 10:13:37 co169 kernel: [65023.192048] FS: 0000000000000000(0000) GS:ffff88013fa00000(0063) knlGS:00000000f6dae700

Older version of Universal forwarder (6.6) works on the same kernel(4.9). From kernel message with flag P, is this crash related to the way splunk licensing works(assuming this changed between 6.6 and 7.1) ?

0 Karma

sakti
Engager

Version 7.0 crashes as well. Downgrading to 6.6 worked, but would like to get support for 7+ versions.

My system information is:

bash-4.3# uname -a
Linux co169 4.9.108.Ar-10738448.4213F #1 SMP PREEMPT Sat Dec 15 12:30:10 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Agentic SOC Triage: Investigating Splunk ES Notables with MCP Server and a Local LLM

The Problem: Too Many Alerts, Too Little Context Security operations teams running Splunk Enterprise Security ...

All Work and No Play? Not at .conf26! Unwind at These Evening Events

Between hands-on technical sessions, keynote reveals, and diving into live architectures, .conf26 is packed ...

Join the Hackathon at .conf26 and build a No-Code AI agent

Join us for the AI Agent Buildathon, an in-person, three-hour hands-on Hackathon where you’ll use Splunk Agent ...