Forward all events/syslog from Splunk 7.2.1 to ESM McAfee SIEM or other 3-party (splunk guide not working)

[evil_security]

Hi,

I've downloaded Splunk 7.2.1 deb package, installed it on the linux machine, add a data source (the server that will generate syslog and sand it to splunk), and up to here everything works.
Now i need to forward / redirect / pass ALL the syslog event received to a third party SIEM (in my case it's a McAfee ESM 10.3.2).

I've read the guide about "Forward data to third-party systems" -> Splunk/7.2.1/Forwarding/Forwarddatatothird-partysystemsd

and from the WebGUI of splunk i've done:
-> setting -> forwarder -> configure forwarder -> add a new forwarder

(the URL is: /it-IT/manager/launcher/data/outputs/tcp/server)

From the CLI i have this file:

root@splunkTest:~# more /opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 172.20.3.97:514

[tcpout-server://172.20.3.97:514]
root@splunkTest:~#

And it's not working.

I can see the comunication with tcpdump:

root@splunkTest:~# tcpdump -i any port 514 -nns 1514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 1514 bytes
16:49:43.201683 IP 10.162.128.37.42824 > 172.20.3.97.514: Flags [F.], seq 2946406031, ack 629729605, win 229, length 0
16:49:43.202493 IP 10.162.128.37.42832 > 172.20.3.97.514: Flags [S], seq 3373985087, win 29200, options [mss 1460,sackOK,TS val 938692088 ecr 0,nop,wscale 7], length 0
16:49:43.223613 IP 172.20.3.97.514 > 10.162.128.37.42832: Flags [S.], seq 1511232163, ack 3373985088, win 29200, options [mss 1460,nop,wscale 10], length 0
16:49:43.223657 IP 10.162.128.37.42832 > 172.20.3.97.514: Flags [.], ack 1, win 229, length 0
16:49:43.224001 IP 10.162.128.37.42832 > 172.20.3.97.514: Flags [P.], seq 1:401, ack 1, win 229, length 400
16:49:43.246353 IP 172.20.3.97.514 > 10.162.128.37.42832: Flags [.], ack 401, win 30, length 0
16:49:43.246401 IP 10.162.128.37.42832 > 172.20.3.97.514: Flags [P.], seq 401:469, ack 1, win 229, length 68
16:49:43.263551 IP 172.20.3.97.514 > 10.162.128.37.42824: Flags [.], ack 1, win 30, length 0
16:49:43.267205 IP 172.20.3.97.514 > 10.162.128.37.42832: Flags [.], ack 469, win 30, length 0
16:49:43.279484 IP 172.20.3.97.514 > 10.162.128.37.42824: Flags [F.], seq 1, ack 1, win 30, length 0
16:49:43.279565 IP 10.162.128.37.42824 > 172.20.3.97.514: Flags [.], ack 2, win 229, length 0
^C
11 packets captured
11 packets received by filter
0 packets dropped by kernel
root@splunkTest:~#

and i've this error in the Splunk GUI: "TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow."

From what i guess i don't have a heavy-forwarder (am i right?) i deployed only the splunk package and started it (which architecture is the default? HF, indexer, forwarder, etc?).

What i'm missing?

[sumitkathpal]

@evil_security @harsmarvania57 quick question what about non syslog devices like windows or qualys. How Mcafee SIEM will parse the logs by forwarding all logs via syslog from Splunk to Mcafee ESA?
I understand from Splunk we can forward the logs via syslogs or there is add-on called CEF format forwarder but we need to forward all sourcetype of logs from Splunk to Mcafee SIEM (mcafee should able to parse the data)...

Any direction will help. Thanks 🙂

[evil_security]

Hi,

I do not know why but it stopped working, it's not sending any data on port 514, i can see al the syslog that the client is sending to splunk (they are indexed on splunk), but from tcpdump point of view i cannot see anymore the forward to ESM.

i've changed the config to

root@splunkTest:~# more /opt/splunk/etc/system/local/outputs.conf
[syslog]
defaultGroup=syslogGroup

[syslog:syslogGroup]
server = 10.0.0.3:514
root@splunkTest:~#

but i can't see any traffic to 10.0.0.3.
From what i suspect, when it was working, the data that was sending to ESM wasn't my echo "hello" syslog message but some strange "SYSLOG user.notice".

From the GUI i cannot see anymore under forwardreceive the device used for forward message. But it's configured under outputs.conf.

What i'm missing?
maybe the free license doesn't allow to forward the message to third party?

Regards,

Stefano

[evil_security]

the issue was that splunk need to check that the forwarded message destination is alive, even with udp, so it check if it can reach the destination. (i see that in the metric log file).

Now that i fixed a fw issue about this i can see all my events forwarded to the syslog-ng and McAfee ESM.

[harsmarvania57 · Accepted solution]

Hi @evil_security,

To send data to 3rd party, can you please let us know whether McAfee ESM 10.3.2 require data in Syslog format ? On which protocol it accept the data (TCP or UDP).

If McAfee ESM require data in syslog format & receive data on UDP then outputs.conf should be like this

[syslog]
defaultGroup=syslogGroup

[syslog:syslogGroup]
server = 10.1.1.197:514

If it require data in syslog format but receive data on TCP then outputs.conf should be like this

[syslog]
defaultGroup=syslogGroup

[syslog:syslogGroup]
server = 10.1.1.197:514
type = tcp

If McAfee receive data on TCP but it is not require in syslog format then you need to send uncooked data from Splunk Enterprise Instance.

So outputs.conf will be

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 172.20.3.97:514
sendCookedData = false

Regarding Heavy Forwarder, Indexer question. There is no separate package for Heavy Forwarder or Indexer and same Splunk Enterprise package work for Heavy Forwarder and Indexer. Only difference is Heavy Forwarder parse data and send it to Indexer to store that data, If you don't have any Heavy Forwarder then Indexer will parse data and store that data. Now it is purely depend on case to case basis and company infrastructure whether Heavy Forwarder is require or not.

When you install Splunk package (Which is Enterprise package) it works as Indexer, Search Head everything. You need to separate functionality based on Splunk Architecture you'll be planning/implementing.

Additionally you need to consider that if Universal Forwarder (Universal Forwarder is lightweight package and sits on most end server on which you want to monitor logs) can achieve the task then avoid Heavy Forwarders. Ref. blog post https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html

[evil_security]

Hi @harsmarvania57,

McAfee accept connection on port 514 upd/tcp in syslog format. After we receive the syslog we can build a custom parser for parse the logs (if they are in Splunk Indexed format)
I've tried the UDP config and i can see from splunk my syslog events, and all events are forwarded as "SYSLOG user.notice", so it's working.

If i use the uncooked config file i cannot see any events on my splunk dashboard (from tcpdump i see that syslogs are arriving on splunk, but they are not elaborated or indexing to splunk), and a lots of message are forwarded to my McAfee siem, but they are only garbage.

Thanks for the clarification for Regarding HV & Indexer, excellent explanation.

Thanks & Regards !

Stefano

[harsmarvania57]

Welcome @evil_security

Thanks,
Harshil