Why am I receiving "splunk-cooked-mode-v3" data from a universal forwarder in the format "\x00\x00\x00..."?

[diewin]

I am receiving data like this from a universal forwarder on Port: 8097:

--splunk-cooked-mode-v3--\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
…
_s2s_capabilities\x00\x00\x00\x00 ack=0;compression=0\x00\x00\x00\x00\x00\x00\x00\x00_raw\x00

Here is my inputs.conf (location: ~/etc/system/local)

cat inputs.conf

[default]
host = ebs-xxx

[splunktcp://8097]
disabled = 0

I receive data only if I configure tcp://8097 and with splunktcp://8097 I see nothing.

What is wrong?

Thanks for some hints!

[vinoth_raj]

Guys, any thoughts on this??

[MuS · Accepted solution]

Hi diewin,

this is the normal behavior if you send cooked data from a forwarder to an indexer over any TCP port.
You should check / verfiy why your splunktcp input is not working correct.
Check the splunkd.log on your indexer for any errors related to this port.
Another thing to remember, did you search all indexes over all time as well?

hope this helps ...

cheers, MuS

[diewin]

splunk@ebs-xxx:~$ netstat -an| grep 8097
tcp 0 0 0.0.0.0:8097 0.0.0.0:* LISTEN
tcp 0 0 172.20.1.50:8097 172.20.1.16:31841 ESTABLISHED

OK. This is the trick: "index=airlock" and search. This works!

Data Summery search does not pick up this index automatically as a search hint.

Guess: Works as designed?
[tcp://8097] shows the hints and [splunktcp://8097] not?

Thanks for your Help!!

[MuS]

you're welcome, feel free to accept this answer 🙂

[diewin]

But I cannot see any messages in the search menu or on the port 8097.
Where does it go?

[MuS]

you said: In the settings menu -> indexes: I can see a slow event count increase for the airlock index.

did you do an all-time search on index=airlock? Do you have the permission to search this index?
Do a tcpdump on the indexer for port 8097, do you get any connections?

[diewin]

This is the service on the forwarder:

root@ebs-v-xxxx:local[400] # netstat -an |grep 8089
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN

I can't connect from the indexer to this port (firewall). Is it needed?
I assume that the forwarder needs to connect to the indexer only and not vice versa.

In the settings menu -> indexes: I can see a slow event count increase for the airlock index.

[MuS]

Indexer does not need to connect to the forwarder. If you can see event count increase on the airlock index you're getting events, this means it is working.

[diewin]

metrics.log looks ok?

Telnet test from the source:

root@ebs-v-airlock03a:~[371] # date; telnet 172.20.1.50 8097
**Tue Jan  6 12:07:18 CET 2015**
Trying 172.20.1.50...
Connected to 172.20.1.50.
Escape character is '^]'.
test
Connection closed by foreign host.

And the metrics.log:

01-06-2015 12:07:18.735 +0100 INFO  StatusMgr - destPort=8097, eventType=connect_done, group=tcpin_connections, sourceHost=172.20.1.16, sourceIp=172.20.1.16, sourcePort=36903, statusee=TcpInputProcessor
01-06-2015 12:07:18.735 +0100 INFO  StatusMgr - sourcePort=8097, ssl=false, statusee=TcpInputProcessor
01-06-2015 12:07:21.363 +0100 INFO  StatusMgr - destPort=8097, eventType=connect_close, group=tcpin_connections, sourceHost=172.20.1.16, sourceIp=172.20.1.16, sourcePort=36903, statusee=TcpInputProcessor

[diewin]

And created an index: (location: /opt/splunk/etc/system/default/indexes.conf)

...
[airlock]
homePath = $SPLUNK_DB/airlockdb/db
coldPath = $SPLUNK_DB/airlockdb/colddb
thawedPath = $SPLUNK_DB/airlockdb/thaweddb
tstatsHomePath = volume:_splunk_summaries/airlockdb/datamodel_summary
disabled = false
...

[MuS]

so your index=airlock is still empty? check splunkd.log on the forwarder for any TailingProcessor message or use the REST API https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus on the forwarder

[diewin]

Is this what you mean? Then i did.

root@ebs-v-xxxx:local[381] # pwd
/opt/splunkforwarder/etc/system/local
root@ebs-v-xxxx:local[382] # cat inputs.conf

[default]
host = ebs-v-xxxx

[monitor:///data/log/airlock/Web-Requests/*.Access]
index = airlock
sourcetype = airlock:web:access

[monitor:///data/log/airlock/Web-Requests/*.Authentication]
index = airlock
sourcetype = airlock:web:authentication

[monitor:///data/log/airlock/Web-Requests/*.Security]
index = airlock
sourcetype = airlock:web:security

[monitor:///data/log/airlock/Web-Requests/*.Usage]
index = airlock
sourcetype = airlock:web:usage

[monitor:///data/log/airlock/System/*.Configuration]
index = airlock
sourcetype = airlock:sys:configuration

[monitor:///data/log/airlock/System/*.default]
index = airlock
sourcetype = airlock:sys:default

[monitor:///data/log/airlock/System/*.Health]
index = airlock
sourcetype = airlock:sys:health

[monitor:///data/log/airlock/System/*.Network]
index = airlock
sourcetype = airlock:sys:network

[monitor:///data/log/airlock/System/*.Security]
index = airlock
sourcetype = airlock:sys:security

[MuS]

Yes, looks good as well....so, did you add any inputs on the forwarder?

[diewin]

Looks ok to me?

splunk@xxxx:~/var/log/splunk$ grep 8097 splunkd.log
01-06-2015 09:54:18.694 +0100 INFO  TcpInputProc - Stopping IPv4 port 8097
01-06-2015 09:54:51.950 +0100 INFO  TcpInputConfig - IPv4 port 8097 is reserved for splunk 2 splunk
01-06-2015 09:54:51.950 +0100 INFO  TcpInputConfig - IPv4 port 8097 will negotiate new-s2s protocol
01-06-2015 10:11:25.311 +0100 INFO  TcpInputProc - Stopping IPv4 port 8097
01-06-2015 10:11:49.546 +0100 INFO  TcpInputConfig - IPv4 port 8097 is reserved for splunk 2 splunk
01-06-2015 10:11:49.546 +0100 INFO  TcpInputConfig - IPv4 port 8097 will negotiate new-s2s protocol

[MuS]

Yes, looks like a splunktcp or a splunk 2 splunk port. Can you telnet to this port from the forwarder? Check your metrics.log if you see any StatusMgr from remote IP's