verify message was received at port using log

[jdepp]

Is there a way to view log files or entries for a specific port where messages should be coming into? The reason I ask is that currently in a dispute with a programmer who insists his program sent messages for a specific thread, yet splunk is unable to find this within the search queries set up for that data source. We have 1500 different threads sending heartbeats to the port but for some reason a couple are not appearing. I have expanded the search over 7 days even and still no results. If there is a log file that I can find that will display all entries for the last 24 hours, then I can verify that this is not an issue with splunk.

Thanks

[DeronJensen]

If you can get to root on the Splunk server, you can use tcpdump or something to see if the packets are at least making it to the server. Most of the time, when someone says they send logs, and no logs are received, this will help. Often there is a firewall or something in the way. Sometimes the IP is being NAT, so it appears to be from a different server.

For example, to see if you are receiving packets on port 9997:
sudo tcpdump port 9997

Or to see all packets from the server 199.99.1.1
sudo tcpdump host 199.99.1.1

Of course you can combine those options to try to minimize the data. Another option is to use the *-w * to create a binary file and then you can pull that off the server and view / analyze the data with wireshark.

[jdepp]

Thanks for your detailed response. It is not that any packets are getting to the port as the data for other stream IDs that are being sent to that port are being indexed in splunk. If the programmer is sending the packet just as he does the others there should be no reason certain packets to the port fail, right? But I will use this to take a look and it may just clarify.

[martin_mueller]

You can rule out timestamping issues by searching like this over all time:

index=foo source=bar _index_earliest=-7d _index_latest=now

As for log files - if you're sending directly to Splunk ports then there will be no log file. You can search index=_internal for errors caused by this source, or for metrics from this source if it is reasonably high volume though.

[jdepp]

Thanks for the response. Newbie with regards to splunk so not sure where to look for inputs.conf. This is the search query I tried now:

source="/fb.activity/tcp/10018" "stream ID"=855 _index_earliest=-7d _index_latest=now

What or where would I find the index name, is that the same as the source?

[martin_mueller]

If you're using all Splunk default settings then the index will be main, and can be left off searches using the default user roles.

If that query returns nothing then nothing was indexed with that source and stream ID in the past seven days, assuming the stream ID is extracted correctly. Try searching for the term 855 without any field name to rule that out - I'm guessing 855 will be a rare term outside of this stream ID.

[jdepp]

sorry could you give me an example of how to search without the field name and just the value?

source="/fb.activity/tcp/10018"  *=855 _index_earliest=-7d _index_latest=now

is this correct? doesn't seem to work.

[martin_mueller]

 source="/fb.activity/tcp/10018"  855 _index_earliest=-7d _index_latest=now

[rphillips_splk]

For the data being monitored, what does the input monitor stanza look like in inputs.conf