Splunk Search

verify message was received at port using log

jdepp
Path Finder

Is there a way to view log files or entries for a specific port where messages should be coming into? The reason I ask is that currently in a dispute with a programmer who insists his program sent messages for a specific thread, yet splunk is unable to find this within the search queries set up for that data source. We have 1500 different threads sending heartbeats to the port but for some reason a couple are not appearing. I have expanded the search over 7 days even and still no results. If there is a log file that I can find that will display all entries for the last 24 hours, then I can verify that this is not an issue with splunk.

Thanks

Tags (1)
0 Karma

DeronJensen
Explorer

If you can get to root on the Splunk server, you can use tcpdump or something to see if the packets are at least making it to the server. Most of the time, when someone says they send logs, and no logs are received, this will help. Often there is a firewall or something in the way. Sometimes the IP is being NAT, so it appears to be from a different server.

For example, to see if you are receiving packets on port 9997:
sudo tcpdump port 9997

Or to see all packets from the server 199.99.1.1
sudo tcpdump host 199.99.1.1

Of course you can combine those options to try to minimize the data. Another option is to use the *-w * to create a binary file and then you can pull that off the server and view / analyze the data with wireshark.

0 Karma

jdepp
Path Finder

Thanks for your detailed response. It is not that any packets are getting to the port as the data for other stream IDs that are being sent to that port are being indexed in splunk. If the programmer is sending the packet just as he does the others there should be no reason certain packets to the port fail, right? But I will use this to take a look and it may just clarify.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

You can rule out timestamping issues by searching like this over all time:

index=foo source=bar _index_earliest=-7d _index_latest=now

As for log files - if you're sending directly to Splunk ports then there will be no log file. You can search index=_internal for errors caused by this source, or for metrics from this source if it is reasonably high volume though.

0 Karma

jdepp
Path Finder

Thanks for the response. Newbie with regards to splunk so not sure where to look for inputs.conf. This is the search query I tried now:

source="/fb.activity/tcp/10018" "stream ID"=855 _index_earliest=-7d _index_latest=now

What or where would I find the index name, is that the same as the source?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If you're using all Splunk default settings then the index will be main, and can be left off searches using the default user roles.

If that query returns nothing then nothing was indexed with that source and stream ID in the past seven days, assuming the stream ID is extracted correctly. Try searching for the term 855 without any field name to rule that out - I'm guessing 855 will be a rare term outside of this stream ID.

0 Karma

jdepp
Path Finder

sorry could you give me an example of how to search without the field name and just the value?

source="/fb.activity/tcp/10018"  *=855 _index_earliest=-7d _index_latest=now

is this correct? doesn't seem to work.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust
 source="/fb.activity/tcp/10018"  855 _index_earliest=-7d _index_latest=now
0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

For the data being monitored, what does the input monitor stanza look like in inputs.conf

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...