Getting Data In

How to find the path for an unknown data source that is sending data to Splunk?

darknetone
Explorer

How can I tell where data is coming from? I have inherited an old Splunk 5.0.1 Enterprise Infrastructure. I can see data on the Splunk head for a specific (IP) server, however, this data is coming into _main. I got on the Windows box where this data is coming from and I could not see a universal forwarder or syslog implementation despite much searching. I do not know how the data is coming into Splunk, which is a problem since I need the data to go into a different index. This leaves me asking, how is the data coming in? Is there a way to trace events all the way back to the origination point AND know what the path that the data took? I there a way to know what process originated the data on the machine?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

I'd check a few things:

  • timestamps on the events - maybe they're old and you're chasing ghosts?
  • host and source of the events
  • receiving enabled on the indexer
  • search index=_internal source=*metrics.log* group=tcpin_connections for info around incoming forwarder connections
  • inputs enabled on the indexer
  • if source and inputs don't line up, check for props.conf/transforms.conf rewrites (TRANSFORMS-foo in props.conf)
  • search index=_internal source=*metrics.log* thruput for clues where the indexer thinks it has throughput
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

A Four-Part Event Series: Full Stack Observability For the AI Era

As AI reshapes applications, infrastructure, and the way teams operate, the traditional boundaries of ...

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Event Series: Level up your SOC: Advancing with Splunk Enterprise Security

AI has fundamentally raised the stakes for security operations, and this three-part series is your guide to ...