Getting Data In

Issues with Ingesting Static File

MikeElliott
Communicator

Hi Team,

I've come across an odd problem, and I'm not sure where to start in troubleshooting.

Once a week, on a Sunday, we ingest a csv file that contains all of our assets (splunk_devices.csv). Recently, we have noticed that there are assets in the asset list that are not present in our asset index.

Last week, for example, our static file contained 28k assets and the data ingested by Splunk only had 24k. I reviewed the list myself and can confirm that the assets were missing from Splunk. I ingested the file into my personal development environment with dev license and had no issues at all - All 28k assets were present and accounted for.

I've found no error messages in Splunk, or any other indicators to start troubleshooting with.

Does anyone have any ideas what we could check? We're using Splunk Cloud, so have no access to indexers or search heads, but can access our forwarder infrastructure.

Tags (3)
0 Karma

rvany
Communicator
  • Which exact command/procedure do you use to ingest the data?
  • Are the missing devices at the beginning/end of your csv file in one big block?
  • You could try to ingest the data into a new (temporary) index on your prod-system if possible?
  • Try the following: export your assets-index into a csv file; load this into a new index on your dev-system; ingest the "splunk_devices.csv" into this index also - does this work?
0 Karma

richgalloway
SplunkTrust
SplunkTrust

What is the exact procedure you are using to ingest the CSV?

---
If this reply helps you, Karma would be appreciated.

MikeElliott
Communicator

Hi Rich,

We have a UF deployed on the asset that generates the list. The list is updated on a daily basis, but only ingested by Splunk on a Sunday morning, around 2am.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What you describe does not sound like typical Splunk processing. Universal forwarders ship data as it is received - they cannot update a list each day and send it to Splunk once a week.
Please provide a detailed explanation of how you get the asset list into Splunk. Without that, we can only speculate about the problem. Include the UF's inputs.conf settings for the CSV and the indexer's props.conf settings for the CSV's sourcetype.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...