Getting Data In

Sourcetypes keep on multiplying.

marquiselee
Path Finder

Anyone know how I can remove the excess sourcetypes and prevent this from happening in the future?



alt text

Tags (1)
1 Solution

sideview
SplunkTrust
SplunkTrust

This is a by-product of how Splunk indexes CSV files when you give the file a sourcetype of "csv". It uses a Splunk feature called "check-for-header", where it checks the header row of the csv file, and then creates config for the data input such that at search-time the fields will all get extracted and named correctly.

The problem is that the way it does this, is although you give it the sourcetype "csv", it ends up with the sourcetype of "csv-N". the reason is that two unrelated CSV files being indexed will in general have different header rows, and when that happens, one will get assigned sourcetype of "csv-7", and the next one "csv-8".

If and when it comes across a csv file that matches a header row it's already assigned a number, it'll of course assign that pre-existing sourcetype to the data.

But anyway, it looks like on your instance you've indexed quite a lot of mutually distinct CSV files, so there's a big pile o sourcetypes.

I'm afraid as far as preventing the proliferation of "csv-N" sourcetypes, the answer is to just not use the "csv" sourcetype at all. You can manually configure the "AutoHeader" rules yourself, and once you get the hang of it, there's not much more to it than pasting the header row plus a little config into props.conf and transforms.conf.

There are lots of other less-than-ideal things about check-for-header indexing. For one thing it doesn't work with any kind of forwarding - the "csv-N" config ends up stranded on the forwarder where it does no good. And for another problem, if you actually assign one of the "csv-N" sourcetypes to a data input, you'll get the wrong field names (the ones from the previously indexed CSV-N file. You have to remember to assign it the "csv" sourcetype, and resign yourself to Splunk possibly autogenerating a "csv-N" for it. It's deeply unsatisfying.

View solution in original post

sideview
SplunkTrust
SplunkTrust

This is a by-product of how Splunk indexes CSV files when you give the file a sourcetype of "csv". It uses a Splunk feature called "check-for-header", where it checks the header row of the csv file, and then creates config for the data input such that at search-time the fields will all get extracted and named correctly.

The problem is that the way it does this, is although you give it the sourcetype "csv", it ends up with the sourcetype of "csv-N". the reason is that two unrelated CSV files being indexed will in general have different header rows, and when that happens, one will get assigned sourcetype of "csv-7", and the next one "csv-8".

If and when it comes across a csv file that matches a header row it's already assigned a number, it'll of course assign that pre-existing sourcetype to the data.

But anyway, it looks like on your instance you've indexed quite a lot of mutually distinct CSV files, so there's a big pile o sourcetypes.

I'm afraid as far as preventing the proliferation of "csv-N" sourcetypes, the answer is to just not use the "csv" sourcetype at all. You can manually configure the "AutoHeader" rules yourself, and once you get the hang of it, there's not much more to it than pasting the header row plus a little config into props.conf and transforms.conf.

There are lots of other less-than-ideal things about check-for-header indexing. For one thing it doesn't work with any kind of forwarding - the "csv-N" config ends up stranded on the forwarder where it does no good. And for another problem, if you actually assign one of the "csv-N" sourcetypes to a data input, you'll get the wrong field names (the ones from the previously indexed CSV-N file. You have to remember to assign it the "csv" sourcetype, and resign yourself to Splunk possibly autogenerating a "csv-N" for it. It's deeply unsatisfying.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Quantify Your Splunk Investment Impact: Introducing Savings Metrics to Value Insights

Building on the foundation established in our initial Value Insights releases, we are introducing the Savings ...

Event Series: Telemetry Pipeline Management

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...