Getting Data In

Why is the data in index from using collect is not being indexed?

caseyra
Explorer

Hello,

I'm trying to take a list of IDs that were given to me and confirm that they are in our system. I imported the data into a lookup table and then use the collect command to put the data into a temporary index that was created some time ago for other reasons. The data ends up in the index, but the data does not appear to be indexed. For example, if I run a query like this, I get results:

index=temp sourcetype=idlist

However, the following query will bring back 0 results:

| tstats count as total where index=temp by ID

ID is the new field that has the ID that we're looking for. Why is the second query not working?

Some additional information: we're doing this on a stand alone search head connected to a indexer cluster. Originally we tried to create a local index for the IDs so that we could easily delete it afterwards, but it looked like it was expecting the index to be in the cluster and not local, hence why we switched to using the temp index.

Thanks.

0 Karma
1 Solution

tiagofbmm
Influencer

Hey

The reason for that is ID is not a Splunk indexed field by default. Only index, source, sourcetype and _time are.

So tstats, working only on indexed fields, can;t give you the results

View solution in original post

tiagofbmm
Influencer

Hey

The reason for that is ID is not a Splunk indexed field by default. Only index, source, sourcetype and _time are.

So tstats, working only on indexed fields, can;t give you the results

tiagofbmm
Influencer

More, if you did not specify a sourcetype when you did the collect command, the sourcetype is stash, and with KV_MODE=none, you will get no search time extractions.

Only the metadata fields are shown (index, source, sourcetype, host, _time)

0 Karma

caseyra
Explorer

I did specify a source type, but time field isn't what we need.

Can we force Splunk to index the ID field? Or, is that something that needs to be configured when you create the index and before you ingest any data?

0 Karma

tiagofbmm
Influencer

To force Splunk to have a new indexed field, you need to do that on your sourcetype.

Follow this doc to do that:

http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Configureindex-timefieldextraction

As collect indexes data in a summary index, you should take a look here to find out how to configure the summary index to have your fields there:

http://docs.splunk.com/Documentation/Splunk/7.0.2/Knowledge/Configuresummaryindexes

0 Karma

caseyra
Explorer

Thanks. I'll check it all out. Right now, it looks like I have an alternative query running faster than expected, so I think I'm good to go for now.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...