Splunk Enterprise Security

Filtering on _indextime in data model tstats search - field appears to be missing from some data?

stroud_bc
Path Finder

I have an alert which uses a tstats accelerated data model search to look for various types of suspicious logins. Since some of our Authentication log sources are in the cloud, logs are ingested in batches, sometimes with several hours of delay. As a result, we schedule this to run hourly with a 24h window (based on event time: _time) but filter the search on _indextime as shown below:

| tstats count(Authentication.app) from datamodel=Authentication by Authentication.app, Authentication.dest, Authentication.src, Authentication.src_user, Authentication.user, sourcetype, _indextime
| fields - count(Authentication.app) 
| sort Authentication.app 
| rename Authentication.* as * 
| where _indextime > relative_time(now(), "-65m@m")

The search shown above works for some of my Authentication data, but not all. It seems like the _indextime field isn't available for all of my data. To illustrate this, compare the below searches with and without "_indextime" in the |tstats ... by clause (in both cases omitting the where clause.)

    | tstats count(Authentication.app) from datamodel=Authentication by Authentication.app, Authentication.dest, Authentication.src, Authentication.src_user, Authentication.user, sourcetype, _indextime
    | fields - count(Authentication.app) 
    | sort Authentication.app 
    | rename Authentication.* as * 

This results in ~250 distinct events over a given time period

    | tstats count(Authentication.app) from datamodel=Authentication by Authentication.app, Authentication.dest, Authentication.src, Authentication.src_user, Authentication.user, sourcetype
    | fields - count(Authentication.app) 
    | sort Authentication.app 
    | rename Authentication.* as *

While this search results in ~2000 distinct events over the same time period.

When comparing the sourcetypes involved in each of these two searches it seems that some sourcetypes ONLY show up in the second search, while some simply have fewer results. Is _indextime considered to be an "indexed" field when it comes to tstats? Is this something I need to configure for my data inputs?

If the above is a non-starter, I would appreciate insight on alternative approaches to the same end!

1 Solution

stroud_bc
Path Finder

I was able to solve this by manually adding _indextime to the data model as an eval field. For anyone else with this problem, my solution is below...

Steps:

  1. Deaccelerate DM
  2. Edit Datasets
  3. Add Field
  4. Eval Expression
  5. Make Eval Expression "_indextime" and both Field Name and Display Name "indextime"; set Type="Number", and leave Flags="Optional" and save
  6. Reaccelerate DM

The two searches below should return the same results (might be off by just a few events since relative times are used.) Note that since the field is part of the data model, it must be referenced as Authentication.indextime instead of just _indextime as before. Additionally you cannot use conditions like "_index_earliest" and "_index_latest" the same way, but must manually filter on the value further on in the search.

With |tstats

| tstats count from datamodel=Authentication by Authentication.user, Authentication.action, Authentication.indextime, _time, sourcetype span=1m
| rename Authentication.* as * 
| where indextime>relative_time(now(), "-65m") 

Without |tstats

| from datamodel:Authentication | bucket _time span=1m
| stats count by user, action, _time, indextime sourcetype 
| where indextime>relative_time(now(), "-65m")

View solution in original post

stroud_bc
Path Finder

I was able to solve this by manually adding _indextime to the data model as an eval field. For anyone else with this problem, my solution is below...

Steps:

  1. Deaccelerate DM
  2. Edit Datasets
  3. Add Field
  4. Eval Expression
  5. Make Eval Expression "_indextime" and both Field Name and Display Name "indextime"; set Type="Number", and leave Flags="Optional" and save
  6. Reaccelerate DM

The two searches below should return the same results (might be off by just a few events since relative times are used.) Note that since the field is part of the data model, it must be referenced as Authentication.indextime instead of just _indextime as before. Additionally you cannot use conditions like "_index_earliest" and "_index_latest" the same way, but must manually filter on the value further on in the search.

With |tstats

| tstats count from datamodel=Authentication by Authentication.user, Authentication.action, Authentication.indextime, _time, sourcetype span=1m
| rename Authentication.* as * 
| where indextime>relative_time(now(), "-65m") 

Without |tstats

| from datamodel:Authentication | bucket _time span=1m
| stats count by user, action, _time, indextime sourcetype 
| where indextime>relative_time(now(), "-65m")

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Until Splunk 8, DMAs cannot be shared among search heads. Only the originating search head can use the acceleration. If you are running 8.0, see https://docs.splunk.com/Documentation/Splunk/8.0.1/Knowledge/Sharedatamodelsummaries

---
If this reply helps you, an upvote would be appreciated.
0 Karma

stroud_bc
Path Finder

I don't think the DMA is to blame. It works on the search head with the same Data Model but with no acceleration, but the search head which DOES have a DMA is not functioning as expected.

0 Karma

stroud_bc
Path Finder

Thanks for the reply. I guess I now AM blaming the DMA, but not because it is not shared-- only because it seems to be excluding the _indextime field.

0 Karma

stroud_bc
Path Finder

I believe this problem may be related to Data Model Acceleration? In our Splunk environment, we have two (non-clustered) search heads directed at the same indexer. One has a number of CIM data models accelerated (including Authentication; for use in ES), and the other does not. I have determined that the search above (the first one) DOES work properly on the other search head. Unfortunately, I need it to work in the case of the accelerated data model.

I have read that _indextime is an internal meta field. Is it implicitly "accessible" to an accelerated data model tstats search, or do I need to explicitly add it to the data model?

0 Karma

stroud_bc
Path Finder

OK, I think I have isolated the problem to the Data Model Acceleration. When removing data model acceleration on the original search head, the search works properly (returning all expected results.) When reaccelerating the data model, it goes back to returning minimal or zero results.

Is _indextime normally available in the tsidx files? I have read other answers on this site that seem to imply it is (e.g., hxxps://answers.splunk.com/answers/540344/how-to-compute-indextime-time-difference-average-w.html)

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!