These two pieces of SPL return two different-looking tables.          index=servicenow sourcetype=incident number=INC5181781 | spath opened_at | spath resolved_at | table number, opened_at, resolved_at, number, _time         Will provide me with different results vs.         index=servicenow sourcetype=incident number=INC5181781 | table number, opened_at, resolved_at, number, _time         In the one with "spath" the table has more values for those values for "opened_at" and "resolved_at". The same number of events are discovered, but the table makes it look like one event is "missing" dimensions. Even if I do these two search, and compare the "Selected Fields" section on the left hand side, the one with spath has more "events" that have the values.    In the props.conf file the "source" has the line INDEXED_EXTRACTIONS = json This may also be impacting my ability to search as well. It seems like I will not get complete results unless I do something like          sourcetype=incident |spath number |spath category |search number=INC5181781 category=Closed         I assume something is not configured as I expect it to be, and I am unsure where else to check. ... View more

I had a report that would run on the first Thursday of every month. It had been working for months. Recently, it startign running more than I expected based on how the cron was written, and the only thing I can attribute it to at the moment is that we upgraded to 7.3.4 from 6.1.1(I think). The cron is: 0 4 1-8 * 4 What it is doing now is running every day from the 1st to the 8th of every month, and running every Thursday at 4 am. ... View more

I do not understand why, but I appear to have missing data. We have 4 indexers in our cluster and in our server.conf filer replication_factor=4 . Stopped splunkd on everything. We backed up each index, on each indexer (each folder under "/opt/splunk/var/lib/splunk/" on each index got backed up). Performed the necessary maint on the cluster. Took everything we backed up on "old indexer1" and put it back on "new indexer1", started splunkd, and had it replicate across the indexers. But when I search the data from a search head, using the SPL, I know there is stuff missing. Beyond going through each of my archives, finding what buckets are missing, and restore them, what am I missing? I assumed since replication_factor=4, that meant I had duplicate copies of all of my data across all 4 indexers. What am I missing? ... View more

In my data, events can have children. There is data in those events that I would want to associate with the parent event. When I do my base search, sometimes I find events where the parent event is created outside of the global time picker window. The only way I can think to solve this is by I can join a sub-search and have the sub-search be a different time window? My existing search is as follows: "base search with sourcetype=A" | rename display_value as parent_num | join type=outer parent_num [search "second search with sourtype=A" | dedup number sortby -sys_updated_on | rename priority as parent_priority, number as parent_num, u_incident_type as type, | table parent_num, parent_priority, type, parent_assignment] | eval type = if(parent!="*", u_incident_type, type) This works, except when, for example, if I am looking at the past 24 hours, and the "parent" was created before. I am sure there are a number of ways to solve this, but this seems most apparent to me. I don't want to exclude the occasions when I am "missing" the data. ... View more