Here's my issue. I'm trying to search data where a single event appears as below. When I use the search:
index=*mobile Action_Name=Page_View user_id="1159314c-25c6-11e4-aa0c-069a72358463"
I am able to see my data, but when I do the search
index=*mobile Action_Name=Page_View | search user_id="1159314c-25c6-11e4-aa0c-069a72358463"
I do not see my data.
Has anyone experienced this before? This has a lot of impact on my search as generally my search would be able to aggregate the data by OS counting the distinct user_id's, and this problem causes a few user_id's not to be counted.
2015-10-18T08:48:32-07:00 Category="Lifecycle" Action_Name="Page_View" Current_Page="ParkingListPanel" vn_app_version="2.3.63" device_ip="192.168.137.49" OS="Android" OS_version="2.3.4" user_id="1159314c-25c6-11e4-aa0c-069a72358463" location="" connection_type="wifi" battery_level=0.890000 page_name="ParkingDetailPanel" previous_page_name="ParkingListPanel"
search | search is a terrible pattern, including all your filters in the initial search gives you not only accurate results but also great performance.
If you have statistics over all data you want to add to detailed searches, consider storing them in a lookup and adding this to your detailed searches as needed. Schedule a search to update the lookup, for example once a day depending on your data. Then you won't have to go over your entire data set again and again, but still get information from your entire data set at ludicrous speed.
Understand this is a terrible idea, but would like to know what is the reason why this would produce inaccurate data. The actual search that I'm running, which this problem is occurring is this:
index=*mobile Action_Name=Page_View |stats count by user_id
But when i do this, the user_id "1159314c-25c6-11e4-aa0c-069a72358463" is missing from the results. Then when i run
index=*mobile Action_Name=Page_View user_id="1159314c-25c6-11e4-aa0c-069a72358463" |stats count by user_id
I DO receive the result. That's the real issue.
That being said, I tried to reproduce the issue with data everyone has in their Splunk. Compare these two searches:
index=_audit TERM(action=splunkStarting) index=_audit | search TERM(action=splunkStarting)
If the second
index=_audit only yielded 10k events to the following
search I should see no events at all, but I see events a year back - for me on Enterprise 6.3.0 standalone at least.
Base setting to return only 10k results could be limiting you. So the primary search brings back 10k results, stops, then then subsearches that and finds no match.
Meanwhile in the first search attempt, you are bringing out only items with the qualifying uid, thus staying under the 10k limit (limit on search results, not events searched).
I have had more than 10K results returned before, but i tried narrowing down the time range i was looking at and the events started to show up, which is GREAT.
But, (and this is an odd 'but'), when i set the date range back to what i originally had, the results were now showing the correct number. It was as if splunk finally recognized these events when i narrowed in, so when i now have larger ranges it continues to recognize them.
Any thoughts on this?