Something strange is going on. I have fields extracted via regex in transforms.conf that have been working fine for over a year. Today I was searching on one of them and it started returning incorrect results. For example, if I search for
field1=AA
over the previous 24 hours, Splunk returns 6 results. I know that there were several hundred events with field1=AA today alone, so something is off. So to prove this, I searched for a string that is in the same events where field1=AA would be true. This time, several hundred events show up and, if I click on field1 in the fields sidebar and look at the statistics they are correct, showing a count of several hundred for the value AA. I try my basic search again and it returns only 6 events.
I tried some other popular values for field1 and found mixed results. One value that should return several thousand results only returned 10. Another one returned the correct results.
I have verified the field extraction, it remains unchanged from when it was first defined over a year ago. Props.conf is also unchanged. This search worked up until around 3:30 pm today, then it just started giving back wrong results. I tried restarting Splunk but nothing changed, and I even tried cleaning and rebuilding my index, but the problem persists. Any ideas, 'O great and noble Splunkers?
EDIT: Tried creating a new field with the same extraction regex as field1 but it shows the same problems.
You have 2 options: Change from using search-time extraction (REPORT-) to index-time extraction (TRANSFORMS-) or add this to fields.conf:
[field1]
INDEXED_VALUE = false
How many of the results you do you get for
field1=A*
Is there a possibility of other hidden characters in the field which are present, but not displayed?
What do you get if you try charting a count?
field1=A* | chart count(field1) by field1
That may give you some indication if what is going on.