Why is Splunk suddenly returning incorrect results with unchanged field extraction?

[wpreston]

Something strange is going on. I have fields extracted via regex in transforms.conf that have been working fine for over a year. Today I was searching on one of them and it started returning incorrect results. For example, if I search for

field1=AA

over the previous 24 hours, Splunk returns 6 results. I know that there were several hundred events with field1=AA today alone, so something is off. So to prove this, I searched for a string that is in the same events where field1=AA would be true. This time, several hundred events show up and, if I click on field1 in the fields sidebar and look at the statistics they are correct, showing a count of several hundred for the value AA. I try my basic search again and it returns only 6 events.

I tried some other popular values for field1 and found mixed results. One value that should return several thousand results only returned 10. Another one returned the correct results.

I have verified the field extraction, it remains unchanged from when it was first defined over a year ago. Props.conf is also unchanged. This search worked up until around 3:30 pm today, then it just started giving back wrong results. I tried restarting Splunk but nothing changed, and I even tried cleaning and rebuilding my index, but the problem persists. Any ideas, 'O great and noble Splunkers?

EDIT: Tried creating a new field with the same extraction regex as field1 but it shows the same problems.

[woodcock]

You have 2 options: Change from using search-time extraction (REPORT-) to index-time extraction (TRANSFORMS-) or add this to fields.conf:

[field1]

INDEXED_VALUE = false

[grijhwani]

How many of the results you do you get for

field1=A*

Is there a possibility of other hidden characters in the field which are present, but not displayed?

What do you get if you try charting a count?

field1=A* | chart count(field1) by field1

That may give you some indication if what is going on.