Missing fields in Splunk that were previously there

[Stefanie]

Hey all,

I have the Splunk add on for unix/linux deployed to about ~70 servers. All was working fine (and has been for years!) up until yesterday.

I'm receiving data into my os index (which is where those logs are stored) but after searching for anything beyond index, host, sourcetype, it does not work.

For example, for a search of 7 days ago I can search for something like:  index=os sourcetype=df host="server1" OR host="server2" | stats max(PercentUsedSpace) as PercentUsed by host,filesystem | sort - PercentUsed | where PercentUsed >=75

It will pull data from 7 days ago up until yesterday.

Searching data for yesterday to now gives me no data. 

If I search index=os host="server1" OR host="server2", I'm receiving logs as normal. The other sources and sourcetypes are there.

So i guess my question is, what happened to my "PercentUsedSpace"? It doesnt show in the interesting fields portion. I can't search for it. It returns blank.

My search for index=os source=df host="server1" OR host="server2" shows my logs. But I can't refine it down further. 

Edit: Now what is interesting in my logs, every now and then, I see that I am receiving a log that is something along the lines of " CPU pctUser pctNice pctSystem pctIowait pctIdle" , "Name rxPackets_PS txPackets_PS rxKB_PS txKB_PS", "memTotalMB memFreeMB memUsedMB memFreePct memUsedPct pgPageOut swapUsedPct pgSwapOut cSwitches interrupts forks processes threads loadAvg1mi waitThreads interrupts_PS pgPageIn_PS pgPageOut_PS"

So it seems that instead of parsing each field as type of field, it is parsing as a log. 

Please assist!

[Stefanie · Accepted solution]

This was resolved. Just updating my post in case someone else in the future has the same issue.

There was a props.conf that was implemented because of Splunk Professional Services' guidance.

Removing the props.conf resolved the issue.