Using extracted fields

[balcv]

I have a data source that I have done a manual regex field extraction and all works fine.  Fields are correct and parsing the data as expected.

A manual search is returning the result I expect when run in verbose mode however, if I run in Fast or Smart modes, the results I get back seem to be approximately 2 hours behind the current data shown in Verbose mode.  All the fields are there and I'm getting data back, it's just old data.

The search is:

index=main host="my.host.name" sourcetype="ProcessedUser*" 
| fields _time,timeStamp,action,src_ip,src_mac,user
| table timeStamp,action,user,src_ip,src_mac
| head 5

While running in a normal search is not a problem as I can switch search modes, however when running in a dashboard is a problem as it does not use Verbose mode.

Any suggestions greatly appreciated.

[inventsekar]

ok, something missing.. @balcv .. you used "rex" or "regex" for manual field extraction? (i think rex only.. regex can not be used for field extraction). 

and on your search you are not using the rex query?!?! am i missing something?!?!

and, more importantly, the smart/verbose/fast modes are nothing but the same. 

verbose - will fetch the logs, while fast mode only fetch you the statistic info tables(it wont bring the logs at the first tab, it only gets you the stats tables, thats why its bit faster and called as it is). the smart mode switches from fast to verbose smartly, nothing much. 

soo, the verbose and fast mode results are nothing but the same. 

and the "Real time" timepicker is for viewing the logs at "Real time".. so, you will always see the latest logs, flowing logs(similar to the movie "the matrix", green color logs, going from top to bottom 😉 )

you are missing some more info. probably everything is right..if you give us some more details, we can resolve this issue. 

(PS - i have given around 700+ karma points so far, received badge for that, if an answer helped you, pls give a karma point!. we all should start "Learn, Give Back, Have Fun")

[balcv]

Hi @inventsekar . In terms of the field extraction, the process I used was to view the original data in a search, expanded one of the rows of data, selected Extract fields from the Event Action dropdown.  I then selected the Regular Expression as the Extract Field method then proceeded to highlight each field and provide a name then saved.

This then makes the required fields available as per my search string previously supplied.

Does that make things any different?

[inventsekar]

Does that make things any different? ///

actually, nope. 

Lets start from basics...

Splunk can extract fields in 2 big broad stages: 
1. at index time (at HF/indexer, you can use props and transforms.conf to make field extractions) 
2. at search time (the method what you followed)

both got some plus and minuses. 
(EDIT)Conversely, as a general rule, it is better to perform most knowledge-building activities, such as field extraction, at search time. 

https://docs.splunk.com/Documentation/Splunk/8.0.6/Indexer/Indextimeversussearchtime#At_index_time

basics documentation:

https://docs.splunk.com/Documentation/Splunk/8.0.6/Knowledge/WhenSplunkEnterpriseaddsfields

so, the search time field extraction is comparatively better than index time, 

the method which you followed, is the "best process " method only. let us know if you have any more questions. 

[balcv]

Extra Info:  I have just discovered if I set the time period to "Real Time" the data current data is shown.  Even in the dashboard.  But if I use any other time period, I get old data.