I have a lot of reports/dashboards about a particular sourcetype that receives data (from a forwarder) one time per day. Now, my requirements changed and I need to send data many times per day. My optimal solution can be "overwrite" the last data with the upcoming data, but I think that this is impossible (the data old remains inside the indexer). How can I only report on the last indexed data? Any other ideas?
If you know the data will be sent at a specific frequency, you could use that in your search. For example, if the index gets data every hour, your could write your search like this
index=yourindex sourcetype=specficsourcetype earliest=-1h@h | rest of your search here
If the frequency is indeterminate, then you can use metadata to find the last time index recd data, like this
index=yourindex sourcetype=specficsourcetype [| metadata types=sourcetypes index=yourindex | eval earliest=lastTime | table earliest] | rest of your search here.
Hi. Thanks! When i ran:
[| metadata types=sourcetypes index=bucle_cm | eval earliest=lastTime | table earliest] ...
I receive an error:
Error in 'metadata': You must specify a 'type' argument to 'metadata', as in 'type=hosts'.
types=sourcetypes is ok?
Well it'll depend upon how your data is. Does all your data have same timestamp? OR if all events are ingested within a certain period like starts at 11 and finished by 11:30 ?Does the source/filename is different between different time it's received?