On 04/13/2012 05:43 PM, Luc Maisonobe wrote:
OK, Issue 89 is now fixed. As part of the fix, I have changed dates
quantization to use long and set up fixed quantum step to 1 microsecond.
This should avoid overflow, but does not prevent the large number of
slots and even a large number or calls to the generator 1589 in a test I
just run!
So there is definitely something to do to avoid this.
Yes, the other case when slots are (unnecessarily) created is related to
the limiting of the range the generator can provide. Adding this fix as
discussed before should fix it (tested it locally).
While debugging issue 89, it appears preventing extra calls is not
trivial. We must make sure we don't prevent generating new entries
before the first existing one or after the last existing one when there
is still some room for potential new entries before generator end. So we
have to compare the current first (or last) entry with generator
boundaries and then decide what to do.
We must also take care that in the general case, slots will be evicted,
so even if some data at boundary has already been generated, it may have
been evicted and may have to be generated again.
This may happen for UTC-TAI, as I think it would be more efficient to
limit the size of the slots to say 1 year, so the search at deepest
level (in the slot) remains fast.
I have looked into this, and the reason (at least for the UTCScale) that
additional call to the generator are performed is a corner case due to
the neighborSize:
- consider you have 40 entries and a neighborSize of 2
- requesting neighbors for a central date that correspond to the last
entry would require the cache to be further filled, as the request
can not be fulfilled: lastIndex + 2 > cache.size()
- thus the generator is asked for new entries
- the UTCScale Generator can not provide more entries, so there is a
safe-guard in TimeStampedCache.Slot.getNeighbors that reduces
lastIndex to fit the available cache size
The problem is that this check is done at the end of the method, after
requesting the generator. To tackle this, two solution come to my mind:
- do a check at the beginning like this:
// if we request neighbors outside the available limit
// and the last entry is already at the generator bound, adjust
// the index of the first neighbor to return to match the current
// available cache entries without requesting more data, as the
// generator will not be able to provide more data
if (firstNeighbor + neighborsSize > cache.size() &&
getLatest().getDate().compareTo(generator.getLatest()) >= 0) {
// we end up with a non-balanced neighborhood,
// adjust the start point to fit within the cache
firstNeighbor = cache.size() - neighborsSize;
}
similar for the lower bound
- add a method to the generator interface (maybe a bad name, but to
get the idea):
public boolean isSingleShot();
which indicates that the generator is providing all its data in one
shot. The Slot.getNeighbors method does a similar check as before:
// same as above, just do not rely on bounds but use an explicit
// indication from the generator
if (firstNeighbor + neighborsSize > cache.size() &&
generator.isSingleShot()) {
// we end up with a non-balanced neighborhood,
// adjust the start point to fit within the cache
firstNeighbor = cache.size() - neighborsSize;
}
Using this the multi-threading test runs through with only 1 generate call.