As explained on the filtering page, the FiltersManager.applyRelevantFilters method works by looping over all registered DataFilter instances and calling their filter method with the current DataSource. If the filter method returns the exact same instance that was passed to it, it means the filter does not do anything. In this case, the FiltersManager.applyRelevantFilters method just continues its loop and check the next filter. If the filter method returns a different DataSource instance that was passed to it, it means the filter does indeed act on the data stream. In this case, the FiltersManager.applyRelevantFilters method sets the current DataSource to the returned value and restart its loop from the beginning.
This algorithm allows the same filter to be applied several time if needed, and it also allows the filters to be applied in any order, regardless of the order in which they have been registered to the FiltersManager.
Users may benefit from this general feature to add their own filters. One example could be a deciphering algorithm if sensitive data should be stored enciphered and should be deciphered on the fly when data is loaded.
As per the way the applyRelevantFilters method works, the filter method must be implemented in such a way that it should check the DataSource passed to it and return its parameter if it considers it should not filter it, or return a new DataSource if it considers is should filter it.
There is one important caveat to understand when implementing custom filter: filters must never open the DataSource by themselves, regardless of the fact they will return the original instance or a filtered instance when their filter method is called. The rationale is that it is the upper layer that will decide to open (or not) the returned value and that a DataSource can be opened only once; this is the core principle of lazy-opening provided by DataSource.
A consequence of this caveat is that a filter cannot peek on the few bytes of the data stream that is referenced by a DataSource, for example in an attempt to look for a magic number in a header. This is the reason why for example the GzipFilter looks for a .gz suffix in the name and does not look for the 0x1f8B magic number at file start.
As applyRelevantFilters restarts its loop from the beginning each time a filter is added to the stack, some care must be taken to avoid stacking an infinite number of instances of the same filter on top of each other. This means that the filtered DataSource returned after filtering should be recognized as already filtered and not matched again by the same filter. If the check is based on file names extensions (like .gz for gzip-compressed files), then if the original DataSource has a name of the form base.ext.gz than the filtered file should have a name of the form base.ext. Another point is that if a filters does not act on a DataSource, then it must return the same instance that was passed to it, it must not simply create a transparent filter that just passes names and data stream unchanged, otherwise it would be considered as a valid filter and added again and again until either a stack overflow or memory exhaustion exception occurs.
The filtering part itself is implemented by opening the data stream from the underlying original DataSource, reading raw data from it, performing the processing on these data (uncompressing, deciphering, …) and returning them as another stream.
The following example shows how to do that for a dummy deciphering algorithm based on a simple XOR (this is a toy example only, not intended to be secure at all).
public class XorFilter implements DataFilter {
/** Suffix for XOR ciphered files. */
private static final String SUFFIX = ".xor";
/** Highly secret key. */
private static final int key = 0x3b;
/** {@inheritDoc} */
@Override
public DataSource filter(final DataSource original) {
final String oName = original.getName();
final DataSource.Opener oOpener = original.getOpener();
if (oName.endsWith(SUFFIX)) {
final String fName = oName.substring(0, oName.length() - SUFFIX.length());
final DataSource.StreamOpener fOpener = () -> new XORInputStream(oName, oOpener.openStreamOnce());
return new DataSource(fName, fOpener);
} else {
return original;
}
}
/** Filtering of XOR ciphered stream. */
private static class XORInputStream extends InputStream {
/** File name. */
private final String name;
/** Underlying compressed stream. */
private final InputStream input;
/** Indicator for end of input. */
private boolean endOfInput;
/** Simple constructor.
* @param name file name
* @param input underlying compressed stream
*/
XORInputStream(final String name, final InputStream input) {
this.name = name;
this.input = input;
this.endOfInput = false;
}
/** {@inheritDoc} */
@Override
public int read() throws IOException {
if (endOfInput) {
// we have reached end of data
return -1;
}
final int raw = input.read();
if (raw < 0) {
endOfInput = true;
return -1;
} else {
return raw ^ key;
}
}
}
}