Counting Bloom filter

A counting Bloom filter is a probabilistic data structure that is used to test whether the number of occurrences of a given element in a sequence exceeds a given threshold. As a generalized form of the Bloom filter, false positive matches are possible, but false negatives are not – in other words, a query returns either "possibly bigger or equal than the threshold" or "definitely smaller than the threshold".

Algorithm description

Most of the parameters are defined same with Bloom filter, such as m, k. m is the number of counters in counting Bloom filter, which is expansion of m bits in Bloom filter. An empty counting Bloom filter is a m counters, all set to 0. Similar to Bloom filter, there must also be k different hash functions defined, each of which maps or hashes some set element to one of the m counter array positions, generating a uniform random distribution. It is also similar that k is a constant, much smaller than m, which is proportional to the number of elements to be added.

The main generalization of Bloom filter is adding an element. To add an element, feed it to each of the k hash functions to get k array positions and increment the counters 1 at all these positions.

To query for an element with a threshold θ (test whether the count number of an element is smaller than θ), feed it to each of the k hash functions to get k counter positions. If any of the counters at these positions is less than θ, the count number of element is definitely less than θ – if it were more and equal, then all the corresponding counters would have been greater or equal to θ. If all are greater or equal to θ, then either the count is really greater or equal to θ, or the counters have by chance been greater or equal to θ. If all are greater or equal to θ even though the count is smaller than θ, this circumstance is defined as false positive. This also should be minimized like Bloom filter.

About hashing problem and advantages, see Bloom filter.

A counting Bloom filter is essentially the same data structure as count–min sketches, but are used differently.

Potential for false negatives

Several implementations of counting bloom filters allow for deletion, by decrementing each of the k counters for a given input. This will introduce the probability of false negatives during a query if the deleted input has not previously been inserted into the filter. Guo et al. present the problem in great detail, and provide heuristics for the parameters m, k, and n which minimize the probability of false negatives.

Probability of false positives

The same assumptions in Bloom filter, which hash functions make insertions uniform random, is also assumed here. In the m pots, kn balls are inserted randomly. So the probability of one of counter in counting Bloom filter counts l is

$b(l,kn,{\frac {1}{m}})={kn \choose l}({\frac {1}{m}})^{l}(1-{\frac {1}{m}})^{kn-l}$,

where b is binomial distribution. A counting Bloom filter determines an element is greater or equal to θ when the corresponding k counters are greater or equal to θ. Therefore, the probability that counting Bloom filter determines an element is greater or equal to θ is

$p_{fp}(\theta ,k,n,m)=(1-\sum \limits _{l<\theta }b(l,kn,{\frac {1}{m}}))^{k}$.

This is different from formal definition of false positive in counting Bloom filter. However, following the assumption in Bloom filter, above probability is defined as false positive of counting Bloom filter. If θ=1, the equation becomes false positive of Bloom filter.

Optimal number of hash functions

For large but fixed n and m, the false positive decreases from k=1 to a point defined $k_{opt}$, and increases from $k_{opt}$to positive infinity.

Kim et al. (2019) shows numerical values of $k_{opt}$within $1\leq \theta \leq 30$. For $\theta >30$ they suggested using the floor or ceiling of ${\frac {m}{n}}(0.2037\theta +0.9176)$.