Probabilistic Data Structures in Go

(antonz.org)

62 points | by mfrw 2 days ago ago

7 comments

  • ncruces a day ago ago

    I'm in the process of adding a few of these to my Go SQLite driver.

    https://pkg.go.dev/github.com/ncruces/go-sqlite3/ext/approx

    They work as window functions, and they also support creating roll-up tables, e.g. you have a table storing a sketch of daily active users and can use that to get weekly and monthly active users.

    Same for quantiles (e.g. median and tail latencies) and next up top-K (e.g. most popular content, highest spenders, etc).

  • donatj a day ago ago

    I'm kind of sad there's no mention of Cuckoo filters. I've generally found them preferable to Bloom filters

  • mfrw a day ago ago

    A related CppCon talk: Fantastic Algorithms and Where To Find Them & the related GitHub Repo:

    - https://youtu.be/YA-nB2wjVcI

    - https://github.com/CppCon/CppCon2017

  • kristianp a day ago ago

    Now I've learnt about blocked Bloom filters:

    > Unsure uses a blocked Bloom filter. The bit array is divided into 512-bit blocks, each matching the size of a CPU cache line. When you add or check an item, it is hashed once: part of the hash selects a block, and the rest selects k bits within that block.

    The article also has a good quality list of references.

  • jpitz a day ago ago

    My introduction to probabilistic data structures was Pugh's skip list, which is probabilistic in structure and performance, but precise with answers.

  • aarjaneiro a day ago ago

    Note a lot of the diagrams seem broken