I know the article says "analytics workloads," but I wonder if this pattern could be used for ex. bursty-write workloads.
For context, I have a use-case where I have a Meilisearch server that has all the data in it updated once daily, and then every single request outside of that one bulk update is a read. The only exception is that once in a great while, something breaks and the entire index has to be rebuilt. Since it's a hobby project, my workload is too expensive to fit into Meilisearch Cloud or similar solutions. I keep finding myself wishing that there was a serverless search engine that would free me from having to self-host one, keep a dedicated server running and updated, etc. If I could "just" run a search engine with Cloudflare Workers + R2 (+ probably ancillary CF services) I'd be very happy.
I don't know enough to actually answer my own question unfortunately; I can't intuit the answer just from reading the docs and database implementations aren't my area of expertise.
> I know the article says "analytics workloads," but I wonder if this pattern could be used for ex. bursty-write workloads.
The key point is that Delta Lake and Iceberg concurrency control is so coarse-grained. As I mentioned in the article, they did this because it seemed they wanted to avoid an external highly-available metadata store (e.g. FoundationDB which is what Snowflake uses).
Just about every Database as a Service (OLTP or OLAP) these days supports storage tiering to cloud blob storage. Delta Lake and Iceberg just took cloud blob storage to the extreme by trying to do everything in it (or as much as possible, Iceberg seems to require a Compare-and-Swap operation which I'm not sure is possible to express with atomic PutIfAbsent alone; seems to require an external HA metadata store for their one current metadata pointer). And in doing this they gave up fine-grained concurrency control.
I wish we generally used solutions like this instead of always reaching for the 200lb database hammer.
however, I think the article would benefit from distinguishing this approach from a 'real' database. that is MVCC gets you a lot more concurrency because it spreads out the exclusionary state in both time and space.
the optimistic concurrency used in the article works well at low contention and then becomes arbitrarily latent as more and more proposers try to jump into same RTT gap with both their initial attempt and their retries.
> however, I think the article would benefit from distinguishing this approach from a 'real' database. that is MVCC gets you a lot more concurrency because it spreads out the exclusionary state in both time and space.
I thought I already was going to be seen as being unfairly critical of the course-ness of the concurrency control.
in any case thanks for writing it up. I was just thinking about doing this in a broadcast domain to provide the same kind of low-throughput bootstrapping consistency that people would normally use zookeeper or etcd for. its certainly very useful in the cases where it fits.
I know the article says "analytics workloads," but I wonder if this pattern could be used for ex. bursty-write workloads.
For context, I have a use-case where I have a Meilisearch server that has all the data in it updated once daily, and then every single request outside of that one bulk update is a read. The only exception is that once in a great while, something breaks and the entire index has to be rebuilt. Since it's a hobby project, my workload is too expensive to fit into Meilisearch Cloud or similar solutions. I keep finding myself wishing that there was a serverless search engine that would free me from having to self-host one, keep a dedicated server running and updated, etc. If I could "just" run a search engine with Cloudflare Workers + R2 (+ probably ancillary CF services) I'd be very happy.
I don't know enough to actually answer my own question unfortunately; I can't intuit the answer just from reading the docs and database implementations aren't my area of expertise.
> I know the article says "analytics workloads," but I wonder if this pattern could be used for ex. bursty-write workloads.
The key point is that Delta Lake and Iceberg concurrency control is so coarse-grained. As I mentioned in the article, they did this because it seemed they wanted to avoid an external highly-available metadata store (e.g. FoundationDB which is what Snowflake uses).
Just about every Database as a Service (OLTP or OLAP) these days supports storage tiering to cloud blob storage. Delta Lake and Iceberg just took cloud blob storage to the extreme by trying to do everything in it (or as much as possible, Iceberg seems to require a Compare-and-Swap operation which I'm not sure is possible to express with atomic PutIfAbsent alone; seems to require an external HA metadata store for their one current metadata pointer). And in doing this they gave up fine-grained concurrency control.
For folks interested in this topic I highly recommend this talk by Cliff Click about building highly scalable hash tables via Compare And Swap:
https://youtu.be/HJ-719EGIts
I wish we generally used solutions like this instead of always reaching for the 200lb database hammer.
however, I think the article would benefit from distinguishing this approach from a 'real' database. that is MVCC gets you a lot more concurrency because it spreads out the exclusionary state in both time and space.
the optimistic concurrency used in the article works well at low contention and then becomes arbitrarily latent as more and more proposers try to jump into same RTT gap with both their initial attempt and their retries.
> however, I think the article would benefit from distinguishing this approach from a 'real' database. that is MVCC gets you a lot more concurrency because it spreads out the exclusionary state in both time and space.
I thought I already was going to be seen as being unfairly critical of the course-ness of the concurrency control.
in any case thanks for writing it up. I was just thinking about doing this in a broadcast domain to provide the same kind of low-throughput bootstrapping consistency that people would normally use zookeeper or etcd for. its certainly very useful in the cases where it fits.