Thank you for your question. You have hit on an important design choice.
SynSQL relies on a strict fan out across all partitions.
- Wide tables: As you can see on the multi_omic live demo, queries include the hash key. And the hash key is always used in an implicit "order by" clause. Each chunk involved executes a query focused on the columns it contains.
- Deep tables: All data brokers execute the incoming query on their data set (partition).
Why fan out even for 'where pk = 1' ? The engine broadcasts to all partitions. It delegates expression evaluation downstream. It is designed for parallel scans. The overhead is perfectly acceptable for its target: analytical workload.
How does SynSQL handle queries that do not include the primary hash key? Does it fan out to every partition, or maintain secondary routing metadata?
Hi Islambaraka.
Thank you for your question. You have hit on an important design choice. SynSQL relies on a strict fan out across all partitions.
- Wide tables: As you can see on the multi_omic live demo, queries include the hash key. And the hash key is always used in an implicit "order by" clause. Each chunk involved executes a query focused on the columns it contains.
- Deep tables: All data brokers execute the incoming query on their data set (partition).
Why fan out even for 'where pk = 1' ? The engine broadcasts to all partitions. It delegates expression evaluation downstream. It is designed for parallel scans. The overhead is perfectly acceptable for its target: analytical workload.
Best, Stéphane.