Linkedin just deprecated Kafka in favor of Northguard [1]. Curious how this actually compares in practice. Is it worth the effort of building yet another distributed log? Is it worth the effort to mantain Kafka if new alternatives are popping? Jepsen found plenty of issues last year [2], so maybe Linkedin is in the right direction by building from scratch...
Distributed systems are tough to get right, and while it's exciting to see new approaches, the space is definitely getting harder to follow.
Besides the implementation effort in building from scratch, it also opens you up to unknown unknowns, while the limitations of mature software are better defined. Just because Jepsen didn't test other alternatives doesn't mean they are free from issues.
Regarding the Kafka issues pointed out:
* 2 only affect you if you are using transactions (and I would be interested in knowing of alternatives that better handle exactly-once semantics);
* Consumer.close() is likely off the critical path of your applications (personally I never encountered these hangs);
* Aborted reads and torn transactions were mentioned as being fixed by KIP-890, of which the mentioned ongoing work was completed since that article was published: https://issues.apache.org/jira/browse/KAFKA-14402
I'm writing a distributed DB rn ; and all the algs and tech they are using, SWIM ; hash ring ; the questions they are asking themselves - are pretty much standard ; it is not easy but it's a great field with different depth ; lot of successful companies don't dive that deep because it's uncharted experimentation failure & document process that is very very expensive to justify in a traditional context. I don't see how I would sell what i'm doing rn without getting it right first - and there is no point in proposing a new technology if there is no operational guarantees - And tbh a lot of people who are using kafka or doing distributed work are mostly doing the equivalent of setting up kafa consumers and publishers and that is enough and *it is fine*.
I'm a big fan of https://nats.io/ as a lightweight alternative to Kafka. I think of it like Nomad vs. Kubernetes: you can get 80% of the functionality for the cost of 25% of the complexity.
Some thoughts:
Linkedin just deprecated Kafka in favor of Northguard [1]. Curious how this actually compares in practice. Is it worth the effort of building yet another distributed log? Is it worth the effort to mantain Kafka if new alternatives are popping? Jepsen found plenty of issues last year [2], so maybe Linkedin is in the right direction by building from scratch...
Distributed systems are tough to get right, and while it's exciting to see new approaches, the space is definitely getting harder to follow.
[1] https://www.linkedin.com/blog/engineering/infrastructure/int...
[2] https://jepsen.io/analyses/bufstream-0.1.0
Besides the implementation effort in building from scratch, it also opens you up to unknown unknowns, while the limitations of mature software are better defined. Just because Jepsen didn't test other alternatives doesn't mean they are free from issues.
Regarding the Kafka issues pointed out:
* 2 only affect you if you are using transactions (and I would be interested in knowing of alternatives that better handle exactly-once semantics);
* Consumer.close() is likely off the critical path of your applications (personally I never encountered these hangs);
* Aborted reads and torn transactions were mentioned as being fixed by KIP-890, of which the mentioned ongoing work was completed since that article was published: https://issues.apache.org/jira/browse/KAFKA-14402
I'm writing a distributed DB rn ; and all the algs and tech they are using, SWIM ; hash ring ; the questions they are asking themselves - are pretty much standard ; it is not easy but it's a great field with different depth ; lot of successful companies don't dive that deep because it's uncharted experimentation failure & document process that is very very expensive to justify in a traditional context. I don't see how I would sell what i'm doing rn without getting it right first - and there is no point in proposing a new technology if there is no operational guarantees - And tbh a lot of people who are using kafka or doing distributed work are mostly doing the equivalent of setting up kafa consumers and publishers and that is enough and *it is fine*.
It is impossible to know what Northguard can and cannot do since it's an internal products.
I'm a big fan of https://nats.io/ as a lightweight alternative to Kafka. I think of it like Nomad vs. Kubernetes: you can get 80% of the functionality for the cost of 25% of the complexity.
Nats is the way