This is a portable format that is used for interchange. The performance may matter elsewhere in your system and yet you may still want to create/access protobuf-packaged data?
Are the only two options SIMD in hand-rolled assembler and unbearably slow?
Often Python, with the most critical parts written in a compiled extension module (like this one), offers acceptable performance with enormously less complexity than writing the whole thing in a compiled language.
Engineer who works on Google Protobuf here, commenting as myself and not as an official statement.
It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.
Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.
Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).
Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into the other buckets shown, affecting the performance of preexisting code is a major concern for our implementations that a greenfield implementation doesn't have.
Wide exposure in critical paths alongside long term support directly causes some quirks: for example some of our APIs followed PEP8 when it was created but PEP8 changed. It looks stupid that we have wrong style APIs but also it would be stupider to break compatibility for style reasons. JavaProto as another example still supports Java8 and the runtime is compatible with 2014 gencode which is a pretty major constraint.
Google Py Proto implementation has one extra interesting choice of the same gencode is reused with 3 different implementations (upb, a complete pure python one, and one that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++), which is why design the way that it is with runtime created classes, the pyi files are readable but the .py files not.
This definitely has pros and cons, and the direct approach taken by Buf here really makes a ton of sense. It's just that Google's maintained implementation falls into a different spot in a larger technical tradeoff space.
If you see things that appear to make no sense with the official implementations, feel free to file an issue on GitHub and we can look, sometimes there is no reason and we can fix it, and sometimes there's a reason which we can explain.
Kudos again to Buf here, I'm fully sure this will solve some set of real business needs better than Google's (but not because Google isn't maintaining our offerings too).
> ... that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++
This is a valid explanation for the odd implementation of the Python protobuf library (the only convincing one I've heard). But, the last time I looked into it (just a couple of years ago), it didn't seem reasonably possible to make use of this.
I can't remember the exact issue I think maybe the C++ library would be statically linked into the Python C extension module, which made it virtually impossible to use it from your own C++ code. Or maybe the issue was just that there was no C++ version prebuilt on pypi (the default is the upb version).
Anyway, it seems a pity to go to such enormous lengths for one feature and then make it essentially unavailable.
After gogoproto I'm hesitant to depend on another non-standard implementation, getting off gogo was a pain. This thing may be better than the one from Google (gogo definitely was), but can we be sure that it will still be around in 10 years?
Hey. I wrote another Python implementation of Protobuf. (protopy https://gitlab.com/doodles-archive/protopy it was a while ago and haven't touched it since). I'm not saying it's better than whatever this is or that it's any good, I just post it as a proof of sorts that I'm familiar with the problem.
So, without further ado: Protobuf isn't a standard. You can't have a non-standard implementation of something that doesn't have a standard to begin with. In reality, you have Google's implementation for C++ and then everything else. Everything else was, for the most part, not written by Google. And it doesn't always align 100% with the C++ Google's stuff.
Furthermore, C++ implementation has a lot of idiosyncrasies specific to that language that can't be translated one-to-one into other languages, or, in some cases, shouldn't be, even if they could (eg. C++ implementation is all about source code generation because generating runtime entities s.a. classes in C++ is very difficult, while in languages like Python, generating classes at runtime is easy.)
Furthermore, C++ implementation has a specific way of parsing the binary payload (lazy: only the top definitions are parsed, the inner structure of messages is parsed on-demand). But, is this how every parser should behave? What if you want a SAX-like parser?
----
In the hindsight, I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage. We, as in the whole programming world, don't have good formats in general, and whenever we come to the point of having to use some, we either go with an existing popular but crooked or roll our own, probably also crooked. The standard you alluded to would've been great (perhaps a refinement of ASN with more attention to parser implementation, more concrete versions etc.?) But we aren't there yet, and there isn't even a work group to try and address the issue.
> I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage.
I am not a fan of Protobuf at all, but it's already demonstrated its ability to ship extremely reliable software with multi-decade lifespans. It's one of the few things Google _hasn't_ deprecated, and it's the backbone of the search and ads stack.
Why does it matter for some Python implementation if the Google C++ implementation has a lazy or eager parser?
The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.
Personally I also prefer code generation over dynamic parsers and generators. This is not an idiosyncrasy of C++, it is just the objectively good way to handle IDLs regardless of programming language.
At least, Google has the resources and the will to support all their implementations in the long run. I don't always agree with what they do there, but at least I can be sure that it will still work ten years from now. At some point it becomes more important than implementation details
The golang implementation of protobuf sucked historically (still does, but is improving) and gogo was an alternative that fixed a lot of problems and was nicer overall. Until its creator burned out and deprecated it. Chasing a constantly moving target that you have no control over is very taxing in the long run.
I think it's less that protobuf is a moving target, and more that gogo tried to add in all the features that google didn't want to maintain, and learned that maintaining a massive feature matrix was impossible.
This is incredible news! I’ve used protobuf in Python, Go, Kotlin and Dart and the Python implementation is totally unusable. I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect. I’ve been waiting for a proper python implementation for years now!
> I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect.
TFA seems to say that they’re just thin proxies over the underlying C++ APIs, which would more than do it, and does not surprise me (the re2 Python bindings are similar, not as bad since they don’t generate Python code but they’re really c++-y — in Google’s flavour too — and uncomfortable).
When I was a novice in protobuf I felt tempted to inspect the generated code, but soon I learned that the documentation is good enough that I don’t need to read the generated code.
Slightly off topic, but is anyone aware of a compile free method to convert protobuf messages to json representation based on a provided .proto file (and the other way around)? All protobuf implementations seem to require a compilation step, which makes it hard to support en-/decoding untrusted content using user provided schemas
There is the --encode (and --decode) options for the `protoc` executable, e.g. `cat myobject.json | protoc --encode MyObject protofile.proto > myobject.bin`
I wish LaunchDarkly and other feature flag providers supported protocol buffers to MN define the feature flag schema. It would be a game changer when you have complex variations and end up reaching for untyped JSON.
Optimizing for the last microsecond and then use Python?
Why?
This. Yes. This is why I clicked on the comments. Exactly
This is a portable format that is used for interchange. The performance may matter elsewhere in your system and yet you may still want to create/access protobuf-packaged data?
Are the only two options SIMD in hand-rolled assembler and unbearably slow?
Often Python, with the most critical parts written in a compiled extension module (like this one), offers acceptable performance with enormously less complexity than writing the whole thing in a compiled language.
> Optimizing for the last microsecond
I don't see that anywhere. I see "Fast where it counts."
Engineer who works on Google Protobuf here, commenting as myself and not as an official statement.
It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.
Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.
Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).
Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into the other buckets shown, affecting the performance of preexisting code is a major concern for our implementations that a greenfield implementation doesn't have.
Wide exposure in critical paths alongside long term support directly causes some quirks: for example some of our APIs followed PEP8 when it was created but PEP8 changed. It looks stupid that we have wrong style APIs but also it would be stupider to break compatibility for style reasons. JavaProto as another example still supports Java8 and the runtime is compatible with 2014 gencode which is a pretty major constraint.
Google Py Proto implementation has one extra interesting choice of the same gencode is reused with 3 different implementations (upb, a complete pure python one, and one that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++), which is why design the way that it is with runtime created classes, the pyi files are readable but the .py files not.
This definitely has pros and cons, and the direct approach taken by Buf here really makes a ton of sense. It's just that Google's maintained implementation falls into a different spot in a larger technical tradeoff space.
If you see things that appear to make no sense with the official implementations, feel free to file an issue on GitHub and we can look, sometimes there is no reason and we can fix it, and sometimes there's a reason which we can explain.
Kudos again to Buf here, I'm fully sure this will solve some set of real business needs better than Google's (but not because Google isn't maintaining our offerings too).
> ... that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++
This is a valid explanation for the odd implementation of the Python protobuf library (the only convincing one I've heard). But, the last time I looked into it (just a couple of years ago), it didn't seem reasonably possible to make use of this.
I can't remember the exact issue I think maybe the C++ library would be statically linked into the Python C extension module, which made it virtually impossible to use it from your own C++ code. Or maybe the issue was just that there was no C++ version prebuilt on pypi (the default is the upb version).
Anyway, it seems a pity to go to such enormous lengths for one feature and then make it essentially unavailable.
After gogoproto I'm hesitant to depend on another non-standard implementation, getting off gogo was a pain. This thing may be better than the one from Google (gogo definitely was), but can we be sure that it will still be around in 10 years?
Buf is well established and maintains a lot of protobuf packages for many languages, including the YAML implementation for go.
Going strong since 2019! It's a wonderful company entirely dedicated to making google's miserable protobuf "somewhat" useable.
Hey. I wrote another Python implementation of Protobuf. (protopy https://gitlab.com/doodles-archive/protopy it was a while ago and haven't touched it since). I'm not saying it's better than whatever this is or that it's any good, I just post it as a proof of sorts that I'm familiar with the problem.
So, without further ado: Protobuf isn't a standard. You can't have a non-standard implementation of something that doesn't have a standard to begin with. In reality, you have Google's implementation for C++ and then everything else. Everything else was, for the most part, not written by Google. And it doesn't always align 100% with the C++ Google's stuff.
Furthermore, C++ implementation has a lot of idiosyncrasies specific to that language that can't be translated one-to-one into other languages, or, in some cases, shouldn't be, even if they could (eg. C++ implementation is all about source code generation because generating runtime entities s.a. classes in C++ is very difficult, while in languages like Python, generating classes at runtime is easy.)
Furthermore, C++ implementation has a specific way of parsing the binary payload (lazy: only the top definitions are parsed, the inner structure of messages is parsed on-demand). But, is this how every parser should behave? What if you want a SAX-like parser?
----
In the hindsight, I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage. We, as in the whole programming world, don't have good formats in general, and whenever we come to the point of having to use some, we either go with an existing popular but crooked or roll our own, probably also crooked. The standard you alluded to would've been great (perhaps a refinement of ASN with more attention to parser implementation, more concrete versions etc.?) But we aren't there yet, and there isn't even a work group to try and address the issue.
> I just think that Protobuf is not a good format for writing reliable software that aims for decades of usage.
I am not a fan of Protobuf at all, but it's already demonstrated its ability to ship extremely reliable software with multi-decade lifespans. It's one of the few things Google _hasn't_ deprecated, and it's the backbone of the search and ads stack.
Why does it matter for some Python implementation if the Google C++ implementation has a lazy or eager parser?
The important part of protobuf is the spec of the wire format. That is what makes the standard an interop format.
Personally I also prefer code generation over dynamic parsers and generators. This is not an idiosyncrasy of C++, it is just the objectively good way to handle IDLs regardless of programming language.
At least, Google has the resources and the will to support all their implementations in the long run. I don't always agree with what they do there, but at least I can be sure that it will still work ten years from now. At some point it becomes more important than implementation details
Honest question - why isn’t the following document [1] a standard? Is it too loosely specified?
[1] https://protobuf.dev/programming-guides/encoding/
Maybe He means because it diesen have an accepted rfc
What was the backstory of gogoproto?
The golang implementation of protobuf sucked historically (still does, but is improving) and gogo was an alternative that fixed a lot of problems and was nicer overall. Until its creator burned out and deprecated it. Chasing a constantly moving target that you have no control over is very taxing in the long run.
This is concerning to hear. What do Protobufs accomplish, that requires them to be a constantly moving target?
https://www.youtube.com/watch?v=HTIltI0NuNg
I think it's less that protobuf is a moving target, and more that gogo tried to add in all the features that google didn't want to maintain, and learned that maintaining a massive feature matrix was impossible.
If I remember correctly, what finally broke the camel's back was the new API that Google introduced. But I may be wrong
This is incredible news! I’ve used protobuf in Python, Go, Kotlin and Dart and the Python implementation is totally unusable. I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect. I’ve been waiting for a proper python implementation for years now!
> I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect.
TFA seems to say that they’re just thin proxies over the underlying C++ APIs, which would more than do it, and does not surprise me (the re2 Python bindings are similar, not as bad since they don’t generate Python code but they’re really c++-y — in Google’s flavour too — and uncomfortable).
I did not know this before... Google protobuf is not one thing, it is three! Same import, but:
* old C++ extension
* upb
* pure Python
upb parses FAST, but then every access is still C->Python and it slows it down. So for many reads the slow python one can win?
This one helped me to dig deeper - https://vectree.io/c/how-python-protobuf-runtimes-work-pure-...
When I was a novice in protobuf I felt tempted to inspect the generated code, but soon I learned that the documentation is good enough that I don’t need to read the generated code.
Slightly off topic, but is anyone aware of a compile free method to convert protobuf messages to json representation based on a provided .proto file (and the other way around)? All protobuf implementations seem to require a compilation step, which makes it hard to support en-/decoding untrusted content using user provided schemas
The buf CLI can do this.
Here's how it looks to convert from encoded protobuf to json (protojson).
And just invert the arguments to convert back.There is the --encode (and --decode) options for the `protoc` executable, e.g. `cat myobject.json | protoc --encode MyObject protofile.proto > myobject.bin`
I wish LaunchDarkly and other feature flag providers supported protocol buffers to MN define the feature flag schema. It would be a game changer when you have complex variations and end up reaching for untyped JSON.
Hope it can auto build a python class from gRPC gateway reflections.