I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.
It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.
I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.
I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.
The article assumes the issue with OTel is slow feature development, which isn't my experience at all. The issue I've had is that the SDKs have terrible performance overhead for instrumentation and are, as you say, highly resistant to integrating the output of better performing (or just preexisting) instrumentation. In Python and Ruby, at least, the CPU cost of all the mandatory abstraction is way too high.
Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for.
As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several pieces and how they work together (hint: convoluted and horrifically inefficiently) to modify any piece, and inevitably you learn that to get what you want, you need to swap out a major portion of it... but doing that while maintaining the auto-registry nonsense is a gigantic effort. If it's even possible.
It is the new poster-child for "design by committee". It's horrific. Unfortunately it's also usually the best option in large setups. I greatly approve of the high level goal, but omfg
It's crazy, IMO, that they didn't simply design otel clients. Making this giant cross language framework is an insane endeavor that just makes everyone unhappy.
Yeah, cross language is generally considered "a protocol", and exists only to cross process boundaries. That's definitely useful! It's even a mostly reasonable one (though with a few weird decisions either due to blindly copying Prometheus' flaws or due to... idk avoiding copying Prometheus on principle? Very strange sometimes, but livable). We needed a grand unification here, even if mediocre, and the time was right.
Trying to make all the supported languages feel similar (beyond sharing concepts which almost directly match the protocol) is foolish in the extreme, and it's why it's such a monstrosity. And worse, they seem to treat that as more important than the bottom-most clients that speak the protocol, so you might be waiting years for any support for a third of the system!
Totally agree. However I am hopeful. We started the first full instrumented project a few years back. It took us a long time to do the whole work including understanding the SDK, mapping the dimensions and getting everything right. Our last project we did the whole thing with agents and they really took away a lot of the pain from the implementation part.
We also use Axiom MCP so when we need some trace or event in the logs the agents look for it and if they don’t find it they’ll add it for the next time. It’s really been a different experience.
Even just basic wire protocol is ass that's PITA to parse, like list of attibutes (which have to be unique) isn't a map but array of maps with some weird way to encode key and type. The whole project is industrial scale mediocrity
What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot.
There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And metrics are something else altogether. Trying to somehow unify them would be a prime example of "wrong abstraction".
> “The next time something like this occurs again, please save me a trace.”
The building blocks for this exist. The observability platform must simply (haha) implement the pattern detectors and use them for sampling decisions.
I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times.
Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging.
All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0".
The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.
This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction.
Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance.
In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure.
"Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems.
Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.
You are conflating the challenges of ingesting and querying at large scale with the what the original comment is about, which is emitting them more easily.
I don't see them as separate issues. Emitting them directly runs into the inherently poor memory locality (and potentially concurrency) of trying to produce logs, metrics, and traces from the same underlying event data representation.
It is only "easy" if performance and scalability don't matter.
> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)
The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.
Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines.
Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.
At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.
In the code define everything as a span with a name, scope (start-end), description and tags... and then you can easily dynamically produce traces, spans, logs or metrics based on what you need.
I don't think OTEL is necessarily "at fault" here. It's a split that's carried all throughout the observability ecosystem. e.g. in the Grafana suite of solutions you have Loki (logs), Tempo (tracing) and Mimir (metrics) to cover storage & querying for all three axis, as all of them have very distinct processing & performance characteristics.
While it may intuitively may look like there is a large overlap in the three areas there is suprisingly little, and for the few parts there are (e.g. trace <-> log correlation), OTEL does offer a standard.
Tracing is the most general of them, and the most expensive unless you're careful with the implementation.
Trace spans are time-delimited units of "stuff that happened", with a tree relationship among the spans, and each span can have arbitrary tags (key/value pairs) and events (time/value).
From that, if you chose, you could derive metrics and logs. The trick is to start with tracing and to actually put it in your program, rather than trying to mostly-automatically tack it on later.
I think it is almost a inevitability where otel came as a standardised aggregate of OpenTracing (which was the same but only for tracing over multiple tracing implementations), logging, and metrics into a single observability standard without alienating all the individual supporting vendors.
Historically, logging and metrics have been different problem domains with different implementations for ages.
Now to your point:
Note that tracing does get the most of love, and that it does include constructs to add logging and metrics into these traces (spans actually). So you could argue that they are trying to develop a single interface.
> “The next time something like this occurs again, please save me a trace.”
Well, if you want this you either need to propagate this predicate to all points that might be involved, or always emit all traces and have the predicate included in the filter. And then you need to be able to dynamically propagate this predicate from the system/ui where you click to where you filter.
This is one of the reasons why we always propagate and emit traces and just post filter it in processing before it lands in the persistence layer.
If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load.
What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.
The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
You're pitching a solution that's incredible brittle and unnecessarily complicated if you think about it in technical terms.
For your feature to work you need bi-directional communication between the otel receiver and your application - that's still doable in general, but now you want a synchronous "upgrade" to traces.
Now we're talking about a massive performance impact - and you need to somehow cache all otel data locally so they're available for the upgrade and only then submit then.
It is a architecture that's not very smart, honestly. And precisely the reason why you'd simply submit everything and let the receiver figure out which samples it wants to keep - as thorian pointed out earlier.
> The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
How does the monitoring system have any of the context to add labels? That would only exist in application memory.
> That may be prohibitively expensive in terms of CPU/network load.
In practice I've not experienced this even on quite high request rates. While it isn't free, exporting everything has been cheap enough that the real cost in dollars spent is basically marginal (it's _storing_ the data that's expensive)
> How does the monitoring system have any of the context to add labels? That would only exist in application memory.
Indeed. If you have a protocol that doesn’t allow exposing that kind of information, then that only lives in application memory. But my suggestion is that it’s exposed.
> If I understand that correctly, it means your app always creates traces
Yes, because otherwise what you propose requires modifying the binary in-place and that's too big of a security hole for lots of (production) environments. Some variants of that could work with an out-of-process method like Dtrace or eBPF, but that means mutating the kernel, even more of a no-no.
Its a shame that the various implementations are pretty horrible. Global state, static methods etc etc.
If you get rid of that, and just pass dependencies around, create some appropriate local abstraction around them.. the tooling, be it datadog or honeycomb does a great job making it useful. Can't really say the same for grafana, but ymmv - depending on budget
OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today:
1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.
2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.
3. Serverless runtimes pay a heavy penalty for cold starts with OTel.
4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.
6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?
> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent.
App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.
Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads.
Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to.
In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.
We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.
A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars).
The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink.
It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet.
> In a runtime like Lambda, how would you avoid the need to run an edge collector?
Here's my setup (in Go, very simplified):
> // Instantiate a new slog logger
> logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger))
> // Use the logger as needed
My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.
If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now.
The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties.
Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.
Because blocking on OTel data to flush before sending a response back is often unacceptable. The layers run a standalone process using the Lambda extension API so they can keep running after your function has responded to a request.
There isn't really a great alternative without vendor lock-in. If you go all-in on AWS Cloudwatch/X-Ray, it's a really easy setup with low effort. If you go all-in on Datadog, it's pretty easy. But if you want to mix Sentry, Langfuse, Datadog, etc, OTel is still probably the best option. It's just a letdown that this is the best there is.
I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.
I looked at the OTel schema generated for recording a single numerical metric. It was like 12 or 13 meta fields in addition to the actual metric fields like timestamp, metric and value.
OTel looks like something designed by a committee of committees, funded by someone who is in the business of selling cloud storage/data warehousing services.
OpenTelemtry is the perfect example of an overengineered mess.
While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped.
A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get.
I think OpenTelemetry turned metrics into a farce. In many situations even self-rolled telemetry works better even with the added stuff. The annoying thing is that OpenTelemtry is that big standard now one kind of has to to add compatibility. So please, if you write software, make sure you don't lock yourself into OTel.
It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host.
Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.
I run OSS Grafana with Loki, Prometheus, and Tempo. I use an Alloy sidecar taking in OTEL and scraping logs.feom my Go services and selfhost the stack. Once you need to scale it gets a bit more complicated but it's all still OSS.
The biggest challenge I have is that each data source needs it's own query language, which DD and the like don't. That's why at my day job they went with DD despite the costs. Still OTEL but the querying is the same. We are also looking at Dash0 but for all of my personal and consulting jobs, OSS LGTM/P works good for me.
Yes, Grafana seems to be the best free querying frontend across multiple data sources right now. But each Victoria product has its own built-in query GUI as well.
I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasant
Grafana is fine to get to a selfhosted basic install. But once you try to actually connect logs, metrics, traces in selfhosted context and perhaps sprinkle some otel in...that sht gets out of hand very fast. It's modular in a way that seems like a win but once you start connecting stuff it starts adding complexity not ease.
Signoz...still pretty early in exploring this and so far it's acceptable, but it too relies on a mix of query languages incl the competitors promql so some panels support it other seem to not to?
The entire thing just seems bewildering to me. Not gonna say "why is this so hard" because I genuinely thing smart people are genuinely trying here...but there result just isn't great.
I think the industry would benefit from some general evangelism for observability. Being able to do distributed tracing was both a "well, duh" and mindblown experience when I first learned about it a decade ago. It made supporting software so much better.
OTel is a fine system for learning observability; it does an okay job of exposing capabilities given how diverse the vendor ecosystem is.
I disagree. I'm an observability geek, and OTel is... fine.
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
Every time I share your blog (and I share it a lot) I tell people:
"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".
'A practitioner's guide to wide events' especially is still my number 1.
I'd make a wager that things would go better smoother faster if folks tried more stuff, ventures forth more on their own. It's obviously not great that there's no semantic convention that's perfect and just works for everything, and yeah it takes a while. I feel like the real data I'd want is who else, how many people show up to say they've tried something. Is that happening? Whether specs are really good enough advance or not, to me, is often whether enough people have tried it to find out.
The net of this is, otel is a very flexible system you can use and adapt in all kinds of ways and while the spec is important, using the toolkit to FAFO yourself, ahead of any beaten path, should really be encouraged. That's the message I'd want to see being radiated out about otel.
We did the same for 1, we have a service that picks up messages from a queue and drops 95% of them. In Python, at least, it was really really ugly. We had the alternative of using refinery, since we're on honeycomb, but didn't want to get stuck with something that only works with one vendor.
I don't understand 2, though, that should be easy? You have access to the span and you can set its status code to 0. With the Python SDK that was trivial for us to do.
The one thing I really hate about the Python SDK is that there's no simple way to say "attach this attribute to the top level span in this app". That way, when you get the user id further down in your app, you can just annotate the top level span with it. Thankfully, honeycomb has any, any2, root, parent (that allows you to filter on any span in the trace, or on the root or parent) but that's slow and again, vendor specific.
OTel has a nice property of working, which sometimes can't be said about vendor stuff.
I think the actual APIs kinda smell at the language level, and when Honeycomb decided to lean into otel and deprecate its Python libs I was super sad, cuz HC's libs were _way_ more usable IMO. Docs are also... painful. Real painful.
I wish that I could get a Python lib which is like "here this is Otel but the config phase isn't weird, and the API just looks a bit better". One of these days.
The biggest trouble I have with Otel recently is getting fixes patched upstream in contrib. Using contribs is super dangeerous, and I would basically recommend people write their own instrumentation and treat the contrib packages as just examples of how to do it
Agreed. Otel itself is fine. The documentation is bad though and full of inconsistent best practices and examples that are flat out wrong and other things.
My life of working with it got easier when I started just looking at the actual code, using network level tools like nc/tcpdump, making extensive use of the debug exporter, and almost ignoring the docs entirely except as a basic summary of what a thing does.
It feels like OTel tried standardizing before the correct design was anywhere close to being settled. It's only time to standardize once there's consensus on all the important points, and what's left is minor details that don't matter for anything other than compatibility.
Speaking only from my experience using their rust crates, they have undergone more “code feng shui” than any of our other dependencies. They’re still 0.x and every point release seems to re-imagine things enough to break everything and require substantial rewriting. They don’t even bother describing the motivation for changes, just, you can’t use this type any more, it’s private now. You can’t configure metadata here any more, you have to do it there now. It’s been the most painful dependency of ours by far.
OTel is very complicated while yeah for example datadog is just dropin. And Graylog support for OTel makes it a second class citizen in the logs (all attributes are prepended with otel_attributes_ which makes searching difficult).
Using is hard, vendors are hostile, it seems like no-one want it to be a first class citizen...
I know sadly very little about otel, it feels “heavy” in a way I am not used to, I am used to simple systems - configured and composed in a way that makes a larger system.
20 years ago, we were doing (what I think) OTel is doing: with “hit IDs” (half way between a session and a request) that were consistently applied when logging the cause a request being fired; along centralised logging and really good timekeeping. Essentially a unique identifier as a tag that followed the request as it passed through the system.
This was enough to debug basically any problem.
We could even measure the distance between requests of the same “hit” and the total wall-time before it managed to return through the load balancer, so we could track our p99 easily.
Though truthfully we didn't make pretty graphs.
I sometimes wonder what OTel gives me more than this, but I work in games now and lots of these things that work well in webdev do not apply at all to our problems.
You are essentially describing a proto-tracing system. At the risk of self-promoting twice in one comments section, I have a post walking through going from what you describe above to OTel-compatible tracing: https://jeremymorrell.dev/blog/minimal-js-tracing/
You are right that what you were doing is very similar! However standardization helps a lot here.
Not the OP, but turning on auto-instrumentation for a Golang app running in Kubernetes breaks the app if the app is either:
- Running an old version of Golang (older than 1.18 if memory serves), or
- has libraries that the eBPF probes don't like.
And while I like OTel, I agree with the OP that you are absolutely going deep-sea diving if you're going to do anything beyond the examples provided (which is very easy to do!)
(I haven't attempted to use opentelemetry-instrumentation-django in at least a year so my information might be dated and my memory is patchy :P)
If I recall the primary issue was the forced loading of the django settings file by otel.
I get that fully automated instrumentation should be turn-key and the current approach kinda works on basic applications.
But most production django applications are monoliths and generally larger apps. They have non-trivial configuration processes which are often multi step and source settings from multiple places.
Otel should not assume it can just randomly load a the django settings at an arbitrary time point in the startup process.
In one of our apps the MIDDLEWARE setting specifically is dynamically generated and re-ordered based on enabled features. That application's startup process also has multiple stages and the initialisation of django occurs much later, after dependant config loaders etc have been initialised.
What would allow us to integrate with opentelemetry-instrumentation-django much more easily is a set of smaller primitives that we can configure and call at the appropriate time.
opentelemetry-instrumentation-django has (had?) a lot of logic hidden inside a large "inject" function which could not easily be extracted into the constituent parts and applied in a compatible manner.
Thanks for the write up, appreciated. A couple of things:
- users are not forced to use auto-instrumentation. People can import the Middleware and use it as they see fit. I see that the instrumentor is configuring the middleware using some private attributes, I guess that can be extracted into a public function so it would be easier to do so
- speaking of the middleware, the chances that it'll become a public symbol are scarce as are the chances that the interfaces will change. So if one has some testing before going to production it should be fine
I don't understand the sentiment. OTEL is better than anything I've ever used before. Do I like every part? No. My personal no-no is the automatic instrumentation which I always bypass and just DI it myself, I don't like the "global" by default approach in Golang and I had to fight team mates who were all for using it. That said, no other observability library that I've ever used was so good overall.
The perf is meh, but tbh if you look at the kind of code we, regular developers write for work, it's probably still vastly better.
> However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on.
This is just plain wrong, binaries of the collector are shipped which are available to use straight away. You can use the builder if you want to create your own version with a selected set of components but it is no way a hard requirement.
The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage.
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone. This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
Prometheus is so easy to add and if you need more scale, there is mimir and a few other options with similar client semantics. I really can't imagine reaching for a framework APK that tries to anticipate every possible thing I would want telemtered, and is inevitably missing all the domain specific derived channels I need. Even prepackaged Prometheus exporters are usually overkill.
Hard Agree on Prometheus. And esp on the complexity - OTel is dizzyingly complex. You can get started ASAP on Prometheus whereas you get lost in analysis-paralysis when dealing with OTel.
Premature instrumentation is the root of all evil. And the source of a significant part of AWS revenue. It should not cost more to monitor an app then run it.
I have always been turned off to attempt to use OTel by the feeling that it is a little bit too over-engineered a that it might be very bad in term of performance/wasted network traffic when you see the data structure that it is using.
It is crazy to me how often people don't grok how to design software well.
1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?
2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?
If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.
otel is amazing and when thoughtfully instrumented, turns out, you can opt out of auto instrumentation btw, it provides a standard that is useful, well maintained, portable to enterprise or self hosted.
It’s modular, but the author is appraising the Ruby shortcomings as a problem while also saying they unfortunately don’t have time to contribute because, you know, they can’t “join the calls” lol
We just got a CTO who loves Ruby and guess what I’m about to do: use AI to fill in the Ruby gaps and open a PR and work a weekend or two and see if they like it and then you won’t write any more articles disparaging a project that I personally love.
It saves our company AT LEAST 10k a month vs having datadog / splunk / enterprise-y bullcrap. You seem so educated, why not roll up your sleeves instead of patronizing the hard working people that make the project work with your “if it were me” just go ahead and say it in their forums.
And, if you work for your paycheck, you’re using an agent. A mature project like Otel? Shit. That’s easy-peasy to feed into an agent, so what’s what is actually the problem? Take the time to learn and help them out if it bothers you so much you want to share it with the world!
Isn’t every single “problem” found in every large and successful open sourced framework?
Throwing out a baity framing like it’s some kind of project going wrong and then kind of just ending the article without making any sort of judgement on where this all leads, proceeding to post on hackernews.. bait!
It’s a cloud native project that is not owned by any company. That’s so rare and worth an article to celebrate open source! What a privilege to stand on the shoulders of giants!
> So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks.
“dizzying” — so I’m lost, did the author remmeber the scope of the project before they started making judgements about it?
And calling the attention of hackernews here: what’s the alternative? Oh that’s right, there isn’t one. Because this is a wag my finger article for attention and aggregating the author on a developer channel to boost their presence. Lame.
I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.
It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.
I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.
I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.
The article assumes the issue with OTel is slow feature development, which isn't my experience at all. The issue I've had is that the SDKs have terrible performance overhead for instrumentation and are, as you say, highly resistant to integrating the output of better performing (or just preexisting) instrumentation. In Python and Ruby, at least, the CPU cost of all the mandatory abstraction is way too high.
Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for.
As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several pieces and how they work together (hint: convoluted and horrifically inefficiently) to modify any piece, and inevitably you learn that to get what you want, you need to swap out a major portion of it... but doing that while maintaining the auto-registry nonsense is a gigantic effort. If it's even possible.
It is the new poster-child for "design by committee". It's horrific. Unfortunately it's also usually the best option in large setups. I greatly approve of the high level goal, but omfg
It's crazy, IMO, that they didn't simply design otel clients. Making this giant cross language framework is an insane endeavor that just makes everyone unhappy.
Yeah, cross language is generally considered "a protocol", and exists only to cross process boundaries. That's definitely useful! It's even a mostly reasonable one (though with a few weird decisions either due to blindly copying Prometheus' flaws or due to... idk avoiding copying Prometheus on principle? Very strange sometimes, but livable). We needed a grand unification here, even if mediocre, and the time was right.
Trying to make all the supported languages feel similar (beyond sharing concepts which almost directly match the protocol) is foolish in the extreme, and it's why it's such a monstrosity. And worse, they seem to treat that as more important than the bottom-most clients that speak the protocol, so you might be waiting years for any support for a third of the system!
Totally agree. However I am hopeful. We started the first full instrumented project a few years back. It took us a long time to do the whole work including understanding the SDK, mapping the dimensions and getting everything right. Our last project we did the whole thing with agents and they really took away a lot of the pain from the implementation part. We also use Axiom MCP so when we need some trace or event in the logs the agents look for it and if they don’t find it they’ll add it for the next time. It’s really been a different experience.
The official Go SDK is a nightmare to use and understand. Just the "getting started"[1] requires adding 10 imports.
[1]: https://opentelemetry.io/docs/languages/go/getting-started/#...
OpenTelemetry reminds me a lot of the bad old days when Java/XML maximalism was fashionable.
I tried to emit metrics from a python app using otel once. Gave up and switched to prometheus. What a nightmare.
Ran into the same issue and didn't find any willingness in the OTEL gods to close this gap.
Even just basic wire protocol is ass that's PITA to parse, like list of attibutes (which have to be unique) isn't a map but array of maps with some weird way to encode key and type. The whole project is industrial scale mediocrity
What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot.
There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And metrics are something else altogether. Trying to somehow unify them would be a prime example of "wrong abstraction".
> “The next time something like this occurs again, please save me a trace.”
The building blocks for this exist. The observability platform must simply (haha) implement the pattern detectors and use them for sampling decisions.
I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times.
https://docs.micrometer.io/micrometer/reference/observation....
The problem is not the instrumentation but the way everyone of them work.
A metric is a point in time. A metric is very small but you have a lot of them.
A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc.
A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation.
How do you decide when to suddenly do the trace and send it? IF you always do the trace, you have to pay for the overhead of that tracing constantly.
Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging.
All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0".
The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.
This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction.
Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance.
In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure.
"Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems.
Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.
You are conflating the challenges of ingesting and querying at large scale with the what the original comment is about, which is emitting them more easily.
I don't see them as separate issues. Emitting them directly runs into the inherently poor memory locality (and potentially concurrency) of trying to produce logs, metrics, and traces from the same underlying event data representation.
It is only "easy" if performance and scalability don't matter.
> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.
No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)
The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.
A metric is not event based.
You don't have a metric 'person logged in' because you would need to scrape the metric at the moment a person logged in.
You have a metric called 'overall people have logged in so far' and you do math on it.
The 'person logged in' is an event you log out.
Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines.
Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.
At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.
Loghandling is quite resource intensive.
All the log ingestion systems i have seen were bigger elastic search clusters.
What? All of this has been solved for a long time. How do you think hyperscalers do this?
Search keyword: "Adaptive sampling"
Adaptive sampling is not tracing, its sampling.
Tracing traces a particular event.
I'm quite aware of the difference between sampling, tracing and profiling.
> How would you represent metrics as traces?
Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.
"free". The observability system would greatly exceed the workload being observed in many cases.
Yup. Not a difficult problem to solve.
In the code define everything as a span with a name, scope (start-end), description and tags... and then you can easily dynamically produce traces, spans, logs or metrics based on what you need.
At some point your monitoring is burning 10x as much CPU as the actual task...
I don't think OTEL is necessarily "at fault" here. It's a split that's carried all throughout the observability ecosystem. e.g. in the Grafana suite of solutions you have Loki (logs), Tempo (tracing) and Mimir (metrics) to cover storage & querying for all three axis, as all of them have very distinct processing & performance characteristics.
While it may intuitively may look like there is a large overlap in the three areas there is suprisingly little, and for the few parts there are (e.g. trace <-> log correlation), OTEL does offer a standard.
Tracing is the most general of them, and the most expensive unless you're careful with the implementation.
Trace spans are time-delimited units of "stuff that happened", with a tree relationship among the spans, and each span can have arbitrary tags (key/value pairs) and events (time/value).
From that, if you chose, you could derive metrics and logs. The trick is to start with tracing and to actually put it in your program, rather than trying to mostly-automatically tack it on later.
I think it is almost a inevitability where otel came as a standardised aggregate of OpenTracing (which was the same but only for tracing over multiple tracing implementations), logging, and metrics into a single observability standard without alienating all the individual supporting vendors.
Historically, logging and metrics have been different problem domains with different implementations for ages.
Now to your point: Note that tracing does get the most of love, and that it does include constructs to add logging and metrics into these traces (spans actually). So you could argue that they are trying to develop a single interface.
> “The next time something like this occurs again, please save me a trace.”
Well, if you want this you either need to propagate this predicate to all points that might be involved, or always emit all traces and have the predicate included in the filter. And then you need to be able to dynamically propagate this predicate from the system/ui where you click to where you filter.
This is one of the reasons why we always propagate and emit traces and just post filter it in processing before it lands in the persistence layer.
You can do that in Lisp, since you can arbitrarily redefine the wrapper to have such or other logic etc.
One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...
If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load.
What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.
The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
You're pitching a solution that's incredible brittle and unnecessarily complicated if you think about it in technical terms.
For your feature to work you need bi-directional communication between the otel receiver and your application - that's still doable in general, but now you want a synchronous "upgrade" to traces.
Now we're talking about a massive performance impact - and you need to somehow cache all otel data locally so they're available for the upgrade and only then submit then.
It is a architecture that's not very smart, honestly. And precisely the reason why you'd simply submit everything and let the receiver figure out which samples it wants to keep - as thorian pointed out earlier.
> The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
How does the monitoring system have any of the context to add labels? That would only exist in application memory.
Grafana went the other way - your app exports all labels, and then you selectively aggregate on ingest: https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...
> That may be prohibitively expensive in terms of CPU/network load.
In practice I've not experienced this even on quite high request rates. While it isn't free, exporting everything has been cheap enough that the real cost in dollars spent is basically marginal (it's _storing_ the data that's expensive)
> How does the monitoring system have any of the context to add labels? That would only exist in application memory.
Indeed. If you have a protocol that doesn’t allow exposing that kind of information, then that only lives in application memory. But my suggestion is that it’s exposed.
> If I understand that correctly, it means your app always creates traces
Yes, because otherwise what you propose requires modifying the binary in-place and that's too big of a security hole for lots of (production) environments. Some variants of that could work with an out-of-process method like Dtrace or eBPF, but that means mutating the kernel, even more of a no-no.
It is very easy way to have your tracing infrastructure cost more than actual infrastructure.
I've never found instrumentation to be a huge issue. Sure it takes more effort but you get a lot more value once you understand _business_ events.
Its a shame that the various implementations are pretty horrible. Global state, static methods etc etc.
If you get rid of that, and just pass dependencies around, create some appropriate local abstraction around them.. the tooling, be it datadog or honeycomb does a great job making it useful. Can't really say the same for grafana, but ymmv - depending on budget
OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today:
1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.
2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.
3. Serverless runtimes pay a heavy penalty for cold starts with OTel.
4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.
6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?
> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent.
App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.
Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads.
Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to.
In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.
(I’m not the person you replied to, but have experience here.)
I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog).
We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.
A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars).
The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink.
It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet.
> In a runtime like Lambda, how would you avoid the need to run an edge collector?
Here's my setup (in Go, very simplified):
> // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed
My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.
In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block.
[0] https://github.com/open-telemetry/opentelemetry-lambda
That lambda layer comes with an incredibly heavy performance penalty.
It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2].
[0]: https://github.com/open-telemetry/opentelemetry-lambda/issue...
[1]: https://github.com/aws-observability/aws-otel-lambda/issues/...
[2]: https://github.com/rotel-dev/rotel
This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.
I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.
If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now.
The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties.
Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.
Why do you even _need_ these "layers"? It's a simple RPC protocol that submits data tagged with Span and Trace IDs.
That's really all there is to it. You can just submit it directly, without involving any layers.
Because blocking on OTel data to flush before sending a response back is often unacceptable. The layers run a standalone process using the Lambda extension API so they can keep running after your function has responded to a request.
So what's the alternative then? (Genuine question, not hypothetical snark.)
There isn't really a great alternative without vendor lock-in. If you go all-in on AWS Cloudwatch/X-Ray, it's a really easy setup with low effort. If you go all-in on Datadog, it's pretty easy. But if you want to mix Sentry, Langfuse, Datadog, etc, OTel is still probably the best option. It's just a letdown that this is the best there is.
I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.
- Paying Datadog $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$, or
- Using and configuring a suite of tools (Jaeger for tracing, Vector or Fluentd for logs, Prometeheus for metrics)
Sounds a lot like K8s. It's not a framework you use, it's a framework to build a framework on top of.
I wish the observability vendors would move to using it under the covers so it's easier to mix and match.
I wish the otel support wasn't super buggy in most of the frameworks and backends.
But then you wouldn’t be locked in!
I looked at the OTel schema generated for recording a single numerical metric. It was like 12 or 13 meta fields in addition to the actual metric fields like timestamp, metric and value.
OTel looks like something designed by a committee of committees, funded by someone who is in the business of selling cloud storage/data warehousing services.
OpenTelemtry is the perfect example of an overengineered mess.
While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped.
A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get.
I think OpenTelemetry turned metrics into a farce. In many situations even self-rolled telemetry works better even with the added stuff. The annoying thing is that OpenTelemtry is that big standard now one kind of has to to add compatibility. So please, if you write software, make sure you don't lock yourself into OTel.
I agree overall, however:
> A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards
By all metrics (hah), Prometheus is the more popular solution and is the de-facto standard, as far as I know.
It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host.
Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.
I run OSS Grafana with Loki, Prometheus, and Tempo. I use an Alloy sidecar taking in OTEL and scraping logs.feom my Go services and selfhost the stack. Once you need to scale it gets a bit more complicated but it's all still OSS.
The biggest challenge I have is that each data source needs it's own query language, which DD and the like don't. That's why at my day job they went with DD despite the costs. Still OTEL but the querying is the same. We are also looking at Dash0 but for all of my personal and consulting jobs, OSS LGTM/P works good for me.
isn't this exactly what the LGTM stack is?
Signoz?
But yes it seemed like OTel was more interested in being a spec than a tool.
There is, it's called VictoriaMetrics/Logs/Traces.
https://victoriametrics.com/
We use victoriametrics, but I believe that's just the collector side of it cuz we also query into it with grafana.
Datadog isn't just a collector, but the whole querying UI as well, right?
Yes, Grafana seems to be the best free querying frontend across multiple data sources right now. But each Victoria product has its own built-in query GUI as well.
clickstack https://clickhouse.com/clickstack
is the closest i've seen to the datadog experience
Signoz
I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasant
What about the experience did you find lacking?
That would be an essay, but in short:
Grafana is fine to get to a selfhosted basic install. But once you try to actually connect logs, metrics, traces in selfhosted context and perhaps sprinkle some otel in...that sht gets out of hand very fast. It's modular in a way that seems like a win but once you start connecting stuff it starts adding complexity not ease.
Signoz...still pretty early in exploring this and so far it's acceptable, but it too relies on a mix of query languages incl the competitors promql so some panels support it other seem to not to?
The entire thing just seems bewildering to me. Not gonna say "why is this so hard" because I genuinely thing smart people are genuinely trying here...but there result just isn't great.
Try datalust/seq
I think the industry would benefit from some general evangelism for observability. Being able to do distributed tracing was both a "well, duh" and mindblown experience when I first learned about it a decade ago. It made supporting software so much better.
OTel is a fine system for learning observability; it does an okay job of exposing capabilities given how diverse the vendor ecosystem is.
I disagree. I'm an observability geek, and OTel is... fine.
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
HN always grumbles about OTel, but I agree. It's fine, and important: https://jeremymorrell.dev/blog/opentelemetry-and-the-value-o...
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
Oh my god. A Jeremy Morrell sighting in the wild.
Every time I share your blog (and I share it a lot) I tell people:
"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".
'A practitioner's guide to wide events' especially is still my number 1.
D'aww, thank you! I'm hoping to find time to write more this year
I'd make a wager that things would go better smoother faster if folks tried more stuff, ventures forth more on their own. It's obviously not great that there's no semantic convention that's perfect and just works for everything, and yeah it takes a while. I feel like the real data I'd want is who else, how many people show up to say they've tried something. Is that happening? Whether specs are really good enough advance or not, to me, is often whether enough people have tried it to find out.
The net of this is, otel is a very flexible system you can use and adapt in all kinds of ways and while the spec is important, using the toolkit to FAFO yourself, ahead of any beaten path, should really be encouraged. That's the message I'd want to see being radiated out about otel.
We did the same for 1, we have a service that picks up messages from a queue and drops 95% of them. In Python, at least, it was really really ugly. We had the alternative of using refinery, since we're on honeycomb, but didn't want to get stuck with something that only works with one vendor.
I don't understand 2, though, that should be easy? You have access to the span and you can set its status code to 0. With the Python SDK that was trivial for us to do.
The one thing I really hate about the Python SDK is that there's no simple way to say "attach this attribute to the top level span in this app". That way, when you get the user id further down in your app, you can just annotate the top level span with it. Thankfully, honeycomb has any, any2, root, parent (that allows you to filter on any span in the trace, or on the root or parent) but that's slow and again, vendor specific.
OTel has a nice property of working, which sometimes can't be said about vendor stuff.
I think the actual APIs kinda smell at the language level, and when Honeycomb decided to lean into otel and deprecate its Python libs I was super sad, cuz HC's libs were _way_ more usable IMO. Docs are also... painful. Real painful.
I wish that I could get a Python lib which is like "here this is Otel but the config phase isn't weird, and the API just looks a bit better". One of these days.
The biggest trouble I have with Otel recently is getting fixes patched upstream in contrib. Using contribs is super dangeerous, and I would basically recommend people write their own instrumentation and treat the contrib packages as just examples of how to do it
Agreed. Otel itself is fine. The documentation is bad though and full of inconsistent best practices and examples that are flat out wrong and other things.
My life of working with it got easier when I started just looking at the actual code, using network level tools like nc/tcpdump, making extensive use of the debug exporter, and almost ignoring the docs entirely except as a basic summary of what a thing does.
It feels like OTel tried standardizing before the correct design was anywhere close to being settled. It's only time to standardize once there's consensus on all the important points, and what's left is minor details that don't matter for anything other than compatibility.
Speaking only from my experience using their rust crates, they have undergone more “code feng shui” than any of our other dependencies. They’re still 0.x and every point release seems to re-imagine things enough to break everything and require substantial rewriting. They don’t even bother describing the motivation for changes, just, you can’t use this type any more, it’s private now. You can’t configure metadata here any more, you have to do it there now. It’s been the most painful dependency of ours by far.
I also think OTel SDKs are a tad bit too prescriptive, but at the same time I can't envision what a better version would look like.
The core of logs and spans are just wide events with some inter-connections, those SDKs and OTel docs make them less obvious.
(I maintain o11ylite https://github.com/o11ylite/o11ylite)
OTel is very complicated while yeah for example datadog is just dropin. And Graylog support for OTel makes it a second class citizen in the logs (all attributes are prepended with otel_attributes_ which makes searching difficult).
Using is hard, vendors are hostile, it seems like no-one want it to be a first class citizen...
I know sadly very little about otel, it feels “heavy” in a way I am not used to, I am used to simple systems - configured and composed in a way that makes a larger system.
20 years ago, we were doing (what I think) OTel is doing: with “hit IDs” (half way between a session and a request) that were consistently applied when logging the cause a request being fired; along centralised logging and really good timekeeping. Essentially a unique identifier as a tag that followed the request as it passed through the system.
This was enough to debug basically any problem.
We could even measure the distance between requests of the same “hit” and the total wall-time before it managed to return through the load balancer, so we could track our p99 easily.
Though truthfully we didn't make pretty graphs.
I sometimes wonder what OTel gives me more than this, but I work in games now and lots of these things that work well in webdev do not apply at all to our problems.
You are essentially describing a proto-tracing system. At the risk of self-promoting twice in one comments section, I have a post walking through going from what you describe above to OTel-compatible tracing: https://jeremymorrell.dev/blog/minimal-js-tracing/
You are right that what you were doing is very similar! However standardization helps a lot here.
I've found their django instrumentation to be kinda useless for larger apps.
The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation.
There is no in-between where I can inject the functionality required in a way that is compatible with the application.
Not the OP, but turning on auto-instrumentation for a Golang app running in Kubernetes breaks the app if the app is either:
- Running an old version of Golang (older than 1.18 if memory serves), or
- has libraries that the eBPF probes don't like.
And while I like OTel, I agree with the OP that you are absolutely going deep-sea diving if you're going to do anything beyond the examples provided (which is very easy to do!)
Could you please elaborate a bit on what is not working for you?
(I haven't attempted to use opentelemetry-instrumentation-django in at least a year so my information might be dated and my memory is patchy :P)
If I recall the primary issue was the forced loading of the django settings file by otel.
I get that fully automated instrumentation should be turn-key and the current approach kinda works on basic applications.
But most production django applications are monoliths and generally larger apps. They have non-trivial configuration processes which are often multi step and source settings from multiple places.
Otel should not assume it can just randomly load a the django settings at an arbitrary time point in the startup process.
In one of our apps the MIDDLEWARE setting specifically is dynamically generated and re-ordered based on enabled features. That application's startup process also has multiple stages and the initialisation of django occurs much later, after dependant config loaders etc have been initialised.
What would allow us to integrate with opentelemetry-instrumentation-django much more easily is a set of smaller primitives that we can configure and call at the appropriate time.
opentelemetry-instrumentation-django has (had?) a lot of logic hidden inside a large "inject" function which could not easily be extracted into the constituent parts and applied in a compatible manner.
https://github.com/open-telemetry/opentelemetry-python-contr...
Thanks for the write up, appreciated. A couple of things: - users are not forced to use auto-instrumentation. People can import the Middleware and use it as they see fit. I see that the instrumentor is configuring the middleware using some private attributes, I guess that can be extracted into a public function so it would be easier to do so - speaking of the middleware, the chances that it'll become a public symbol are scarce as are the chances that the interfaces will change. So if one has some testing before going to production it should be fine
I don't understand the sentiment. OTEL is better than anything I've ever used before. Do I like every part? No. My personal no-no is the automatic instrumentation which I always bypass and just DI it myself, I don't like the "global" by default approach in Golang and I had to fight team mates who were all for using it. That said, no other observability library that I've ever used was so good overall.
The perf is meh, but tbh if you look at the kind of code we, regular developers write for work, it's probably still vastly better.
> However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on.
This is just plain wrong, binaries of the collector are shipped which are available to use straight away. You can use the builder if you want to create your own version with a selected set of components but it is no way a hard requirement.
The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage.
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone. This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
Prometheus is so easy to add and if you need more scale, there is mimir and a few other options with similar client semantics. I really can't imagine reaching for a framework APK that tries to anticipate every possible thing I would want telemtered, and is inevitably missing all the domain specific derived channels I need. Even prepackaged Prometheus exporters are usually overkill.
I’ve built custom Prometheus metrics very easily and had node exporter pick up the .prom files. Python and bash scripts reading and translating.
Node exporter runs on my Prometheus server next to Blackbox Exporter. Blackbox Exporter handles TLS expiry metrics.
Hard Agree on Prometheus. And esp on the complexity - OTel is dizzyingly complex. You can get started ASAP on Prometheus whereas you get lost in analysis-paralysis when dealing with OTel.
OTEL metrics are a bit awkward, but they work just fine with Prometheus.
Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.
What, so people don't like OTel, but they like Jaeger, which implements an OTel spec? (I'm a noob to this subject, if that wasn't obvious.)
Jaeger doesn't really implement an otel spec - Otel wrapped itself around Jaeger.
Yep.
Kinda like people hating Obamacare but loving the ACA.
Jaeger does not implement all the OTEL features, though. It's specifically focused on traces rather than metrics.
I believe you can still use Zipkin with Jaeger
And Vector for logs, which is also freaking great.
I wish otel was never there. It is badly designed abstraction and due to otel the code gets very very messy and bad.
Skill issue
Premature instrumentation is the root of all evil. And the source of a significant part of AWS revenue. It should not cost more to monitor an app then run it.
(TFA author is not me.)
I have always been turned off to attempt to use OTel by the feeling that it is a little bit too over-engineered a that it might be very bad in term of performance/wasted network traffic when you see the data structure that it is using.
It is crazy to me how often people don't grok how to design software well.
1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?
2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?
If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.
otel is amazing and when thoughtfully instrumented, turns out, you can opt out of auto instrumentation btw, it provides a standard that is useful, well maintained, portable to enterprise or self hosted.
It’s modular, but the author is appraising the Ruby shortcomings as a problem while also saying they unfortunately don’t have time to contribute because, you know, they can’t “join the calls” lol
We just got a CTO who loves Ruby and guess what I’m about to do: use AI to fill in the Ruby gaps and open a PR and work a weekend or two and see if they like it and then you won’t write any more articles disparaging a project that I personally love.
It saves our company AT LEAST 10k a month vs having datadog / splunk / enterprise-y bullcrap. You seem so educated, why not roll up your sleeves instead of patronizing the hard working people that make the project work with your “if it were me” just go ahead and say it in their forums.
And, if you work for your paycheck, you’re using an agent. A mature project like Otel? Shit. That’s easy-peasy to feed into an agent, so what’s what is actually the problem? Take the time to learn and help them out if it bothers you so much you want to share it with the world!
Isn’t every single “problem” found in every large and successful open sourced framework?
Throwing out a baity framing like it’s some kind of project going wrong and then kind of just ending the article without making any sort of judgement on where this all leads, proceeding to post on hackernews.. bait!
It’s a cloud native project that is not owned by any company. That’s so rare and worth an article to celebrate open source! What a privilege to stand on the shoulders of giants!
> So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks.
“dizzying” — so I’m lost, did the author remmeber the scope of the project before they started making judgements about it?
And calling the attention of hackernews here: what’s the alternative? Oh that’s right, there isn’t one. Because this is a wag my finger article for attention and aggregating the author on a developer channel to boost their presence. Lame.
(Thumbs down)
[dead]
[flagged]