> I still think the world really needs a solid, immutable-first and functional-first scripting/glue language, and Crystal isn’t that.
The author might be interested in Roc [1] by Richard Feldman et al, which seems to check all the boxes, including scripting with type inference and glue with good FFI to other languages. It is still in early stage, though, since it is a very ambitious project.
We use Crystal extensively at https://heiioncall.com/ for the high-performance stuff, and it’s terrific. :)
As the author notes: the typing system is very pleasant. And the standard library is great!
Generics work fine.
The testing story is very good (almost as good as rspec).
Have some experience with the Threads but won’t comment extensively as that’s about to get some big changes in the next minor release or two.
Overall: highly productive for high-performance backend API servers and background task daemons. Way more fast and pleasant to write than Rust or Go (IMHO).
I got to work on a crystal project years ago and loved it. I was exposed to a bunch of people who were really into crystal and it made me come to the completely incorrect conclusion that crystal was up and coming and I’d be able to find more work using it.
I was wrong. I’ve never encountered it again. It was a great language, super productive. I helped some people develop a pet insurance service. So weird. I’d work with crystal again for sure.
I loved writing straight Ruby. It is by far my favorite scripting language. It makes me sad that Ruby on rails was the one big thing done in Ruby and that that gave it somewhat of a bad rap.
Were it not for Rails, most people wouldn't have heard of Ruby. It is unfortunate that Ruby doesn't get more general computing use, but Rails is really just the tip of the iceberg in a big project that uses Ruby. (You still get to do a lot of nice work that isn't Rails specific, at least if you structure your system well.)
Did you have to work around some rough edges of the language in that job?
To me it feels like it never quite reached maturity, although in a better world it would have gotten more support and conquered all territory that is now ruled by the Gophers.
I guess to make a successful programming language, you either need corporate backing or extend an existing one nowadays, otherwise it's really hard to build a stdlib of modern proportions.
> Can you magically return/break out of iterator functions like you can in Ruby?
Sure can.
$ cat break.cr
(1..).each do |x|
break if x > 2
puts x
end
$ crystal break.cr
1
2
$
The answer to Crystal questions that end in "like you can in Ruby" is usually "yes." The language really is just Ruby with static typing, warts and all. Most random snippets of Ruby will run just fine when compiled as Crystal.
I agree this is weird, but I think it's largely an analogy issue: Ruby and Crystal both explicitly document blocks as not being closures, even when they can be converted into closures. They're their own weird thing.
Crystal also documents that it will fail if the block contains return or break statements[1].
I'm a core developer of Crystal.
Note that there are two different flavours of blocks in Crystal: inlined and captured ones. `return` and `break` are supported in inlined blocks.
The bit about iterating over a directory struck me as a bit of a misunderstanding and therefore the rant following it seemed a bit misplaced. The code example makes no sense at all, so you can’t really make sense of the argument being made.
Without that, it just looks like the author has an insatiable love affair with rust and their arguments lose credibility. Of course if they’re trying to smash Rust semantics into any other language they’re going to be disappointed.
nothing against the language but it's a bit suspicious that the top sponsors include "buy Google reviews" and "buy Instagram followers" (if you navigate to the actual Crystal language site)
I've come across some legitimate open source projects where all the listed sponsors are dodgy gambling sites, I doubt they actually use the tech, it's just a way to get relatively cheap backlinks from a legitimate website.
Not sure what you're supposed to do as the project maintainer, might also be a bit strange to selectively only display some sponsors that you think are "trustworthy"?
I know it's easy to say from the outside, but a project is under no obligation to accept a sketchy sponsor. If someone offered money to post a link to something obviously illegal, it would be illegal to accept that deal. If they offer money to post a link to something dubiously legal, it is likewise dubious to accept that deal.
That they're starving open source developers doesn't remove the obligation to vet their sponsors before posting their links.
That's a bit of an indictment on how poorly projects are supported. If those sponsors feel like the eyeballs they get from supporting this project is worth it, probably at least some of the owners of those eyeballs are getting enough value from the project to overbid.
Is it an artifact of the Individual/Organisation split of OpenCollective? I see the top 10 individuals that have put in $1000 or more.
How does the team feel about selling Google reviews and Instagram followers? How do 84codes and Manas.Tech feel about being placed alongside these sketchy sponsors?
I get that money is hard to come by, but I hope you realize that you might be doing more than $1500/mo in damage to the project by accepting these ad placements.
That's an interesting form of guerrilla advertising. Find a project with decent website traffic that displays sponsors, and commit just enough to be displayed in the list of top contributors (which includes some delicious SEO-friendly backlinking juice too). $750/mo and it's already getting the companies mentioned in the currently-top comment of a thread on the front page of HN!
We've been wondering about why it's worth the money for these sponsors. The links ideally shouldn't have much effect because they're marked as `sponsored`.
Anybody got any ideas?
Either way, it's money that helps the project so we're fine with accepting it.
OO language with a static type system that feels as low-friction as a dynamic one
Sounds nice, but does the performance act like a dynamic language or is it super speedy? I'd be happy with half the speed of C.
I'm still on the lookout for a language that feels as low-friction as a dynamic language but could do some form of iterable.map in parallel on all cores transparently.
I'm a core developer of Crystal.
Crystal's dynamic nature is only in the source code. It generates highly optimized code thanks to LLVM codegen.
So performance is generally comparable to other compiled languages like Go, Rust or C.
Of course there are differences depending on specific use cases.
A workload with lots of memory allocations means a high workload for GC which could be less efficient than an implementation that uses no garbage collection. But even that depends.
> I'm still on the lookout for a language that feels as low-friction as a dynamic language but could do some form of iterable.map in parallel on all cores transparently.
If you can get past the typical "ew parenthesis instead of curly-brackets" and prefix/Polish notation, Clojure is great for this. You'd normally do `(map a-func my-elements)` but if you want the map processing to be parallel, you'd change `map` to `pmap` without any other changes needed.
I know almost nothing about Crystal, so I can't make a fair comparison between them. Julia is used in scientific computing, where parallel clustered processing is essential.
I don't like subthreads which hijack a Language A post to talk about Language B, so I'll resist making an elaborate pitch, but Julia does fit the bill for what you're on the lookout for.
Super speedy. It is in fact about half the speed of C. It has a GC so that’s probably the main slow down, but hey, no memory errors or raking your brain over a barrow checker.
The GC in particular is excessively slow. It's not a production implementation. I don't need JVM level GC engineering everywhere but it really is painful.
I'm a core developer of Crystal.
Looks like something went very wrong there. The GC may not be super optimised, but it's still practical.
I have never heard about such drastic performance issues. And I'm aware of quite a few companies who use Crystal in heavy production loads for exactly the web server + db use case without such issue reports.
So I'd suggest the root cause might be something else then the GC implementation.
It is using Boehm/libgc. Just a simple webserver should not have the described behavior. The GC is not incremental though, so having a big heap would trigger it. But that is typically not the case for the described use case. Likely the issue is with doing something that involves more allocations than necessary.
There are works in libgc to allow incremental collection, but it is not yet ready for the needs of crystal (or at least it wasn't the last time I investigated).
Conceptually, I think the correct time to do garbage collection is when your web server process is idle.
My Crystal implementation of idle-time garbage collection is here: https://github.com/compumike/idle-gc though please note that its idle detection mechanism only works for single-threaded Crystal programs.
An analogy is to imagine a single employee (thread) operating a convenience store. If there are customers waiting in the checkout line (latency-sensitive requests), the employee should priorities serving the customers ASAP! But once the line is empty (thread is idle), that might be a good time to start sweeping.
Right now, with automatic garbage collection, the employee only decides to start sweeping the entire store while in the middle of serving a customer! (Because that's when mallocs are happening, which may trigger automatic GC.) Pretty ridiculous!
With idle-time GC, the sweeping happens entirely or mostly while there are no customers waiting. This may not show latency improvements in an artificial benchmark where the system is running flat-out with a full request queue, but in the real world, it changes GC from something that happens 100% of the time in the middle of a request is being served (because that's when mallocs happen and trigger automatic GC), to something that only rarely or never happens while a request is being served.
Even better would be to combine idle-time GC with incremental GC, so that the employee could put down the broom when a new customer arrives without finishing sweeping the entire store. :)
I spent some time experimenting with the Crystal language, and it seemed very promising. But the absence of capabilities for parallel programming (as of 2022) made me abandon the attempts to learn better Crystal, as it felt pointless.
I've tried Crystal half a dozen times but always ran into some inconsistency or bug that became a show-stopper. Sigh. I hope maybe the next time it will be reliable and usable.
> Oops, looks like its tempfile lib is written in library code in the style of mktemp(3), instead of calling the OS’s mkstemp(3) or equivalent. So it just finds a filename that doesn’t exist and then opens it separately, letting an attacker create a new file which they can read at that location between those two steps. Uh, have fun with your temporary files leading to exploitable race conditions guys.
Good catch. The API should be opening the tempfile atomically (the best option), or at least should use a CSPRNG for its random suffix. It doesn’t, currently.
I don't think that complaint is even true. A quick glance at the Crystal stdlib source[1] shows that the mktmp implementation uses the O_EXCL mode, which will cause the open call to error out if the file already exists, and the implementation here will handle that and try again with a different filename.
I think the worst thing an attacker could do here if they could somehow predict the generated names before the file is created is DoS the mktmp call by racing ahead of it 100 times until it gives up.
Seconded. My use case might be running my own Ruby scripts faster, but they don't need to run any faster so it's not worth the trouble of setting up a development environment, especially one that requires a compilation step.
It would be a totally different thing if it could run Rails, which is one of the things I do in my job. The speedup would probably justify using it, especially when running tests on my laptop. My customers are perfectly happy with the performance of Rails on their servers. Same with the ones using Python with Django.
> It would be a totally different thing if it could run Rails, which is one of the things I do in my job.
At https://heiioncall.com/ we combine the two and use Rails for the human-facing parts, and Crystal for the machine-facing parts (inbound API server calls, and outbound HTTP probes). For us, the number of machine-facing requests per second is so many orders of magnitude higher than human-facing requests that Crystal's performance and lower footprint is valuable there. While on the other hand, the conveniences of Ruby on Rails are still great for a conventional human-facing web and mobile application.
Yeah,I've found that for the vast majority of Rails apps, the bottleneck is the database, not Ruby. Even a 100x speed up in the application code would have very little end user impact on performance in most cases.
I work with Ruby every day at work & have always been curious about Crystal. This was a very helpful dive into some overarching concepts & small details. Thanks for sharing!
> I still think the world really needs a solid, immutable-first and functional-first scripting/glue language, and Crystal isn’t that.
The author might be interested in Roc [1] by Richard Feldman et al, which seems to check all the boxes, including scripting with type inference and glue with good FFI to other languages. It is still in early stage, though, since it is a very ambitious project.
[1] https://www.roc-lang.org/
It seems that Clojure ticks all this boxes
JVM is a hard dealbreaker for a scripting/glue language.
The JVM is a pretty heavy dependency for scripting/glue, if just for startup times.
Babashka it when its done if it depends on libraries not in Babashka?
... or most Schemes/Racket
We use Crystal extensively at https://heiioncall.com/ for the high-performance stuff, and it’s terrific. :)
As the author notes: the typing system is very pleasant. And the standard library is great!
Generics work fine.
The testing story is very good (almost as good as rspec).
Have some experience with the Threads but won’t comment extensively as that’s about to get some big changes in the next minor release or two.
Overall: highly productive for high-performance backend API servers and background task daemons. Way more fast and pleasant to write than Rust or Go (IMHO).
I got to work on a crystal project years ago and loved it. I was exposed to a bunch of people who were really into crystal and it made me come to the completely incorrect conclusion that crystal was up and coming and I’d be able to find more work using it.
I was wrong. I’ve never encountered it again. It was a great language, super productive. I helped some people develop a pet insurance service. So weird. I’d work with crystal again for sure.
I loved writing straight Ruby. It is by far my favorite scripting language. It makes me sad that Ruby on rails was the one big thing done in Ruby and that that gave it somewhat of a bad rap.
Were it not for Rails, most people wouldn't have heard of Ruby. It is unfortunate that Ruby doesn't get more general computing use, but Rails is really just the tip of the iceberg in a big project that uses Ruby. (You still get to do a lot of nice work that isn't Rails specific, at least if you structure your system well.)
Did you have to work around some rough edges of the language in that job?
To me it feels like it never quite reached maturity, although in a better world it would have gotten more support and conquered all territory that is now ruled by the Gophers.
I guess to make a successful programming language, you either need corporate backing or extend an existing one nowadays, otherwise it's really hard to build a stdlib of modern proportions.
> Can you magically return/break out of iterator functions like you can in Ruby?
Sure can.
The answer to Crystal questions that end in "like you can in Ruby" is usually "yes." The language really is just Ruby with static typing, warts and all. Most random snippets of Ruby will run just fine when compiled as Crystal.I agree this is weird, but I think it's largely an analogy issue: Ruby and Crystal both explicitly document blocks as not being closures, even when they can be converted into closures. They're their own weird thing.
Crystal also documents that it will fail if the block contains return or break statements[1].
[1]: https://crystal-lang.org/reference/1.13/syntax_and_semantics...
I'm a core developer of Crystal. Note that there are two different flavours of blocks in Crystal: inlined and captured ones. `return` and `break` are supported in inlined blocks.
[dead]
The bit about iterating over a directory struck me as a bit of a misunderstanding and therefore the rant following it seemed a bit misplaced. The code example makes no sense at all, so you can’t really make sense of the argument being made.
Without that, it just looks like the author has an insatiable love affair with rust and their arguments lose credibility. Of course if they’re trying to smash Rust semantics into any other language they’re going to be disappointed.
nothing against the language but it's a bit suspicious that the top sponsors include "buy Google reviews" and "buy Instagram followers" (if you navigate to the actual Crystal language site)
I've come across some legitimate open source projects where all the listed sponsors are dodgy gambling sites, I doubt they actually use the tech, it's just a way to get relatively cheap backlinks from a legitimate website.
Not sure what you're supposed to do as the project maintainer, might also be a bit strange to selectively only display some sponsors that you think are "trustworthy"?
I know it's easy to say from the outside, but a project is under no obligation to accept a sketchy sponsor. If someone offered money to post a link to something obviously illegal, it would be illegal to accept that deal. If they offer money to post a link to something dubiously legal, it is likewise dubious to accept that deal.
That they're starving open source developers doesn't remove the obligation to vet their sponsors before posting their links.
That's a bit of an indictment on how poorly projects are supported. If those sponsors feel like the eyeballs they get from supporting this project is worth it, probably at least some of the owners of those eyeballs are getting enough value from the project to overbid.
Is it an artifact of the Individual/Organisation split of OpenCollective? I see the top 10 individuals that have put in $1000 or more.
I'm a core developer of Crystal.
All sponsorship contributions pay some development hours, so they're valuable to improve the project.
Sponsors presentation is only based on their _current_ contribution, not the accumulated total.
How does the team feel about selling Google reviews and Instagram followers? How do 84codes and Manas.Tech feel about being placed alongside these sketchy sponsors?
I get that money is hard to come by, but I hope you realize that you might be doing more than $1500/mo in damage to the project by accepting these ad placements.
That's an interesting form of guerrilla advertising. Find a project with decent website traffic that displays sponsors, and commit just enough to be displayed in the list of top contributors (which includes some delicious SEO-friendly backlinking juice too). $750/mo and it's already getting the companies mentioned in the currently-top comment of a thread on the front page of HN!
I'm not sure there's a lot of SEO-friendly back linking juice. They're marked as rel="nofollow sponsored"
I'm a core developer of Crystal.
We've been wondering about why it's worth the money for these sponsors. The links ideally shouldn't have much effect because they're marked as `sponsored`. Anybody got any ideas?
Either way, it's money that helps the project so we're fine with accepting it.
In my opinion, if it gets interesting projects funded I'm all for it. Otherwise, who knows if this language might get the support to thrive.
Oh... and only for 750$ a month. The other sponsors pay more, but have to appear alongside those.
What's the performance like?
The comment at the end
OO language with a static type system that feels as low-friction as a dynamic one
Sounds nice, but does the performance act like a dynamic language or is it super speedy? I'd be happy with half the speed of C.
I'm still on the lookout for a language that feels as low-friction as a dynamic language but could do some form of iterable.map in parallel on all cores transparently.
There's a blog post on paralellism in Crystal https://crystal-lang.org/2019/09/06/parallelism-in-crystal/ but that's a fair few years ago now.
I'm a core developer of Crystal. Crystal's dynamic nature is only in the source code. It generates highly optimized code thanks to LLVM codegen. So performance is generally comparable to other compiled languages like Go, Rust or C. Of course there are differences depending on specific use cases. A workload with lots of memory allocations means a high workload for GC which could be less efficient than an implementation that uses no garbage collection. But even that depends.
10x compared to Ruby for my use cases.
> I'm still on the lookout for a language that feels as low-friction as a dynamic language but could do some form of iterable.map in parallel on all cores transparently.
If you can get past the typical "ew parenthesis instead of curly-brackets" and prefix/Polish notation, Clojure is great for this. You'd normally do `(map a-func my-elements)` but if you want the map processing to be parallel, you'd change `map` to `pmap` without any other changes needed.
It's possible that the language you're looking for is Julia. https://docs.julialang.org/en/v1/manual/parallel-computing/
I know almost nothing about Crystal, so I can't make a fair comparison between them. Julia is used in scientific computing, where parallel clustered processing is essential.
I don't like subthreads which hijack a Language A post to talk about Language B, so I'll resist making an elaborate pitch, but Julia does fit the bill for what you're on the lookout for.
Super speedy. It is in fact about half the speed of C. It has a GC so that’s probably the main slow down, but hey, no memory errors or raking your brain over a barrow checker.
The GC in particular is excessively slow. It's not a production implementation. I don't need JVM level GC engineering everywhere but it really is painful.
https://en.m.wikipedia.org/wiki/Boehm_garbage_collector
What use cases do you see that in? And do you mean throughput or latency is slow?
Web server with DB backend.
Pause times can exceed 2000ms. No doubt some code changes could help but the fact remains the GC is not an optimised one.
I'm a core developer of Crystal. Looks like something went very wrong there. The GC may not be super optimised, but it's still practical. I have never heard about such drastic performance issues. And I'm aware of quite a few companies who use Crystal in heavy production loads for exactly the web server + db use case without such issue reports. So I'd suggest the root cause might be something else then the GC implementation.
What is roughly the gc algorithm?
It is using Boehm/libgc. Just a simple webserver should not have the described behavior. The GC is not incremental though, so having a big heap would trigger it. But that is typically not the case for the described use case. Likely the issue is with doing something that involves more allocations than necessary.
There are works in libgc to allow incremental collection, but it is not yet ready for the needs of crystal (or at least it wasn't the last time I investigated).
Conceptually, I think the correct time to do garbage collection is when your web server process is idle.
My Crystal implementation of idle-time garbage collection is here: https://github.com/compumike/idle-gc though please note that its idle detection mechanism only works for single-threaded Crystal programs.
An analogy is to imagine a single employee (thread) operating a convenience store. If there are customers waiting in the checkout line (latency-sensitive requests), the employee should priorities serving the customers ASAP! But once the line is empty (thread is idle), that might be a good time to start sweeping.
Right now, with automatic garbage collection, the employee only decides to start sweeping the entire store while in the middle of serving a customer! (Because that's when mallocs are happening, which may trigger automatic GC.) Pretty ridiculous!
With idle-time GC, the sweeping happens entirely or mostly while there are no customers waiting. This may not show latency improvements in an artificial benchmark where the system is running flat-out with a full request queue, but in the real world, it changes GC from something that happens 100% of the time in the middle of a request is being served (because that's when mallocs happen and trigger automatic GC), to something that only rarely or never happens while a request is being served.
Even better would be to combine idle-time GC with incremental GC, so that the employee could put down the broom when a new customer arrives without finishing sweeping the entire store. :)
See also "Idle Time Garbage Collection Scheduling" in Google Chrome (2016): PDF at https://static.googleusercontent.com/media/research.google.c...
I spent some time experimenting with the Crystal language, and it seemed very promising. But the absence of capabilities for parallel programming (as of 2022) made me abandon the attempts to learn better Crystal, as it felt pointless.
As far as a REPL for Crystal goes, the interpreter is available via "crystal i"
Also for finding shards/libraries/documentation, I'd recommend checking out https://shards.info/, https://shardbox.org/, and https://crystaldoc.info/
The awesome crystal repo is also a good: https://github.com/veelenga/awesome-crystal
I've tried Crystal half a dozen times but always ran into some inconsistency or bug that became a show-stopper. Sigh. I hope maybe the next time it will be reliable and usable.
> Oops, looks like its tempfile lib is written in library code in the style of mktemp(3), instead of calling the OS’s mkstemp(3) or equivalent. So it just finds a filename that doesn’t exist and then opens it separately, letting an attacker create a new file which they can read at that location between those two steps. Uh, have fun with your temporary files leading to exploitable race conditions guys.
Good catch. The API should be opening the tempfile atomically (the best option), or at least should use a CSPRNG for its random suffix. It doesn’t, currently.
I don't think that complaint is even true. A quick glance at the Crystal stdlib source[1] shows that the mktmp implementation uses the O_EXCL mode, which will cause the open call to error out if the file already exists, and the implementation here will handle that and try again with a different filename.
I think the worst thing an attacker could do here if they could somehow predict the generated names before the file is created is DoS the mktmp call by racing ahead of it 100 times until it gives up.
[1] https://github.com/crystal-lang/crystal/blob/release/1.13/sr...
Also, good catch! :)
Crystal is beautiful. I wish I had a use case for it.
Seconded. My use case might be running my own Ruby scripts faster, but they don't need to run any faster so it's not worth the trouble of setting up a development environment, especially one that requires a compilation step.
It would be a totally different thing if it could run Rails, which is one of the things I do in my job. The speedup would probably justify using it, especially when running tests on my laptop. My customers are perfectly happy with the performance of Rails on their servers. Same with the ones using Python with Django.
> It would be a totally different thing if it could run Rails, which is one of the things I do in my job.
At https://heiioncall.com/ we combine the two and use Rails for the human-facing parts, and Crystal for the machine-facing parts (inbound API server calls, and outbound HTTP probes). For us, the number of machine-facing requests per second is so many orders of magnitude higher than human-facing requests that Crystal's performance and lower footprint is valuable there. While on the other hand, the conveniences of Ruby on Rails are still great for a conventional human-facing web and mobile application.
Yeah,I've found that for the vast majority of Rails apps, the bottleneck is the database, not Ruby. Even a 100x speed up in the application code would have very little end user impact on performance in most cases.
I would see plenty of use cases as a Micropython substitute if at least a subset of it was ported to microcontrollers.
I work with Ruby every day at work & have always been curious about Crystal. This was a very helpful dive into some overarching concepts & small details. Thanks for sharing!
[flagged]
[flagged]
“Oh it’s nice but it’s not Rust, why everything can’t be just like Rust”
The audacity of Rustaceans is so thick.