This is one of my biggest pet peeves! I hate reading shell scripts which uses small options. It is okay to use short options when interactively using shell, but there is no excuse to not spend time and try converting a small options shell script to use long options if its going to be read by someone else, especially if you anyway have to write comments explaining it.
The biggest part i hate is that, some utilities don’t have long option counterpart!
Another pain is when there's no long options to use and you want to leave comments to document the short options, bash doesn't have inline comments and multiline commands break if you try to comment a line. There's awkward and verbose workarounds like https://stackoverflow.com/questions/9522631/how-to-put-a-lin... but open to tips!
On the flip side, not everyone uses GNU coreutils (though I do on every machine that runs Git), and I wouldn't have been confident that "du -c" and "du --total" are fully interchangeable prior to reading the manual just now. For Git, he could have confidently used "git --message" instead of "git -m", but then again, I'm sure it would have surprised many Git users ("is that -m?").
I'd argue that short options aren't particularly more likely to be portable. Years ago I was debugging a CI issue that turned out to be caused by a script hard-coding the short option for base64 decoding, which was not compatible across the GNU and BSD versions (one of them used -d and the other used -D, I can't remember which was which, and that's basically the same issue honestly). I was trying to figure out if I needed to check MacOS versus Linux (I think I was fixing the newly-added MacOS job at the time) when I realized that they both used --decode as the long option, which meant that the long option being used originally would have prevented the issue in the first place.
I guess my point is that I don't really care what the standard says if it doesn't cover enough of what I'm doing, and in practice, I can't recall a single time where I had a script that was incompatible with where I ran it because of a long option where a short one would have been compatible.
Yeah, we had "always use long options in scripts as self-documentation" in a style guide at my last place. (Sure they were often less familiar - "we don't expect you to know, we expect you to learn" was also developer guidance. Though I think we had one or two exceptions listed, I don't remember them off the top of my head...)
I played with making a git alternative that was streamlined according to workflow. I haven't completely abandoned it, but my core idea was that you can use ast path to compress the data further.
Like you said git is really efficient, and even though I went in with some criticism because of the work(mess) that is chromium and the deep hacking I did there to fix stuff between versions without forcing a complete rebuild
I came to the conclusion that as a compression alternative, my approach wasn't worth it. Git did a better job!
There are some attempts at this (semanticdiff, etc) and they seem very good. But I was looking for a tool that could actively edit a set portion of code, maintaining a small diff above a moving target
In that case merging from "main" starts to be extremely difficult
Git was designed with assumptions about local file storage which make it challenge when storing data in a distributed file system. We built our distributed filesystem, HopsFS, to store data in S3. To support high performance git in FUSE, our writes hit remote NVMe disk(s) and asynchronously sync to S3 (you can replicate on NVMes or just do failure recovery). Previously, we used remote NVMe disks as a write-through cache, but that doesn't work with git, S3 latency is too high.
Maybe worth adding is that those 250 files are stored as a single blob object since their content and consequently their hash is identical. So in this example, a lot of storage is actually reused.
No. You can configure the compression level but that's at far as it goes.
You can configure loose object and pack compression separately (core.compression and pack.compression) so loose object compression could be switched to an other algorithm as it's completely local[1] (likely a low-compression zstd), but pack files are part of the network exchange, so things are a bit more complicated there. Also an annoyance there is that git uses raw zlib there is no identifying header or anything.
There are old threads on the mailing list, as well as a gitlab issue.
Although you can also tune compression per pack when you create them explicitly, on $DAYJOB's repository in order to speed up gc some I have .keep packs which segregate images uncompressed in their own packs, because there's little point wasting cycles on zlib-compressing jpegs and pngs.
[1]: unless you're using the "dumb http" protocol, or performing (n)fs-mediated clones
I'm a bit surprised Github or some other big git hoster didn't push for such a protocol improvement. Since I assume that smaller and cheaper to create packs would reduce their costs noticeably.
my local experiments showed any tricky scheme I could come up beaten by plain large window zstd. both on speed and resulting size (memory needed for the window is another matter completely). zstd also was beating the git xdelta+deflate.
The most "free lunch" actually is compressing fast export (or similar format) — if the goal is packing repo as small as possible, then all the SHA in commits and tree objects are derivative incompressible data, and plain references like in fast export can even halve archive size
It can get even smaller when loose objects get packed into packfiles because of delta compression: a whole bunch of identical files can be stored in minimal overhead over storing one copy.
Right, packfile size is really what matters in the long-term. A tiny change to a file in the short term just adds the whole thing again (compressed), whereas a tiny change in a packfile shouldn't add much at all. Unless you've got very limited disk space, the long-term is the only size that matters.
> The du options are: s to summarize and b to display bytes.
Small nitpick: Use long options and your code samples become self explanatory!
This is one of my biggest pet peeves! I hate reading shell scripts which uses small options. It is okay to use short options when interactively using shell, but there is no excuse to not spend time and try converting a small options shell script to use long options if its going to be read by someone else, especially if you anyway have to write comments explaining it. The biggest part i hate is that, some utilities don’t have long option counterpart!
Another pain is when there's no long options to use and you want to leave comments to document the short options, bash doesn't have inline comments and multiline commands break if you try to comment a line. There's awkward and verbose workarounds like https://stackoverflow.com/questions/9522631/how-to-put-a-lin... but open to tips!
On the flip side, not everyone uses GNU coreutils (though I do on every machine that runs Git), and I wouldn't have been confident that "du -c" and "du --total" are fully interchangeable prior to reading the manual just now. For Git, he could have confidently used "git --message" instead of "git -m", but then again, I'm sure it would have surprised many Git users ("is that -m?").
I'd argue that short options aren't particularly more likely to be portable. Years ago I was debugging a CI issue that turned out to be caused by a script hard-coding the short option for base64 decoding, which was not compatible across the GNU and BSD versions (one of them used -d and the other used -D, I can't remember which was which, and that's basically the same issue honestly). I was trying to figure out if I needed to check MacOS versus Linux (I think I was fixing the newly-added MacOS job at the time) when I realized that they both used --decode as the long option, which meant that the long option being used originally would have prevented the issue in the first place.
That reminds me of a really horrible afternoon in pre-gpt devops when I learned to add
```bash
if [[ $OSTYPE == “darwin” ]]; then
fi Assuming you have run `brew install gnutools` on your Mac that should enable local dev for scripts that run on Linux. Apple’s bsd sed fork is strange.Not all short options are portable, but all POSIX options are short.
I guess my point is that I don't really care what the standard says if it doesn't cover enough of what I'm doing, and in practice, I can't recall a single time where I had a script that was incompatible with where I ran it because of a long option where a short one would have been compatible.
I always use long options when possible. I don’t need terseness, computers do. I wanna help future me figure out how to run something.
Yeah, we had "always use long options in scripts as self-documentation" in a style guide at my last place. (Sure they were often less familiar - "we don't expect you to know, we expect you to learn" was also developer guidance. Though I think we had one or two exceptions listed, I don't remember them off the top of my head...)
I played with making a git alternative that was streamlined according to workflow. I haven't completely abandoned it, but my core idea was that you can use ast path to compress the data further.
Like you said git is really efficient, and even though I went in with some criticism because of the work(mess) that is chromium and the deep hacking I did there to fix stuff between versions without forcing a complete rebuild
I came to the conclusion that as a compression alternative, my approach wasn't worth it. Git did a better job!
Anyway here's the attempt: https://github.com/pankon/gat
Very interesting! Maybe the AST part is more useful for diffs and merges than for compression.
There are some attempts at this (semanticdiff, etc) and they seem very good. But I was looking for a tool that could actively edit a set portion of code, maintaining a small diff above a moving target
In that case merging from "main" starts to be extremely difficult
It is not public is it?
Good call, made it public :)
Thanks!
Great name btw lol
thanks :)
Git was designed with assumptions about local file storage which make it challenge when storing data in a distributed file system. We built our distributed filesystem, HopsFS, to store data in S3. To support high performance git in FUSE, our writes hit remote NVMe disk(s) and asynchronously sync to S3 (you can replicate on NVMes or just do failure recovery). Previously, we used remote NVMe disks as a write-through cache, but that doesn't work with git, S3 latency is too high.
We built similar at Swarmfile, and it's compatible with Git too
In newer versions of Git you check this by using the new command `git repo structure`[1]
[1] https://git-scm.com/docs/git-repo#Documentation/git-repo.txt...
Maybe worth adding is that those 250 files are stored as a single blob object since their content and consequently their hash is identical. So in this example, a lot of storage is actually reused.
Does git support better compression algorithms than deflate, like zstd or brotli?
No. You can configure the compression level but that's at far as it goes.
You can configure loose object and pack compression separately (core.compression and pack.compression) so loose object compression could be switched to an other algorithm as it's completely local[1] (likely a low-compression zstd), but pack files are part of the network exchange, so things are a bit more complicated there. Also an annoyance there is that git uses raw zlib there is no identifying header or anything.
There are old threads on the mailing list, as well as a gitlab issue.
Although you can also tune compression per pack when you create them explicitly, on $DAYJOB's repository in order to speed up gc some I have .keep packs which segregate images uncompressed in their own packs, because there's little point wasting cycles on zlib-compressing jpegs and pngs.
[1]: unless you're using the "dumb http" protocol, or performing (n)fs-mediated clones
I'm a bit surprised Github or some other big git hoster didn't push for such a protocol improvement. Since I assume that smaller and cheaper to create packs would reduce their costs noticeably.
Packed objects are compressed individually. I expect the costs and benefits of compression are generally dwarfed by delta-ification.
my local experiments showed any tricky scheme I could come up beaten by plain large window zstd. both on speed and resulting size (memory needed for the window is another matter completely). zstd also was beating the git xdelta+deflate.
The most "free lunch" actually is compressing fast export (or similar format) — if the goal is packing repo as small as possible, then all the SHA in commits and tree objects are derivative incompressible data, and plain references like in fast export can even halve archive size
It can get even smaller when loose objects get packed into packfiles because of delta compression: a whole bunch of identical files can be stored in minimal overhead over storing one copy.
Right, packfile size is really what matters in the long-term. A tiny change to a file in the short term just adds the whole thing again (compressed), whereas a tiny change in a packfile shouldn't add much at all. Unless you've got very limited disk space, the long-term is the only size that matters.
> whereas a tiny change in a packfile shouldn't add much at all
Well to the extent that git is able to find a suitable delta, which is the difficult part.
I think the 250 files all containing just "foo" are stored as a single blob, too, even before creating a packfile.
I forgot about that, delta compression still can save quite a bit when you have nearly-identical files.
Exactly, a tiny change in a large file produces a full new blob, but is stored very efficiently as a delta in a packfile.
du -sh
Human readable, no explanation required.
Did you do —apparent-size?
The -b option is equivalent to '--apparent-size --block-size=1'
[flagged]
[dead]