I appreciate the note on virtual memory not being reliable. This is what Windows task manager reports by default and it's terrible. Resident size is the most reliable metric. Anything else can be wrongfully inflated by things like harmless memory mapped files that won't actually hurt anything. eg. memory map 2GB of logfiles, it'll only be paged in if reading that portion of the logfile so isn't really using memory but users look at the processes and claim "OMG why does this app use so much memory". It doesn't. It uses very little. You're reading the memory usage wrong. Chrome actually had this problem for a while and they moved away from using memory mapped files. Not because memory mapped files are a bad thing but because users will read the memory usage and go crazy over what they see even though it's not really using that much actual physical memory.
There's actually guides out there on the web that tell people judge usage by virtual memory allocated too :(. At least this article gets it right :).
If you use memory-mapped files, cached pages count towards the resident set size of your process. If you use ordinary file I/O, they don't. That behavior has amusing consequences in HPC clusters that monitor the memory usage of each job and kill them if they use more memory than they requested.
2 Settings I change on every htop which makes a HUGE difference.
1. I disable user threads. Those mostly just clutter up the htop view while providing no useful information.
2. I enable the process tree view. Very frequently, where a process comes from is much more important than other information. It also lets you see and track things like a compiler process which is eating through a bunch of files.
IMO, both these things should be the default behavior of htop.
When I read stuff like this, I come to the realization that even after daily driving Linux for 20+ years I still barely utilize its full potential. Great article.
For top if you use the > character it will sort by memory usage. I use that sometimes to figure out why my host is becoming laggy. Also you'll see swapd is taking up CPU.
For the ones that don't know "nmon", have a look at it as well! (press "h" to see the list of available monitors - press it again to make it go away, press "q" to quit)
Nowadays most of my processing happens on the GPU, so htop/top better evolve or become mostly irrelevant because a tool that will support both CPU __and__ GPU will replace it.
> Nowadays most of my processing happens on the GPU, so htop/top better evolve or become mostly irrelevant
If you’re a 3D rendering designer, an ML engineer or a crypto bro, then sure.
Here are the common workloads (for the average SWE on HN) that use CPU/RAM:
- compilation/builds
- language servers and IDEs
- test suites
- local containers
- local databases
- node tooling
- browsers
- data processing
- compression and encryption
- searching/indexing
I've relatively recently migrated over to using btop[0], and it's the kind of modern interface, useful and informative, that I needed.
As others mention it - it seems to shows the Watts used as well :) (and network, and GPU, and disks,....)
[0]: https://github.com/aristocratos/btop
Yup, btop zealot here, it even replaced iStatMenu on my brand new MacBook ..
Oh wow, now I gotta check it out.
I appreciate the note on virtual memory not being reliable. This is what Windows task manager reports by default and it's terrible. Resident size is the most reliable metric. Anything else can be wrongfully inflated by things like harmless memory mapped files that won't actually hurt anything. eg. memory map 2GB of logfiles, it'll only be paged in if reading that portion of the logfile so isn't really using memory but users look at the processes and claim "OMG why does this app use so much memory". It doesn't. It uses very little. You're reading the memory usage wrong. Chrome actually had this problem for a while and they moved away from using memory mapped files. Not because memory mapped files are a bad thing but because users will read the memory usage and go crazy over what they see even though it's not really using that much actual physical memory.
There's actually guides out there on the web that tell people judge usage by virtual memory allocated too :(. At least this article gets it right :).
If you use memory-mapped files, cached pages count towards the resident set size of your process. If you use ordinary file I/O, they don't. That behavior has amusing consequences in HPC clusters that monitor the memory usage of each job and kill them if they use more memory than they requested.
2 Settings I change on every htop which makes a HUGE difference.
1. I disable user threads. Those mostly just clutter up the htop view while providing no useful information.
2. I enable the process tree view. Very frequently, where a process comes from is much more important than other information. It also lets you see and track things like a compiler process which is eating through a bunch of files.
IMO, both these things should be the default behavior of htop.
I like the process tree view, but it stops the dynamic updates and reordering of process list.
When I read stuff like this, I come to the realization that even after daily driving Linux for 20+ years I still barely utilize its full potential. Great article.
For top if you use the > character it will sort by memory usage. I use that sometimes to figure out why my host is becoming laggy. Also you'll see swapd is taking up CPU.
I prefer using the more memory friendly M (uppercase) for memory and P (uppercase) for CPU
Anyone else feel as if HN is healing? I hope this isn't the walking-ghost era of HN.
3 AI related articles on the front page, but one is busting slop. I'm hopeful.
For the ones that don't know "nmon", have a look at it as well! (press "h" to see the list of available monitors - press it again to make it go away, press "q" to quit)
https://nmon.sourceforge.io/pmwiki.php
Especially disk throughput and I/O (keys "d" & "D") can be very useful.
I've had this bookmarked since 2016, and have referred to it many times over the years.
This is really good!
I use htop often but pretty much only use it to find pid or cpu-culprits, and never really understood the rest.
For pid I find pgrep to be the better suited tool
Very interesting topic,Cool.
s/htop/btop/
You'll be glad you did.
A bit silly that you can see a load average but not the amount of Watts used by your system.
powertop to the rescue!
https://github.com/fenrus75/powertop
Nowadays most of my processing happens on the GPU, so htop/top better evolve or become mostly irrelevant because a tool that will support both CPU __and__ GPU will replace it.
Irrelevant for you does not mean irrelevant for others
Nails and hammers are great but most of us have moved on to screws and screwdrivers.
What good does it do to stick your head in the sand?
CPUs are great for orchestrating work, GPUs are great for actually doing the work.
>CPUs are great for orchestrating work
Right, and wouldn't it be really nice if we could check on our orchestrators to make sure their not bottlenecking ops?
"How come we can fully load the GPUs?" "Idk boss, amelius said htop et al were irrelevant so we can't really investigate"
Did you write this comment using your gpu?
>"What good does it do to stick your head in the sand?"
Get the fuck out. I do write for GPU as well. One does not replace the other.
For high performance work, gpus have replaced cpus a long time ago.
Not for all definitions of HPC, though.
No one's doing database management on GPUs. No one's scraping data on GPUs. Can't run VMs on GPUs. Can't run web servers on GPU...
Stupidest comment ever.
> Nowadays most of my processing happens on the GPU, so htop/top better evolve or become mostly irrelevant
If you’re a 3D rendering designer, an ML engineer or a crypto bro, then sure.
Here are the common workloads (for the average SWE on HN) that use CPU/RAM:
Ok sure, top/htop is totally irrelevant now /s