Not supposed to be a judgemental question. Not every repo has to be a labor of love for sure. But how do you get to that place where there are 120 unused dependencies? I'm sure there are many different pathways where things get a bit out of control. People doing the equivalent of pip freeze? Or (too) many cooks?
There are several methods you can end up with extra/redundant NPM libraries.
At my last company, we had a cleanup project which ended up with a similar number of libraries removed over the epoch which cleaned up+upgraded NPM dependencies.
(1) the number might be multiplied if one library is removed from each package of a monorepo. Eg. If there are 10 packages in the monrepo, then you only need to remove on average 12 NPM libraries.
(2) sometimes updating the upstream libraries to newer versions reduces the size of the upstream dependency tree. Similarly, it was more common for NPM libraries to declare a dependency, whereas more mature libraries might move those to devDependencies or peer dependencies.
(3) removing unnecessary / unused code can unblock you so you can remove a library which is no longer referenced in your repo.
(4) sometimes programmers over engineer early. Sometimes your product requirements change. Reviewing your codebase after each of these conditions allows you to decide if you still need all of the code/dependencies.
(5) as the NodeJS runtime changes, sometimes you can migrate from a 3rd party library to use the built-in tool (we did this with BluebirdJS / Promises).
Older versions of node especially had problems resolving mutual secondary dependencies efficiently, though all versions struggle to an extent. Sometimes you have to declare a dependency at the top level in order to force the most common one to be installed at the root, so the less common one gets installed in all of the places it is needed rather than sharing the single one at the root.
The likelihood you ever get back to zero copies of that module are fairly low, unless it has become persona non grata for some reason (leftpad). So it's easy to forget to back out the disambiguation entry at the top level. Especially if git blame doesn't make it dead apparent (see also people manually adding entries in non-alphabetic order so that the next 'npm install --save' rearranges five lines).
I have witnessed a lot of kinda spurious pinning going on. Or like "ok we need to fix the bounds for this transitive dependency for a bit" and then it just sticks around.
Over a decade that's once a month, which is a lot though!
I think sometimes people will hear advice like "pin your deps" and do a `pip freeze | requirements.lock.txt`, without really absorbing that pinning transitive dependencies like this is generally not what you want.
You want a lock file! But you tend to want transitive dependencies that aren't locked down to get upgraded when you upgrade your direct dependencies. But it's a subtlety that can get lost in the noise.
1- I want my dependencies to define a range for the libraries that they work with, so we don't have 20 different versions of some common library because our dependencies are hyper specific
2- I want every install of my project- be it on a dev machine or deploy machine- to have the exact same versions of dependencies, including transitive ones. I don't want to deal with bugs caused by surprise version changes
3- if I upgrade a dependency or remove it, I want the transitive dependencies managed automatically. I don't want orphaned transitive dependencies. In fact, I don't even want to think about them at all other than know that they work and aren't adding bloat or security risks.
You need a package manager with more than a passing thought for handling lock files. For the longest time, npm wasn't it. I'd argue that it still isn't, because "npm install" should NOT be the command used for both "set up a project for the first time" and "add a new package". In the first case, I want a reproducible, deterministic result of a known state. In the second case, I want to modify the dependency graph, producing a new state.
PSA: pnpm is fundamentally superior to npm or yarn; used properly, it provides reproducible / deterministic builds, with efficient control of the dependency graph.
No tool can prevent all footguns, but standard / idiomatic / out-of-the-box pnpm usage beats even expert use of npm or yarn. It's different, and better.
> I want a reproducible, deterministic result of a known state
Yes, that's `npm ci`.
With a fresh `npm install` you get the latest packages that satisfy dependencies in the ranges spec'd in package.json. That's basically a crap shoot because even if you specifically pin the versions of all your dependencies, chances are that at least some of your dependencies did not have the good sense to do the same for their dependencies. This is the path to hell.
With a fresh `npm ci` on the other hand, you get back exactly to the known state specified in package-lock.json, where everything is pinned, all the way down. This is the path to happiness.
The official tutorial at https://nodejs.org/en/learn/getting-started/an-introduction-... says to use `npm install` and makes no mention of `npm ci` at all. Further, the name of the command (though not the clean-install alias) shows that it was tacked on top of a fundamentally broken base.
> That's basically a crap shoot because even if you specifically pin the versions of all your dependencies, chances are that at least some of your dependencies did not have the good sense to do the same for their dependencies.
As I mentioned in my first comment, I don't really want my dependencies to pin their dependencies. I want them to specify a range they work with to minimize the number of redundant copies of common transitive dependencies.
I assure you that hearing "pin you dependencies" is not why people create a requirements file using pip freeze. It's simply because that has long been how people have said to generate a requirements file, because Python spent much of its life lacking proper project dependency management.
And now it's extremely hard to get people to stop. There's so much info out there on the internet that says to use pip freeze that people are going to continue to run into and continue to learn to use.
my usage of pip freeze has been of the "make sure to know what CI used when it build the image" (used to be "make sure the installation on the server uses the same packages that we used in CI" pre-docker-image world).
It happens when devs and businesses reach a point of tolerant negligence w.r.t their codebases. I suspect this is true for the vast majority of codebases, a "labor of love" codebase in the enterprise is a unicorn.
Remember not every project is for some amazing product, with a great agile way of working. Some projects are a withdrawals system internal to an insurance company, for example. No one gives a shit there.
Motivating for any kind of tech debt work on such a project is unlikely to be met with approval, and all the devs who really care about such things will likely have left the project long ago.
It sounds like they are counting transitive dependencies. If so, that means they deleted far fewer than 120 different lines in their own config to end up with that level of reduction.
We have a vendors package in our monorepo to centralize both tracking and versioning of our production dependencies on the frontend.
We pre build the dependencies and upload them to a CDN as well, which we can then at build replace the import URLs with the stable CDN ones, it works really well.
We have thin wrappers around each dependency to give a stable interface as well.
For what backend TypeScript projects we do have we created typed wrappers around the server framework instead of pre-building because those both deploy differently and have different constraints but this still lets us track and upgrade versions centrally while keeping APIs stable.
Using this approach has had a positive impact on performance across the board as we naturally strive to keep our dependency count low because the maintenance burden is more obvious
Just incompetent cooks. In one of my jobs, I had to cleanup a bunch of dependencies like "npm" and "install" because the guy typed "npm install" and then pasted "npm install react" or something like that. And that was not noticed for several months.
in a decade+ old fintech monorepo w/ microfrontends + BFFs + mixed stacks/design systems (react, angular, express, apollo, chakra, tw) the cruft builds up fast.
every time we migrate stuff (like the recent shift to next.js), old apps/libs don’t always get cleaned out. after years and years it’s an npm landfill. knip was basically the bulldozer.
Unfortunately, that's been the reality in most places where I was engaged to maintain a platform. Usually in-experienced developers do this and it just sticks and the list keeps growing and tech debt increasing until the whole thing is no longer maintainable or even operational in some cases.
There's never enough time to prune the software just like there's never enough time to consider the long term consequences of any decision. The incentive system is usually set up where the focus is always on the next quarter and the short term. And the result is exactly what you'd expect it to be.
I could easily imagine someone introducing a dependency because it's needed for something, then a few months later someone refactoring to remove the code that used it but not being aware that the dependency wasn't used anywhere else. Repeat this pattern a bunch, and the number of unused dependencies will start climbing. Keep in mind that the more dependencies you have, the less obvious one specific unused dependency in the config files will be (in addition to it likely being easier to introduce yet another one; it's a lot harder to object to dependency N + 1 when you didn't object to the N beforehand).
Without changing the policy around dependencies themselves for a project (which presumably would be an uphill battle for anyone feeling strongly enough about this), the only way to avoid this scenario from happening without tooling would be for people to manually verify that a dependency is still used any time they remove a usage of it. Asking the person who introduces the dependency to keep track of every time someone else introduces another usage of it to the codebase would be burdensome even if you could assume that they wouldn't ever switch jobs/stop contributing to the open source project/take vacations when code is merged in their absence, and realistically I doubt that asking people to notice whenever they remove a use of a dependency and manually verify that the dependency is still used elsewhere would be particularly effective. I don't think there's any way to make this obvious in the absence of tooling, which unfortunately isn't super common in my experience even in projects that make ample use of linting other static checks in CI.
I don't totally agree with the takes that this is from devs not caring about their projects or otherwise phoning it in. Often the projects I've seen struggle with things like this (and even things that are a lot more worrisome than this, like missing or flat-out wrong documentation around stuff like testing locally due to the team relying on unwritten shared knowledge of process) have developers who care deeply about what they're doing, but are stressed, burned-out, and desperately trying to tread water to avoid drowning in the sea of things they'd like to do to improve things. That's not to say that every situation like this is blameless, but unless my experience is greatly out of the ordinary, I'd think that quite a lot of us have been in or at least seen situations like this enough that we should have enough empathy not to jump to conclusions about the state of a project being a direct reflection of how much the devs care. (To be clear, I don't think this is what the parent comment I'm replying to is saying, but it does seem like overall there's a bit of a sentiment similar to this in the thread as a whole, so it feels worth calling out).
yeah you can definitely point it at lodash (or any utility lib) and it’ll surface spots where it’s imported but never really used. but for “we could just rewrite this tiny helper ourselves” type stuff, knip won’t judge code quality. it’s more about flagging dead deps/files than saying “ditch lodash.” that call still needs a human.
Not supposed to be a judgemental question. Not every repo has to be a labor of love for sure. But how do you get to that place where there are 120 unused dependencies? I'm sure there are many different pathways where things get a bit out of control. People doing the equivalent of pip freeze? Or (too) many cooks?
There are several methods you can end up with extra/redundant NPM libraries.
At my last company, we had a cleanup project which ended up with a similar number of libraries removed over the epoch which cleaned up+upgraded NPM dependencies.
(1) the number might be multiplied if one library is removed from each package of a monorepo. Eg. If there are 10 packages in the monrepo, then you only need to remove on average 12 NPM libraries.
(2) sometimes updating the upstream libraries to newer versions reduces the size of the upstream dependency tree. Similarly, it was more common for NPM libraries to declare a dependency, whereas more mature libraries might move those to devDependencies or peer dependencies.
(3) removing unnecessary / unused code can unblock you so you can remove a library which is no longer referenced in your repo.
(4) sometimes programmers over engineer early. Sometimes your product requirements change. Reviewing your codebase after each of these conditions allows you to decide if you still need all of the code/dependencies.
(5) as the NodeJS runtime changes, sometimes you can migrate from a 3rd party library to use the built-in tool (we did this with BluebirdJS / Promises).
Older versions of node especially had problems resolving mutual secondary dependencies efficiently, though all versions struggle to an extent. Sometimes you have to declare a dependency at the top level in order to force the most common one to be installed at the root, so the less common one gets installed in all of the places it is needed rather than sharing the single one at the root.
The likelihood you ever get back to zero copies of that module are fairly low, unless it has become persona non grata for some reason (leftpad). So it's easy to forget to back out the disambiguation entry at the top level. Especially if git blame doesn't make it dead apparent (see also people manually adding entries in non-alphabetic order so that the next 'npm install --save' rearranges five lines).
I have witnessed a lot of kinda spurious pinning going on. Or like "ok we need to fix the bounds for this transitive dependency for a bit" and then it just sticks around.
Over a decade that's once a month, which is a lot though!
I think sometimes people will hear advice like "pin your deps" and do a `pip freeze | requirements.lock.txt`, without really absorbing that pinning transitive dependencies like this is generally not what you want.
You want a lock file! But you tend to want transitive dependencies that aren't locked down to get upgraded when you upgrade your direct dependencies. But it's a subtlety that can get lost in the noise.
1- I want my dependencies to define a range for the libraries that they work with, so we don't have 20 different versions of some common library because our dependencies are hyper specific
2- I want every install of my project- be it on a dev machine or deploy machine- to have the exact same versions of dependencies, including transitive ones. I don't want to deal with bugs caused by surprise version changes
3- if I upgrade a dependency or remove it, I want the transitive dependencies managed automatically. I don't want orphaned transitive dependencies. In fact, I don't even want to think about them at all other than know that they work and aren't adding bloat or security risks.
You need a package manager with more than a passing thought for handling lock files. For the longest time, npm wasn't it. I'd argue that it still isn't, because "npm install" should NOT be the command used for both "set up a project for the first time" and "add a new package". In the first case, I want a reproducible, deterministic result of a known state. In the second case, I want to modify the dependency graph, producing a new state.
PSA: pnpm is fundamentally superior to npm or yarn; used properly, it provides reproducible / deterministic builds, with efficient control of the dependency graph.
> used properly
That phrase is frequently way more load-bearing than it should be. Is pnpm's default behavior the correct one? (yarn user here, but open to switching)
Yes.
No tool can prevent all footguns, but standard / idiomatic / out-of-the-box pnpm usage beats even expert use of npm or yarn. It's different, and better.
> I want a reproducible, deterministic result of a known state
Yes, that's `npm ci`.
With a fresh `npm install` you get the latest packages that satisfy dependencies in the ranges spec'd in package.json. That's basically a crap shoot because even if you specifically pin the versions of all your dependencies, chances are that at least some of your dependencies did not have the good sense to do the same for their dependencies. This is the path to hell.
With a fresh `npm ci` on the other hand, you get back exactly to the known state specified in package-lock.json, where everything is pinned, all the way down. This is the path to happiness.
The official tutorial at https://nodejs.org/en/learn/getting-started/an-introduction-... says to use `npm install` and makes no mention of `npm ci` at all. Further, the name of the command (though not the clean-install alias) shows that it was tacked on top of a fundamentally broken base.
> That's basically a crap shoot because even if you specifically pin the versions of all your dependencies, chances are that at least some of your dependencies did not have the good sense to do the same for their dependencies.
As I mentioned in my first comment, I don't really want my dependencies to pin their dependencies. I want them to specify a range they work with to minimize the number of redundant copies of common transitive dependencies.
Does anybody disable `npm install`? For yourself? For a team?
I assure you that hearing "pin you dependencies" is not why people create a requirements file using pip freeze. It's simply because that has long been how people have said to generate a requirements file, because Python spent much of its life lacking proper project dependency management.
And now it's extremely hard to get people to stop. There's so much info out there on the internet that says to use pip freeze that people are going to continue to run into and continue to learn to use.
my usage of pip freeze has been of the "make sure to know what CI used when it build the image" (used to be "make sure the installation on the server uses the same packages that we used in CI" pre-docker-image world).
Glad that we have better tooling nowadays!
It happens when devs and businesses reach a point of tolerant negligence w.r.t their codebases. I suspect this is true for the vast majority of codebases, a "labor of love" codebase in the enterprise is a unicorn.
Remember not every project is for some amazing product, with a great agile way of working. Some projects are a withdrawals system internal to an insurance company, for example. No one gives a shit there.
Motivating for any kind of tech debt work on such a project is unlikely to be met with approval, and all the devs who really care about such things will likely have left the project long ago.
It sounds like they are counting transitive dependencies. If so, that means they deleted far fewer than 120 different lines in their own config to end up with that level of reduction.
We have a vendors package in our monorepo to centralize both tracking and versioning of our production dependencies on the frontend.
We pre build the dependencies and upload them to a CDN as well, which we can then at build replace the import URLs with the stable CDN ones, it works really well. We have thin wrappers around each dependency to give a stable interface as well.
For what backend TypeScript projects we do have we created typed wrappers around the server framework instead of pre-building because those both deploy differently and have different constraints but this still lets us track and upgrade versions centrally while keeping APIs stable.
Using this approach has had a positive impact on performance across the board as we naturally strive to keep our dependency count low because the maintenance burden is more obvious
> Or (too) many cooks?
Just incompetent cooks. In one of my jobs, I had to cleanup a bunch of dependencies like "npm" and "install" because the guy typed "npm install" and then pasted "npm install react" or something like that. And that was not noticed for several months.
This is a good reason why even cursory checking for whether dependencies actually get used is valuable.
in a decade+ old fintech monorepo w/ microfrontends + BFFs + mixed stacks/design systems (react, angular, express, apollo, chakra, tw) the cruft builds up fast.
every time we migrate stuff (like the recent shift to next.js), old apps/libs don’t always get cleaned out. after years and years it’s an npm landfill. knip was basically the bulldozer.
The ecosystem is setup to encourage this type of pollution.
In my experience it’s usually engineers who run install commands until it builds locally, then check in whatever they’ve got.
I cannot imagine the work environment where that’s acceptable lmao
Unfortunately, that's been the reality in most places where I was engaged to maintain a platform. Usually in-experienced developers do this and it just sticks and the list keeps growing and tech debt increasing until the whole thing is no longer maintainable or even operational in some cases.
There's never enough time to prune the software just like there's never enough time to consider the long term consequences of any decision. The incentive system is usually set up where the focus is always on the next quarter and the short term. And the result is exactly what you'd expect it to be.
I could easily imagine someone introducing a dependency because it's needed for something, then a few months later someone refactoring to remove the code that used it but not being aware that the dependency wasn't used anywhere else. Repeat this pattern a bunch, and the number of unused dependencies will start climbing. Keep in mind that the more dependencies you have, the less obvious one specific unused dependency in the config files will be (in addition to it likely being easier to introduce yet another one; it's a lot harder to object to dependency N + 1 when you didn't object to the N beforehand).
Without changing the policy around dependencies themselves for a project (which presumably would be an uphill battle for anyone feeling strongly enough about this), the only way to avoid this scenario from happening without tooling would be for people to manually verify that a dependency is still used any time they remove a usage of it. Asking the person who introduces the dependency to keep track of every time someone else introduces another usage of it to the codebase would be burdensome even if you could assume that they wouldn't ever switch jobs/stop contributing to the open source project/take vacations when code is merged in their absence, and realistically I doubt that asking people to notice whenever they remove a use of a dependency and manually verify that the dependency is still used elsewhere would be particularly effective. I don't think there's any way to make this obvious in the absence of tooling, which unfortunately isn't super common in my experience even in projects that make ample use of linting other static checks in CI.
I don't totally agree with the takes that this is from devs not caring about their projects or otherwise phoning it in. Often the projects I've seen struggle with things like this (and even things that are a lot more worrisome than this, like missing or flat-out wrong documentation around stuff like testing locally due to the team relying on unwritten shared knowledge of process) have developers who care deeply about what they're doing, but are stressed, burned-out, and desperately trying to tread water to avoid drowning in the sea of things they'd like to do to improve things. That's not to say that every situation like this is blameless, but unless my experience is greatly out of the ordinary, I'd think that quite a lot of us have been in or at least seen situations like this enough that we should have enough empathy not to jump to conclusions about the state of a project being a direct reflection of how much the devs care. (To be clear, I don't think this is what the parent comment I'm replying to is saying, but it does seem like overall there's a bit of a sentiment similar to this in the thread as a whole, so it feels worth calling out).
Can this also detect low-hanging fruit like say, 5 instances where we can add some code and give up Lodash?
yeah you can definitely point it at lodash (or any utility lib) and it’ll surface spots where it’s imported but never really used. but for “we could just rewrite this tiny helper ourselves” type stuff, knip won’t judge code quality. it’s more about flagging dead deps/files than saying “ditch lodash.” that call still needs a human.
The constantly renamed titles and general witch hunt against clickbait is really annoying.
Because you re-read the same article? It looks like the title here is the same as the URL indicating it hasn't changed.