Deduplicating JS Bundles
Introduction
![cover image](/assets/C3X4rNLj-cover.webp)
The Duplicate Dependency Dilemma
As your applications grow, duplicate dependencies tend to pile up. Each dependency listed in your package.json file has its own set of dependencies, resulting in hundreds of installed packages for a single project. In fact, it's not unusual to find that some of these dependencies are duplicated.
For example, an application with nearly 100 dependencies (excluding devDependencies) might install 305 unique libraries, or 352 if we count duplicates.
Reducing the size of your JavaScript payloads is crucial to improving Core Web Vitals metrics like First Input Delay (FID) and Interaction to Next Paint (INP). In short, smaller bundles are always better.
Deduplication in Package Managers
Imagine you've installed four libraries as your application's dependencies:
- editor 1.0.0 (depends on button@1.3.0)
- modal-dialog 1.2.0 (depends on button@1.4.0 and icon@2.0.0)
- button 2.5.0
- icon 3.0.0
Your node_modules folder would look like this:
.
└── node_modules
├── editor
│ └── node_modules
│ ├── button-1.3.0
│ └── icon-1.1.1
├── modal-dialog
│ └── node_modules
│ ├── button-1.4.0
│ │ └── node_modules
│ │ └── icons-1.99.0
│ └── icon-2.0.0
├── button-2.5.0
└── icon-3.0.0
To deduplicate, you can start by modifying the lock file.
![dependencies 1](/assets/Dj3B2KdF-dependencies-1.webp)
Deduplication in Yarn
Modify yarn.lock
"button@^1.3.0", "button@^1.4.0":
version "1.4.0"
resolved "link-to-down-button-1.4.0"
dependencies:
"@wtlin1228/icon" "1.99.0"
Deduplication in npm
Modify package-lock.json
"editor": {
"version": "1.0.0",
"resolved": "link-to-down-editor-1.0.0",
"requires": {
"button": "^1.3.0"
},
"dependencies": {
"button": {
"version": "1.4.0",
"resolved": "link-to-down-button-1.4.0",
"requires": {
"icon": "1.99.0"
}
},
"icon": {
"version": "1.99.0",
"resolved": "link-to-down-icon-1.99.0",
"requires": {}
}
}
},
Deduplication in Webpack
Even if dependencies are "deduped" at the yarn.lock
or package-lock.json
level, each package with button@1.4.0 and icon@1.99.0 as dependencies will still install its own copies.
So, how does Webpack handle this situation? Unfortunately, it doesn't (there was a Webpack dedup plugin in the past, but it was removed after Webpack 2.0).
How Webpack Works
Webpack resolves dependencies using enhanced-resolve and the module resolution algorithm behind the scenes. In short, every time a file in the editor imports Button
from 'button', Node.js will search for this button in the nearest node_modules
folder.
This results in the following requests for buttons:
<rootPath>/node_modules/editor/node_modules/button/index.js
<rootPath>/node_modules/modal-dialog/node_modules/button/index.js
<rootPath>/node_modules/button/index.js
And for icons:
<rootPath>/node_modules/editor/node_modules/icon/index.js
<rootPath>/node_modules/modal-dialog/node_modules/button/node_modules/icon/index.js
<rootPath>/node_modules/modal-dialog/node_modules/icon/index.js
<rootPath>/node_modules/icon/index.js
Resolving to the Same Module
There are two identical button@1.4.0
and two identical icon@1.99.0
. To further deduplicate them, we can take advantage of Webpack's extensive plugin interface. Webpack uses tapable internally, providing a plethora of hooks that we can tap into.
In this case, we need the normalModuleFactory.hooks.beforeResolve
hook. We can replace the resolveData.request
right before Webpack resolves one module.
![webpack](/assets/s9_um51j-webpack.webp)
Custom Webpack Plugin
The plugin is available here: https://github.com/wtlin1228/deduplicate-webpack-plugin
Simply add it to your Webpack plugins, and it will work seamlessly:
import DeduplicateWebpackPlugin from "deduplicate-webpack-plugin";
// webpack config
const config = {
// ...
plugins: [
// ...
new DeduplicateWebpackPlugin({
cacheDir,
rootPath,
packageManager: "yarn", // or npm
}),
// ...
],
// ...
};
Wrapping Up
Setting up the playground from scratch can be tedious. Feel free to use my repository directly if you want to give it a try: https://github.com/wtlin1228/webpack-bundle-duplications
Without the DeduplicateWebpackPlugin, Webpack bundles four icons and three buttons by default.
![icon bundle duplications](/assets/BFKlh5ju-icon-bundle-duplications.webp)
![button bundle duplications](/assets/DnXQPlov-button-bundle-duplications.webp)
With the DeduplicateWebpackPlugin, there are only three icons and two buttons in our bundle.
![icon bundle deduplications](/assets/CgcI6QLA-icon-bundle-deduplications.webp)
![button bundle deduplications](/assets/BtuM4Xxm-button-bundle-deduplications.webp)
The total bundle size of node_modules/@wtlin1228
is reduced by 20.5%. This optimization could be quite beneficial for more complex projects. Jira, for example, used this strategy to shrink their overall bundle size by approximately 10% and achieve about a 300ms improvement in Time to Interactive (TTI).
By deduplicating your JavaScript bundles, you can significantly improve your application's performance and provide a better user experience. So, give deduplication a try and see how it can optimize your project!