Deduplicating JS Bundles

Outline

.
├── The Duplicate Dependency Dilemma
├── Deduplication in Package Managers
│   ├── Deduplication in Yarn
│   └── Deduplication in npm
├── Deduplication in Webpack
│   ├── How Webpack Works
│   ├── Resolving to the Same Module
│   └── Custom Webpack Plugin
└── Wrapping Up

The Duplicate Dependency Dilemma

As your applications grow, duplicate dependencies tend to pile up. Each dependency listed in your package.json file has its own set of dependencies, resulting in hundreds of installed packages for a single project. In fact, it's not unusual to find that some of these dependencies are duplicated.

For example, an application with nearly 100 dependencies (excluding devDependencies) might install 305 unique libraries, or 352 if we count duplicates.

Reducing the size of your JavaScript payloads is crucial to improving Core Web Vitals metrics like First Input Delay (FID) and Interaction to Next Paint (INP). In short, smaller bundles are always better.

Deduplication in Package Managers

Imagine you've installed four libraries as your application's dependencies:

editor 1.0.0 (depends on button@1.3.0)
modal-dialog 1.2.0 (depends on button@1.4.0 and icon@2.0.0)
button 2.5.0
icon 3.0.0

Your node_modules folder would look like this:

.
└── node_modules
    ├── editor
    │   └── node_modules
    │       ├── button-1.3.0
    │       └── icon-1.1.1
    ├── modal-dialog
    │   └── node_modules
    │       ├── button-1.4.0
    │       │   └── node_modules
    │       │       └── icons-1.99.0
    │       └── icon-2.0.0
    ├── button-2.5.0
    └── icon-3.0.0

To deduplicate, you can start by modifying the lock file.

Deduplication in Yarn

Modify yarn.lock

"button@^1.3.0", "button@^1.4.0":
  version "1.4.0"
  resolved "link-to-down-button-1.4.0"
  dependencies:
    "@wtlin1228/icon" "1.99.0"

Deduplication in npm

Modify package-lock.json

"editor": {
  "version": "1.0.0",
  "resolved": "link-to-down-editor-1.0.0",
  "requires": {
    "button": "^1.3.0"
  },
  "dependencies": {
    "button": {
      "version": "1.4.0",
      "resolved": "link-to-down-button-1.4.0",
      "requires": {
        "icon": "1.99.0"
      }
    },
    "icon": {
      "version": "1.99.0",
      "resolved": "link-to-down-icon-1.99.0",
      "requires": {}
    }
  }
},

Deduplication in Webpack

Even if dependencies are "deduped" at the yarn.lock or package-lock.json level, each package with button@1.4.0 and icon@1.99.0 as dependencies will still install its own copies.

So, how does Webpack handle this situation? Unfortunately, it doesn't (there was a Webpack dedup plugin in the past, but it was removed after Webpack 2.0).

How Webpack Works

Webpack resolves dependencies using enhanced-resolve and the module resolution algorithm behind the scenes. In short, every time a file in the editor imports Button from 'button', Node.js will search for this button in the nearest node_modules folder.

This results in the following requests for buttons:

<rootPath>/node_modules/editor/node_modules/button/index.js
<rootPath>/node_modules/modal-dialog/node_modules/button/index.js
<rootPath>/node_modules/button/index.js

And for icons:

<rootPath>/node_modules/editor/node_modules/icon/index.js
<rootPath>/node_modules/modal-dialog/node_modules/button/node_modules/icon/index.js
<rootPath>/node_modules/modal-dialog/node_modules/icon/index.js
<rootPath>/node_modules/icon/index.js

Resolving to the Same Module

There are two identical button@1.4.0 and two identical icon@1.99.0. To further deduplicate them, we can take advantage of Webpack's extensive plugin interface. Webpack uses tapable internally, providing a plethora of hooks that we can tap into.

In this case, we need the normalModuleFactory.hooks.beforeResolve hook. We can replace the resolveData.request right before Webpack resolves one module.

Custom Webpack Plugin

The plugin is available here: https://github.com/wtlin1228/deduplicate-webpack-plugin

Simply add it to your Webpack plugins, and it will work seamlessly:

import DeduplicateWebpackPlugin from 'deduplicate-webpack-plugin';
 
// webpack config
const config = {
  // ...
 
  plugins: [
    // ...
 
    new DeduplicateWebpackPlugin({
      cacheDir,
      rootPath,
      packageManager: 'yarn', // or npm
    }),
 
    // ...
  ],
 
  // ...
};

Wrapping Up

Setting up the playground from scratch can be tedious. Feel free to use my repository directly if you want to give it a try: https://github.com/wtlin1228/webpack-bundle-duplications

Without the DeduplicateWebpackPlugin, Webpack bundles four icons and three buttons by default.

With the DeduplicateWebpackPlugin, there are only three icons and two buttons in our bundle.

The total bundle size of node_modules/@wtlin1228 is reduced by 20.5%. This optimization could be quite beneficial for more complex projects. Jira, for example, used this strategy to shrink their overall bundle size by approximately 10% and achieve about a 300ms improvement in Time to Interactive (TTI).

By deduplicating your JavaScript bundles, you can significantly improve your application's performance and provide a better user experience. So, give deduplication a try and see how it can optimize your project!