Npm scripts - running commands with globs expansion for each file

Running npm command in npm scripts with Glob expansion and executing commands multiple times for multiple matched files.

• This article is the result of: stackloverflow -> an answer i was writing -> terser -> terser issue, and question-> github terser issue -> generalization, npm, commands of one to one file process output -> Me investigating some tools -> Seeing a one good option -> testing it -> finding an issue -> fix pr (fix quick. manually tested. PR ? Wait => testing => another issue => no pure tests. => spent more time fixing the tests from no pure (randomly failing). To fully pure (consistent). => Also i got to discover more terser issues. And how to fix them.
  ◌ I set a list of useful links at the end of the story. (CMD + ⇩)

The problem

Many CLI commands that process files, don’t support running on multiple files across a whole directory, or by files selection with glob matching. As we are used to, with many CLI tools (typescript (tsc), eslint , swc, babel, …)

So How do you manage running those commands with glob matching, or cross a whole dir. And with the following requirements:

• Cross-platform option (shells do offer a lot, but not cross-platform)

Examples

terser command

• https://github.com/terser/terser/issues/544

terser file1.js file2.js -o someCombiningFile.js # terser is made with bundling in mind
terser src/file.js -o dist/file.minified.js

• Terser CLI works on one file only at a time. And it doesn’t support multiple files to multiple files outputting
• It does support multiple files to => one bundled file.
• So if you have for example a typescript project where there are multiple files. And after transpiring typescript to javascript in an dest output directory. And want to terser all of the files. How would you do it ??
  ◌ We can use a nodejs script with glob , fast-glob , … packages and programmatically iterate and execute in each iteration the terser command using exec . But what about a one-liner in npm scripts? Something so easy to set up, and very productive. And you can use in any situation.

The solution and options

Besides programmatic usage with some glob packages. The best is to have and use a CLI runner package that does glob expansion. Just like npm-run-all simplifies running multiple npm scripts ( I use npm-run-all in mostly all of my projects. It’s so productive)

Having Something similar for glob expansion-based execution is a great thing.

Packages

We can build our own. But packages like this already exist. And a quick search has shown (foreach-cli and glob-run).

• ✨ foreach-cli does support parallel execution. And does support exposing matching variables and their expansion so that you match files and you can then construct your command. Many commands require source and output. Many times you may need to change the output format ….
• ❌ While glob-run executes things in sync. And expand directly. Support more than one glob. But again no parallel . no matching vars, for managing more than one option param with dynamic vars.
  ◌ That makes it not an option. Except for a one case. I want to run in sync and only glob expansion.
• And surely for Terser, what makes sense is parallel execution.
• What about other packages?
  ◌ ❌ There is also glob-env. not sure is something maintained and so advisable to use).
  ◌ ⚠️ I tried through npm search. Those are. all the packages I was able to find and no others. At least for now.

foreach-cli seems the only good option (at least for now) ✅

• foreach-cli seems a good option given the 4k download.
• It's also very flexible with the path patterns it does expose. It works better than simple glob expansion.
• For terser it fully does the job well. And in general, it does the job for almost every case scenario. Mostly.

Example for terser

"scripts": {
  "build": "npm-run-all -s build:*",
  "build:ts": "tsc",
  "build:terser": "npm-run-all -s build:terser:*",
  "build:terser:clean": "rimraf dest/minified && mkdirp dest/minified",
  "build:terser:terse": "foreach -g \"dest/js/**/*.js\" --concurrent -x \"mkdirp dest/minified/#{reldir} && terser #{path} -o dest/minified/#{reldir}/#{name}.min#{ext} -c \"ecma=6,toplevel\" -m \"toplevel,eval\" --mkdir\"",
}

Notes:

Quotes

• within any npm script. If you you want to use double quotes “ , then you’ll have to escape them \” .
• What about inside the already escaped quotes of foreach ?

"foreach -g \"...\" -x  \"cmd <what if here i need part of the command quotes>\""

◌ with for each. You can keep using \” and not double escape like \\\” , like with some systems.

• What about commands that map files to a directory. What about nested directories ?
  ◌ Normally foreach-cli had an answer for that. And that’s the reldir placeholder, that it offer. You can see me i used it below

terser #{path} -o dest/minified/#{reldir}/#{name}.min#{ext} -c \"ecma=6,toplevel\" -m \"toplevel,eval\" --mkdir\"

◌ But mmm. There is a bug. It’s not working. mmm => reldir is wrongly set, not allowing nested dirs flow => I created a PR and fix for it. Hopefully it will be merged.
◌ https://github.com/danielkalen/foreach-cli/pull/11
◌ Full issue details
◌ The PR is interesting in the part of No pure tests and how i fixed them. Check the PR description
◌ With the PR there is no issue

Terser issues

• Only file to file processing (no globs, or dir) — already mentioned — You need to loop through them yourself — executing multiple times.
• Doesn’t create the nested directories if not already exists. => you have to run mkdirp yourself.
  ◌ You can see me i did this with

mkdirp dest/minified/#{reldir}

◌ Only works with my fix. But in case you are outside foreach-cli . Know you need to assure the folders existence by yourself. Programatically you can do it with mkdirp or fs.mkdir with recursive: true option.

The ideal tool

• For a general-purpose tool. I guess supporting more than one glob loop can be interesting.
• Personally, if I fall into a case and need that requires more than foreach-cli. i would directly implement such a CLI package. And implement glob expansion like foreach-CLI, but with multiple globs expansions looping (cartesian product). I don't see another need besides that, that foreach-cli doesn't cover it.
• Another idea would be to add helpers function that you can use in transformations within the shell strings.

Why do we care and what about programmatically?

• Productivity.
• Establish an easy fast accessible flow.
• Yes too. We can have our flow. With nodejs scripting (create a one nodejs script, using glob or fast-glob package). And we can be productive by having a fast accessible snippet. A base snippet for the core of glob looping and command execution, with full parallelism. Or synchronously. A snippet for every case. And then more snippets for every specific case (one for Terser, one for every tool u would fall on and use often).

Example:

/**
 * npm install fast-glob
 */
const { spawn } = require('child_process');
const fastGlob = require('fast-glob');
const path = require('path');
const fs = require('fs');
const fsp = fs.promises;

const rootDir = path.resolve(__dirname, '..');

/**
 * Relative path to root
 */
function root(relPath) {
  return path.resolve(rootDir, relPath);
}

/**
 * Resolve alias
 */
function r(..._path) {
  return path.resolve(..._path);
}

/**
 * File selection
 */
(async () => {
  const glob = './dest/js/**/*.js';
  // const globIgnore = '';

  const files = await fastGlob(root(glob));

  files.forEach(async (file) => {
    // Note: file is absolute (fastGlob default)
    console.log(`file: ${file}`);
    const fileParse = path.parse(file);
    const { name, ext, dir } = fileParse;
    const reldir = path.dirname(path.relative(root('dest/js'), file));

    // console.log(`reldir: ${reldir}`);

    const outFile = root(`dest/minified/${reldir}/${name}.min${ext}`);
    console.log(`outFile: ${outFile}`);

    // Pre-commands
    await fsp.mkdir(path.dirname(outFile), { recursive: true });

    // Commands
    command = `npx terser ${file} -o ${outFile} -c "ecma=6,toplevel" -m "toplevel,eval" --mkdir`;

    spawn(command, { stdio: 'inherit', shell: true });
  });
})();

For snippets productivity => check this deep great article I wrote

• To make great snippets. You have to meet the following
  ◌ good smart trigger keyword -> so that you can easily search for it and find it
  ▶︎ Even when you fully forget and come a year later (design for any future time, even after years)
  ▶ The best way is to use something like glob_cmd_run_parallel_spawn_base_placeholders
◌ Wait a second, are you serious? Yes i am! It’s like tags, for search. You can do what you want and what works for you. But know that vscode is efficient. So long is no issue. Also know that intelisense in vscode have efficient fuzzy search. You only write part of the whole Example globplac , that alone can get the long one above. Make sure you test and keep in mind what would allow you to even faster. The above long keyword. I can fast access in more than one way. I would see what works too fast.
  ◌ Create a snippet that is productive. And productive comes from speed of operation. So design them for speed of operation. Here down some ideas
  ▶︎ Add install packages commands in a top comments section
  ▶︎ Make sure you cover up the need efficiently
  ▶︎ You can have comments. Or extra things
  ▶︎ You can have multiple snippet variations. Keep same naming for trigger keyword. With more different parts at the end regex_search, regex_search_with_group, regex_search_with_lib_dragon
  ▶ ✨ Here a great one. Add critical notes that help you remember things. In the snippet above i did add // Note: file is absolute (fastGlob default). In a year i’m not sure i can remember what is the default for fastGlob. And if fastGlob default is the inverse of glob package. So when it’s absolutely critical a note comment is so powerful and valuable.
  ▶ A list of helpers that you may need or not easy to delete. (You can have a neat variant with no helpers and another with)
  ◌ You are free. Have your snippets powerful and useful to you and what you stand for. Otherwise, Packages dependencies, critical notes, and documenting. A list of helpers that you may need or not easy to delete. You can have for documentation, a section on top or bottom. Easy to delete. anything easy to delete wouldn’t take you time.
  ◌ Keep your snippets neat. Organized, production ready. And remember you can always improve them later on.
  ◌ Get familiar or design your snippet with productivity in mind. Vscode have many features. Including CMD + D for duplicate selection. You can use such things for more efficiency. Here an example

◌ Ask yourself about what other vscode features i can use to be productive.

To work greatly with snippets

• Use a snippets extension. That allows you to move fast. And quickly create them. With zero friction. And no downtime. The vscode default way can have a bit of friction. {I’ll let you search and try the best one that works for you. Otherwise, I may update this part to add some extension suggestions. I personally for the moment using this extension, am not fully 100% happy with it. I believe it could have been better. But so far it does help me be productive. I’m planning to make my own soon enough when I'll get to it}

Tests and demonstration repo

• I did create one for you to check
  ◌ https://github.com/TheMagicianDev/tutorial-running-terser-npm-glob-programmatic-file-one-to-one
  ▶ terser, npm-run-all, foreach-cli, programmatic way

foreach-cli, reldir issue to keep track

• danielkalen/foreach-cli#10
• Fix: danielkalen/foreach-cli#11 that you can check.
• The PR does explain and show a good example of no pure tests. A mistake that many may fall on.︎

npm-run-all

• fully unrelated to the topic. But one of those awesome runners. 2m+ weekly downloads …
• If you never used it before. I suggest you check it out. It is so neat and productive. I picked it up years ago. And it’s part of all my tasks setup.