Yes, the npm ecosystem is at fault

Posted on Tue 27 November 2018 in Programming • Tagged with npm, Javascript, open source, package manager, securityLeave a comment

Even if you are like me and don’t use any (server-side) Javascript, or node.js, or npm, it may feel like every other week there are news about yet another npm snafu.

The latest incident could be the most severe one yet, at least from the security standpoint. In a nutshell, the owner of event-stream — a popular package even by npm standards — had transferred ownership of it to another person. The new owner would then publish a new version of the package which contained a targeted backdoor, intended to steal large amounts of Bitcoin from the users of one particular wallet provider.

Quite a step up from the left-pad fiasco, isn’t it?…

Like it usually happens with any major incident, lots of people are eager to put the blame on someone, or something, as quickly as possible. Rather unsurprisingly, the original owner of event-stream came up as a pretty obvious target. Without his abdication of ownership rights, the argument goes, the entire ordeal wouldn’t have happened in the first place. More broadly, as a maintainer of a popular package, he owes it to the community to make sure it remains available and safe to use, now and for the foreseeable future.

But if those are the responsibilities, then what are the rewards?… Well, in the author’s own words, “you get literally nothing from maintaing a popular package”. Looks like once the fun of it wears off, it’s just all work and no play, for little to no tangible benefit.

This problem is of course not specific to Javascript or npm. Here’s, for example, a good take on the issue from the creator of Clojure.

However, this doesn’t mean every other package manager is equally susceptible to the sort of issues that have shaken npm time and time again. To say the same thing could’ve happened to Python/Ruby/Go/Haskell/etc. is vacuous. While factually true, it’s an instance of the Fallacy of Grey: a claim that because nothing is ideal, everything must be equally flawed.

In reality, the Javascript ecosystem facilitated by npm is singularly vulnerable to problems of this kind. It follows directly from how granular the npm packages are, and how wide and deep their dependency trees get.

Indeed, it would be quite a fascinating exercise to quantify the difference numerically, by comparing the average size of a dependency graph between npm, PyPI, Rubygems, Hoogle, and so on. It’s rather obvious npm would come as a “winner”, likely with Rust’s Cargo not very far behind.

Apparently, this not unintentional either — the node.js community seems to like it this way. But as the yesterday’s incident has exposed, this state of affairs relies on the distributed conscientiousness of very many parties — in this case, npm package authors. When one inevitably falters, the impact reaches far and wide, rippling through the massively intertwined npm registry.

While it may be satisfying, then, to blame the package owner as the immediate culprit, it’s way more productive to consider how the damage could have been mitigated.

We can, for example, look at the unimaginably dense dependency graph of npm — if anyone ever dared to visualize it in its entirety — and realize it shouldn’t really have so many edges. Had it been little sparser, more similar to the graphs of PyPI or Ruby, then removing (i.e. compromising) a single leaf node would’ve had proportionally smaller impact.

So yes, it’s true that all package managers are exposed to the risk of compromised packages. It is, however, something that can be estimated and compared — and unfortunately, it is clear that for npm, this risk is currently the largest.

Continue reading

let: binding for Knockout

Posted on Wed 18 November 2015 in Code • Tagged with Knockout, JavaScript, web frontend, data bindingLeave a comment

Knockout is a JavaScript “framework” that has always lurked in the shadows of other, more flashy ones. Even in the days of its relative novelty, the likes of Backbone or Ember had seemed to garner more widespread interest among frontend developers. Today, this hasn’t changed much, the difference being mainly that the spotlight is now taken by new actors (*cough* React *cough*).

But Knockout has a loyal following, and for good reasons. Possibly the most imporant one is why I’ve put the word “framework” in quotes. Knockout is, first and foremost, a data binding library: it doesn’t do much else besides tying DOM nodes to JavaScript objects.

This quality makes it both easy to learn and simple to integrate. In fact, it can very well live in just some small compartments of your web application, mingling easily with any server-side templating mechanism you might be using. It also interplays effortlessly with other JS libraries, and sticks very well to whatever duct tape you use to hold your frontend stack together.

Lastly, it’s also quite extensible. We can, for example, create our own bindings rather easily, extending the declarative language used to describe relationship between the data and UI.

In this post, I’m going to demonstrate this by implementing a very simple let: binding — a kind of “assignment” of an expression to a name.

From Knockout with: bluff

Out of box, Knockout proffers the with: binding, a quite similar mechanism. How it may be somewhat problematic is analogous to the widely discouraged with statement in JavaScript itself. Namely, it blends several namespaces together, making it harder to determine which object is being referred to. As a result, the code is more prone to errors.

On the other hand, freeing the developer from repeating long and complicated expressions is obviously valuable. Perhaps reducing them to nil is not the right approach, though, so how about we just shorten them to a more manageable length? Well, that’s exactly what the let: binding is meant to do:

<div data-bind="let: { addr: currentUser.personalInfo.address }">
  <p data-bind="text: addr.line1"></p>
  <!-- ko if: addr.line2 -->
    <p data-bind="text: addr.line2"></p>
  <!-- /ko -->
  <p>
    <span data-bind="text: add.city"></span>,
    <span data-bind="text: add.region"></span>
  </p>
  <p data-bind="text: add.country"></p>
</div>

Making it happen turns out to be pretty easy.

Binding contract

To define a Knockout binding, up to two things are needed. We have to specify what the library should do:

  • when the binding is first applied to a DOM node (the init method)
  • when any of the observed values changes (the update method)

Not every binding has to implement both methods. In our cases, only the init is necessary, because all we have to do is modify the binding context.

What’s that? Shortly speaking, a binding context, is an object holding all the data you can potentially bind to your DOM nodes. Think of it as a namespace, or local scope: whatever’s in there can be used directly inside data-bind attributes.

let: it go

Therefore, all that the let: binding has to do is to extend the context with a mapping passed to it as an argument. Well, almost all:

ko.bindingHandlers['let'] = {
    init: function(element, valueAccessor, allBindings, viewModel, bindingContext) {
        var innerContext = bindingContext.extend(valueAccessor);
        ko.applyBindingsToDescendants(innerContext, element);
        return { controlsDescendantBindings: true };
    }
};

The resulting innerContext is a copy of the original bindingContext, augmented with additional properties from the argument of let: (those are available through valueAccessor). Once we have it, though, we need to handle it in a little special way.

Normally, Knockout is processing all bindings recursively, pasing down the same bindingContext (which ultimately comes from the root viewModel). But since we want to locally alter the context, we also need to interrupt this regular way and take care of the lower-level DOM nodes ourselves.

This is exactly what the overly-long ko.applyBindingsToDescendants function is doing. The only caveat is that Knockout has to be told explicitly about our intentions through the return value from init. Otherwise, it would try to apply the original bindingContext recursively, which in our case would amount to applying it twice. { controlsDescendantBindings: true } prevents Knockout from doing so erroneously.

Continue reading

Optional loading of RequireJS modules

Posted on Tue 29 September 2015 in Code • Tagged with JavaScript, RequireJS, modules, Web Workers, DOM, AJAXLeave a comment

RequireJS is a module loader for JavaScript. Similar to its alternatives such as Browserify, it tries to solve an important problem on the web front(end): dividing JavaScript code into modules for better maintainability while still loading them correctly and efficiently without manual curation of the <script> tags.

Once it’s configured correctly (which can be rather non-trivial, though), modules in RequireJS are simply defined as functions that return arbitrary JavaScript objects:

define([
    'jquery',
    'lodash',

    'myapp/dep1',
    'myapp/dep2',
], function($, _, dep1, dep2) {
    // ... all of the module's code ...

    return {
        exportedSymbol1: ...,
        exportedSymbol2: ...,
    };
});

Before executing the function, RequireJS loads all the specified dependencies, repeating the process recursively and asynchronously. Return values from module functions are passed as parameters to the next module function, and thus the whole mechanism clicks, serving as a crafty workaround for the lack of proper import functionality1.

Relative failures

If, at some point in the chain, the desired module cannot be found or loaded, the entire process grinds to a halt with an error. Most of the time, this is perfectly acceptable (or even desirable) behavior, equivalent to an incorrect import statement, invalid #include directive, or similar mistake in other languages.

But there are situations when we’d like to proceed with a missing module, because the dependent code is prepared to handle it. The canonical example are Web Workers. Unlike traditional web application code, Web Worker scripts operate outside of a context of any single page, having no access to the DOM tree (because which DOM tree would it be?). Correspondingly, they have no document nor window objects in their global scope.

Unfortunately, some libraries (*cough* jQuery *cough*) require those objects as a hard (and usually implicit) dependency. This doesn’t exactly help if we’d like to use them in worker code for other features, not related to DOM. In case of jQuery, for example, it could be the API for making AJAX calls, which is still decidedly more pleasant than dealing with bare XMLHTTPRequest if we’re doing anything non-trivial.

Due to this hard dependency on DOM, however, Web Workers cannot require jQuery. No biggie, you may think: browsers supporting workers also offer an excellent, promise-based Fetch API that largely replaces the old AJAX, so we may just use it in worker code. Good thinking indeed, but it doesn’t solve the issue of sharing code between main (“UI”) part of the app and Web Workers.

Suppose you have the following dependency graph:

The common module has some logic that we’d want reused between regular <script>-based code and a Web Worker, but its dependency on jQuery makes it impossible. It would work, however, if this dependency was a soft one. If common could detect that jQuery is not available and fall back to other solutions (like the Fetch API), we would be able to require it in both execution environments.

The optional plugin

What we need, it seems, is an ability to say that some dependencies (like 'jquery') are optional. They can be loaded if they’re available but otherwise, they shouldn’t cause the whole dependency structure to crumble. RequireJS does not support this functionality by default, but it’s easy enough to add it via a plugin.

There are already several useful plugins available for RequireJS that offer some interesting features. As of this writing, however, optional module loading doesn’t seem to be among them. That’s not a big problem: rolling out our own2 plugin turns out to be relatively easy.

RequireJS plugins are themselves modules: you create them as separate JavaScript files having code wrapped in define call. They can also declare their own dependencies like any other module. The only requirement is that they export an object with certain API: at minimum, it has to include the load method. Since our optional plugin is very simple, load is in fact the only method we have to implement:

/* Skeleton of a simple RequireJS plugin module. */

define([], function() {

function load(moduleName, parentRequire, onload, config) {
    // ...
}

return {
    load: load,
};

});

As its name would hint, load carries out the actual module loading which a plugin is allowed to influence, modify, or even replace with something altogether different. In our case, we don’t want to be too invasive, but we need to detect failure in the original loading procedure and step in.

I mentioned previously that module loading is asynchronous, which JavaScript often translates to “callbacks”. Here, load receives the onload callback which we eventually need to invoke. It also get the mysterious parentRequire argument; this is simply a regular require function that’d normally be used if our plugin didn’t stand in the way.

Those two are the most important pieces of the puzzle, which overall has a pretty succinct solution:

/**
 * RequireJS plugin for optional module loading.
 */
define ([], function() {


/** Default value to return when a module failed to load. */
var DEFAULT = null;

function load(moduleName, parentRequire, onload) {
    parentRequire([moduleName], onload, function (err) {
        var failedModule = err.requireModules && requireModules[0];
        console.warn("Could not load optional module: " + failedModule);
        requirejs.undef(failedModule);

        define(failedModule, [], function() { return DEFAULT; });
        parentRequire([failedModule], onload);
    });
}

return {
    load: load,
};

});

The logic here is as follows:

  • First, try to load the module normally (via the outer parentRequire call).
  • If it succeeds, onload is called and there is nothing for us to do.
  • If it fails, we log the failedModule and cleanup some internal RequireJS state with requirejs.undef.
  • Most importantly, we define the module as a trivial shim that returns some DEFAULT (here, null).
  • As a result, when we require it again (through the inner parentRequire call), we know it’ll be loaded successfully.

Usage

Plugins in RequireJS are invoked on a per-module basis. You can specify that a certain dependency 'bar' shall be loaded through a plugin 'foo' by putting 'foo!bar' on the dependency list:

define([ 'foo!bar'], function(bar) {
    // ...
});

Both 'foo' and 'bar' represent module paths here: the first one is the path to the plugin module, while the second one is the actual dependency. In a more realistic example — like when our optional loader is involved — both of them would most likely be multi-segments paths:

define([
    'myapp/ext/require/optional!myapp/common/buttons/awesome-button',
], function(AwesomeButtonController) {
    // ...
});

As you can see, they can get pretty unreadable rather quickly. It would be better if the plugin prefix consisted of just one segment (i.e. optional!) instead. We can make that happen by adding a mapping to the RequireJS config:

requirejs.config({
    // ...
    map: {
        '*': {
            'optional': 'myapp/ext/require/optional',
        }
    }
})

With this renaming in place, the loading of non-mandatory dependencies becomes quite a bit clearer:

define([
    'optional!myapp/common/buttons/awesome-button',
], function(AwesomeButtonController) {

// ...
if (!AwesomeButtonController) {
    // ... (some work around) ...
}

});

Of course, you still need to actually code around the potential lack of an optional dependency. The if statement above is just an illustrative example; you may find it more sensible to provide some shim instead:

AwesomeButtonController = AwesomeButtonController || function() {
    // ...
};

Either way, I recommend trying to keep the size of such conditional logic to a minimum. Ideally, it should be confined to a single place, or — better yet — abstracted behind a function.


  1. An actual import statement has made it into the ES6 (ECMAScript 2015) standard but, as of this writing, no browser implements it. 

  2. Most of the code for the plugin presented here is based on this StackOverflow answer

Continue reading

mailto: URLs in JavaScript

Posted on Tue 15 September 2015 in Code • Tagged with JavaScript, email, URLLeave a comment

Though not as popular as back in the days, mailto: URLs are sometimes still the best way — and most certainly the easiest — to enable users to send emails from a web application. Typically, they are used in plain regular links that are created with the <a> tag:

<a href="mailto:john.doe@example.com">Send email</a>

Depending on the operating system and browser settings, clicking on such a link could invoke a different mail client. In the past, this could cause much confusion if the user has never used its local email application (e.g. Outlook), but nowadays browsers are increasingly doing the right thing and presenting a choice of possible options. This includes the various web applications from popular email providers. User can thus choose — say — Gmail, and have all subsequent mailto: links take them to a Compose Mail view there.

Given those developments, it’s possible that soon those URLs won’t be as quaint as they used to be ;) In any case, I think it’s worthwhile to have a few mailto: tricks up our sleeve for when they turn out to be an appropriate solution for the task at hand.

Triggering them in JavaScript

In the example above, I’ve used a mailto: URL directly in some HTML markup. For the most part, however, those URLs behave like any other and support a lot of the same functionality that http:// URLs do:

function sendEmail(address) {
    window.location.href = 'mailto:' + address;
}

Predictably, calling this sendEmail function is equivalent to clicking on a mailto: link. This additional level of indirection, however, may be enough to fool some web crawlers that look for email addresses to spam. Its effectiveness depends largely on how radically we’re going to obfuscate the address argument in the actual invocation.

More importantly, though, a function like that can be part of some more complicated logic. The only limitation is the same one that restricts where and when it’s possible to window.open another browser window. Namely, it has to be called in a direct response to a UI action triggered by the user (a button click would be a good example).

Customization

In their simplest form, mailto: URLs aren’t terribly interesting, since their only parameter is the address of a sole recipient. Fortunately, this isn’t where their capabilities end: it’s possible to customize other email fields as well, such as the subject or even body1.

Somewhat surprisingly, all this can be done in a manner that’s more typical for http:// URL: with query string parameters. Right after the recipient’s address, we can drop a question mark and add some ampersand-separated key-value pairs:

<a href="mailto:john.doe@example.com?subject=Hi&amp;body=Hello%20John!">
  Say hello to John!
</a>

A more programmatic approach is of course also possible. One thing we need to remember, though, is the proper escaping of input values in order to make them safe to put inside an URL2. The encodeURIComponent function can be used for this purpose:

function getMailtoUrl(to, subject, body) {
    var args = [];
    if (typeof subject !== 'undefined') {
        args.push('subject=' + encodeURIComponent(subject));
    }
    if (typeof body !== 'undefined') {
        args.push('body=' + encodeURIComponent(body))
    }

    var url = 'mailto:' + encodeURIComponent(to);
    if (args.length > 0) {
        url += '?' + args.join('&');
    }
    return url;
}

In theory, almost any of the numerous email headers can be specified this way. However, support may vary across browsers, especially since some of the headers (like the Bcc: one) are subject to certain security or privacy considerations. Personally, I wouldn’t recommend relying on anything besides subject= and body=, with a possible addition of cc= for the Cc:header .


  1. By RFC 2368, only text/plain email body can be specified this way. 

  2. Note that this is unrelated to the HTML escaping of & character as &amp; entity in the previous example. 

Continue reading

Interruptible fade-out of DOM elements

Posted on Sun 30 August 2015 in Code • Tagged with HTML, JavaScript, DOM, jQuery, animation, promisesLeave a comment

Here’s a nice animated effect you may have seen in some applications, quite often games. An ephemeral UI element is displayed, either in a corner or pretty prominently, and after a short while it slowly fades out to transparency over several seconds:

"Ding" with full opacity
"Ding" after fading out

If you hover your mouse over it, though, it turns opaque right away and stays like that until you move away with the cursor. It then tries to disappear again, lest you stop it once more and restore to full opacity — and so on. Leave it for a moment, though, and it eventually vanishes.

Cool, eh? Wouldn’t it be nice to have a similar mechanism for, say, notifications displayed within a web app? Sure it would be! And it doesn’t even seem all that complicated, so let’s give it a try.

A promising API

Before delving into implementation details, it’s a good practice think for a moment about an API that we’ll expose to the user. Since it looks like a relatively simple problem, we don’t need no classes or frameworks. All that’s necessary is a simple function:

function interruptibleFadeOut(element, options) {
    // ...
}

Surely, however, whoever calls it would like to be able to tell when the animation finishes (e.g. to remove the element they’ve passed from the DOM tree). Simply waiting a predefined number of seconds is not sufficient, because — as described above — the effect could be restarted arbitrary many times through mouse interaction.

We could pass a callback, of course, but the modern JavaScript way of dealing with asynchronicity and concurrency is through promises. Following this pattern, our function shall return a promise that resolves when the effect ends. Disregarding the possible user interaction for now, a minimal implementation of the fade-out alone would simply use the jQuery.fadeOut method and look something like this:

function interruptibleFadeOut(element, options) {
    if (typeof options === 'number') {
        options = {fadeOut: options};
    }

    element = $(element);
    return new Promise(function(resolve) {
        element.fadeOut(options.fadeOut, function() {
            resolve();
        });
    });
}

The constructor of Promise is admittedly a little weird, as it employs a level of abstraction and indirection that is indicative of (more) functional programming languages. You pass it a function that’s invoked immediately with an argument that itself is a function. (Actually two arguments, but we ignore the other one here). That input function, resolve, should be called whenever the new promise “completes”, in whatever sense it’s appropriate for the particular use case.

Here, we’re resolving the promise when our animation ends. This enables callers to react to this in a very straightforward way:

var elem = $('#foo');
interruptibleFadeOut(elem, 400 /* ms */).then(function() {
    elem.remove();
});

As a small aside, the Promise class is a ECMAScript 6 feature, which means it may not be available in all the browsers. The open-source Q library is often used as a compatibility shim, among many other libraries implementing the Promises/A+ recommendation.

Interaction handling

There’s little value in just wrapping an existing jQuery method, so let’s make it more interesting through mouse events. We want hovering on the element to pause the animation, while moving the cursor away should resume it:

return new Promise(function(resolve) {
    element.on('mouseenter', function() {
        element.stop(true);  // true == discard remaining animation stages
        element.css('opacity', 1.0);
    });
    element.on('mouseleave', function() {
        element.fadeOut(options.fadeOut, function() {
            resolve();
        });
    });

    element.fadeOut(options.fadeOut, function() {
        resolve();
    });
});

That will mostly work even with the above code, but there are several problems that need to be addressed immediately:

  • The mouseenter and mouseleave event handlers are should only fire while the fade-out effect is being played. Currently, they “leak”, and remain bound even after it finishes, interfering with given DOM element for as long as it exists.
  • The whole concept of hovering is contingent upon the user having a pointing device. In today’s mobile world, this excludes a significant fraction of clients. We need an alternative interaction pattern for touchscreen devices, and one possible option is restart the effect on click/tap.
  • There is a fair amount of repetition here, and it will only get worse once the previous problems are dealt with.

Cleaning up

A correct (and cleaner) version is quite a bit longer, but shouldn’t be too hard to follow:

return new Promise(function(resolve) {
    // Event handlers used by the effect.
    var events = {
        mouseenter: function() { interruptAnimation(); },
        mouseleave: function() { delayedFadeOut(); },
        click: function() {
            interruptAnimation();
            delayedFadeOut();
        },
    };
    Object.keys(events).forEach(function(name) {
        element.on(name, events[name]);
    });

    // Helper functions.
    var interruptAnimation = function() {
        element.stop(true);
        element.css('opacity', 1.0);
    };
    var delayedFadeOut = function() {
        if (options.delay) {
            element.delay(options.delay);
        }
        element.fadeOut(options.fadeOut, function() {
            Object.keys(events).forEach(function(name) {
                element.off(name, events[name]);
            });
            resolve();
        });
    };

    // Start the animation.
    if (options.fadeIn) {
        element.fadeIn(options.fadeIn);
    }
    delayedFadeOut();
});

The main part is of course the delayedFadeOut function. It is invoked right at the beginning, as well as whenever the animation has to be started again.

We also put the event handlers in a “map” (JavaScript object). This makes it easy to initially bind them to the element, and — more importantly — unbind them when the effect ends.

As a final touch, we also allow the caller to specify how long the element should remain at its full opacity before it starts to fade out (delay), and to optionally add a fadeIn stage at the very beginning to make everything look a little nicer. (I recommend no more than 300-400ms for that part, though).

Practical example

A complete code sample for the interruptibleFadeOut function can be seen in this gist. To use it in practice, you’d most likely need at least a little bit of CSS and additional JavaScript to position the element appropriately:

<div id="toast">Hello, world!</div>
#toast {
    position: fixed;
    display: none;

    /* These are arbitrary */
    top: 100px;
    width: 80%;
    height: 50px;
}
var toast = $('#toast');
toast.offset({
    left: Math.round($(window).width() - element.outerWidth() / 2),
});
toast.show();
interruptibleFadeOut(toast, { fadeIn: 200, delay: 1000, fadeOut: 3000 })
    .then(function() { toast.remove() });

Styling it to look like a level-up ding is left as an exercise for the reader :)

Continue reading