An Ajax-Include Pattern for Modular Content

Posted by Scott on 03/28/2012

Topics:

While developing the front-end of the BostonGlobe.com site last fall, one of the toughest challenges we faced was delivering roughly the same content to all devices (and connection speeds) while ensuring the most important content on a page was usable as soon as possible. We approached this challenge with a variety of techniques, such as only loading the most essential JavaScript up-front (weighing roughly 4-6kb tops) and lazy loading the rest, dynamically injecting advertisements, and loading "nice-to-have" content via JavaScript—all after the initial content was delivered and usable.

Bullet-proofing our Content Includes

The idea of conditionally or lazy loading content is not a new one; indeed, Jeremy Keith covered a similar pattern last year, and many sites already employ similar techniques. These techniques help a great deal when optimizing a page, but—particularly when dealing with conditionally loading HTML content—we need to be careful to use them in ways that do not introduce barriers to universal accessibility.

In order to lazy load nonessential content on BostonGlobe.com without introducing access barriers, we developed a simple, markup-driven “Ajax-include” pattern that enhanced certain links on a page by including a fragment of content from a URL related to their linked resource. For example, the header of BostonGlobe.com contained a link to the Weather section of the website, and that same link is also used as a marker for including a fragment of content from that section that is used to create a weather widget.

Initial State:

initial Boston Globe weather widget

Expanded, Ajax-enhanced state:

expanded Boston Globe weather widget

With this pattern in place, we found a nice balance between loading performance and accessibility: the content begins lightweight, and accessible without Javascript, and it is enhanced into a richer experience on capable devices—all without holding up the initial page load.

From a technical standpoint, the approach was simple: any element in the page could reference an external HTML fragment by URL through a small set of predefined HTML5 data- attributes; our JavaScript would simply loop through those elements, Ajax-request their referenced content, and inject it into the page (either before, after, or in place of the referencing element, depending on which data- attribute was in play).

Pseudo-markup for the particular scenario above, which the JS would enhance by including the referenced fragment:

<a href=”fullpage.html” data-include=”fullpage-fragment.html”>Full Page</a>

The technique worked well, and we began using it in several places to lighten the Globe’s initial load. Around that time, we also tweeted a gist snippet of the technique as it stood (crediting Scott Gonzales for some of the initial thinking behind the idea).

A tipping point

Unfortunately, as we began to use this technique more within our templates, its effectiveness began to decrease: each Ajax-include incurred a separate HTTP request, and with more than a few includes on a page at a time, those requests often clogged the time it took for a basic experience to become enhanced. On many mobile devices, this problem could quickly become serious: you only need to read a couple Steve Souders posts to know that the number of concurrent requests allowed by devices can be quite low. With varying connection speeds, it often resulted in our "nice-to-have" content taking too long to arrive.

This limitation meant restricting our usage to a few particularly worthwhile components per page, and continuing to strive to keep the total page weight as low as possible. (Which is, of course, a good plan in any case). Still, we wondered if the concept could be extended somehow without the drawbacks.

Getting-by with a little help

Skip forward a few months to yesterday: Nicolas Gallagher and I were chatting on Twitter about dynamic content inclusion, particularly the non-essential parts:

After a bit of hacking, it appeared that our existing Ajax-include approach could be extended to handle all content includes via a single HTTP request. Not long after that, a Gitub repo was pushed, which offers the code for you to use anywhere you’d like, and perhaps help us improve too (if you please).

A Quick Demo

The page linked below contains with 3 links to hypothetical, non-essential content. Data-after attributes in the page reference content by URL as well, which is pulled in dynamically and automatically with Ajax. If you use a browser inspector tool, you’ll notice that the 3 Ajax-include’d sections of the page arrive via one request.

Demo page: http://filamentgroup.com/examples/ajax-include/demo.html

Github code repo https://github.com/filamentgroup/Ajax-Include-Pattern/

How it works

The difference between the initial approach and this single-request version is not major: instead of requesting each piece of content via separate HTTP requests, the JavaScript makes a single request to a server-side helper script (included in the code repo), telling it which pieces of content to pack into a single response back to the UI.

Technically, all this means is a PHP-or-otherwise script is configured to accept a query string of comma-separated URLs and spit out the contents of those files in a single response. To identify the separate pieces of content within the response, the PHP proxy wraps each one in an invented <entry> element with a url attribute referencing the location from which it was fetched (note: this was formerly a <page> element, but now that our demo is using quickconcat, it's using <entry>. As such, the JS is now updated to look for <entry> element elements.). When this response arrives back to the UI, the JavaScript notifies all of the Ajax-include elements on the page, and the proper subset of content for each element is injected into that element's location in the page.

The proxy, by the way, is optional; without it, the script will revert to multiple requests. Depending on the use case, this might be preferable as well. For more information on the proxy that we're using in the demo (and that is included in the code repo) check out the quickconcat project on Github.

As far as dependencies go, the current script uses jQuery, but it could probably be done pretty concisely without it (though the event handling might make for some fun workarounds). It could also easily be built without PHP. We would love to see a pull request or two with alternatives!

Using the code

To use the technique yourself, grab the code from the Github repo above and...

  1. Include jquery.js and jquery.ajaxinclude.js in your page (anywhere’s fine)
  2. Add attributes to elements in your page where nonessential fragments of content can be included from an external URL, using jQuery-api-like qualifiers like data-after, data-before, and data-replace:
    
    <h2 data-after="/related-articles/45?fragment"><a href="/related-articles/45">Related Articles</a></h2>
    
  3. When the DOM is ready, select the Ajax-include elements on the page and call the ajaxInclude method on them, passing the URL of the server-side proxy and expected query string:
    
    $( "[data-after]" ).ajaxInclude( "quickconcat.php?wrap&files=" );
    

Note: an ideal use case might have an anchor link somewhere in the page referencing a full HTML document, and a data-attribute somewhere else in the page referencing a smaller HTML fragment to include into the page (pseudo-coded above with the ?fragment in the URL). You might also employ smarter server logic to simply return the fragment when the page is requested via Ajax instead of regular HTTP, but let’s not get ahead of ourselves...

For more information on using the code, check out the project readme.

Taking this pattern further

The idea of a building websites in a more modular, container-independent manner is a hot topic, and for good reason: every day we’re dealing with more diversity across the devices that access our sites, and we want to deliver the best experience we can to all.

With the downsides of this pattern tamed, perhaps we could even start to consider ways to extend our notion of what’s essential in a webpage...

Perhaps a news article’s initial HTML delivery consists of nothing but the article itself and some links to the top-level sections of the website, while the rest of the page is deferred to Ajax-includes... or maybe a landing page consists of nothing but links to the top-level sections of a website, and then the rest of the content (headlines, article teases, etc) is included via ajax in a separate request.

In both of these examples, the page would deliver light-weight and lightning fast, allowing you to prioritize the truly essential bits that should be there at the start, and let the slow stuff come in moments after.

You might even take it farther to apply a DRY-like pattern across all pages on a site or application, where each piece of content truly has only one home, and pages can borrow freely from the content of others to create mashup pages that resemble the sorts of pages we use today. (And perhaps a slightly smarter server-side helper could find the fragments within a page by their identifiers to facilitate this....)

A related sketch from my notes during last year’s Mobilewood meetup (click image for full size):

a concept diagram of a modular webpage

Thoughts? Ideas?

Whether you do or don’t use this particular script, we’d love to hear what you think of the pattern. We’re pretty excited by the potential optimizations that could come out of it, so please drop us a note, or even fork the code and extend it further!

Book cover: Designing with Progressive Enhancement

Enjoy our blog? You'll love our book.

For info and ordering: Visit the book site

Comments

This is fascinating. One small nit. Why would you use a made up element like ? Shouldn’t you use a data- attribute? Something like <div data-role="page" id="..">content</div>.

Comment by Raymond Camden on 03/28  at  08:53 PM

Thanks Ray. Good point. The element never makes it into the DOM, so really it could be anything. That said, we haven’t tested the proxy very broadly yet, as this is sort of a first-pass idea, and it may be that a div with a data-attribute would behave better across older browsers. We’ll take a look!

Comment by Scott (Filament) on 03/28  at  08:57 PM

hinclude.js is an interesting library for handling this: http://mnot.github.com/hinclude/

Comment by Kevin Bond on 03/28  at  10:24 PM

Good thoughts. I’m doing something similar with my HTMLDecor project (http://github.com/shogun70/HTMLDecor) and it also reminds me of Mark Nottingham’s hinclude (https://github.com/mnot/hinclude).

This also seems to be the sort of thing Mike Migurski is appealing for in http://mike.teczno.com/notes/bandwidth.html

I suppose the obvious place for discussions such as this is microformats.org, but that seems to be mostly dormant these days.

Anyway, it’s all good.
Sean

Comment by Sean Hogan on 03/28  at  10:27 PM

Great concept here, I really like the ease with which this could be applied to a page.

Could there perhaps be an opportunity to publish an event when all the resources have been loaded and inserted into the DOM?  This would allow JS to hook into the loaded content for further interactivity (eg: click listeners to trigger analytics events).

By allowing the page that requested the extra content to add its own JS interactivity to it would mean that the same HTML fragment that is requested could behave in different ways depending on which page requested it.  This would be in keeping with the DRY principles you outlined above.

Great article.

Comment by Christopher Imrie on 03/29  at  03:05 PM

As much as I would love that idea and as much as i hate SEO for that reason. Wont exactly those whine about having content loaded via Javascript? Not for the pure textual content but images etc?  Any downside here?

Comment by Sam Doodle on 03/29  at  04:33 PM

This is a great idea for limiting the html to an individual request, but what about keeping css and js modular? Surely you’ll still have to load those files individually or as two big minified files with all the content in? ...which would then perhaps defeat the point of loading the content via ajax anyway?

I’m currently doing something similar to this, actually it’s the basis of a framework for which other applications will be built into, and we can notice the performance of several multiple requests taking it’s hit at times. Your suggestion would be a good approach to speed things up, but we’d still want to keep css/js modular so as not to increase the footprint every time a new module is created.

Thoughts?

Comment by Ryan Beard on 03/29  at  06:28 PM

I love the idea of the weather widget, since it enhances the link while making it still work as a normal link. For instance you could open that link in a new tab and the full weather page would be shown. Include content this way doesn’t incur an increase in the page size because you load in on demand when needed.

However, going through all the elements that are referring a partial content, and request all them at once, will make you load a lot of content that is unlikely to be used.

I would propose a different lazy-loading approach consisting on returning that partial content in the main page body in such a way that code is not parsed and resources are not requested. It would be similar to how Gmail defers JS execution http://www.stevesouders.com/blog/2009/09/26/mobile-gmail-and-async-script-loading/

This content could be rendered inside a hidden textarea whose id matches the data-<after|before|append|replace>attribute of the link. Or it could be rendered commented out inside a hidden div tag with that given id.

Thus, we are saving one request while not delaying the page load. Only the resources referred by the injected partial content will be loaded precisely in the moment when that content is being appended to the DOM tree.

Comment by Jose M. Perez on 03/29  at  10:46 PM

This is what Facebook does as part of their Big Pipe approach as well, the single request for amalgamated content items.

Comment by Jesse Beach on 03/30  at  02:49 AM

Jose: great ideas. In our case, we were less concerned with long rendering time of already-delivered HTML and more concerned with long request time due to the initial response including large amounts of “nice to have” HTML. Keeping all that data in the initial response meant a lot heavier delivery than we needed for essential content. That said, for JS, I think the gmail pattern is interesting. It still makes for a heavy page, but not blocking page load for initial JS execution is a big improvement (especially if the JS is large).

Comment by Scott (Filament) on 04/06  at  12:57 PM

Hi Ray and all, I’ve updated the demo and code repo to use our newly pushed “quickconcat” file combiner, which is hosted here: https://github.com/filamentgroup/quickconcat. It’s php-based, and we’re looking to accept other language translations too, as long as the admittedly simple API remains 1-to-1.

The only change this brought to the JS is the invented “page” elements are now “entry” elements. As such, the JS is adjusted to look for those in the Ajax response instead of page.

We’ll post about quickcontat separately soon as well.

Thanks!

Comment by Scott (Filament) on 04/06  at  01:01 PM

“Perhaps a news article’s initial HTML delivery consists of nothing but the article itself and some links to the top-level sections of the website”

In other words, “Real Content First”.

One benefit is that “PushState Assisted Navigation” doesn’t require full and trimmed versions of a page (because the page is already trim).

It can also save a bit of bandwidth if the non-page-specific content (site content) can be shared between several pages.

And it potentially allows the decisions of page layout and non-page-specific content to be deferred to within the browser. You could have the same real content but different layout and auxilliary content, depending on:

- media queries
- whether the page is loaded in an iframe (or another context?)
- user preferences

Anyways, that’s what HTMLDecor is about, which I referred to above.

Comment by Sean Hogan on 04/06  at  02:45 PM

I’ve written a ColdFusion version of quickconctact. If you want to include it, be my guest. This has NOT been heavily tested.

https://gist.github.com/2320964

Comment by Raymond Camden on 04/06  at  05:52 PM

Commenting is closed for this post.

Book cover: Designing with Progressive Enhancement

Enjoy our blog? You'll love our book.

For info and ordering: Visit the book site