Archive Page 2

A number of users have reported a conflict between the latest version of WebMynd (0.9.1) which was released on addons.mozilla.org on Tuesday and various toolbars. The main symptom is that custom buttons from other addons that are usually displayed in the urlbar disappear after install.

We’ve now fixed this problem and you can download a fixed version of  0.9.1.1 by clicking here.

Prior versions of WebMynd don’t have this issue. We will release WebMynd 0.9.2 with this fix included on addons.mozilla.org next week after our testing is complete. That release will also include great new sharing features and performance improvements as well as bug fixes.

For the technically minded, the problem was caused by including the jquery 1.3.2 library in our addon overlay xul file. For some reason this causes toolbar buttons to disappear. jquery 1.6.2 works fine and we found the fix to allow us to continue using jquery 1.3.2 here:

http://chrispederick.com/forums/viewtopic.php?id=2374


As the RedesignGoogle gallery has continued to grow, we’ve been working on ways to make our tools more flexible for designers and more compelling for users. So far, RedesignGoogle has only allowed designers to modify the style sheet of the Google results page, but we’re taking that a huge step further: designers can now embed JavaScript in the page, and the truly excellent jQuery library is included in the add-on. Rather than simply reskinning Google results, designers now have the power to remake every aspect of the user experience, and we’re excited about the possibilities this opens up.

Below are a couple of simple examples (and code snippets) of what Google redesigners can create with jQuery. But first, a quick note about the RedesignGoogle competition: over the last couple of months, we’ve been running a contest to find the best user-submitted design and award the designer a MacBook Air. To make sure entrants get to make the most of the new features, we’re extending the contest deadline to January 1, 2010, and we’ll announce the winner two weeks later. You can read the full rules here.

Infinite Scroll

It makes a lot of sense for Google to only load ten results at a time — the fewer results are loaded, the faster your search — but there’s not a whole lot of sense to the way they force you to click through results pages. More than anything, it’s a vestige of last-century web design that no one’s bothered to change. What if instead of needing to click “next” to go to a separate second page results, the second group of ten results simply loaded in place when you scrolled down to the bottom of the page?

First, you need some code to detect when the user has reached the bottom, as well as a div to load the new results into:

$(document).ready(function() {
  $(window).scroll(function(){
    if ($(window).scrollTop() == $(document).height() - $(window).height()){
      loadMore();
    }
  });

  start = 0;

  $('<div id="tempresults" class="moreloaded"></div>').insertAfter("#res");
});

The code above calls the loadMore() function when the user scrolls to the bottom of the page. It also appends a new div to the results section, and initializes a global variable, “start”, which will come into play later on.

Here’s what loadMore() looks like:

function loadMore() {
  start += 10;
  var nextlink = $("#nav tr td:last a").attr('href');
  var currenturl = window.location;
  url = currenturl + '&start=' + start;
  $("#tempresults").load(url + ' #res');
  $("#tempresults").attr('id', 'oldtemp');
  $('<div id="tempresults" class="moreloaded"></div>').insertAfter(".moreloaded:last");
}

The function begins by incrementing the “start” variable by 10; the variable signifies how many results have been loaded already. The results are loaded into the tempresults div via an AJAX request, and a new tempresults div is created.

You can see infinite scroll in action by installing this design.

Find As You Type

We thought it would be cool if Google updated its results as you type, the same way that, say, iTunes does. This design will do exactly that. It works by binding two functions to the “focus” and “keypress” events on the search field:

$(".lst").bind("focus", function(e){
  var currentTime = new Date();
  thisKeystroke = currentTime.getTime();
});

$(".lst").bind("keypress", function(e){
  var currentTime = new Date();
  lastKeystroke = thisKeystroke;
  thisKeystroke = currentTime.getTime();
  timeSinceLast = thisKeystroke - lastKeystroke;
  if (timeSinceLast < 600 && searchEvent != null) {
    clearTimeout(searchEvent);
  }
  searchEvent = setTimeout("updateResults();", 600);
});

Basically, every time you enter or delete a character in the search field, the search function, updateResults(), will be told to execute in 600 milliseconds. If you press another key before the search executes, the search is canceled and a new search is scheduled. We need this to make sure we don’t send too many search requests — sending lots of requests per second will make Google think you’re an automated search robot, and it’ll temporarily shut off service.

updateResults() is pretty simple:

function updateResults() {
  var url = '/search?' + $('#tsf').serialize();
  $.get(
    url,
    {},
    function(data, textStatus) {
      $("#res").replaceWith($("#res", data));
    }
  );
}

This function uses jQuery’s built-in AJAX and form serialization tools to execute a search and insert the new results into the page.

Hopefully these examples demonstrate how easy it is to make major changes to Google’s search interaction with just a few lines of code. We’re really looking forward to seeing what the community can come up with. If you need help with the details, check out the jQuery docs. Happy coding!


One of the most common bits of feedback we’ve heard from our users is that the WebMynd add-on makes search results slower to load. Nothing is more important to search than speed,  and this has been a source of irritation for us too. So it was the first thing we set out to improve with the new version of the Firefox add-on.

We’ve totally revamped the search sidebar and moved the customizable search tools to a separate pane in your browser:

The sidebar pane automatically opens when you do a search, and hides itself when you navigate away from the search results page. Google will now be as lightning-quick as always, and your WebMynd tools will load noticeably faster as well.

Install WebMynd 0.9.1 to try out the new sidebar, and be sure to let us know what you think.


The rules for the RedesignGoogle competition are now up. You can read them here.


RedesignGoogle

16Sep09

One of our key goals at WebMynd is to give you more control over your search experience. Our first steps toward that were to outfit our add-on with our own set of customizable search interfaces. Now we’re excited to announce our next step: RedesignGoogle.

RedesignGoogle lets you completely customize the look of Google Search by installing user-submitted designs from an online gallery. A stripped-down version of the WebMynd add-on applies your installed stylesheet whenever you’re on a Google search results page. (You can enable WebMynd’s other features if you like, but they’re turned off by default.) Currently, RedesignGoogle is available for Firefox, and we’ll be releasing versions of the add-on for other browsers soon.

If you’re a designer, this is your chance to revamp the most widely used service on the web. Anything that can be manipulating using CSS can be changed with RedesignGoogle; here are a couple of examples of what’s possible. We offer a web-based CSS editor for creating and previewing your designs, but you’re free to use your preferred tools — just paste your stylesheet into the editor when you’re ready to apply it to your own searches and publish it to the Gallery.

Most exciting of all: we’ll be holding a competition to find the best user-submitted designs. The winner will receive a new MacBook Air; runners-up will get something cool too. Designs will be judged by Paul Graham, the WebMynd team, and a few others TBA. Stay tuned for more details and official rules.


Available in the new 0.9 release of the WebMynd Firefox add-on are two new experimental search interfaces, Manhattan and Osaka:

These new interfaces were designed based on some of the feedback we received on our first experimental interface, Phoenix. In contrast to Phoenix, which minimizes visual clutter by only presenting the top three results from each of your selected sources on the main screen, Manhattan and Osaka display all your results within scrollable frames in a magazine-style layout. The idea is to let you browse the most possible information with the least possible interaction.

We’re just getting started with these features, and there’s a lot of work yet to be done. But we hope that by offering new kinds of search experiences we can help improve one of the most important (and often most frustrating) aspects of using the web.

Install WebMynd 0.9 to try our new interfaces on your own searches.


We’ve been working on some exciting new features over the last few weeks. One of them is Phoenix, an experimental new search interface that offers the functionality of Google and the WebMynd sidebar with a clean and engaging look and feel.

Phoenix is the first of several new interfaces we’re working on. Our hope is that each will offer a unique experience, so that you can choose the one that’s best suited to the way you search. We think Google’s interface has grown stale from a lack of innovation and there’s a wide range of design possibilities that have yet to be explored. Our lead designer, Imran Zaidi, has written an article outlining the thinking behind Phoenix and its underlying design framework.

Phoenix is included in the new 0.8.3 release of the WebMynd extension for Firefox.


Relational databases, and the object-relational mapping layers which abstract them, are not particularly well suited to storing large blobs of data: images, videos, pictures, compressed files and so on.

Far better than streaming megabytes of binary to the database is to instead keep a reference into a separate store, better suited to the task of saving and serving files.

At WebMynd, we use SQLAlchemy as our ORM and Amazon’s Simple Storage Service (S3) to store our files. We’ve used Boto to create a convenient, transparent way to store a file in SQLAlchemy, with the actual data of the file actually residing in S3. These files can then be served directly from S3, decreasing database size and I/O load, and potentially reducing bandwidth costs.

Transparent changes to file content

Suppose the objects we wish to be backed by S3 have a content attribute, which is the file body itself. What we’re aiming for is to be able to do something like:

file = session.query(File).get(file_id)
file.content="new content"
session.save_or_update(file)
session.flush()

This can be achieved by creating a property on the SQLAlchemy model class:

    def _set_content(self, cont):
        s3     = boto.connect_s3(aws_id, aws_key)
        bucket = s3.get_bucket(s3_bucket)
        key    = bucket.get_key(self.key)
        if not key:
            key = Key(bucket=bucket, name=self.key)
        key.set_contents_from_string(cont)
        # if you want to serve files directly from S3:
        key.make_public()
    def _get_content(self):
        s3     = boto.connect_s3(aws_id, aws_key)
        bucket = s3.get_bucket(s3_bucket)
        key    = bucket.get_key(self.key)
        if not key:
            pass # complain
        else:
            return key.get_contents_as_string()
    content = property(_get_content, _set_content)

Cleaning up S3 artifacts

The task of keeping S3 synchronised with the database state seems like it would be awkward, perhaps involving database triggers and queues of reconciliation tasks. I was pleasantly surprised to find that SQLAlchemy has an excellent MapperExtension class, which gives you a bunch of hooks to hang custom code off. For example, to delete an S3 key when a SQLAlchemy File object is deleted, you would do something like:

class CleanupS3(MapperExtension):
    def after_delete(self, mapper, conn, inst):
        s3     = boto.connect_s3(aws_id, aws_key)
        bucket = s3.get_bucket(s3_bucket)
        key    = bucket.get_key(inst.key)
        if key:
            key.delete()
        else:
            pass # complain
        return orm.EXT_CONTINUE

mapper(File, file_table, extension=CleanupS3())

A script with a working example can be found here. It requires Boto, SQLAlchemy and some AWS configuration. In real-world usage, you’d want some more error-checking, handling of mime types and you may choose to stream in the file content with Boto’s set_contents_from_file method. You’ll also note that we connect to S3 for every method invocation; if you have frequent changes to file content, using a connection pool for Boto might help improve performance.


Most software projects start with a nice, clean, compartmentalised architecture, whether real or imagined. As implementation progresses, the lines between components tend to blur as unforeseen dependencies emerge and edge cases are dealt with.

However, by the time it comes to deployment, you’ll probably still have a number of separate packages, with some (hopefully acyclic) dependency graph binding them together.

At WebMynd, the web tier runs on Turbogears, a Python web framework. Turbogears is by its nature very modular, with various options for “plugging in” alternative tools and extensions, which has led us to be quite modular with our own code.

Dependencies between these packages is managed via the install_requires setuptools parameters, e.g.:

    install_requires=[
        "TurboGears",
        "SQLAlchemy",
        "MiniMock >= 1.2.2",
        "Boto >= 1.5",
        "Sphinx",
        "WMQueueLib",
        "WMModel",
    ],

Here, the “WM…” packages are internal, and we don’t really want to share them on PyPI. So how best to get them installed onto the machines where they’re required?

One option is to grab the source code directly, build and install it into place. Even if your code is in a DVCS, this process can complex, and you’re going to have to store somewhere the URLs and/or levels that each package depends on from the others. But this information is already encoded in a much more concise and flexible way: the install_requires declarations!

We’ve found it convenient to take advantage of this version-controlled dependency graph by hosting our own little package index internally. It’s nice and easy: all that’s required is some easy_install configuration like this:

[easy_install]
find_links = http://internal_server.webmynd.com/packages/

Our internal_server is only accessible from a restricted set of IPs, but you could use other security measures – I’ve just tried basic HTTP authentication and it works: just prepend username:password@ to the domain.

There are a few places you can put this configuration, but we include it in setup.cfg in all our packages, so that install dependencies just take care of themselves, with no hassle and no changes required on the machines. Installing a package is as simple as:

easy_install WMWebTier

Rather than making sure that the right source is pulled down on the right machine at the right time, now you can safely push all your good builds up onto your internal package index and trust that the client selects the right one. You’ve already encoded dependencies in your package metadata – relax and let easy_install do the hard work for you!


Last week we removed the ‘Ask Twitter’ feature from WebMynd’s interface on search results pages. The idea of the feature was that, when you search, as well as being shown results from your favourite sources on the right-hand side of the results page, you could also ask your Twitter followers for help with your search.

It seemed to be an exciting idea when we first thought of it. By putting the ability to post to Twitter right into your search workflow, we thought to improve Twitter’s utility for you and make your search more social. Early beta testing by the WebMynd team seemed to validate this – I found a great fish and chip shop in Covent Garden, London by asking my Twitter network after I had searched unsuccessfully. It would never have occurred to me to ask had the interface not been right there on the search page.

We removed this feature last week and I’ll cover the reasons. It also got me thinking in more general terms about the circumstances under which you would consider removing a feature, since this isn’t the first time we’ve done it: in early versions of WebMynd last summer had a feature allowing offline access of your web history, and a feature which let you publish parts of your web history for the world to see as you browsed.

The data looked bad

We looked at how many unique users were posting to Twitter from our interface. For a start, this number was at most 10% of the total number of new signups that day. And while we did not break down our retention stats down to that level of granularity, the numbers were low enough that we could simply observe by the individual posts that very rarely did any user post more than once from the interface. The low proportion of new users trying it could be explained by the feature as it stood only being applicable to the overlap between Twitter users and WebMynd users. But the terrible retention was more worrying.

The fact that the data looked bad after a couple of months of tracking it was a great big warning light. When thinking about the causes of the poor data, I asked myself the following questions:

Did the feature satisfy the intended use case?

While it allowed users to post a question to Twitter about their search query we had no built-in way to collate the answers. We assumed that the user, if already also a Twitter user, would have their own way to keep alerted on @replies or posts from people they followed, by for example, using TwitterFox. So we didn’t want to replicate those features in WebMynd and we had no automatic way to correlate replies to the question with the question itself.

Also, we observed that many of the users who did try out the feature appeared to be brand new Twitter users by the fact that their post from WebMynd was their first post. These users did not have many, if any followers, and probably had not yet figured out what set of habits and applications allowed them to keep track of replies.

So it seemed quite likely that the feature as it stood was not a complete enough solution.

Did we promote the benefits sufficiently?

We thought that putting the ‘Ask Twitter’ box onto our interface on Google and other major search engines would make it sufficiently high profile for a lot of our users to try out – our users saw that interface many times per day on average. But the data showed this wasn’t the case. We did not make a big splash on our website about the feature since we did not want to distract from our main use case with an experiment. We hoped that people would click on the ‘Ask Twitter’ link to open up the posting interface and so discover the feature and its usefulness.

It became clear to me that in order to really to a good test of the feature and use case, we would have to work much harder on promoting the benefits as well as making our implementation much more complete.

Could a different feature satisfy the use case better?

But that changed when we launched the WebMynd ‘Dock’ earlier this month. This allowed users to share links and post to Twitter as well as several other tools (Facebook, Delicious, Digg, Reddit, Hacker News) from a sidebar in the browser.

Within days of launching that feature, we had thousands of people actively using it, on average, 4 times per day. And while we had launched it for the purpose of sharing links as you browsed, it seemed so easy to also be able to make general posts to Twitter, including asking questions right from that interface.

For me that made the decision – we had another feature which satisfied the use case and more, where the data was showing great up-take by users. There seemed no sense in investing the effort to make the ‘Ask Twitter’ feature more complete and to promote that when the alternative feature was taking off so well.

Did the intended use case exist?

Ultimately it is possible that the ‘Ask Twitter’ feature was not well used simply because the use case simply didn’t exist. Maybe people don’t find the ability to easily ask their Twitter network questions related to their search useful. I don’t think the experiment we ran was sufficient to conclude that but it obviously was not working in the form we had originally tried.

What now?

It’s been about a week since we removed the feature, we have yet to receive a comment or complaint, and there have been no repercussions in terms of the usage of other features. So I think it is fair to conclude that it was the right choice. We should of course ask the 5 whys in our next product roadmap iteration to see what we can learn from the feature and how we might improve our decision making process.

We now have two interfaces into Twitter – the search widget that WebMynd puts on the right-hand side of Google and other search engines when the user selects it. And the Dock which allows users to post to Twitter and share links easily. I think those two features are just asking to be combined in new and interesting ways, and I very much doubt that the ‘Ask Twitter’ feature will our last experiment in this area.

In general, I don’t think we should be afraid of removing features, because if we cannot do that then we will become more reluctant to try out new ideas in the future and run the risk of them not working out. WebMynd did not start out embedding search results on the right-hand side of search engines. But when, almost in a whim, we just stuck some web history results up there to see what would happen, we discovered the most popular feature in our product. It changed our direction as a company.