Announcing Testlio: A partner for agile development teams


Today I’m pleased to announce our investment in Testlio, a mobile app testing service tackling the ever-increasing demand for world-class quality assurance. QA has for some time been an underinvested (and always underappreciated) segment of the workforce. Much of it has to do with the adoption of web-enabled technologies and processes, which downplayed the importance of QA. However, with the adoption of native mobile applications and IoT devices, we are seeing renewed demand for this category. As we dug into the investment opportunity in Testlio, the reasons became obvious.

Agile vs Mobile

Agile development and DevOps are two of the hottest terms in enterprise technology today.

Unfortunately, they’re also rapidly running into the limitations of the Web-based technology ecosystem that they emerged from. As we move into a more mobile-centric world, particularly one where a wide variety of connected devices surround us (the “Internet of Things”) and every company we interact with has a native app they want us to use, agile and DevOps are unfortunately already dated.

Agile is optimized for a Web-centric technology environment where you can push updates to your application at a high rate of speed, sometimes many times a day. If there’s a glitch in the Web app, no problem: Just apply a patch as soon as you discover it and everyone will instantly be using the fixed version.

Not so with mobile apps or IoT devices. If you push an update to your app and discover belatedly that it has a critical security flaw or another problem, your options are much more limited. You may have to wait several weeks for a company in Cupertino to approve your update, if it’s an iOS app. And even once the new version of the app is out there, it will take time for every current user to upgrade. It could be weeks or months before the flawed version is no longer a significant presence in the wild.

I recently had a discussion with a top global bank. Over half of their online transactions take place via their mobile app, which has over 30 million users. So it’s a mission-critical component of their customer strategy.

This bank, which used to do a handful of updates to their software every year, now does five or six major releases in any given month. Getting each of those releases right is critical—there’s no room for an agile, “we’ll just push an update” mentality. Decisions on major UI components can’t be relegated to A/B testing.  If there’s a problem with a mobile app, there’s egg on everyone’s face, including the CEO — the mobile app represents a brand’s direct interface to its consumers.

The bank is not alone. For every major enterprise with an app—and that’s virtually every major consumer brand—the stakes of pushing a new update are much higher than they ever were in the Web world. In many ways, today’s mobile and IoT developers need to go back to the era of client-server computing to learn some best practices. Clearly mapping out how users will experience an application, and testing it before deployment, have become critical again.

A brief history lesson

Before the internet, sometime in the late 1980’s, client-server technologies evolved as the personal computing tidal wave ran into the needs of large enterprises to centralize control and management of critical data. PCs were already in the enterprise, with employees often bringing in their own TRS-80s or IBM PCs to work on spreadsheets in Lotus 1-2-3. Computing could no longer be confined to the white-floored, air-conditioned datacenters that housed the old mainframes—but PCs also didn’t offer the power, or the security, needed for advanced corporate work. To make that tradeoff work, the client-server architecture distributed computing power between very capable servers and somewhat less capable clients.

In this environment, development was characterized by long release cycles (six to 18 months) and extensive testing. When you were shipping software on physical media (floppy discs, and later CD-ROMs) mistakes were expensive and crippling. You had to get it right.

As we moved into the Web era, we built a new model around extremely thin clients — the web browser. That necessitated the creation of application servers, connecting web pages to databases for dynamic page generation and transactions. Over a period of ten or so years (roughly 1998 to 2008) the dominant enterprise architecture shifted from client-server to three-tier architectures utilizing dynamically interpreted languages such as PHP and Python.

The browser-centric model enabled an extremely rapid, iterative development methodology, now known as agile. After all, when your application runs on a server and is accessed only through a browser, correcting mistakes is simple: You just push a hot fix and all your users are immediately using the updated application. In that environment, iteration was cheap, so it called for an approach to development that was far nimbler than the client-server era that had gone before, pushing updates to code as frequently as daily, or even many times per day.  “Move fast and break things” became the motto of the era.

The mobile upheaval

With the advent of native mobile applications, we’re moving back to an era of compiled code, through “native” apps that are distributed via app stores.

But the mobile era is only the tip of the iceberg. We’re also seeing the onslaught of the Internet of Things: Connected devices that surround us, in everything from cars to thermostats to light bulbs. All of these devices will be running code, most of it compiled, much of it mission-critical, and all of it pushed via sporadic releases rather than continuous updates.

There are currently an estimated 7.7 billion connected devices in the world—more than there are humans on the planet. Thanks to the growth of IoT, there will be almost 21 billion connected “things” online by 2020.

That means radically slower development cycles than in the Web era, because it’s harder to push updates. In the case of iOS and Android, penetration of new releases into the market can take months due to user behavior. And for IoT, upgrades are dependent on the devices being connected to the network (and power)—and, as Nest has learned, you really don’t want to screw up the upgrade of someone’s thermostat or smoke alarm.

Yet development cycles cannot afford to go back to the leisurely pace of the client-server era. Consumer and enterprise expectations are much higher, thanks to two decades of life on the Web. Apps need frequent updates to remain competitive, and also to address security vulnerabilities that can emerge with disconcerting frequency. The trick for developers is to find a new balance between rapid development are careful vetting of updates in order to avoid embarrassing and costly mistakes in deployment.

Takeaways for the future

To survive in this new era, developers need to shift their thinking beyond “agile.” Yes, frequent releases have become the norm and that’s unlikely to change. But developers need to balance the need for speed with a more careful, considered deployment process that takes into account some of what we learned in the client-server era.

Quality assurance and quality control are necessary investments.   Every release needs to go through a structured testing process where its features are tested and validated on a variety of platforms.

Tools for automating testing are critical. With the vast number of platforms that companies need to manage in the mobile era, manual testing alone will not be able to deliver the results you need. There will always be someone with an older version of Android running an odd device you haven’t heard of. Automated testing tools can help uncover incompatibilities that you might have overlooked.

Security is a particular concern. Hackers are growing increasingly adept at exploiting whatever vulnerabilities present themselves. Sometimes those vectors are through improperly designed apps, or APIs. Security testing prior to release is just as important, if not more important, than ensuring that an app or IoT device functions as expected.

As the internet scales from 5 billion to 50 billion connected devices in the next decade, we see fundamental challenges in how processes such as dev/QA/ops evolve.  We continue to seek fresh approaches to these age old problems, and are delighted to have partnered with Testlio as a starting point.