The computing world is changing rapidly, and the change may be most obvious in the area of software testing. Internet commerce has created complexities for testing that weren't imagined five years ago.

This article compares testing of traditional localized software with multilingual Web testing.

Testing Traditional Software

Testing localized software has always been a bit of challenge, since it takes place at the end of the project and may be seen as holding up shipment. Trying to decide who should do what can be confusing. How much testing is the localization vendor responsible for? How much can you do yourself? Under what circumstances should you consider outsourcing functional testing to external test vendors?

Despite the challenges, the software industry has managed to work out some effective processes.

Types of Testing

The basic testing categories are described below. You may know them by different names.

Internationalization testing checks for problems with character set support, time/date/currency formats and externalization of resources.

Linguistic review checks for language problems.

User interface validation checks for visual problems such as text truncation or overlap, graphics issues or other visual problems.

Functional testing ensures that no functionality was broken during the localization process and that the localized product works the same way as the source language product.

Interoperability testing ensures that the software interacts properly with targeted platforms, operating systems, applications (and versions), equipment and so on.

Usability testing evaluates how easy a system is to use. (Note: This article focuses on performance testing rather than usability.)

Key Testing Issues

I spoke with a software test vendor project manager for a large Fortune 500 company who manages localization testing efforts and acts as the liaison between internal software quality leads and localization test vendors.

He described some of the critical issues for managing the functional testing of localized products. First of all, the localization vendor must have clear instructions for what strings need to be localized and how to successfully build and unit test the software. Good instructions help reduce problems during the testing phase, when change is expensive and can lead to schedule delays.

Secondly, the localization vendor must deliver complete glossaries to the test team. This reduces disagreements about how something was supposed to be translated.

The test team must create test case procedures and expected results for accessing all graphical user interface screens and error message dialogs. This ensures complete test coverage in order to validate quality.

The last critical issue is that there needs to be open and continuous communication between the localization vendor and the test team throughout the corrective action process. Without good communication, it is more likely that you will have schedule slips or will end up shipping products which contain bugs.

Testing Web-based Software

Web testing is very similar to client-server testing, but with more complexity and a wide range of load variability. Finding problems is much more complicated because the potential interactions between technologies can be so varied. In addition, managing change is a significant challenge.

Gerard Roche, the lead SQA engineer at SimulTrans, LLC, compares traditional software testing to Web testing and notes that one of the biggest challenges relates to the rate of change. "Web sites are updated regularly, so the testing cycle becomes an ongoing task. This constant change can lead to version control issues, which can cause huge problems during testing."

Steve Nemzer, vice president of the Lionbridge VeriTest Business Unit, agrees: "When testing a Web-deployed application, the fundamental challenge to the QA manager is developing a process to cope with the ‘continuous change' environment."

When developing a testing approach for your Web site, you should keep this concept in mind.

Testing Scenarios

In addition to dealing with the rapid rate of change for Web sites, you need to consider a variety of issues in order to structure a testing level which makes the most sense. You can start by asking yourself the following questions.

How critical is the site? Mission critical sites should obviously have a high degree of emphasis on testing.

Is the site live or still under development? This impacts how testing can be conducted.

Is the site used internally within an organization (and will probably, therefore, have a fast connection speed and be accessed by similar browsers) or Internet-wide (with a wide variety of connection speeds and browser types)?

Is the site new or has it been up and running for a while? New sites are a higher risk.

What components work together? Are there server-based functions, database access requirements or legacy system tie-ins? The higher level of complexity, the more opportunities there are for things to go wrong.

What character sets need to be supported? Are all of the languages Latin-based, or are you working with Asian or bidirectional character sets?

How important is the marketing message to the site? If it is critical, you should probably conduct some testing which focuses on regional acceptance and usability.

How will the targeted regional audiences access the site? From PCs, mobile phones or kiosks?

Preparing for Testing

After you have answered the questions listed above, you should create a configuration plan to support the system you outlined. This plan can be in the form of a chart, which lists all of the language versions of browsers, operating systems, IMEs and applications that you plan to support for your site.

The chart will act as the underlying structure for planning and coordinating the testing.

Types of Testing

Web sites require the same types of testing described for traditional software, but they may be applied differently. Multilingual sites also have a number of additional testing requirements. Some specifics for Web testing are described below.

Static HTML testing. When sites are localized, testing needs to ensure that links weren't broken, that images still work and that critical content is intact. Static HTML testing involves tests which do not execute software. These tests perform inspections of documents, identification of HTML syntax errors, link checks and content checks.

Link checking tools find all of the linked objects on a page and then try to download them. Tools should check for links both within the site and outside of it. Most authoring tools like Microsoft FrontPage offer link checking as part of their standard package.

Functional testing. As with traditional software, functionality is the core of what an application is supposed to do. Functionality on your site may differ between regions. For example, an e-commerce site may have steps for processing special offers based on regional holidays or other events. If this is the case for your site, then test plans and cases must reflect the functional differences. Automation under these circumstances can be a challenge.

Functional testing can be broken down into functions performed by the browser and functions which run on the server.

Browser testing. Browser tests check the objects that execute within the browser, but don't deal with server-based components.

Browsers can introduce problems in user interactions. For example, what happens if a user presses the back, forward or refresh button in the middle of a transaction? Will transactions be repeated?

Handling of cookies is another issue which must be tested. Will transactions take place accurately if a user disables cookies on his/her machine or if they are removed? Your test suite should include cases for these types of instances.

Server testing. Server tests check objects which execute on the server, but which are initiated by forms-based user interactions on a browser. Tests of server functions may be for CGI components, Java, JavaScript, VB and so on. Some examples of these transactions are catalog searches, order placement and payment processing.

Potential internationalization problems at this level could relate to sorting and indexing routines for non-Latin character sets, international payment issues, delivery methods and instructions and so on.

End-to-end transaction testing. This type of testing looks at the entire communication chain, from the process of a user filling out an on-line form to the form getting processed and a response being generated and sent back to the user. While other types of testing focus on individual tasks, transaction testing takes a look at the overall process. It focuses on the interfaces between components.

Potential internationalization problems that can occur during this testing would be that data can't be transferred well between components due to character set problems. Interactions with legacy system components may be an area of particular concern. Another issue to look at related to the overall transaction is to see what happens if there is a loss of connection to the server. Are disrupted transactions handled gracefully?

Is translation between encoding methods working correctly? This would apply if you store data using one encoding method, but deliver it in others to meet regional user preferences. Each translation between encodings must be tested.

Herb Bauer, testing services manager for Sykes Enterprises, Incorporated, suggests, "If at all possible, test your site from Web clients that are installed in-country; use native ISPs and various Internet connections." This is because many errors show up after a Web site is launched in the real world, outside of a controlled test environment. Bauer also says, "It pays to test using the least compatible combinations. Your site may shine on Windows 98 with IE 5.0, using a monitor set to 800x600 resolution. But how will it work under less favorable conditions?"

Security testing. You will need to ensure that the site is secure, particularly if conducting product sales through the Web. Issues to consider for security testing are firewalls, encryptions, passwords and so on. Some products (such as downloadable software) may require encryption in order to be sold in certain regions, so security issues could differ based on the target locales.

Regression testing. Regression testing looks at the impact of changes on all of the areas covered by the other testing types.

The biggest challenge of Web testing may be the high degree and frequency of change. How do you determine what has changed and then base testing on those changes? Given the continuous-change model of Web development, doing a good job of identifying a regression testing approach is key.

Load testing. Load testing emulates a specified number of users on the Web site at once. Typically, the more users you have, the more problems you'll discover. When taking a site multilingual, it can be difficult to estimate what the load will be.

Another issue to consider is that peak loads will vary by region. Access hours will be different from those in the United States, holidays are different, sales may be different and so on.

Do multiple servers share the load properly? If your site will be using multiple servers in tiers, the transaction testing should involve all levels in order to get an accurate sense of how things will work.

What happens if your site prospers beyond your wildest expectations? Can it handle the load? What happens to response times?

High volumes of http requests to test out loads are generated by "http torture machines." This testing can bring to light a number of problems, but it doesn't mimic the way real-world users would make hits to your site.

When planning load testing, it's best not to worry about normal days; instead, consider what the volume might be if all your expectations were exceeded.

Performance testing. "Performance" is a concept that is entirely based on customer perception. When establishing performance baselines, you should consider who your users are. Home-based and small-office-based users may have slower Web access. Corporate users may have higher-end access capabilities.

While access time is dependent on the entire communication chain from the user to your server, speed of page loading is very closely tied to the complexity of the design. For example, large graphics files can introduce "last-mile" quality problems in regions with low bandwidth.

Before creating tests, you'll need to set a target for what you want your user to be able to expect for performance. This baseline may vary by locale depending on the technology issues faced regionally, the distance from the site host and so on.

You should also consider how response times might vary by time of day, load and usage.

Key Issues

Some of the key issues related to Web testing include distributed content development, differences among browsers, clarification of responsibility and the issues of localization and internationalization.

Distributed content development. Another challenge which is specific to Web testing is the fact that there are multiple content authors and multiple methods for creating content. Establishing rules, style guidelines, process and workflow is critical for minimizing problems during testing.

Browser differences. As described earlier, the first thing you need to do is determine which browser versions you want to support. Your choices should be based on regional usage differences in your target markets. A site viewed on Microsoft Internet Explorer can look quite different from the same site when viewed on Netscape Navigator, even without issues related to character set support. You may find that colors of standard objects change, that centering and scaling of objects is different and that tables are handled differently.

If your site supports a variety of character sets, you need to be even more concerned about browser versions in order to deliver pages which your intended audiences can understand.

Fonts and preferences can be an additional challenge, since users can make a wide variety of choices and browsers offer varying levels of support.

As Bauer suggested, you might want to consider running some portion of your tests under the worst possible configuration to see what comes up. In the real world, people will use outdated technologies and work under bad conditions, so it's best if you know what to expect before going live.

Clarification of responsibility. Depending on how a site is set up, the localization vendor may not have access to the live server and therefore can't test server-driven content. Well-structured sites store text in a database, and localization vendors may be asked only to supply translations in string tables or in static HTML pages. This means that they can't really verify the impact of the translated text on the site. In these cases, the client must be responsible for conducting all of the appropriate testing to ensure that no problems exist.

Internationalization and localization issues. As with all software, the standard internationalization issues must be checked. Issues such as time, date, monetary formats, currency handling, searching and sorting features, cultural correctness and so on must be handled correctly.

Nemzer of Lionbridge noted that externalizing strings to resource files or bundles can be difficult for popular scripting applications such as VB and JavaScript. He also commented that there tends to be inconsistent use of HTML meta tags such as CHARSET and FONT FACE, along with untranslated ALT tags.

For many Web sites, strategic decisions are made regarding what sections, if any, to keep in English. If portions of your site are not translated, this information must be made clear to the testers so that you don't generate a bunch of faulty bug reports.

If advertising is included on the site, it needs to be regionally focused, since ads for the wrong region will be essentially useless.

Web Testing Tools

Web sites are becoming critical elements of many companies' business strategies, and the need for testing is therefore increasing. This need is bringing about the development of all sorts of new tools and services. Web testing tools can be an effective way to conduct a good portion of your testing.

Various types of testing are covered by the new tools, from HTML validation to load and reliability testing. Tools can range in price from free (shareware) to high-end packages costing over $250,000.

Some tools automate test functions by recording scripts, and you will need to discuss how these scripts can be modified for use on multilingual systems. For example, if a user on an English system has his/her transactions recorded for later reuse through a test tool, is there some way to replace the English text with the target text required in order to mimic the text in the native environment?

The smooth operation of a Web site can be critical to a business' success. Because of the changing nature of Web site content and functionality, this can be a challenge even for single-language sites. Ensuring the quality of multilingual sites is a task that should be approached methodically and with advanced planning.

Web site testing is becoming big business for a number of software developers and service providers. The Software QA/Test Resource Center (www.softwareqatest.com) has an extensive list of free and for-purchase testing tools, but here are a few providers to get you started.
COAST Software Inc. (www.coast.com) Web testing solutions.
CompuWare Corporation (www.compuware.com) Web optimization tools.
eValid, Inc. (www.soft.com/eValid/evindex.html) Web site quality checking products.
Mercury Interactive Corporation (www.merc-int.com) Web testing through virtual clients and through a hosted ASP testing service.
NetMechanic Inc. (www.netmechanic.com) Tools for assessing speed and static testing of HTML.
RadView Software, Inc. (www.radview.com) Integrated Web application testing through virtual clients for functional and load testing.
Rational Software Corporation (www.rational.com) Wide variety of testing tools.
RSW Software, Inc. (www.rswsoftware.com) Web application testing software.
Segue Software, Inc. (www.segue.com) Web reliability products.
W3C HTML Validation Service (http://validator.w3.org) Checks for conformance with W3C standards and recommendations.
Watchfire Corporation (www.watchfire.com) Products which measure and report on Web sites as they run.