Challenges with Automation projects

The testing team in our company is a small group responsible for testing all the applications developed in-house, with all the team members involved in manual testing having a minimal knowledge in automation.

Initially in our company, there were mainly three applications developed. Each of this application had very minimal features. Manual testing approach worked great.  But with the growth of the company, many new features were added to the existing applications as well as many new applications were developed to support client’s business requirements.

With the size of these applications, manual testing effort increased from one week to three weeks of regression testing. In certain cases, regression testing cycle exceeded three weeks. With a waterfall based software development life cycle, manual testing still seems to be working fine with quarterly and monthly release cycles. This challenge became one additional reason for our team to consider agile software development and related testing practices.

Agile Testing

Our team is modeled on the agile testing principles, most accurately captured by Lisa Crispin in her Agile testing Quadrants concept. In brief, most, if not all software development projects have both manual and automated testing needs. In addition to this, projects have business facing needs as well as technology facing needs. These needs are captured as four dimensions of the quadrant. Each quadrant’s needs can be addressed by various tests that include performance testing, unit testing, functional testing and exploratory testing. Our team embarked on unit tests and automated tests, while continuing to rely on manual tests and outcome based tests during this transition.

 Short Release cycles

Due to the nature of our companies business, time to market and client feedback were critical in order to release a valuable product to our advisors. Realizing the fact that long release cycles did not help address business needs, our engineering team decided to move from monthly release cycle to weekly releases. Now manual testing has become a big challenge. We can no longer afford three weeks of regression testing.

So we have decided to change the testing approach by initiating the automation project. Our goal is to automate all fully loaded applications to reduce the regression time and the resources required in testing these applications. This also helps testers to spend less time on testing and more time on understanding companies business, learning new skills to help in aiding companies’ growth.

What we tried at Fortigent:

Our team initially tried to explore the available test automation tools in the market and to find the best testing tool supporting our applications. Below are the tools that have been tried out with the pros and cons of each of these;

Selenium IDE: Started automating our client facing web application by using Selenium IDE which is an open source tool. Selenium IDE operates as a Mozilla Firefox add-on and provides an easy to use interface for developing and running individual test cases or entire test suites. Selenium-IDE has a recording feature, which will keep account of user actions as they are performed and store them as a reusable script to play back.

We were able to automate most of the static test cases and simple dynamic scenarios using the IDE;

Drawbacks in using IDE:

• IDE is compatible with only Firefox browser. We had to switch to selenium RC to run the tests created in Firefox on other browsers
• IDE doesn’t have an integrated spreadsheet for data driven testing (parameterization), always need to use external files for it which is a main drawback, i.e. does not work well with complex dynamic scenarios
• To deal with browser compatibility and complex scenario issues, we had to upgrade to RC. Since we did not have qualified resources to upgrade the scripts from selenium IDE to Selenium RC, we could not proceed further with Selenium IDE tool.

Our next Heavy featured application is a windows application developed using Dev Express controls. It was a big challenge for us to find the automation tool that supports Dev express grids.  During our research process, we found that Test Complete tool made by SmartBear software supports dev express control grids.

TestComplete  is an automated testing tool, developed by SmartBear Software which aims to allow testers to create software quality tests. Tests can be recorded, manually scripted or created manually with keyword operations and used for automated playback and error logging. TestComplete is used for testing many different application types including Web, Windows, WPF, HTML5, Flash, Flex, Silverlight, .NET and Java.[1] It automates front-end UI/functional testing and back-end testing like database testing. One very important feature of TestComplete however is the ability to access native properties of .Net Objects in the AUT.

We started automating the .NET Windows Forms applications that uses a grid based on the control from Developer Express using Test Complete. We were able to automate close to 100 test cases using TC. But we faced many issues while re running the tests. Ex: TC was stuck in recognizing the cell values in grids. We also implemented textEdit and some other controls on gridView. All cells contain lookupedit controls remain unrecognized when the table checkpoints were used (or property checkpoint). TC can only partly recognize some cell’s value.

Drawbacks in Test Complete:

• Neither GridLookUpEdit nor XtraGrid are fully supported by TestComplete.TC records any object by coordinates. So when an object (ex: combo box) order changes or width changes, related tests fail
• Cannot create Data Driven Tests  with coordinate points
• Some times TC was stuck in recognizing the cell values in a grids
• We have redesigned the application to use Virtual scrolling approach by keeping the same user interface. We were not able to run any of the existing tests as the positions of objects are slightly changed in new grids.

With the above failures, we started looking for an automation tool that can support both web and client based applications at a low cost. With our new automation engineer on board, we purchased an automation tool called “EggPlant” that supports both web and client/server apps.

EggPlant: Eggplant is a black box GUI test automation tool, originally developed by Redstone Software starting in 2002. Redstone was acquired in 2008 by a UK-based company called TestPlant.

Eggplant is a two-computer system, consisting of a controller machine, where scripts are authored and executed, and a system-under-test which runs on a VNC server. Eggplant connects to this VNC server via its built-in viewer via TCP/IP. The system-under-test can be any system that has a VNC server for it. Scripting is accomplished using a proprietary scripting language called SenseTalk, an English-like language that is easy to use. SenseTalk is used in conjunction with a “guided record” mode, meaning the scripter directs Eggplant to navigate a system and verify a set of test steps.

Advantages:

Easy to use – Anyone with minimal programming skills can learn EggPlant as the language used to code in SenseTalk, an English-like language  that is easy to use.

Image capture – uses image matching technology as opposed to looking to the object-level of the application being tested. This allows for system-wide automation of a system-under-test as opposed to an application-specific solution. It also means that the technology used to build the application to be tested does not matter, nor does the system that the application runs on

Drawbacks:

• As EggPlant uses image matching technology, images captured in one operating system cannot work on other OS.

Ex: Tests with Images captured in windows XP cannot be  run on windows 2k8.We need to capture images again in other operating system for tests to run successfully. Requires a dedicated resource to maintain the test scripts whenever  we update Operating system or change the computers.

• Tests fail even if the resolution of system is changed while running tests. Image has to match 100% for any test to run .
• Expensive compared to other competitive tools in the market
• EggPlant is not very popular in market. So it is very difficult to find the resource having programing skills to work on Eggplant automation when compared to other competitive tools like Selenium Webdriver, QTP etc.

Due to above drawbacks, we failed one more time to continue our automation project with EggPlant tool.

What we are trying now at Fortigent:

• With so many hurdles in our journey, we finally found the right tool to automate our client facing web application using Selenium Webdriver
• Selenium Webdriver: WebDriver is designed to provide a simpler, more concise programming interface. Selenium-WebDriver was developed to better support dynamic web pages where elements of a page may change without the page itself being reloaded. WebDriver’s goal is to supply a well-designed object-oriented API that provides improved support for modern advanced web-app testing problems.
• Page Object Model:  This model provides a separation between actual test scripts and test page objects.  Creating Selenium test cases can result in an unmaintainable project.  By using the page object model we can make non-brittle test code and reduce or eliminate duplicate test code. Besides that it improves the readability and allows us to create interactive documentation. Last but not least, we can create tests with less keystrokes.
• Selenium Webdriver is an open source tool . Very popular and widely used in market. It is very easy to find resources interested to work on Selenium webdriver tool.
• Selenium webdriver allows to write tests in a number of popular programming languages, including C#, Java, Groovy, Perl, PHP, Python and Ruby. The tests can then be run against most modern web browsers

Our engineering team started using Selenium webdriver using page objects model to automate web application. This approach helped developers also to use page objects for writing Unit tests. We are getting additional help from developers as well in building page objects. Webdriver automation seems to be working well so far not just for testing team as well as for developers – Test/Dev Automation Approach

What we are doing now to test for weekly releases:

While automation tasks are undergoing , with Agile process, we still have to test and release fixes/enhancements every week .So we have found the temporary solution to test the fixes included in weekly releases.

Focused Test Approach:

1. Analyze the fixes included in next release
2. Check with the developer /technical team to find what are the different areas impacted as a result of the fixes.
3. Define a scope for regression cycle by including all the modules impacted as a result of fix. This is a much focused approach. Ex:  Even if the application has 30 modules. Our regression scope may include only 6 modules as a result of code change since last release
4. Verify all progression fixes and perform regression only on impacted modules
5. This seems to be a very effective testing approach to stick to weekly releases cycles

Drawbacks:

–          This approach works fine for the weekly releases. But if there is a hot fix needed in between the release, then the resource has to spend some time on testing and releasing the hot fix. This impacts the regular release cycle as we may end up spending more time or adding additional resource to stick to weekly release cycle.
–          Need to rely on developer to find the impacted modules. There are chances developer might miss on providing information on all the code changes, the testing team will then end up not performing regression testing in that area . As a result, bugs/defects may occur in the production.

What can be done to solve these problems?

We still need to explore and find the right tool to automate our client-server application which uses dev express control grids. If we are not able to find one, we would need to hire additional experienced automation testers to explore the tool and automate the windows application.Our challenge here is not just to find the right resource with all the skills required for automating any project, but also to find the right tool to automate our applications.

We are currently progressing very well in automating client faced web application using Selenium Webdriver. Our goal is to automate 90% of all our applications. As 100% is not possible in automation without manual interference. We know that it is not an overnight project. But we are trying our best to reach this goal.