Setting up your environment for Flash/Flex Automation

Before going into details about some of the tools mentioned in the previous post, we thought it would be a good idea to start with what you need to do to get your environment ready for Flex/Flash automation. There's a lot of information out there but there isn't a place that walks you through the entire process - so we decided to write everything down and try to make it easier for other users.

Here is what you need to get started:

1. A Web Server

You will need to install and configure a web server or make sure you have access to one where you can upload your .html and .swf files. This is the server that you will direct your test scripts to as well.

We won't go into details here – the most obvious choice, if you don't already have access to one, is to install Apache.

However, if you prefer to run all your tests locally, you will need to change your Flash Player settings to allow local Flash content to connect to the internet. Here's how you do that:
This link will bring up your Flash Player Settings Manager.. Go to the “Global Security Settings”, go to “Edit Locations” and add the local .swf file that you want to test.

2. Flash Builder or another IDE of your choice

There are several ways (depending on the tools you plan to use as well) to setup your automation environment. All of them, though, require compilation of a Flex project, so you will need an environment that allows you to do that. At first, you can download the Flash Builder and use the 60 day trial. Note that if you plan to use Adobe's Automation Framework you will only be allowed to use 30 actions/test script, unless you have a FlexBuilder Pro Licence. The Framework for Flex 2 used to be available within the LiveCycle Data Services package, for which, as far as we know, you can get a free Licence by registering on the site if you only need it for development purposes. But since Flex 3, the Framework is no longer provided in LCDS.

Depending on the preferred approach and the chosen automation tools, there are several ways to start your flex/flash automation.

Let's say you want to test a Test.swf application.

Option 1 – Using Adobe's Automation Framework and an Automation tool of your choice

To automatically test your Test.swf application, you need to either have the source code for it and recompile it (probably the best approach) or load your automation libraries at run time.

Here are the instructions for both cases:

Option 1a – Recompiling Test.swf with automation support

Once you have the source code for your Test.swf, you need to recompile the Flex Project after including your automation libraries by using the include-libraries compiler option (in Flash Builder, you just have to go to Project > Properties > Flex Compiler and add them to the “Additional Compiler arguments” field).

You will have to include:

automation.swc
automation_agent.swc
automation_charts.swc (only if your application uses charts)
automation_dmv.swc (for charts or the AdvancedDataGrid classes)

and any other libraries required by your test automation tool. For example, if you want to use FunFX, you will also have to add the following libraries:

funfx-0.2.2.swc (the current FunFX automation agent)

automation_rb.swc

Option 1b – Loading your automation libraries at run time

If you don't have the source code, you will have to load your automation libraries at run time before you can test the .swf file that was compiled without automated testing support.

To do this, you need to create an new Flex project and use the SWFLoader to load your Test.swf and also add all the required libraries:

1. Create a new Flex project (let's call it Wrapper) and have the following MXML file:

1 2 3 4

<?xml version="1.0" encoding="utf-8"?> <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" <mx:SWFLoader source="Test.swf" width="100%" height="100%" </mx:Application>

2. Include all the libraries mentioned at Option 1a as compiler arguments.

3. Compile the project. The Wrapper.html file will then load your Test.swf application and will also load the automation libraries at run time.

4. Direct all your test scripts to Wrapper.html to access the objects in Test.swf.

Note: In the above option the name of the test application is hard-coded inside the wrapper, but there is also the option to pass the name of the application as a parameter.

For example, we have a FunFXLoader (compiled with all the required automation libraries) that we use anytime we want to test a Flex application using FunFX.

The .mxml looks like this:

1 2 3 4 5 6 7 8 9 10 11 12

<?xml version="1.0" encoding="utf-8"?> <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" creationComplete="initVars()"> <mx:Script> <![CDATA[ private function initVars():void{ FunFX.source=Application.application.parameters.testswf; } ]]> </mx:Script> <mx:SWFLoader id="FunFX" width="100%" height="100%"> </mx:Application>

while here you can see what the index.template.html file contains.

Now, to load your Test.swf, you will have to access the following page:

http://mytestsite/FunFXLoader.html?testswf=”Test.swf”

Of course, this can be used with any .swf file as long as you copy it on your web server (relative paths can be used to specify the file location).

Option 2 – Using an ActionScript “Framework”

If you want to use an already developed ActionScript Framework - I guess if you develop one of your own you know how to use it :) - for example SeleniumFlex.as, you must refer the ActionScript file in your main MXML file, like this:

1	<mx:Script source="YourFramework.as"/>

Option 3 – Using other Automation methods/tools like Test Complete

Automation tools that don't use Adobe's Automated Testing API, like Test Complete, have other methods of exposing Flash Objects.

With TestComplete, for example, to access internal objects of Flex applications, you have to compile your Flex application with the Generate accessible SWF file option enabled and make TestComplete recognize Flash elements as MSAA objects. However, this approach only lets you test Flash elements residing on web pages that are shown in Microsoft Internet Explorer or the Microsoft WebBrowser control.

Flex/Flash Test Automation Tools

For a while now, I've been involved in testing a complex media web application developed in Flex, and I got to the point where some help from a tool would have been much appreciated :). I have to admit that this was my first RIA project, so I decided to dig a little bit into it, especially regarding the options one would have in automating some of the functional tests.

Scope
I don't want to argue about the ROI of an automated GUI test suite, or about the fact that maybe it would be much more useful to automate some other types of testing like unit and performance (there are specific tools for each of these).

All the info from this post can be found on the internet, but it takes a lot of time and trouble to gather and analyze it. This post will provide a list of tools I found that support Flex Automation, hoping it will help you if you're thinking about functional testing automation for Flex based applications.

The Flash in the blackbox
The main issue with RIA Flex applications is that the Flash player is basically a runtime, which makes its objects invisible to its container (e.g. browser's html or javascript code), unless explicitly exposed from within.
I was able to identify three ways to add automation support for Flex applications:

1. using the Flex Automation Framework (+ExternalInterface). Adobe has developed this framework to provide a way for developing ActionScript automation-agents that facilitate the communication between the automation tools and the Flex objects. This is possible using the ExternalInterface API. I couldn't find a way of using the Automation Framework without the ExternalInterface, although it is not specified as the only option for establishing the communication. Even though the Flex SDK is open-source, the Automation Framework is not: the trial version can be used for scripts with up to 30 actions. One would have to pay 499€ (VAT not included) for the FlexBuilder Pro license in order to create larger scripts.
   


2. by developing your own framework in ActionScript and make it visible through the ExternalInterface - more info about the API can be found here.
   


3. IAccessible interface and MSAA. Not all the objects support the IAccessible interface by default; here you can find more info on how it can be added.
   

The Tools
Below you can find a table with the tools that could be useful. I say this because I found some tools that claim to support Flex Automation, but they are just basic macros (mouse click at a certain coordinates on the screen), i.e. iMacros from iOpus.



We plan to take a closer look at some of these tools, especially at the open-source ones but also at the ones that have a decent price. I am pretty sure that companies like IBM, Borland or HP have their own research and analysis on flex automation already available. Plus, we are not a big corporation, so we don't plan to acquire any of them in the near future :).

Here you can find some useful info on how to setup your environment and get started with automating your applications.

Stay tuned, we'll post some new info soon.

ISTQB Certification Survey Results

We had a total of 81 responses to our ISTQB certification survey. The analysis below will first look at the responses provided by the 57 testers that are certified and will then show the same graphs with the answers provided by the 24 testers that are not yet certified.

(click on the graph to enlarge picture)

ISTQB Certified Testers and their opinions

To find an answer to our initial question, have a look at the graph below. Though arguable, this graph tells us that ISTQB really is worth it:

(click on the graph to enlarge picture)

More than 90% of the certified testers believe ISTQB is useful and would recommend it to others, while more than 60% of them believe that having an ISTQB certificate helped them to get a better job.

Moreover, as you can see below, more than 90% of them also learned new things while studying for the exam:

(click on the graph to enlarge picture)

Finally, the pie below shows which is the most common ISTQB certificate:

(click on the graph to enlarge picture)

Testers that are NOT (yet) certified and their opinions

If you think the answers provided by the certified testers are surprising, please have a look at the ones provided by the testers that don't have a certificate. More than 80% of the the people who said they don't have the ISTQB certification and don't plan to take the exam believe they would learn new things if they studied for it:

(click on the graph to enlarge picture)

Even more surprising, more than 70% of them believe that ISTQB is useful (and only 1 of them said it would be useful for someone else) and would recommend it to others:

(click on the graph to enlarge picture)

Why don't they plan to get certified in the future? Is it the price of taking the exam? The price of the courses and their value? Doubt that self-study is enough for passing the exam?

As observed by one of the specialists who filled in the questionnaire, another interesting aspects that this survey doesn't cover is weather the content of the ISTQB certification complies with industry standards.

These are all questions that could be included in another survey. Until then, please feel free to send us your own thoughts and let us know what you make of these results.  

Disclaimer: Survey results are based on data collected from members of Software Testing and QA LinkedIn groups as well as testers and test specialists from our own personal network. Information on which the survey results were based was not audited or verified. We do not guarantee the accuracy or completeness of the survey or of any information presented here. The information is presented without warranty, express or implied. We assume no liability for loss or damage as a result of errors or omissions in the information presented on this website, or for damages resulting from use or misuse of data presented on this page. Persons or entities using these survey results do so at their own risk. 

ISTQB Certification

We’ve heard many different opinions about the efficiency and value of an ISTQB certification (Foundation or Advanced level) so we decided to upload this survey on our blog. 

We will gather the results and post them here later on to see if you think getting an ISTQB certification is really worth it. 

What are the common misconceptions regarding test consultancy?

I sometimes hear opinions like: “Test consultants are hired to just click on some links/buttons and they don’t have advanced technical skills”.

The main source for this misconception is the general opinion regarding testers - employees or consultants – generated by software companies that don’t really focus on quality and prefer to hire unskilled/low qualified testers with no knowledge of test planning, design and execution.

The latest experience I had was with a UK company that hired a team of test consultants for their final acceptance testing. At the end of the test round, they were puzzled by the fact that the test consultants were more efficient than the offshore test employees and found a lot of new defects on a product version that was considered stable and ready for release. Most probably, they shared the same misconception, as they were not expecting the test consultants to perform better than their own employees, but to be less efficient and to simply execute some tests.

Another misconception is that test consultants don’t really belong to test team and they tend to be marginalized – they are not invited to all meetings, they don’t have the same working environment, etc. Companies should pay a lot more attention to this and stop calling them “externals”. Consultants are more expensive and they are supposed to perform better than the employees, so companies should not restrain them from doing their job by differentiating them form their own employees in terms of access to all needed information, desks, computers and so on.

The third misconception is that they are too expensive. However, if the quality of the product is too low and the client is not satisfied they may sue you or refuse to pay and find another provider. This will most probably be more costly than hiring a team of test consultants to measure the quality and take the right actions to correct any issues. Test consultants are too expensive only when they are not used properly.

What companies should not hire test consultants?

Most companies would benefit from either test management services that would improve their testing processes, test specialists that would target and test very specific parts of a product or test execution services that would bring fast and objective results. There are a few companies, however, that would be better off not hiring test consultants.

Some companies are so resistant to change and so confident in their own processes and practices that hiring test consultants would not bring any value to their software development activities and their quality.

Software testing alone cannot make software better. Software testing services will help companies identify risks and measure the quality of a product using a set of relevant testing metrics - but still the company/project manager needs to act to improve the level of quality. Doing all the testing tasks in an efficient manner may be a waste of time and money if their results are not taken into account.

Another situation that works against test consultants is that of long projects with no clear schedule and deadlines. In this case, companies should consider recruiting and training a test team rather then calling a test consultancy company, unless this provides recruiting and training services as well.

When should you hire a test consultant?

You always need a test team. However, here are a few situations when you should think about hiring a test consultant:

1. When you have tight deadlines and you need (additional) resources very fast
   
   
   
   Don’t waste time employing resources that may only be needed for one particular project and for a short period of time. If you are sure that the initial schedule cannot be changed and you cannot move the deadlines, the best approach is to quickly hire a test specialist who will start working on the project right away and will have all the necessary skills for testing your product. However, make sure that everything that has already been done for the project is well documented or you may find out that you are not buying extra time by hiring a test consultant.
   
   
   
2. When you need a person that is highly specialized
   
   
   
   It takes time and money to train one of your employees to get really specialized on a certain test area. The testing training courses are not cheap. You might also find it difficult and expensive to recruit a specialized person. The most common practice in cases like this is to get a test consultant that will also train your team to use or update the already existing scripts/test cases for the next projects and releases once their project is done and their task completed.
   
   
   
3. When you want to have an objective measurement of the quality of your product
   
   
   
   You should get test consultants to design and execute tests in System Testing or User Acceptance testing when you could benefit from having an “internal auditor” verify and validate your application before shipping it to the client. As a client, you may want someone to confirm that the level of quality is the same as the one stated by the vendor before paying the bill and marking the contract as successfully completed.
   
   
   
4. When you need resources or certain skills for a short period of time
   
   
   
   Whenever you need a test specialist or test manager to do a certain task for a determined period of time and you know that you will not need that resource afterwards, you should pay for a test consultant. You will get someone who can start right away and cut down the costs with re-training an employee for a new position after the task is done. 
   
   
   
5. When you want to have your current  testing processes and practices improved
   
   
   
   If you want to change or improve the current practices in your testing department, you should get a consultant specialized in test management. You will benefit from the consultant’s extensive experience in choosing the optimal and the most suitable process for your project and your company and you will give your other testers a chance to learn and improve their skills. Test management consultants will also bring their own templates for test strategy and test plan documents, as well as test reports and lessons learned presentations that you can re-use in later projects.