Two days of Eclipse

Day one, Demo Camp

At 5pm on the 28th June, BREDEX is organizing a demo camp in Braunschweig. Confirmed presenters include Kai Tödter, Ed Merks and Marcel Bruch - and we're working on a couple more  BREDEX will also be doing a demonstration of Jubula. Registration is now open via the Eventbrite site - so sign up for a great evening!

Day two, Eclipse 4.2 Training

The following day (29th June), Kai Tödter is holding a one-day training course on Rich Client Development with Eclipse 4.2. This is an excellent opportunity to get up to speed on the latest developments in Eclipse without having to travel to far-off conferences or workshops. A full description of the training day is on the BREDEX website, including a link to the registration. Places are limited, so register early.

• training
• events
• Eclipse Demo Camp
• Eclipse

Eclipse Demo Camp Braunschweig

On 28th June, there will be a Demo Camp in Braunschweig starting at 5pm. The Eclipse Wiki page gives more information on the event, including a link to the registration page and the opportunity to sign up as a presenter. There are still some spaces left, so come and show us what you're doing with Eclipse!

Eclipse Testing Day

The third Eclipse Testing Day will be held on the 5th September 2012 in Darmstadt. The call for papers has started; please send your submissions by the 31st May 2012.

We're also pleased to announce our current sponsors: GFB Softwareentwicklungsgesellschaft mbH, Diaz & Hilterscheid GmbH, and Verit Informationssysteme GmbH. We are also collaborating with Professional Tester Magazine and Testing Experience Magazine. There are still sponsoring positions available, and of course we're looking forward to hearing about the Eclipse community's experiences and adventures in testing as well as meeting testers, developers and project managers on the day.

• events
• Eclipse Testing Day
• Eclipse Demo Camp
• Eclipse

What we're looking for

The topic for this year's testing day is "Testing and Beyond". The program committee are putting together a format that will consist of traditional talks on the topics below, as well as a set of lightning talks from areas "beyond" testing and a panel discussion with experts from the Eclipse and testing communities. 

The topics for talks could be:

1. Testing Eclipse applications
2. Testing within the Eclipse Ecosystem
3. Testing on Eclipse Projects
4. Design for testability in Eclipse
5. Case studies of testing projects
6. Eclipse tooling and technology for the test process
7. Continuous integration and testing for Eclipse applications

As in the past years, demonstrations and case studies are very much welcome but product demonstrations will not be accepted. Submit your abstract (in English, 100-200 words) and your biography to testingday at bredex dot de by 31st May 2012. More information on talk submissions and length is on the wiki.

Be a sponsor!

The Eclipse Testing Day is a not-for-profit event. Costs are primarily covered by the organisors and our sponsors. We have a range of sponsoring opportunities available - sign up to support this great event and meet Eclipse users and testers on the day.

Come to the event

The registration will open shortly after announcing the program. If you would like to be notified when registraion opens, then send an email to testingday at bredex dot de.

We look forward to an exciting event!

• events
• Eclipse Testing Day
• Eclipse

Test result reporting

We've had the Reporting Perspective for some time now, and it's one of my favourite additions over the last few years. In almost every team I've been a part of, it takes time to learn how to deal with test results and to define how to react to them as a part of the process. That's one of the reasons why we keep adding new features to this area - that and the fact that we also use the reporting capabilities on a daily basis. So what's new this time?

Web dashboard

An added feature in GUIdancer from 6.0 is that we've made the reporting perspective available in the browser thanks to the magic of RAP. Now you can do your test result analysis from the browser - which is excellent if you're not at your desktop, or if you 'just' want to perform analysis and don't need an installed GUIdancer.

More test result details

One of the new features available in both Jubula and GUIdancer is more information in the test result reports. We've added the cumulative duration to each node in the result report as well as the data used for the node. Instead of having to expand the test result to see these details, they are instantly visible at each level, so now you can easily see how long each part of a test is taking as well as which data sets were used.

Create Mylyn task

As an extension to the Mylyn integration in GUIdancer, we now also have the opportunity to create a task in external systems (like Trac or Bugzilla for example) directly from a Test Result Report. So if you open up your test result and find that the Test Suite failed due to an error in the software, then you can create a ticket for it directly from GUIdancer.

Working with test data

Alongside the extras in test reporting, we've also made some improvements to actually writing tests. One of the directions we've gone in is test data.

Functions

One of the aspects of test data that has been missing for a while is being able to use functions. Well, those days are over. Jubula 1.2 and GUIdancer 6.0 both support the use of functions as test data. There are a few basic math functions (add, subtract, multiply, delete, round and truncate) and some date functions (now, formatDate, modifyDate and parseDate) that we've already written, but new functions can be added via an extension point as well.

In-editor decoration for missing data

Often its the small things that really make a difference. We've added decoration within editors to show you if you've forgotten to enter data. No more saving and then realising that you've forgotten something - now you can see instantly whether the data in the editor you're working on is complete or not.

Other really cool stuff

So we've got some exciting things that fit into the areas of result reporting and data. We didn't stop there though, there's more cool stuff in other areas as well:

Metrics framework

In both Jubula and GUIdancer, there's now a new framework for Metrics within the project. We've added three example metrics to it at the moment, and we'll be expanding the possibilities in later versions. At the moment though, you can already

• count various things (Test Cases, Event Handlers, Test Steps etc) in the project,
• calculate the ratio of specific actions and general actions used throughout the test, 
• and show any redundant-looking hierarchies that you have created

Categories in the Test Suite Browser and Central Data Sets Editor

The Test Suite Browser and Central Data Sets editor just got prettier: now you can categorise your Test Suites and Test Jobs as well as any central data you have in your project.

New action: store property value

Sometimes you don't know how many rows you have in a table, but you do know that after searching it should be less than before. A new action to store the property value lets you do this. Combine it with the "check string values" or "check numerical values" actions to perform the checks you need.

Save As New Test Case

I've deliberately left this until last because it's both exciting and terrifying. Since its conception, GUIdancer (and now Jubula) have been staunch opposers of copying and pasting. We've always said - if you need something twice you should make a module and reuse it. And we still say that. But based on some talks with customers, and some enhancement requests and forum entries, we've relented just a little bit for a specific use case.

Imagine you have a wonderfully structured, modular test. It's perfect as it is, nobody could accuse you of being lazy with your modularisation. Nevertheless, you need other use cases that consist of variations of the modules you have in this test. For one use case, you need them all with an extra one in-between. For another you need most of them but not the last one, and so on. In this case, you'll be thankful for the new option "Save As New Test Case". You can use it to create a new Test Case that already contains the modules you select from an existing one and so save time looking for the same modules again.

Sounds great, where can I get it?

GUIdancer 6.0 is available from the BREDEX GUIdancer page. Jubula 1.2 is available from the Eclipse Project Page. Bear in mind that the new version of Jubula is just the standalone at the moment. The features described here won't be in the Eclipse Feature until the Juno release, so look at the standalone as a preview of what's to come.

• Test Suite Browser
• Test results
• Mylyn
• Metrics
• Jubula
• GUIdancer
• Eclipse

Note that the actual version number for GUIdancer is definitely not 0.4.0 (it's actually 5.2, going on 6.0)! And yet, the plugin.properties file wouldn’t lie:

aboutText=GUIdancernVersion: 0.4.0nn(c) Copyright Bredex GmbH 2004 - 2011

It turned out that I had inadvertently broken the script that automatically replaces the build number in all manifest files as well as the plugin.properties file based on a timestamp. Since a part of the release engineering changes separated the version numbers of GUIdancer’s bundles (such that they don’t have to all have the same version number), I had removed the reference to the script in the build process and neglected to replace it with some other form of automation. Our build tool (Tycho) takes care of updating the manifest files, replacing the ending “qualifier” string with an appropriate timestamp, but what about the as-yet-overlooked plugin.properties file? I was appalled at the idea of manually changing the version number in the plugin.properties file before every release. Above all, what if we forgot to perform the change before releasing?! We’d have multiple versions of GUIdancer out in the wild, all with the same version number displayed in their respective About Dialogs! The end result? Massive confusion for users and support teams:

Image ©caesararum,

http://www.flickr.com/photos/29707865@N05/2780508266/sizes/z/in/photostream/ licensed under Attribution 2.0 Generic (CC BY 2.0)

Not wanting to have that on my conscience, I started thinking about a way to automatically update the version number presented in the About Dialog. I settled on a solution combining plugin.properties, about.mappings, an Activator, and a Java Property. The relevant lines of the relevant files follow:

plugin.properties:

aboutText= My Foo Bar ApplicationnVersion: {0}

about.mappings:

0=$foo.bar.application.version$

activator:

public void start(BundleContext context) throws Exception {     super.start(context);     System.setProperty("foo.bar.application.version",             context.getBundle().getVersion().toString()); }

plugin.xml:

<extension id="product" point="org.eclipse.core.runtime.products">     <product application="foo.bar.application" name="My Foo Bar Application">         <property name="aboutText" value="%aboutText">         </property>     </product> </extension>

All of these file belong to what I would call the “Product” bundle (the bundle that defines the org.eclipse.core.runtime.products extension point), which is then also the bundle that defines the version number displayed in the About Dialog. Using this tactic, I can once again believe what my Product says in its About Dialog.

• Eclipse

Thanks to the folks at RCP Vision, we're happy to be able to offer the Jubula Training Day the day before the Eclipse Day Florence, so if you're planning on going to one of the events, we'd definitely recommend participating in the other one as well!

More information and registration is available on the Eclipse Day Florence pages. If you're reading this and thinking you'd like to work with BREDEX to make a Jubula Training Day happen near you, then please get in touch!

• Test design
• Test automation
• Jubula
• events
• Eclipse

The visit to Knochenhauer after the talk let us carry on with our questions to Manuel and our dicussions of our work with Eclipse. There seemed to be a lot of great conversations going on, so thanks again to Manuel for coming!
The next Eclipse Event with BREDEX will be a Demo Camp in June. If anyone is interested in showcasing their work feel free to contact us.

• Modelling
• events
• Eclipse Stammtisch
• Eclipse

Identifying coverage and risk

I’m sure that we’re not the only team who, once having established a good base of automated regression tests, wanted to know something about their effectiveness. How much do they cover? Where are our risks? Do new tests actually test new areas of the software? How does our safety net look when more code / more tests are added?

It seemed like a fair assumption that the simplest way of getting this kind of information automatically would be to add code coverage monitoring to our tests. As it seemed like something that could interest other Jubula / GUIdancer users, we also added it to the tool, and about a year ago, it became a standard feature of GUIdancer.

Using the JaCoCo integration in GUIdancer

The code coverage feature in GUIdancer uses JaCoCo (a Java Code Coverage library available under the Eclipse Public License). Code coverage monitoring can be added to an Application Under Test (AUT) in the AUT Configuration dialog for Swing and RCP AUTs.

Once code coverage has been activated, a few extra settings should be defined to declare:

1. the installation directory for the AUT,
2. the place where the source code for the AUT is (so that the code coverage report can be viewed line-for-line)
3. a package pattern so that only your own code (and not every single library you use) is monitored

Once that’s done, you can run your test and get a nice shiny code coverage value and report that can be viewed in the Reporting Perspective:

The philosophical part…

So now you’ve got your report. It’s got some green lines, some red lines and some numbers. And if you’re in a similar situation to our team, you quickly have someone asking about these numbers and wanting to know what they mean. (I’m going to try not to joke about managers in *every* blog entry, but I think we all know who will be asking these kinds of questions ).

One of the first things I read about code coverage was this blog entry. Having now had some experience in various projects, it has become my favorite way of explaining to customers that the numbers gathered by code coverage are not the aim of the exercise. The absolute values delivered by a monitoring report can’t be used as a guide to quality, nor should they be formulated as an aim for the team.

It’s not about the absolute numbers

What code coverage numbers or percentages tell us is only what lines, methods, classes or decisions have been executed. It tells us nothing about whether they were executed correctly, or whether the test that caused these things to be executed was a good one. It can be far too easy to attach too much importance to a simple number removed from its context – and this can drive the wrong kind of behavior in our processes.

So what is it good for?

It’s nice to have nice shiny percentages for code coverage, but it’s not the absolute values of the individual percentages that we should be looking at. Instead, there are two other things we should be looking at.

The trend of the lines

The first thing of interest is to move away from individual numbers and look at the trend, most especially the development over time of amount of tests : code coverage : amount of code. The graph generated by GUIdancer shows the first two numbers as lines over time. The report below shows how code coverage can increase with the addition of tests.

Using graphs such as this, we can see whether a new test added increases the coverage or not. If so, then if we assume no large additions or subtractions from the amount of code, we can reason that the test has executed other lines than executed by previous tests. If the code coverage value does not increase, it could mean that new code has been added, thus reducing the overall percent of the code coverage. Looking at the detailed report would tell us whether new code was executed or not (more on this in the next section). Even if the amount of code had remained the same, and therefore new lines of code weren't actually executed, that doesn't necessarily mean that the new test isn’t performing an important use case or workflow through the application, just that the lines executed have remained the same.

What is being missed

Another use of code coverage over and above isolated values is the ability to look at what is not being tested. Don’t look at the green; look at the red. The red areas tell us what parts of the software we’re missing. We can look at e.g. class or method names and decide whether we can design tests that should cover these areas.

When a new test is added we can check whether it covers the areas we thought it would. Instead of just looking at a simple number, we should look at the details.

New questions

As it turns out, the questions about effectiveness and risk can’t really be answered by code coverage. What it can answer though, is "where are we not performing any testing?", "where are we performing too little testing?" and "what effects have my last changes (more/less code, more/less tests) had on the trend?". Based on the answers to these questions, we can design new tests. Code coverage can certainly help us to get better quality software, but perhaps not in the way that is often imagined.

Stay tuned…

The next blog entry in this series will be on Mylyn Integration for task-based working to reduce the amount of context-switching we have to do when automating tests. Until then, happy testing!

• Test automation
• Jubula
• GUIdancer
• Eclipse
• code coverage

Pain points

Over the years, our team has got quite good at defining user stories that are visible and value-bringing for the customer. We've even got good at making them nice and "thin" so that we can see progress quickly and aren't dependent on everything being ready to make it worthwhile. Nevertheless, like every team, we have our pain points. For us these are most often forgetting details (what needs to be tested, what aspects of the development can't be forgotten when implementing this story, etc.) We also sometimes found it difficult to see whether everything was going to plan in terms of time or not. This was often compounded by "sneaky stories" joining the sprint after it had started, essentially increasing what we had to do in the same amount of time. (An entry on why sneaky stories are dangerous anyway is here).

Throwing out the board

About three sprints ago, we decided to change some fundamentals of how we were doing things. Up until then, we'd been using a storyboard.

On the one hand, this was nice and easy to use and it was very visual. However, there were some aspects of our process for which it wasn't supporting us, and some aspects in which it was helping us to be bad...

The board was a bad influence

First of all, cards are just so temptingly easy to add to the storyboard once the sprint has started. There was no automated collection of how many story points we had planned, no way of seeing that this had happened, and no way of seeing how often within a sprint. Also, if you know you can change things, maybe you don't plan so well in the first place.

Lack of details

The cards themselves were also problematic. They let us write nice user-stories, but never really had enough room to write individual development activities (update model, show new info in problems view, documentation etc), or acceptance criteria. Details sometimes got left out, sometimes forgotten. At the same time, we had no good way of managing a product backlog where we could always go back and see what we'd already planned, what we'd estimated and what other things are of interest.

On a purely logistical level, our wall was also getting too small for a growing team.

Time for a tool?

So, three sprints ago, we decided to try out a tool we use at the company for other aspects of our work (booking hours, calendars, project planning, writing invoices etc) which proclaimed to have support for agile processes. We were naturally somewhat sceptical, because we thought that it would limit us in our work. Nevertheless, one of the great things about agile teams is their willingness and ability to try out new things, so we launched a pilot sprint for the tool.

Three sprints on, and we're actually really enjoying it and gaining benefits from specific areas of additional structure it brings us.

Hierarchy: from story to technical details

First of all, epics, stories and activities are created and edited as hierarchical structures. We can keep the customer perspective we've got so good at thinking of while also specifying more technical activities for the developer, so our stories are more well-rounded and also well-estimated. Because everything we've done gets saved into the backlog, we don't have to do any work twice. If we've discussed a feature, written partial or complete stories for it, done mock-ups etc, then those data are still all there later. We can use old stories to help us plan new ones, and we can go back to not-yet-fully-planned stories and carry on where we left off - even if it was a while ago because something else got prioritized instead.

Prioritization

Speaking of prioritization, that's become easier too. Without a well-maintained central list of what the customers want (we have multiple sources for requirements), then it's easy to prioritize something more highly for an upcoming sprint and forget that other things are maybe just as (if not more) important. Now, we see exactly what we're leaving out to include something else and can make sure that that's a decision the customer(s) want(s) to make.

Keeps us on the straight and narrow

At the beginning of the sprint, we add product backlog items to the sprint backlog in the tool. Once the sprint has started, it's very much a "holy" object. An automatic burn-down chart gets created every day, and we have a representation of our old story board on our screens. It is "difficult" to add new items to the sprint, and impossible to do so without noticing that our planned story points have increased. So we don't do it. This has advantages for our transparency with our customers as well - we can tell them exactly when a sprint is beginning, and exactly what it will contain.

Tool must support the team

Now, working with the tool wasn't the only change we made - we also started doing more regular, but shorter, planning meetings and changing the way we work with known or new issues within a sprint. So our general planning has changed a bit too, but even here its nice to know that the results of our planning are safe in the backlog, with all the information it contains.

I wouldn't want to say by any stretch of the imagination that the tool is perfect, or even that our way of working with it is always ideal. Nor would I want to challenge the idea that tools are secondary to interaction. But used in the right way, a tool can help a team to improve. We've now had three good consecutive sprints, and that's worth celebrating .

• story board
• stories
• agile team
• agile process
• agile

Back in early 2011, when we were deciding where and how to split GUIdancer into two tools, we put a lot of consideration into what we wanted to achieve with our Open Source activities. We firmly decided that Jubula users should have all they need to write, execute and analyze tests. We wanted to have a good, well-rounded tool as our Open Source contribution, not something that is missing necessary features. Nevertheless, we did want to save some nice features for GUIdancer, the idea being that for a small amount of money, you can add some nice aspects to your testing project and process.

As it turns out, many of the features we kept closed source and as part of the commercial tool are things that are likely to become nice-to-haves once you’ve got more than just a few tests up and running. Once tests start getting bigger, then aspects like Test Style (like Checkstyle) and Mylyn integration really start getting interesting. And if you’re using Jubula in CI processes and your tests are gaining in importance, then it would be nice to have reporting capabilities (you know, for the managers ;) ) and to get some information on code coverage. If you’ve already used Jubula successfully in one project, then maybe you’ll think about starting testing even earlier next time with the Model-Based approach.

But these are things that come later, and we often get asked the question by newer users – what can I do with all of this? Well, in a short blog series, we’re going to describe how the various aspects of GUIdancer are designed to be used, and how they help us with our work in the Jubula project and in customer projects. This first entry will look at how we use BIRT reports in our daily, weekly and monthly work.

GUIdancer and BIRT – the basics

Jubula contains various options to analyze single test runs – a test result report appears dynamically during test execution, and individual test reports can be reopened including information on error-types and screenshots. This is indispensable in a test tool – I need to be able to see what went wrong so it can be fixed. However, what this doesn’t allow me is any kind of view over time. Questions such as:

• How many tests ran successfully / failed / didn’t run this week / month?

• How many tests have been added over the last period of time?

• How has the code coverage developed since adding the new tests?

can’t be answered with single test runs; they need to be cumulated and displayed in an easily readable manner. This is the aim of the BIRT integration in GUIdancer.

The existing BIRT reports

When we added the BIRT feature, we added some example reports that we imagined we would like to use and that may be useful for customers. New reports can be added by anyone who wants to, but it is nice to have a selection out-of-the box. Of the reports that we offer, there are three that we use very extensively. They have become such an integral part of our project that it’s worth describing how we use them and what they tell us.

Report 1: Comments

The report we use the most frequently is the “Comments” report. The first activity our team has every morning is to analyze the test results. As we have in excess of 70 GUIdancer / Jubula Test Suites (functional tests, performance tests, tests for our actions, tests for starting AUTs – all multiplied by a total of 5 platforms!), we wanted something that would help us to report the test results at our stand-up meeting. For any Test Suite that failed overnight, we use the “Edit Comment” function in the Reporting Perspective to add a short description of the reason for the failure to the test run. This also gives us a comparison over time and lets us find patterns in even sporadically occurring errors.

Once the comments have been entered, the “Comments” report is generated. The report displays a table of all failed Test Suites in a specified period of time (we choose “yesterday”) that don’t have the word “BROKEN” in their name (we have specific “BROKEN” Test Suites containing tests that show known, uncritical bugs). For each Test Suite, the comment is shown in the table. The report is printed and brought to the stand-up meeting so that any necessary action can be discussed.

Report 2: History and Coverage

The second report that we use on a regular basis shows the amount of expected Test Steps versus the amount of executed Test Steps as well as the Code Coverage value as three points for each test run. Again, by entering a time frame (we usually take 2 weeks for our weekly “Show&Tell” meeting, but sometimes look over a whole year for a better comparison), we can see how the tests are progressing. The report below shows a period of two months during which new tests are frequently added (the red line), the code coverage improves (the green line), and after an initial period of stability, tests often failed (the blue line). This pattern improved on the day the report was printed however (the last point on the graph).

This report is one of my favorite ones, because it can potentially show us so much. One the one hand, the amount of expected Test Steps can show how many tests have been added recently, and what sort of an effect it has had on the Code Coverage (I’m looking forward to writing the blog entry on Code Coverage, because it is fascinating to try and decide whether it actually tells us anything, but that’s another story…).

On the other hand, the amount of executed Test Steps can give us some hints about the type of errors we’ve been having:

• drops to almost 0 can either mean that something went terribly wrong at a central place (no database connection could be established for example), or that the error handling is bad (a small error resulted in the whole test stopping)
• a zig-zag graph is usually a good sign of test environment problems: sometimes the tests run, sometimes they don't. Test environment problems are particularly nasty - nobody really wants to feel responsible for them, yet they can be just as damaging as software errors or errors in the test specification.

• A flat graph means that nothing has been changed. No new tests, possibly no new versions of the software, and no new fixes. It's important to analyze why - in this example, it was simply because the tester was on holiday and there was no cover for him!

• Small dips in the executed test steps can either mean relatively uncritical errors, or a good error handling.
• Reports that never manage to get to 100% execution also tell us something about our reaction times. The status quo should always be that we are executing 100% of our expected tests – otherwise we have gaps in our knowledge. This means reacting quickly in the team, and it also means that our tests have to be able to keep up with the pace (I wrote this entry on test discipline a while ago). Errors will happen, so there should never be any questions about why 100% isn’t achieved constantly. But we can get a feeling for our general stability if we’re seeing too many drops that take too long to fix.

Report 3: Histogram

This report gets used to publish our test results on the Jubula website. It currently has a couple of advantages over the History report – it can show days when tests didn’t run at all (broken build, environment problems), and it also splits the Test Steps into successful / failed / not-executed. Otherwise, it shows a pretty similar view of the world to that of the History except that the colors are nicer :).

This first diagram shows tests that are generally stable, but no new steps are being added.

This diagram, on the other hand, shows a test for a problematic area of the software that our team decided make a priority. We wanted to add more tests and fix any issues they found quickly. This shows the last year. We've steadily been adding more and more tests, making them work by fixing bugs, then adding more tests. As you can see, at the time of writing, we're entering a stage of fixing ;)

Other reports

We tend to add new reports as we realize we need them to support our process. The possibilities are endless – percentage of static waits in the tests and test duration are just two that come to mind. At the moment though, we feel quite at home with the three I’ve described. What we’d like to hear is how you use reports in GUIdancer, if you do already. Contact us with your examples and receive a goodie to say thank you!

Until next time

The next entry in this series will be about Code Coverage, so stay tuned. In the meantime, happy testing!

• Test automation
• Jubula
• GUIdancer
• Eclipse
• BIRT Report

Even though it's getting a little late to wish people a Happy New Year, I'd like to do it anyway on behalf of all of us here at BREDEX! We're starting what will hopefully be an exciting year of development, testing and Eclipse events with a regional meet-up (Stammtisch) in our offices on the 9th February at 7pm.

We're pleased to welcome Dr. Christian Bartelt from the Technical University in Clausthal as the speaker for the first Stammtisch of the year. He'll be talking about the Cooperative Modelling project that he is involved in at the university. After the talk, we'll head on over to our local restaurant, the Knochenhauer, to chat over some food and a beer or two

As always, registration and arrival details are on the BREDEX website. If you can't make the Stammtisch, you can already start planning your participation in the spring series of Demo Camps - contact us if you'd like to give a demo!

• Modelling
• Eclipse Stammtisch
• Eclipse

My week started with Gojko Adzic's Specification By Example workshop on the Monday. Gojko has a great range of stories to exemplify the way different teams have adapted the approach to suit them, and I'm sure I wasn't the only one who went away with new ideas of how we can derive scope from goals and ensure that everyone has the same understanding of the system and the new features.

The second day for me was very much about test automation. I attended one talk on specification by example through the GUI, and then presented a Jubula demonstration in the afternoon to give some insights into how we fit into agile processes. The feedback from the participants was very useful, and we're talking about incorporating some very high level description possibilities into the tool to make it even more accessible to business users. Watch this space!

The second evening was the main social event of the week, giving everyone the chance to chat in a relaxed (and very entertaining!) atmosphere. It was great to see Gojko receive an award, and to chat with people about their processes, problems, solutions and experience. That kind of evening really sums up this kind of event for me - I've come back with a better understanding of people and processes, a list of books I should really get around to reading, and with a renewed enthusiasm for all things testing. I'll be demonstrating that enthusiasm later on this week at the Eurostar conference in Manchester - see you there!

• Successful testing
• Software testing
• Jubula demonstration
• Jubula
• Agile Testing Day

A few relatively simple constraints are already built in:

• a column does not contain null values (Not-Null)
• a column does not contain a value twice (Unique)
• a set of columns uniquely identifies any row (Primary key)
• all values of a column point to another table (Foreign key)
• the values of a row are within a certain range (Check-constraint)

It’s up to the developer to translate business rules into those checks. Most are quite obvious, once you’re familiar with relational modeling. But some rules are more complex, sometimes too complex to be expressed by the simple constraints.

Examples:

a) Foreign key cycles: Table PUPIL has a foreign key to table CLASS. CLASS has a foreign key back to PUPIL, to nominate one representative as spokesman. How do we guarantee that the spokesman is from that particular CLASS?

b) Defaults per category: If there is no explicit CLASS table and the PUPILS only have a VARCHAR column for the class, the spokesman will likely be modeled as a NUMBER(1) column IS_SPOKESMAN. A combined unique constraint over both columns can guarantee that there are never two spokesmen within the same class. But how can we verify that each class has exactly one spokesman?

c) 3 dependent tables: ANIMAL, CAT and DOG each with an ID-column. CAT.ID and DOG.ID point to the ANIMAL.ID to represent the inheritance that any DOG or CAT is an ANIMAL. How can we make sure that no ANIMAL is left after deleting a DOG?

d) How can we check that CATs and DOGs don’t share the same ANIMAL?

As you can see, there are quite a few cases where the built-in database constraints won’t suffice. On the other hand it’s not possible to check everything that comes to mind, as checks slow down any write operation. Most often an additional check is justified either by the potential loss or by the developer time to fix the data later on. Sometimes it’s too late anyway and you have to deal with the corrupted data. But if you're in the situation where you have to write some SQL to fix a problem, it’s a good idea to add a check to ensure that it won’t happen again.

For the example problems this might look like:

a)

select * from CLASS c inner join PUPIL s on c.fk_spokesman = s.id where s.fk_class != c.id;

b)

select * from   (select p.class, sum(p.is_spokesman) as numSpoke from PUPIL p group by p.class)  where numSpoke != 1;

c)

select * from ANIMAL where ID not in ((select ID from CAT) union all (select ID from DOG))

d)

(select ID from CAT) intersect (select ID from DOG);

If the necessary columns are covered by indices, the checks will be reasonably fast; fast enough that the end user won’t detect any difference. Just one question remains: Who’s going to run the checks? Answer: A trigger!

create or replace trigger <TRIGGER_NAME>  after insert or update or delete on <TABLE_NAME>  declare i integer; begin     select count(*) into i from <CHECK_CONDITIONS>;     if i > 0 then         RAISE_APPLICATION_ERROR(-20123, i || ‘ rows invalid’);     end if; end;

This straightforward approach has one small drawback. For example b) you cannot change the spokesman ever again. If you set the old spokesman to zero first, the check will immediately complain that there has to be at least one. If you set the new spokesman to one first, the check will immediately complain that there cannot be two spokesman for the same class. Both values must be updated in one single update statement. Good luck trying to convince any persistence layer like Hibernate to do this for you!

What we need is the trigger to fire only once on commit and not once per update statement. Fortunately we can work around this limitation by introducing an intermediate table:

Create table RESULTS (     ID integer primary key,     VIOLATION integer ); alter table RESULTS add constraint NN_VIOLATION check (violation = 0) deferrable initially deferred;

The trigger would end accordingly:

    ...     -- Maintain a single row per Check within the RESULTS table.     delete from RESULTS where id = -20123;     insert into RESULTS(id, violation) values(-20123, i); end;

A single transaction can update the violation count multiple times. Only the last update before the commit to write back a zero into column violation is crucial. Otherwise the deferred constraint NN_VIOLATION will stop the transaction altogether and thereby prohibit entry of invalid data. The trigger must always run on insert, update and delete. Inconsistency might have been introduced by previous statements that get deleted again before the commit.

In addition, multiple triggers on several tables can reuse the same RESULTS table. Sadly almost the same trigger might have to be copied to multiple tables if they are all involved in the check. E.g. example a) must not only listen to changes of CLASS.fk_spokesman but also to changes of PUPIL.fk_class. But the actual check-code can be moved to a function in a package to facilitate its maintenance.

• Data integrity
• SQL
• integrity of the internal structure
• database

We at BREDEX are obviously interested in improving the state of web testing - especially for modern applications that use complex UI components. And as the developers of RAP (Rich Ajax Platform), EclipseSource are the perfect partner for collaboration in this area. Together, we'll be working on a solution to implement the ARIA standard for screen reader component recognition into our products, RAP and GUIdancer, to enable and improve automated testing for RAP and AJAX applications.

You can chat to us at Eclipse Con Europe this week in Ludwigsburg for more details, or wait till Eclipse Con in America early next year to see what we've been working on.

• Eclipse
• testing solution
• AJAX
• RAP
• web testing
• GUIdancer
• Eclipse Con Europe
• Eclipse Con

1. Install the latest version of Jubula (1.1). You can use either the standalone version or Jubula as a feature. Both are downloadable from the Jubula Download Page. Please make sure you have 1.1, not 1.0 or 0.9!
2. Download the test projects. EclipseConTest_1.0.xml is the basis project, EclipseConTest_Complete_1.0.xml is how our projects are going to look after the tutorial. You can download the package for the tutorial from here.
3. If you are using Jubula as a feature in your Eclipse Installation, then you will also need to get hold of the example AUT. At the moment, this is only available as a part of the standalone application, so you should also follow the instructions in step 1 .

We will, of course, have some USB sticks available at the tutorial, but we know that you want to get down to the interesting stuff as quickly as possible - so come prepared!

Missing the tutorial?

If you're missing Eclipse Con Europe, or can't make it to the tutorial, then you might want to join in with the next training course at BREDEX at the end of November (28th-30th). Have a look at the BREDEX website for more information. If you want a more personal approach (like learning how to use Jubula based on your own application instead of on example applications), then you might want to think about a workshop.

Other Jubula activities at ECE

There are a few other talks being held by members of the Jubula team, you can come and chat to us in the exhibition, and we're also holding a Jubula Programming Competition on the Thursday night. Sign up to participate and win great prizes!

Looking forward to seeing you all next week!

• Eclipse
• Jubula Tutorial
• Jubula
• Eclipse Con Europe
• Jubula Programming Competition

Since starting at BREDEX earlier this year, one of my roles has been setting up automated smoke tests for the Eclipse Packaging Project. To do this, I’ve been using the Eclipse for Testers package to write Jubula tests.

Testing the EPP
The Eclipse Packaging Project is a reasonably complex bear in terms of testing. For each build, various dependencies have to be tested for completeness, and the correct integration of their component parts must be checked. In a typical release cycle, that means once per package for each release candidate, multiplied by six for the different platforms. Not a job anyone wants to have to do manually, really. Nevertheless, there is currently no common strategy to automate tests for this work.

My job has been to find out whether such a common strategy is possible over the different Eclipse flavors in the EPP, and to implement it as a proof of concept with Jubula.

Come to the talk
The results have been very promising so far. So promising, even, that I’ll be talking about Testing the EPP at Eclipse Con Europe in November. Basically, there are features common to all packages that can reuse the same Test Cases, features whose testing is only minimally different (so they reuse the same Test Cases but just add a few details), and features that only exist in specific distributions. I’ll be explaining how this is set up with Eclipse for Testers, and be showing what sort of things are being tested.

Participate!
This is all very well and good, but you’ll have noticed the title of this entry – we’re looking for community involvement in testing the EPP. BREDEX will obviously be continuing to write tests for the Eclipse for Testers package with Eclipse for Testers itself, but it would be a shame to miss out on the chance to get something collaborative going that benefits all the packages. To do this though, we need your help. We need people to define what tests should be performed for each package, to write those tests and to work with us on getting a continuous build and test process set up at Eclipse.

So here is our call: come to the talk, join in on the mailing list, speak to me or one of the members of the Jubula team at Eclipse Con Europe and let’s get some tests up and running!

• Eclipse
• Software testing
• automated tests
• Eclipse for Testers
• Eclipse Packaging Project
• Jubula
• Release
• Eclipse Con Europe

What’s new?
This time it’s really worth talking about what’s new for Jubula, because it’s our first release since the first release (…if you see what I mean). So last time, everything was new. This time we have a nice list of shiny new never-before-seen features to present. We’ve added support for testing applications started via Launch Configurations, a new embedded AUT Agent (which is also a part of the Jubula Feature / Eclipse for Testers distribution), and we’ve done some work on the object mapping editor.

The crystal ball of component recognition
One of the things that’s interested us for a while is being able to see into the future. We’ve put a lot of work into our object recognition so that changes in an application’s GUI or internal hierarchies don’t necessarily lead to components not being found at runtime. However, after various small changes (which may, individually, not bother the object recognition), it could be the case that the next change is the straw that breaks the camel’s back. These are the tricky changes, because they’re the ones testers don’t get told about, or they are the ones that we don’t expect to have an effect. It’s a shame when a test encounters a “component not found” error that could have been avoided, so we’ve added some help to do just that.

Now, when you collect a component in Jubula or GUIdancer, a handy colored dot shows you how well this component can be found in the current state of the application. GUIdancer users can also use a new BIRT report to see how easily components were found during each test run. The idea is that we can keep an eye on components whose recognition values are decreasing, and remap them before they actually need to be, and maybe preventing a test error.

I wouldn’t go so far as to say that we can predict the future, but these new aspects certainly mean that we have some more information on what the future could look like. Armed with that kind of information, we may be able to prepare ourselves better

Other new stuff
There’s a variety of other nice new features, like a new Test Case replacement wizard, and a clean-up wizard for the object mapping editor. Check out the New and Notweworthy section on the Jubula page, or the release notes installed with GUIdancer and Jubula for a full overview.

You can get your GUIdancer version 5.2 from the GUIdancer Shop and Jubula from the Eclipse Jubula download page.

• Eclipse
• Jubula
• GUIdancer
• Release
• component recognition

- An entry is selected from a menu that can cause a hint dialog to appear.
- If the dialog appears, we want to click “proceed”. If it doesn’t appear, we want to carry on regardless.
(This is achieved by performing a check – we check that the dialog isn’t there, and deal with it if it is. We use a retry Event Handler to let the test be successful regardless of which variation is required.)

- The test continues

Our test was failing on the steps following the action to deal with the dialog. On the screenshot, we could see that the hint dialog was either still there, or had appeared again. However, looking at the execution log for the test up until that point showed that our check for the existence of the dialog had reported that the dialog wasn’t there… This could have been due to a synchronization problem (the check was occurring before the dialog appeared), or it could have been something else.

Watching this part of the test and checking manually quickly discounted the synchronization theory though. Feeling rather stumped, I had a look at the test specification – and found the problem.

Our check for the non-existence of the dialog used the “proceed” button in the dialog as its clue:

• If the “proceed” button is found at runtime, then the dialog is present and the button should be clicked to close the dialog.
• If the “proceed” button is not there, then the dialog is not there and the button doesn’t need to be clicked.

The problem with this approach is that if the GUI has changed, resulting in a new “proceed” button, then the test will deliver a false positive. Although the dialog was indeed on screen, neither of the buttons in it corresponded to the object we were looking for. Had we tried to click the button, we would have received a “component not found” error and all would have been instantly clear.

The solution? I changed the check. Now the test checks whether the window is there based on the title of the window, so we’ll always know for definite if the window is actually there or not. The test execution log now shows that the window is indeed being discovered, that the button (which I also remapped) is clicked, and that the check to make sure that the window is then gone is also successful.

If the name changes, our test will fail – but I can read the name change from the screenshot, or even be warned in advance that the name change is coming.

So this is my new best practice for dealing with occasional dialogs in GUIdancer and Jubula tests: react to the window title, not a component within it

• regression test
• automated tests
• Software testing
• Successful testing
• dealing with occasional dialogs
• component recognition
• Jubula
• GUIdancer

This year we had a wide variety of topics over the course of the day, focusing on two main aspects: process and technology. The mix seemed to be a good one – the feedback forms show that different participants enjoyed different talks, so I feel like we managed to give everyone something to take away from the day, regardless of their interests and background.

Talk 1: Requirements to Test Cases – A long and difficult road?

The day began with an introduction from the process perspective – Mirko Drobietz from FSS Consulting GmbH gave his talk on requirements, quality and testing. After looking at the types of requirements we should be gathering, he advocated the use of checklists to make sure that requirements are not being forgotten. He also spent some time exemplifying the intricacies of language and their effect on how requirements can be ambiguously interpreted. One very interesting aspect of his talk was his support for paper prototypes – they can be used to clarify requirements without giving the impression that the implementation is production-ready or set in stone (dangers that easily result from programming prototypes). His conclusion was that our tests and quality are only as difficult as our planning (or lack thereof) allow them to be – if we plan well, then the road is much easier.

Talk 2: Fokus!MBT and ModelBus – Heading Towards Test Automation

Armed with this introduction, we moved straight onto and into the depths of some of the technology available for testing. Max Bureck from the Fraunhofer Institute gave a presentation of Fokus!MBT and ModelBus for generating test cases, attaining traceability between Test Cases and requirements, and early specification testing using models. Fokus!MBT and ModelBus use and integrate with Eclipse technology, and can be used to make the start of testing much earlier, as well automating parts of the test process.

Talk 3: UX-Testing – Ergonomics and User Experience

After a coffee break, we continued with a very different view on testing, presented by Alexander Klein from BeOne Stuttgart GmbH. His focus was on User Experience and ergonomic aspects of testing – both facets that tend to be overlooked. As well as reiterating what Mirko Drobietz had said about the importance of paper prototyping, he made us aware of ways to perform UX tests. Although there is very little to no automation available for the discipline as yet, there are techniques such as user observation and user stress testing (a concept I find quite amusing and yet ethically questionable at the same time ) that can be used to assess the usability of a product.

Test automation – more than just writing Test Cases

The final talk before lunch was presented by two students, Jonas Wolniczak and Markus Sagurna, from the Hochschule Bremerhaven, who spoke of their experiences in a project about test automation they had completed for their Bachelor studies. Despite the focus on test automation, the students involved were responsible for all aspects of the project lifecycle including requirement gathering from the customer, implementation, project organisation (Scrum), test environment and the continuous build process. They reported that the project allowed them to see how tests and automation fit into the development process, how easily test aspects (or testing itself) can be forgotten, and the consequences of starting testing later on. I must admit I had a personal interest in this talk, as I knew about the project they were completing – Achim Brede from BREDEX played the role of their customer and they used GUIdancer as their automation tool. It was great to talk to
Jonas and Markus afterwards (I was one of many who did!) and hear about how positive and helpful their experiences were doing a full-scale project for the first time.

Talk 5: Automated GUI Tests in functional levels

Lunch came and went, and we sat down for the second half of the day. The afternoon began with a talk that combined technology and process, given by Patrick Möller from Bitmarck Software GmbH. Their interesting predicament is the collaboration and communication between their field test team and the automation team. They outlined their current process and situation before describing steps they have implemented with the aim of receiving unambiguous regression Test Cases for automation. Their story is very much ongoing, and I will be interested to hear a follow up on it at a future date.

Talk 6: Understanding Eclipse-Plug-in Test Suite

The talk before the afternoon break was held by Micheala Greiler from the TU Delft. Michaela is person behind the Eclipse Study for testing and her name is highly associated with testing in the Eclipse ecosystem. She presented ETSE – Eclipse Plugin Test Suite Exploration – a tool for analysing which plugins, features, extensions and services are addressed by a test run. The demo showed that the visualisation of such complex content is clearly modeled, and I for one am excited to see how ETSE could be integrated into existing Eclipse test tools (Jubula comes to mind, for example ).

Talk 7: Testing different variants: Testing the Eclipse Packaging Project

The penultimate talk of the day was given by Felix Ziesel from BREDEX. He has been working on a proof of concept for automated GUI tests for the Eclipse Packaging Project using Eclipse for Testers. He demonstrated the structures used to create general Test Cases (applicable to all packages), parametrised Test Cases (that require data for specific packages) and individual Test Cases (to test functions only available in single packages). He concluded with a demonstration of an automated test for Eclipse for RCP and RAP developers making use of all three types of Test Cases.

Talk 8: Beyond “Works-On-My-Machine” with PDE Build and Git

After starting the day with requirements, it seemed fitting that it should end with a talk about build and release processes. Matthias Kempka from EclipseSource spoke of how to avoid the “Works on my Machine” certificate. His talk was choc full of tips, tricks, tooling, patterns and information sources about getting a build up and running with PDE and Git. He’s written a blog entry on the topic as well if you want to read in more detail.

After eight talks about testing, it was time for a well-deserved beer and some snacks to finish off the day. We BREDEX guys got back yesterday (Thursday) and immediately began looking at the feedback forms. The messages were generally very good – everyone seemed to find something that was relevant to them, and the breaks were definitely better structured this year. Next year we’ll maybe be looking for some interactive content for the afternoon to have a change of pace after a few talks, so we’ll be excited to hear your ideas for what you’d like to do.

This has been a long blog entry, but before I finish up, a final thank you to all involved in the day. From the sponsors and organisers (Eclipse, BREDEX GmbH, Xored, GFB Softwareentwicklung and sepp.med) to the supporting organisations (Testing Experience, ASQF, Professional Tester and EclipseSource), to the Commundo Tagungshotel Neuss and ultimately, to the participants – thanks for a great, interesting and thought-provoking day. The slides will be going up in the near future on the Eclipse Wiki, and we look forward to seeing you next year!

• Eclipse
• Eclipse Testing Day
• Successful testing
• Software testing
• Requirements to Test Cases
• Fokus!MBT
• ModelBus
• UX-Testing
• automated tests
• Test automation
• Automated GUI Tests in functional levels
• Eclipse-Plug-in Test Suite

On Thursday night at ECE, we want to see your extensions, add-ons or integrations for and with Jubula. You’ll have five minutes to demonstrate your work to a community audience, who will vote on their favourite idea at the end. The focus is definitely on innovation and fun – we want to see your ideas and proofs of concept for how you could provide Jubula integration or adaption for your project, process or tool-chain. The first prize will be an AR.Drone, and we’re pleased that the Eclipse Foundation is sponsoring some swanky Eclipse fashion items for runner-up prizes

Competition slots are limited, so register your entry soon! Details (prizes, ideas, tips, etc.) are on the registration page:
http://pimpyourjubula.eventbrite.com

Happy thinking and coding, and we’ll see you in Ludwigsburg!

• Jubula
• Eclipse Con Europe
• Eclipse
• Jubula Programming Competition

Most of us know that our testing could be improved. The question is how? Do we need to find the right people, make our processes better, or just use the right tools? Maybe it’s a combination of the three…

The best way to find out what has (not) worked for others, and to get concrete ideas for making your testing give you better results, is to come to the Eclipse Testing Day on 7th September in Neuss!

This is the second year of the Eclipse Testing Day, and we have a program including talks on test automation, testing Eclipse, build processes, requirements, test case design and writing ergonomic software.

Our speakers are practitioners, researchers and experts. Come and learn from them, and from the discussions with other participants and sponsors in the breaks.

Registration is open at: http://eclipsetestingday2011.eventbrite.com. Tickets cost 50 € or 40 € for Eclipse and ASQF members. We also have discounted tickets for students.

Get signed up for an excellent day of testing talk: Your quality – and your customers – will thank you for it!

• Eclipse
• automated tests
• Errors
• Eclipse Testing Day
• Software testing

This year we have eight talks over the course of the day. Topics range from acceptance testing and usability testing to build processes and test processes (requirements, test case definition…). In short, if you’re looking for a chance to hear from the experts for testing, test processes, and the tools and technology that go hand in hand with them, then you should come along!

Alongside the talks you’ll also have the chance to chat to our sponsors and see what they have to offer in terms of testing and process. BREDEX are hugely grateful to all of our co-sponsors and supporters – including the Eclipse Foundation, Xored, GFB Softwareentwicklungsgesellschaft mbH, ASQF, EclipseSource, Testing Experience Magazine and Professional Tester Magazine. If you want to be added to this illustrious list, then there are still sponsorship opportunities available.

Full details (including the agenda) are available on the Eclipse Wiki at:

http://wiki.eclipse.org/Eclipse_Testing_Day_2011

Registration is open now via our Eventbrite site. Tickets cost 40€ for members of the Eclipse Foundation or the ASQF, and 50€ for non members. We also have a limited number of student tickets at a reduced price. The ticket cost includes catering on the day.

Looking forward to seeing you all there!

• Eclipse Testing Day

The task for today is “Creating GUIdancer reports by command line” to be run by Hudson.

If you are an experienced BIRT (http://www.eclipse.org/birt) user, this might be obvious, but for the others I will describe the steps:

The short version is to do the following: download the BIRT runtime, find the GUIdancer report design files and run the report.

The long version (tested with Linux and Windows) is below:

1. Detect the BIRT version

Look for org.eclipse.birt.* in the GUIdancer installation directory.
I found: guidancer\plugins\org.eclipse.birt.report.viewer_2.6.1.v20100913
The BIRT version is 2.6.1.

2. Download the Birt Runtime

Download 2.6.1 from http://download.eclipse.org/birt/downloads/ (look for “Report Engine”).

3. Find the report design files

Look for *.rptdesign file in the GUIdancer installation directory.
One of the report definitions I found was guidancer\plugins\com.bredexsw.guidancer.reporting.birt.viewer_5.0.00340\reportsGuidancerHistoryAbsoluteAndCoverage.rptdesign

4. Unzip the runtime and define BIRT_HOME

Set the environment variable BIRT_HOME to runtime directory.

5. Execute genReport script.

For windows:

genReport.bat -f PDF -o outputfile.pdf -F insert_path\GuidancerShort.rptdesign

Output:

SEVERE: Required parameter P_START is not set.

You will find a genReport.sh script for unix.

6. Define the report parameters

The report parameters are configured in file GuidancerLIB.rptlibrary.
We need

• P_START : Report start date in mm/dd/yyyy (format is locale dependent)
• P_END : Report end date
• DB_USER: database user
• DB_PASSWD: database password
• P_PROJECT: name of GUIdancer Project
• P_TS_NAME: name of GUIdancer Test Suite

7. Optional: Deploy jdbc driver

The runtime will need a jdbc driver to connect to the test database.
Looking for *jdbc*.jar I found:

• postgresql-8.4-701.jdbc3.jar
• ojdbc14.jar
• oda-jdbc.jar

I need the Oracle driver and therefore copy ojdbc14.jar to birt-runtime-2_6_1\ReportEngine\plugins\org.eclipse.birt.report.data.oda.jdbc_2.6.1.v20100909\drivers

7. Run the report

Example for PDF:

genReport.bat -p “P_START=01.02.2011″ -p “P_END=17.06.2011″ -f PDF -o outputfile.pdf -F GuidancerShort.rptdesign

To create a HTML report use -f HTML.

• Eclipse
• GUIdancer
• BIRT Report
• Jubula
• automated tests
• Creating GUIdancer reports by command line

The Eclipse Jubula Team is pleased to announce that Jubula version 1.0 is now available! It has been a year of challenges, the occasional surprise and, sometimes, a steep learning curve to get to this place, but the hard work (at BREDEX and at the Eclipse Foundation – thanks guys!) has paid off. Since making the decision to follow the release train, this date has been … erm … looming in the distance looking rather stern. We’re proud to be able to say we made it. Not only that, we also graduated from incubation (hence the 1.0 version) and managed to get GUIdancer 5.1, Jubula’s twin brother, released at the same time.

So now it’s hot off the press, you probably want to get your hands on it

You can get hold of Jubula in three ways:

1. As a Feature via an Update Site
2. Automatically as a part of the new Eclipse for Testers package
3. As a Standalone Application

All links are on the Jubula download page.

If you’re planning on working with Jubula as a Feature (Eclipse for Testers or via Update Site), then you’ll also need to make sure you’ve got the following:

• the AUT Agent (server component) which is necessary to run tests, do object mapping etc. We’re already working on having this as a Feature, but you’ll have to take the AUT Agent from the Standalone application for now.
• the database drivers (for anyone wanting to work with other databases than the embedded database). These are available via the Eclipse Marketplace Client.

Anyone wanting to use Jubula for HTML testing should probably use the standalone for now – the plugins for web testing are not currently available via the Marketplace.

Remember you can get in touch with us via mailing lists, Bugzilla and the forum if you’ve got any questions. The next training course at BREDEX is also coming up soon (25th -27th July), so if you fancy getting a thorough introduction to testing with Jubula or GUIdancer, then just drop us a line!

• Standalone Application
• Jubula
• GUIdancer
• Eclipse for Testers
• Eclipse
• Release

The use case for doing this is more often than not some kind of shared data. Application A enters the data and application B should receive it. If someone is working on the data in application A, then a user in B should only have read privileges etc.

Working with multiple applications for tests in Jubula requires the use of two constructs: Test Jobs and the autrun command.

Test Jobs – an overview

A Test Job is actually nothing more than a collection of Test Suites to be executed after each other. You can create a Test Job via the context-menu in the Test Suite Browser.

Once the Test Job has been created, you can open its editor via double-click and add Test Suites to it via Drag&Drop. In the Properties View for each referenced Test Suite, you’ll have to select an AUT ID. This is the unique identifier used to be able to find the correct application at runtime. Every AUT you start has (or receives) an ID – more on IDs later.

What types of applications?

The first point to look at is whether the application(s) to be tested use the same toolkit (GUI library). If so (i.e. both are RCP applications), then there are no limitations on the interactions you’ll be able to perform in either application.

If the applications use different toolkits (i.e. one application is Swing and the other is RCP), then you will have to set the toolkit for the Project to concrete.

This will allow you to start and test both applications but will mean that the RCP-specific actions usually offered in the modules for RCP and SWT are not available. You’ll not be able to perform actions such as working with a GEF canvas or tree tables, for example. The “unbound_modules_concrete” actions (which are valid for Swing and RCP, and contain most of the necessary actions to work with applications) are available for writing tests.

Getting the applications started

Before we get to the various ways of starting applications for testing, a tip in advance: If you will be starting the same application twice, then use two AUT Configurations (they will share an Object Map so you don’t have to repeat the mapping). If the applications you are testing are different, then create a new AUT Definition for the second.

Once that’s cleared up, you can choose one of the ways below to make your applications available for testing:

Testing interactively

If you plan on starting the test manually (i.e. by pressing play in the Integrated Test Environment), then you can write AUT configurations for each application to be started, and start each application individually via the Start AUT button on the toolbar. The AUT IDs as defined in each AUT configuration can be selected for the individual Test Suites in the Test Job. As all the applications are started manually before the test, you don’t need to worry about starting any other applications via autrun. Having said that, this approach is usually only interesting when you are trying out your tests. Normally, it’s preferable for tests to be started via a CI tool (in our case, one that starts our Test Executor with the necessary parameters).

Testing in unattended builds via autrun

If you want to use the Test Executor to run your Test Job, then you’ll need to have some way of making sure that your applications get started. Unlike for Test Suites, the Test Executor cannot (currently) start an AUT from a configuration. The way to start an AUT without requiring an AUT Configuration is to use the autrun command. This is available from Java 5 for applications that can be started by an executable file.

autrun is a small program that can be found in the autrun folder in the installed (standalone) version of Jubula. Navigate to this folder in a command line client and enter the following:

autrun.exe
–a <the hostname of the agent you want to connect to>
-p <the port number for the agent>
-w <the working directory for the AUT>
-i <the ID that that AUT should receive when it is started>
-<TOOLKIT>
-e <the executable file>


The options available for the toolkit parameter are –rcp, -swing and –swt (HTML applications can’t be started using the autrun option). For RCP applications (i.e. Eclipse), you will also need the parameter –k .

Bear in mind that AUT IDs are unique throughout the Project. Start your AUT via autrun with a different AUT ID than those already used in the Project.

An example autrun command for an Eclipse application could look like this:

autrun.exe –a localhost -p 60000 -w C:\Programs\Eclipse -i Eclipse_autrun -rcp –k en_US -e Eclipse.exe

If you are currently connected to the Agent on the port number specified, then you should see an entry in the Running AUTs View in Jubula when the AUT starts. If the AUT is not already declared in the project, then it will be displayed as unknown:

For any AUT started by autrun, you need to enter an AUT Definition (not a Configuration) in the Project Properties. There is a specific section for AUTs started by autrun and you can open the dialog by selecting the (as yet unknown) AUT and selecting “Create AUT Definition” from the context-menu.

Create AUT definition

Fill in the necessary details and click OK. Now this AUT ID is known and can be used for this Project. You can select the AUT ID in the Test Job Editor for the necessary Test Suites.

Selecting the AUT ID

In terms of your test, you can now do the following:

1. For the first AUT, make sure that it is started from your CI tool as a part of the test setup. This is a prerequisite for the Test Job.
2. For any other AUTs in the test, you have a choice:
   • You can call autrun from your CI tool for each AUT required
   • Or you can start a small script from within the test itself (via Execute External Command).

The former option has the benefit that the AUT gets started even if specific parts of the test don’t run, but does mean an overhead in the CI scripts. (You will, however, have to start the first AUT from the CI script, as the Test Job cannot run otherwise). Starting an AUT from within a test is usually lower overhead, but it does introduce a dependency between separate Test Suites in a Test Job. However, the very idea of a Test Job is to link dependent Test Suites, so the dependencies exist anyway.

Applications that start other applications

Some use cases have a different set up entirely. The AUT itself causes another AUT to be started. Jubula can deal with this situation based on the following assumptions:

1. The AUT Agent has been started in Lenient Mode. This allows AUTs that are started by other AUTs to connect to the Agent. You can switch to lenient mode for the AUT Agent via the context-menu on the system tray icon, or you can start the Agent with the parameter –l
2. The AUT that is started has the same toolkit as the AUT it was started from (e.g. RCP)
3. If multiple AUTs are started in this way, their order for the test is known.

If these three assumptions hold true, then you can define and use AUT IDs for applications started in this way by declaring the newly started AUTs as ID+1 ID+2 etc.

Switching between applications

Once you’ve:

• made sure your applications will all be started
• declared the AUT IDs they will use in the Project
• created a Test Job containing the necessary Test Suites
• selected the correct AUT ID for each Test Suite (this is even necessary when you are testing different instances of the same AUT)

… then you can start your test.

What you will notice fairly quickly, however, is that you will most likely need some way of making sure the correct application is in the foreground. Jubula cannot force this, as there are no clear-cut platform- and window-manager-independent commands to do so. It is up to your genius as a tester to make a module that will always bring the correct application to the foreground.

As luck would have it, we’ve worked with this construct a few times and have found the following module to be useful:

The Test Case performs a click in the active window – or tries to. If the window that receives the click is not the expected application (based on the AUT ID), then an “Action Error” is thrown. The Event Handler reacts to an Action Error and performs “Alt+Shift+Tab” using “External Key Combination”. This brings the next application to the front. The Event Handler is specified with the Reentry Type “Retry”, meaning that the failed Test Step (the click) is retried once the Event Handler has been executed. If the application that is now in front is the correct one (i.e. has the right ID), then the Test Step “click in active window” is marked as successful. The “Number of Retries” lets you define how often to do this loop. Once the AUT you require is in front, then the test is marked as successful and your actual tests can begin.

If you’re working on Mac systems, then it is advisable to alter your settings to make Alt+Shift+Tab switch between your applications in order (Alt+Tab alone switches between two applications, and Mac systems use another key combination by default to tab through a list of applications).

This little module lets you have as many windows and AUTs running as you like – you’ll always get to the right one eventually (just make sure the retries are set high enough)!

Test results and variables

Between each Test Suite, the test results for that Suite are written to the database (no accumulation of results for the whole Job as yet). However, if you want to work with variables between Test Suites, that’s no problem. A value stored as a variable in one Test Suite is available for the rest of the Test Job.

Testing multiple applications is one of the more complex use cases to automate. Hopefully these steps will make it easier to get started. At some point I’d like to see support for starting AUTs from configurations for a Test Job and a cumulative test result for the whole Test Job, but the current system already let you start your AUTs and then run and analyze your multi-AUT tests to get those tricky use cases automated.

• Test variables
• Test Suite Browser
• Test Suite
• Test results
• Test Job
• Test Executor
• Switching between applications
• Software testing
• Jubula
• Jubula Tutorial
• Errors
• GUI library
• Eclipse
• automated tests
• AUT Configurations
• Applications that start other applications
• Testing in unattended builds via autrun
• Testing interactively