Exploratory testing - a rookie's thoughts (part 3)

Q: What other testing activities have you done besides pair testing?

A: Well, we also did something I named parallel testing, which involves less teamwork but can be just as engaging as pair testing. I’ll try to explain what this means using a similar analogy I used for pair testing. I compared pair testing to a car ride, where one tester is the driver, while the other is a passenger in the same car. Parallel testing is similar, with both testers being drivers (both with their own keyboard/computer), in slightly different cars (this can vary between using different browsers/OSs/computer configurations), driving parallel to each other (testing the same piece of software/section of application), with a communication link between them (both testers are within earshot of each other - in same office or room).

One aspect of this method is that each person tests the same section or even features that overlap with one other. This makes it possible to test things in more detail, knowing that there is another person that is doing tests on the same piece of software.

Instant feedback is another aspect to consider. A different tester is basically testing the same part of an application you are, so when you would normally ask someone to help test something the entire introduction to what you are doing can be skipped. Of course there are some cases where you just end up diverting their attention to something less significant you want them to try out and they might miss something important...

Googling parallel testing turned up finds pointing to a different procedure than what I was referring to, (testing an old version of software in parallel with a new version, checking for inconsistencies between the two), so the name might not be the most appropriate, but it will have to work until someone suggests a better one. :)

A mindmap of my Eurostar 2011 experience

Having just returned from my first Eurostar conference, I've made a mindmap of all the sessions and tutorials that I attended and tried to include all the notes I had (if any) on each of them:

(opening the image in a new tab will allow zoom-in)

The mindmap is also shared online here: http://www.xmind.net/share/rucindrea/eurostar-2011/

I also want to thank all the people that I met in Manchester this year and that made this first Eurostar experience a very special one. 

-Ru

Exploratory testing - a rookie's thoughts (part 2)

Q: How about reporting issues? How did you go about logging bugs while pair testing?

A: Logging the bugs we found took longer than expected. This means, that we found something, determined that it was a bug, investigated its cause, then spent too much time logging the said bug (even if we agreed on the cause and the effect of the bug, as well as on the steps to reproduce it); I guess you could say we didn’t agree on what information to include in the report, what order was the most appropriate, and what was relevant as far as that particular bug was concerned.

I don’t have a concrete example for this, but I seem to recall logging a tricky bug that overlapped with another one and my testing partner suggested adding information from one bug in the other’s report, while it was clear - to me at least - that the piece of information they wanted to add was not relevant to the bug report at hand. I realize this is quite biased, seeing that this is only my side of the story I suppose….

Bug reporting may have been slow because when I write things down, I constantly revise it until it starts to resemble a coherent thought (this fails at times though... miserably even). Or it may have been because my level of English was different than that of my testing partner. It was probably a little bit of both. :)

Or maybe this is a personal thing, I don’t know; I tend to be a perfectionist when it comes to putting my words on digital paper and I consider committing anything to words to be mainly a solitary activity, after all it’s your ideas on digital paper. Besides, two people can have very different ways of writing the same thing down, even if the main idea is identical. So you would have a train of thought and your partner would have a different one, and oftentimes they would rather collide instead of combining into a cohesive bug report.

All in all, my conclusion is that having someone next to you can help a lot during testing, while it could be a hindrance when logging bugs.

Exploratory testing - a rookie's thoughts (part 1)

Here are some of my thoughts in the form of questions and answers, which mostly come from feedback I gave Alex and Oana on exploratory testing when I first started out as a tester, and although much has changed since, I still have a lot to learn in order to become better at my craft… Enjoy! :D

Q: So… pair testing; comparing the experience to testing alone, what are the things you did differently when testing with someone else?

A: Pair testing? Err… don’t you mean peer testing? Hm…

*thought about this for a while, then googled a bunch of stuff regarding peer/pair testing*

Okay, let me try to explain why naming it peer testing makes sense to me: if someone is your peer that means the person is of the same level as you (viewed as your equal, if you will). So if two peers sit down together to test a piece of software, there is no hierarchical difference between them, one tester does not simply oversee the other tester’s progress, both testers are interchangeable. I’m not saying pair testing implies inequality between the two participants, but it does not exclude that possibility either (one could be a developer or a business analyst - according to the Wikipedia page on Pair testing, anyway). I also did a Google search on peer testing, turns out it’s also used in the way I’ve used it, but pair testing is more widespread, so I guess for the sake of familiarity, I’ll change the name.

Regardless of how you call it, one thing is certain: there are far more questions raised by two people looking at the same piece of software than going at it alone. Instead of sitting in front of a monitor and wondering what this feature does, you engage in conversation. The testing partners enter a kind of dialogue in which the third participant is the application itself.

Another thing is you bounce ideas off each other about how a user might act in a certain situation or what might be considered a bug in what situation. There are also occasions where you just look at each other perplexed and wonder what had just happened and how you could make the application do that again.

The way I see it, the person sitting at the keyboard focuses on the task (or scenario) while their partner takes notes and can look at the overall changes happening in the application. It’s kind of like driving a car really, the driver, as in the person at the keyboard, focuses on the road ahead, and is basically in control of what the application is doing, and sure there are times when the person next to them tells them what to do or where to go, but backseat driving is never popular. :P

Meanwhile the passenger can look around freely, without any focus in particular. They can look at different things that happen throughout the application, not just what the “driver” is looking at. This is where most of my ideas for possible tests came from. When you are focusing on one thing happening (or waiting for some particular thing to happen), you can’t see all the possibilities for bugs just waiting to happen. Of course some of the ideas are a little far-fetched, and wouldn’t necessarily discover bugs in most situations.

Letsdoitromania - a great testing exercise

This weekend Altom participated for the second time in LetsDoItRomania! The program is part of http://www.letsdoitworld.org/, started by the Estonians in 2008, that has the target to clean the country by involving a big number of people. This year we didn't have the time to get involved in the organization of the event, but we managed to gather a team of 5 people: Ionela, Levi, Ramona, Oana and myself :D.

We got the garbage bags and the plastic gloves from the organizers, we bought two pairs of industrial work gloves  and we borrowed a fork and shovel (lessons learned from last year), we chose two garbage sites near Valisoara, a remote village near Transilvania Highway, and on Saturday morning, 24th of September, we left Cluj to clean them :D.

This whole experience reminded me a lot of the software projects I'm involved in as a tester. I think this was a great example of how someone can learn about software testing by doing something not related to work, and I will try to explain why.

Delayed start

On Friday we had decided to meet the next day at 9:00 AM and leave for the selected sites, but on Saturday morning when we looked for the sites details (pictures, GPS coordinates, description) the application was not showing any of the needed information. We found out that the organizers had set up a local call center, we called them (we had to wait until we were able to get someone on the other line) and someone was able to give us the GPS coordinates for the selected locations. 

All this hassle delayed our departure with around 20 minutes. One thing we could have done to minimize the risk of this happening was to look for the information on Friday evening, but we were too optimistic and left this important task for the last moment :).  

Has that never happened to you when starting a new testing project: realizing that you didn't have important information, and in order to get it you had to call someone on the client side/development team which delayed the start of your work?

Our village doesn't have garbage

Once we found the information we needed, we entered the GPS coordinates and we left for the 1st location. We arrived in the village, and even if we had the coordinates, we stopped to talk to an old man and get some inside information :). 

I first asked him if he knew about areas used by the village people to dump the garbage, and his answer was that the village was clean, as they had a garbage collection system and that took care of the problem. I continued by asking: "well, but before this system was put in place, where did you use to dump the garbage?", and his answer was: "well, but that was a long time ago...".

This short discussion reminded me of lots of developer's / engineering manager's answers I got while testing: "our code doesn't have bugs, as we use this system/process (please read garbage collector, unit tests, code review) that makes sure everything is taken care of."

I told the old man that it had been reported that there were areas in the village where garbage was dumped, and his first reaction was: "Who said that? Was it someone from the village?". I said "No, not someone from the village", then he replied "Well, if they're not from the village, then they don't know"

Working as a consultant I often noticed this attitude: "if you're not from the organization, how dare you  say we have a problem?"

Changes you don't know about

We left the old man, and continued our way to the first garbage site. We stopped where pointed by the  GPS coordinates, but unfortunately there was no dump yard there. Instead, we found some men who were trying to build something. I got off the car and went to talk to them. One guy came towards me and told me that he had bought the area one year ago, and that he had cleaned most of the garbage.

It happened to me while testing to find out that parts of the application or functionalities had been removed, but the documentation had not been updated in the meanwhile. This was the case with this location too: the garbage site had been reported last year, but no one cared /had the time to check and update the details of the site.

We phoned the call center again, and asked them to assign us another site somewhere in the neighborhood. Once we got the new details, we headed to the new site which was at the other end of the village.

Understanding the impact

The site was on the water stream, which is quite a common location. For some reason people think that it's OK to throw their garbage in the stream / river, as the water takes it away, but they don't think that the people up the stream can, and will, do the same thing or that their garbage will harm the people down the stream. 

The way we perceive the impact of our actions always amazes me. As the villagers don't understand that the animals (domestic or wild) will drink from that water and that they will later eat the meat or they'll drink the milk and get sick. 

It often happens that we don't understand or minimize the risks introduced by the changes done in the software application we develop.

Estimating the amount of garbage

The procedure when mapping the garbage sites is the following: someone identifies the location, gets the GPS coordinates, takes some pictures and enters some details describing the type of garbage that is found there and estimating the amount of garbage bags needed to clean up the location. This was the third time I took part in a cleaning action (last year I participated both in the pilot event and in the national event), and every time the estimates were way off. For this year's location, the estimates were 25 garbage bags. We collected 44, and we could have had more if we had continued until sunset.

Another thing I noticed was that the garbage bags were not identical: some had more garbage than  others, they contained different things (plastic bottles, clothes, rubber, glass bottles...), some were heavier that others.

This reminded me of a project I worked on that used some algorithm to estimate the number of bugs that should be found in the application, and afterwards reviewed testing based on bug numbers...

Coverage

After approximately four hours of intense fight with the garbage and nettles, we decided to stop. We had to call the organizers and tell them the exact location where we had left the garbage bags so pickup trucks could come to take them. 

One of the questions they asked me was the coverage percentage for that site. I told them that the estimates were 25 bags and we collected 44, so what kind of percentage did they want?

One thing I want to specify about garbage dumps is that they pile up between plants and the earth brought by the water. It's like thinking of different layers of an application. So if one continues to dig, there's a big chance that s/he'll find more garbage. This is why I thought that providing a percentage was kind of useless. In order to know the percentage, we needed to know the total amount of the garbage that was located on the site, something we couldn't know for sure.

The girl on the phone insisted I give her a number, and I said 70%. While I'm writing this I question my answer and wonder if that was the right thing to do... If a client asks me for something, and I explain them that this number is useless but they still ask for it, is it OK to give them my best guestimate? I've offered them my advice, but they are the ones paying, after all... For an interesting view of this issue please see the first 5 minutes of this TEDTalk - the Japanese tea story: http://goo.gl/I64c.

One of the things that I like most about being a tester is that I often find similarities between my work and day-to-day events, which makes learning more interesting and fun :).