In this module we start to put together some API tests to check the REST Api that is delivered as part of a Rocket Chat install. The moment the Selenium tests complete Jenkins will trigger SoapUI to run a number of REST based API tests from one of the Windows AWS client machines. In setting this up we’ll cover a few key areas:

1. Looking at the Rocket Chat API calls
2. Installing and writing SoapUI REST API tests
3. Driving SoapUI from Jenkins
4. Using the Jenkins xUnit plugin to display test results

We’ll run in to a few small challenges whilst setting this up. For example the rocket Chat Api returns some security credentials after the initial login. In order to successfully run all the other REST tests we need to pass those credentials between all the different test cases. We’ll look at how you can configure this in SoapUI.

Once we have SoapUI running the tests we’ll see how we can save the results in JUnit format. Then using the Jenkins xUnit plugin you’ll learn how to capture those test results and display them from within Jenkins. This is a great way to introduce you to an important plugin that has a multitude of uses when tracking xUnit results from various sources.

What You’ll Learn

In this module we’re going to look at creating and running some REST API tests. We’re going to use the open source tool SoapUI to run these tests. Of course these tests will all be initiated from Jenkins once the build of the Application Under Test (AUT) is complete. We’ll look to run these SoapUI API tests in serial from the same test machine that we’re running the Selenium tests from. The Selenium tests will complete then the SoapUI tests will be kicked off.

Once this is configured we’ll have these pieces in place on our test automation rig:

This is an important module because we’re starting to build out the bredth of our test coverage now. You’ll find the configuration of Jenkins quite straight forward now. However, we’ll show you how to overcome a few key challenges like passing parameters in SoapUI and displaying JUnit results within Jenkins.

The Tools We’ve Chosen

We’ve picked SoapUI. Mainly because of it’s popularity but also because it’s GUI is easy to get started with. It’s also simple to covert the tests we write into a test suite that can be run from the command line – which is what Jenkins needs.

SoapUI also provides us with the capability to record test results in JUnit format. This format can be read by the Jenkins xUnit plugin which allows us to display all of our REST Api test results from within Jenkins.

Worth pointing out here that the Rocket Chat API is a REST api. SoapUI might imply support for SOAP api’s but it does support REST too.


If you’ve followed Modules 1, 2 and 3 you should already have your Amazon Virtual machine environment up and running along with Jenkins and Selenium. This setup gives us the 3 machines we’ll need to use in this module.

1. Windows Master machine: this is running Jenkins and controlls all our other machines (including the installation of the AUT and the exection of our Selenium tests). This machine will be responsible for kicking off our SoapUI API tests

2. Linux Client machine: this Ubuntu linux machine is run up on demand by Jenkins and then has the AUT (Rocket Chat) autoamtically installed on it. This machine provides the web interface for the Rocket Chat application and the API for the Rocket Chat application.

3. Windows Client machine: this Windows Server 2012 machine has 3 browsers (Ie, Firefox and Chrome) installed on it along with Selenium. The Windows master machine kicks off the Selenium tests that drive the 3 different browser tests against the Rocket Chat application.

Check the Status of your AWS Machines

Both of the windows machines should already be running. The Linux machine may or may not be running. The Linux machine is run up automatically by Jenkins so it’s fine if it’s not running. Whatever the state of the linux machine you should see the Windows machines status in the AWS console as follows:



Open an RDP Terminal Session on the Windows Master Machine

With these windows machines running you’ll need to open an RDP session on the Windows Master machine. This is where we’ll be configuring Jenkins.




Then enter the password (you may need your .pem private key file to Decrypt the password if you’ve forgotten it) to open up the desktop session.

Start the Linux Client Machine

IF the Linux machine isn’t running with the AUT installed then we need to start it. We can get Jenkins to do this for us. Once you have an RDP session open to your Windows Master machine you should have the Jenkins home pages displayed. If not open a browser on this machine and go to this URL:


From here you can start the ‘BuildRocketChatOnNode’ job and start up the AUT.




Once RocketChat is up and running we’ll need to know the host name that Amazon has given our new Linux instance. We save this in our ‘publicHostname.txt’ file that is archived as part of our build job. So if you go to this directory using Explorer

C:\Program Files (x86)\Jenkins\jobs\BuildRocketChatOnNode\builds\lastSuccessfulBuild\archive

You should find this publicHostname.txt file…




Open this with notepad and make a note of the hostname. We’ll need this while we setup our API tests.

Open an RDP Terminal Session on the Windows Client Machine

As we’re utalising our existing Clicent Windows 2012 R2 test machine we don’t need to configure any new Amazon instances or machines. We have what we need running already. We just need to open an RDP terminal and install the software we need. To open the RDP terminal:



Again enter the password (you’ll need the .pem private key file again if you need to Decrypt it) to open up the desktop session.

At this point you should have…

1. An RDP session open to your Windows Master machine
2. An RDP session open to your Windows Client machine
3. Your Linux Ubuntu machine running with Rocket Chat installed

From here we’ll start to setup our Windows test machine and configure Jenkins on the Master machine to run the tests.

Part 1: Install SoapUI

We’re going to use our existing Windows client machine for this. So once we have the RDP session open we can download SoapUI and install it on our Windows Server 2012 R2 64 bit machine.

Only a few steps to follow here then, all to be carried out on the Windows client machine:

1. Open a browser

2. Enter the following address:

3. Download the ‘Windows Installer (64 bit)’ package



4. Run the installer, accept the agrements and accept all the default options…



Once you’ve finished you should be presented with the SoapUI user interface:



Now we have SoapUI running we can check the connection to the Rocket Chat Rest API.

Part 2: Connect to the Rocket Chat API

So Rocket Chat, our AUT, comes installed with a REST Api. This API is documented here:

On your install of Rocket Chat you should be able to connect to the Api directly with a browser here…

http://<your-amazon-aws-instance >:3000/api/

Remember earlier we said you could get your host name from the publicHostname.txt file on the Windows Jenkins master machine. So if you replace the <your-amazon-aws-instance> text with your public hostname you should be able to connect to this Api from any remote machine. You’ll see something like this:



If you have trouble connecting from outside your Amazon Virtual Private Cloud you’ll need to check your Security Group settings in your AWS control panel. If all else fails connect from the Windows Client machine (the one with SoapUI now installed).

One final check before we start writing some SoapUI tests. Lets check we can the version of Rocket Chat that’s running from the API. We’ll do this from the Windows client machine. So in the Windows Client machine RDP session…

1. open Chrome (it must be Chrome)
2. browse to this URL

http://<your-amazon-aws-instance >:3000/api/version

3. check the Rest response

You should see this Json structure returned in the Rest response:



The reason we had to use Chrome there is that Ie isn’t that happy displaying Json without messing around a bit. Easier just to use Chrome.

We’ve confirmed that we can connect to the Rocket Chat API and that we can get a valid response. Lets start writing some tests.

Part 3: Creating Some REST Requests

Now we have access to the Rocket Chat API we can start creating some REST requests. These requests will be written in SoapUI and executed on the Windows Client machine. Eventually we’ll have these tests kick off automatically from Jenkins.

For now though lets write some requests in the SoapUI gui and get them working. We need to create a new REST project and …

1. create a new REST project and enter the Rest URI



and then enter the URI for your Rocket Chat Rest API and click ok

http://<your-amazon-aws-instance >:3000/api/version

This should give you something like this…



The navigator section here showing you the hierarchy of project, endpoint, resource, method and finally request. We’ll look at each of these in detail next.

2. Define the request

In the request window we’ll configure a request to check the version number. So set the following:

Method: GET
Endpoint: (already set)
Resource: /api/version

It should look like this:



This particular request doesn’t have any parameter or body send. So we’ve defined enough now to check this. Before we move on though a few basic points about SoapUI

In SoapUI we basically have 3 main panels for each request:

1. Request details: the method (eg. GET, POST, etc), Endpoint, Resource and Parameters

2. Request body: any additional content (e.g. body or parameters) that need to be included in the request

3. Response: the response that comes back from the Endpoint after the request has been sent

These different panels are shown here…



In addition to this we have our green ‘Submit Request’ button that sends the request we define. Let’s see how this works.

3. Test an Initial Request and Check the Response

We can now click on the ‘Submit Request’ button …



And check the response. The result? Probably not very exciting. You probably got the error message…

“The content you are trying to view cannot be viewed as XML”

We already know, from our earlier call using a browser, that the response is in Json format. Just need to click the ‘Json’ tab to view the response then:


We can also check the raw response and make sure we received an Http 200 Ok response code:



Next then we’ll add a few more requests to login, list the rooms and log out again. From there we can add all these requests to a test suite and define some assertions.

4. Add More Requests

At this point we need to add three more requests. Requests to login, list the rooms and log out again. To add each of these 3 requests we’re going to repeat these steps:

1. Clone Resource



2. Name Resource



3. Open the request (double click)



4. Set the Method



5. Set the resource



6. Set the parameters


You’ll need to click in the parameters field here then the dialogue box opens up where you can enter the parameters


7. Specify the Post data

On some requests you need more than just the end point and resource. You’ll need to send additional data (e.g. like a browser does when you complete and html form and press the submit button). You’ll need to define that data and pass the data using a specific content-type


8. Set the header

You only need to set these values once you’ve completed the login request (this is NOT needed on the login request). Basically you run the login request, an auth token and userId are returned from that request. In all sebsequent requests you’ll need that ‘auth token’ and ‘userId’ to complete the requests.



You’ll need to repeat the above steps and create these three requests:

Request 1 – Login

Resource Name: Login
Method: Post
Resource: /api/login
Data (1): user=admin
Data (2): password=tester123
Post QueryString: <checked>

Create this request then run it.

You should have something like this after you’ve configured and run the request:



And the Json response containing data like this:

“status”: “success”,
“data”: {
“authToken”: “-7GGh68UQ5B0zbuLAEj29DO95H3uNg2HvRgZg63-nMF”,
“userId”: “bNitLrJ3rhZq5uLMR”

You need then to take the authToken and userId and use them in the next requests.

Request 2 – List the Chat Rooms

So go back and repeat steps 1 to 8 above using this data:

Resource Name: publicRooms
Method: Get
Resource: /api/publicRooms
Data: <remove data fields>
Post QueryString: <unchecked>
Header(1): X-Auth-Token=<authToken>
Header(2): X-User-Id=<userId>

Create this request then run it.

You should get a response like this:



And Json data equivalent to this:

“status”: “success”,
“rooms”: [ {
“_id”: “GENERAL”,
“ts”: “2016-03-01T10:57:10.758Z”,
“t”: “c”,
“name”: “general”,
“usernames”: [“admin”],
“msgs”: 0,
“default”: true

Request 3 – Logout

So go back and repeat steps 1 to 8 above using the data below. Make sure you clone the ‘publicRooms’ resource as this one already has the headers defined:

Resource Name: logout
Method: Post
Resource: /api/logout
Data: <remove data fields>
Post QueryString: <unchecked>
Header(1): X-Auth-Token=<authToken>
Header(2): X-User-Id=<userId>

Create this request then run it.

You should get a response like this:


And a Json response containing this data:

“status”: “success”,
“data”: {“message”: “You’ve been logged out!”}

Don’t forget if you want to expand all these tests and create your own you can refer to the list of supported requests here:

We now have a few requests that work. We just need to convert them to tests and create some assertions that check the responses.

Part 4: Converting the REST Requests to Tests

From here then we have 4 REST requests that all give us valid responses. With SoapUI the next step is to convert them to tests and create the assertions that validate the responses. Two ways you can do this:

1. Create individual test cases for each Request

NOTE: you can do this one by one (as listed below) or you can jump to section 2 below and convert the whole lot in one go. The choice is yours.

i. Right click on the ‘request’
ii. Select ‘Add to TestCase’


iii. Create the Test Suite (just give it a test suite name)
iv. Create the Test Case (just give it a test case name)
v. Accept the default options for adding the test case



Easier to use method 2 though…

2. Create test cases within a test suite for all resources.

i. Right click on the ‘end point’
ii. Select ‘Generate Test Suite’



iii. Select <create> test suite and create ‘One Test Case for each Resource’



iv. Give the Test Suite a name (e.g. RCTestSuite)

In the navigator panel you should then see something like this…



We now need to update each Test Case so that it has some assertions that check the results from the test run. We’ll also need to look at passing some values between test cases. We’ll need to pass values because the login test case response has the ‘authToken’ and ‘userId’ values that subsequent test cases need.

3. Add the Test Case Assertions

We can select the first test case (the one that gets the version of Rocket Chat) and add our first assertion. Make sure you have the Request work space open:



Then click on the assertion button to open the assertion panel:



Then we’ll add our first assertion. For this test case it’ll just be that the response is a valid HTTP Status Code 200 Ok:



After you’ve clicked the ‘Add’ button just enter 200 as the code:



And then click okay. You can confirm that this works by clicking the green ‘Submit’ button for the request and then check that your assertion passes:

I’ll show you one more example of an assertion that checks the content of a Json response. Then you’re on your own for adding a range of assertions for all of the test cases.

Next select the Login TestCase Request and open the work space (double click the request):


This time you’ll notice that I’ve run the request before I’ve created the assertion. You can see the Json response that I want to create the assertion for now.

Click the add assertion button and select the ‘JsonPath Count Match’:



We know that if the login is successful then we’ll see this in the response…

“status”: “success”,
“data”: {
“authToken”: “-7GGh68UQ5B0zbuLAEj29DO95H3uNg2HvRgZg63-nMF”,
“userId”: “bNitLrJ3rhZq5uLMR”

What is absolutely crucial is that we get back this ‘data’ structure because we need the ‘authToken’ and ‘userId’. We need these for the subsequent requests we’re going to send. So let’s create this assertion so that it checks for the ‘data’ structure. We’ll enter these values in the dialogue box:

JSONPath Expression: $.data
Expected Result: 1

You can click the test button to check this in the dialogue box too:



From here you know enough to start experimenting with adding further assertions to the other requests. Just follow exactly the same process we’ve covered above.

What’s important now is to string these Test Cases together and pass the ‘authToken’ and ‘userId’ from one Test Case to the next.

NOTE: might be worth saving your project at this point. File menu, Save All Projects. You can save it with the default name of “REST Project 1”

4. Passing Parameters to our Test Cases

What we need to do now is work at the Test Suite level, link all these Test Cases together in a chain and pass some parameters between the Test Cases. So double click on the ‘Test Suite’ node and open the ‘Test Suite workspace’. Then click on the ‘Properties’ button to open up the properties panel.



We need to create a couple of properties at the Test Suite level. So click the ‘plus’ button and add ‘userId’ and ‘authToken’ properties.


These properties will be set by the Login test case. Then these properties can be used by the other test cases as part of the Headers to enable access for each request. Next step is to set these Test Suite level property values.

Right click on the ‘Login’ TestCase and add a ‘Property Transfer’ test step to this test case. You can give this test step the default name of ‘Property Transfer’:



Then open the workspace for this Property Transfer Test Step (double click). Then add 2 Transfer values, userId and authToken:


Now all we need to do is set the source and target for these property transfers. The source will be the Login Test Case response that returns the ‘userId’ and ‘authToken’. The Target will be the Test Suite level properties (which then makes these values available to ALL test cases in the suite).

Set the authToken first by setting these values…


Source: Request 1
Property: Response
Path Language: JSONPath
Path: $.data.authToken

Target: TestSuite
Property: authToken
Path Language: JSONPath

It should look like this…



Then set the userId …


Source: Request 1
Property: Response
Path Language: JSONPath
Path: $.data.userId

Target: TestSuite
Property: userId
Path Language: JSONPath

Which should look like this…



We can check this setup by clicking on the ‘Run All Property Transfers’ button. From the Transfer log we should see these values picked up from previous responses from the login request:



And now the properties are available at the Test Suite level we can use them to run the other test cases. Just need to tweak the other test cases where the ‘authToken’ and ‘userId’ are currently hard coded.

Edit the ‘publicRooms’ test case and click on the ‘Header’ button so that we can add these TestSuite parameters to the Header values. We’ll need to add these property strings to get the Header values to use our TestSuite properties:

X-User-Id: ${#TestSuite#userId}
X-Auth-Token: ${#TestSuite#authToken}

So replace the hardcoded values with these strings so that you have this:



You can check this setup by clicking the green submit request button for this test case. Assuming it works what we now have is the ability to run the Login test, get the dynamic authentication variables that Rocket Chat passes us back in the response, and use these variables in all the other test cases.

All you need to do now is update the Logout Test Case to. The logout test case should look like this:



Next step then is to get these TestCases to all run in series, end to end.

5. Running the Test Suite

Well this bit is probably easier than you’re expecting. Just open the TestSuite workspace and click on the ‘Run’ button. You should see each test case within the Test Suite run in turn. Each test case highlighted in green to indicate that it’s passed.



All we need to do now is get this running from the command line and get Jenkins to kick the API test run off.

Oh! Probably worth saving your SoapUI project at this point!

Part 5: Running the SoapUI Test Suite from the Command Line

There is an easy way to work out how to run this Test Suite from the command line. You can initiate a command line run from the SoapUI gui and then copy the command line that the GUI gives you.

1. Launch the TestRunner

To do this, from the ‘Project’ menu select ‘Launch TestRunner’:



This will give you the test runner dialogue box. One the ‘Basic’ tab all we need to define is the path to the TestRunner. When we run these tests from the command line we’re basically running a windows batch file called ‘testrunner.bat’. You just need to set the path to this file to:

TestRunner Path: C:\Program Files\SmartBear\SoapUI-5.2.1\bin

The version number in the directory may change for you but it should be like this…



Then on the ‘Overrides’ tab we just want to select our Endpoint. We need to see how to pass the value in on the command line so setting it here will get SoapUI to use this value as a parameter.



Then click ‘Launch’ to run this. A successful run should look like this…



A few exceptions in there with this release of SoapUI. Nothing to worry about. We can fix these by stopping SoapUI trying to load these plugins.

2. Clear the Exceptions (optional step)

To clear all these unwanted exceptions open explorer and go to this directory


Close SoapUI after you’ve saved your project. Then rename the plugins directory to something like ‘plugins-old’. Like this…


3. The Command Line

What we’re really interested in though is the command that was run by the TestRunner. If necessary re-run the test runner from SoapUI. Then click ‘OK’ and scroll back up to the top of the Test Runner windows. You need to locate this…


From this you can pick out and copy the command line you need. Like this…

directory: C:\Program Files\SmartBear\SoapUI-5.2.1\bin
command: cmd.exe /C testrunner.bat -ehttp://<your-aut-host>:3000 C:\Users\Administrator\REST-Project-1-soapui-project.xml

You’ll want to modify this slightly so that you have this format…

cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -ehttp://<your-aut-host>:3000 C:\Users\Administrator\REST-Project-1-soapui-project.xml

Note the path in front of the bat command and the inverted commas. This is what we’ll test before we add it to a Jenkins job. So lets check this from a command line prompt. Open a command window and run this command:



When it’s completed check the error level with a command like ‘error %errorlevel’:



If all the tests passed then the error level should be ‘0’. If any failed then the response should be ‘-1’. You can test this if you like by slightly modifying one of your test cases in SoapUI to make it fail (change the assertions on one of the TestCases or something). Then run the command line command again and check the error level is ‘-1’.

Nearly there now. Just need to plug this all into a Jenkins job.

Part 6: Running the SoapUI Test Suite from Jenkins

First we’ll need that SoapUI command from the previous step. So you’ll have something like this…

cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -ehttp://<your-aut-host>:3000 C:\Users\Administrator\REST-Project-1-soapui-project.xml

Back on to our Windows Master machine that’s running Jenkins (open the RDP session if necessary). Then bring up the Jenkins home page. From here we need to create a new job / new item:



We’ll make this a ‘Freestyle Project’ and call the job ‘RunApiTests’

We have a few key things to configure on this new job:

1. Restrict where this project can be run: SeleniumTestClient

2. Build Triggers:

Build after other projects are built
Projects to watch: RunSeleniumTests
Trigger only if build is stable

Then we need to add 3 new build steps. To create each one click on the ‘Add build step’ button, select the build step type and set the parameters (as set out below)

1. Copy artifacts from another project
We need this to pull in the publicHostsname.txt file from the ‘BuildRocketChatOnNode’ job. This way this build job will know where Rocket Chat is running.

2. Inject environment variables
We’ll take the contents of the publicHostname.txt file, that contains the host name where Rocket Chat is running, and use this value in the next batch command job.

3. Execute Windows batch command
This is where we’ll run our SoapUi command on the remote machine (the machine labled SeleniumTestClient). Note that we don’t need to hard code the host name. We can use the %PUBLIC_HOSTNAME% variable that’s set from the publicHOstsname.txt file. So enter your command like this:

Using this command:

cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -e%PUBLIC_HOSTNAME% C:\Users\Administrator\REST-Project-1-soapui-project.xml

No post build actions at this stage. So we can just save this job and return to the Jenkins home page.

Before we run this though lets setup things up so that Jenkins can actually report on the status of the tests run by SoapUI. To do this we’ll need a new plugin installed in Jenkins.

Part 7: Reporting the SoapUI Test Results in Jenkins

What we want to do now is see the individual test case results displayed in Jenkins. SoapUi can save the test results in a JUnit style report format. Then we can get the Jenkins ‘xUnit’ Plugin to display the test results. Lets install the plugin first.

1. Install the xUnit Jenkins Plugin

Just go to the Jenkins ‘Manage Plugins’ page, select the available tag and search for “xUnit”.



Click install without restart. Once the plugin is installed we can configure our SoapUI project to save the test results in JUnit format.

2. Modify the Jenkins ‘RunApiTests’ Job

Back on the Jenkins home page configure the ‘RunApiTests’ job.

What we need to do now is configure our SoapUI command line so that it save the test results in a JUnit format. We can do this by changing our ‘Execute Windows batch command’ from this:

cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -e%PUBLIC_HOSTNAME% C:\Users\Administrator\REST-Project-1-soapui-project.xml

to this…

cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -e%PUBLIC_HOSTNAME% -sTestSuite -j -fc:\jenkins\workspace\RunApiTests\ C:\Users\Administrator\REST-Project-1-soapui-project.xml

Note that we’ve added three parameters to this command:

-sTestSuite: defines the TestSuite to run (which we called ‘TestSuite’)
-j: Turns on exporting of JUnit-compatible reports
-fc:\jenkins\workspace\RunApiTests: which specifies the directory to save the report in

So now when we run this job we should get this file created on the client machine that will contain the individual test results:


It’s important to save this in the Jenkins workspace on the test machine. This way Jenkins will automatically make the JUnit files available to the master Jenkins machine for processing.

Don’t save this job yet. Still one more thing to configure. We have to get Jenkins to read those results and publish them.

3. Publish xUnit Test Results

Whilst we’re still editing the ‘RunApiTests’ job we can add a post build action. Click on the ‘Add post-build action’ button and select ‘Publish xUnit test result report’

Once this is added we need to add an xUnit type of report. Click on ‘Add’ and select ‘JUnit’

From here we can set the file names Jenkins needs to look for where the JUnit test results reside. Well SoapUI will be creating a JUnit file called TEST-TestSuite.xml. So we’ll need to configure this ‘Publish xUnit test result report’ to look for TEST*.xml files. So set these parameters:

JUnit Pattern: TEST*.xml
Skip if there are no test files: un-checked
Fail the build if test results were not updated this run: checked
Delete temporary JUnit files: checked
Stop and set the build to ‘failed’: checked

And that it’s really nothing else to set here (you don’t even need to set any of the test thresholds).

Save this job now and lets run the full set of Jenkins jobs.

On the Jenkins home page click the “Run now” button for the ‘BuildRocketChatOnNode’ job. This should end up running our full set of 5 jobs. And once complete we can check the console output for our ‘RunApiTests’ job.

That console ouptput should show you something like this at the end:

c:\jenkins\workspace\RunApiTests>exit 0
[xUnit] [INFO] – Starting to record.
[xUnit] [INFO] – Processing JUnit
[xUnit] [INFO] – [JUnit] – 1 test report file(s) were found with the pattern ‘*.xml’ relative to ‘c:\jenkins\workspace\RunApiTests’ for the testing framework ‘JUnit’.
[xUnit] [INFO] – Check ‘Failed Tests’ threshold.
[xUnit] [INFO] – Check ‘Skipped Tests’ threshold.
[xUnit] [INFO] – Setting the build status to SUCCESS
[xUnit] [INFO] – Stopping recording.
Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered
Triggering a new build of CloseClientRDPSession
Finished: SUCCESS

If you get warnings about clocks being out of sync check the next section. If there results are okay then move on to ‘Reviewing the JUnit Results’.

4. What to do if Clocks are Out of Sync

The ‘xUnit’ plugin depends on the clocks, between master and slave nodes, to be in sync. If they’re not in sync then you might see an error like this in the console log for the ‘RunApiTests’ job

[xUnit] [ERROR] – Clock on this slave is out of sync with the master, and therefore I can’t figure out what test results are new and what are old.
Please keep the slave clock in sync with the master.
[xUnit] [INFO] – Failing BUILD.

If that’s the case take a look at the Jenkins configuration to confirm this.


So indeed on this system the client machines clock is running 17 seconds behind. To resolve this you’ll need to run these 4 commands on the ‘Client’ machine:

net stop w32time
w32tm /config /syncfromflags:manual /manualpeerlist:”,,,
w32tm /config /reliable:yes
net start w32time


Like this…


Then wait a minute or two. Go back to the Jenkins master machine and refresh the status:

And clocks are all back in sync.

5. Reviewing the JUnit Results

We’ve set the xUnit reporter to pass the build/job if all the tests fail. What happens if we see a failure though? Well if you first check the workspace for the job:

You’ll see the TEST-TestSuite.xml file and the other log files that were saved from the SoapUI test run:

From here if you look at the drop down menu for the last test run you’ll see an option for viewing the test results:

Click on this and you can then start drilling down through all of the unit test results:


And if you click the ‘History’ item in the right hand menu bar you’ll see a track record of all the SoapUI JUnit test results.



Well nothing too difficult on the Jenkins side of things this time round. By now you should be experienced enough to configure the Jenkins build job easily to handle the execution of the SoapUI test run. The only new thing we introduced here for Jenkins is the capability to process JUnit test results and display them. A key capability for tracking test results in many other tools controlled by Jenkins too.

We delved into a lot of SoapUI features to create a handful of REST based Api test cases. The only real complexity here being the passing of parameters between test cases. Once you’ve seen how to do this it not difficult to repeat the process and build up a good pack of Api test cases. Key to the success of this module though is the ability to run the SoapUI tests from the command line and get Jenkins to pick up the JUnit test result file. As long as you know where to pick up the SoapUI test runner application and configure the xUnit plugin none of this should have been too challenging.

At the end of this module then we have a test automation environment that covers installing our appliation under test, running a suite of GUI based browser tests with Selenium and now running a handful of API test cases using SoapUI. All started and executed with ‘one click’ on a ‘Run Now’ button.

In the next module we’ll look at how we can build in some performance tests with JMeter driven from Jenkins too.

Free Test Automation Framework Course
Learn the Practical steps to building an Automation Framework. Six modules, six emails (each email a short course in it's own right) covering Amazon AWS, Jenkins, Selenium, SoapUI, Git and JMeter.

As we look at driving a Windows client machine from Jenkins, to run Selenium based browser tests things get a little bit more involved. In this 3rd module of our Building The Test Automation Framework course we step things up a little bit. Nothing we can’t handle though as we take you through each step in detail.

We’re going to cover four key areas:

1. Setting up a Windows Jenkins salve machine
2. Install the test software on the slave machine (e.g. Python and Selenium)
3. Creating a basic Selenium script
4. Running the Selenium Scripts from the Jenkins Master Machine

Once you’ve completed all of these steps you’ll have setup a client Windows AWS machine that will run Selenium tests against our application under test (AUT). As always we’re looking to automate the whole process. So we’ll be configuring our Jenkins server to start the client machine and then control this client machine through the use of the Jenkins slave application.

A few complexities in this module though as Windows User Account Controls (UACs) do their best to prevent us from achieving our goals. We’ll show you some tips and tricks that get round a lot of these issues. Tips and tricks you’ll find invaluable when you come to creating your own Jenkins environments.

What You’ll Learn

In this module we’ll go through the 12 parts listed below looking at the practical aspects of setting up Selenium.

In this session we’re going to configure our system so that we automatically start a Windows test machine that we’ll run a set of Selenium browser tests from. This client windows machine will be automatically started by Jenkins. Once running the Selenium scripts will be started by another Jenkins project.

Once complete we’ll have added the Selenium components to our system as shown below:

However, in order to run these Selenium scripts on a client windows machine we need a desktop terminal session that the scripts will run in. We want to run the test scripts in a real environment and see them running. We don’t want to run them in some headless mode that isn’t a true simulation of what an end user would be doing with the application under test.

Getting Jenkins to create RDP terminal sessions is quite tricky due to the way Windows implements user account control (UAC). So we’ll learn some tricks to get round this and how to really create a set of tests that represent real user actions.

One final thing to mention before we get started. This is a course on building the test automation framework using jenkins. It’s not a course on writing Selenium scripts. So the Selenium scripts we’re using are pretty basic. They server the purpose of implementing this framework and are easy to build on if you want to expand and plugin your own scripts later.

The Tools We’ve Chosen

We’re testing a browser based application so it seems sensible to choose Selenium as the tool to write our automated test cases in. We can get Selenium drivers for a range of different browsers so we’ll be able to provide test coverage against IE, FireFox and Chrome (we could do more but this’ll be adequate for now).

It’s important to point out though that this isn’t a Selenium course. So the Selenium scripts we’re using here are simple, linear and, well basic. This gives the added advantage that it makes things easier to understand. You just need to be aware that in order to scale the actual test cases you’ll probably want to look at improving the way these test cases are written (something we’ll get into in a later course).

So if we’re using Selenium the next question is what language are we going to write our test scripts in. Well “to stop the should it be C# or Java discussion” we discounted those two languages straight off. And because I’m keen on Python that’s what I chose. Then there’s the fact that it’s efficient, fast and easy to learn. Oh and there’s the point that the demand for testers that can code in Python is growing exponentially. So Python it is.


Right, we need to make sure we have everything setup correctly before we start. First we need to be sure we’ve launched our Windows master machine and that we have an RDP session open.

Make Sure your Windows Master EC2 Instance is Running

If you don’t have your Windows master machine up and running yet then you’ll need to revisit Module 1 and take a look at the section on Running up the Windows Instance.

With the Windows 2008 master machine configured then we’ll need make sure the instance is running and that we have an RDP session open. You can complete these steps with

1. Open your AWS management console, and view your list of Instances (making sure you have the correct Region selected).

2.Once started open an RDP session on this Windows machine.

Then enter the password (you may need your .pem private key file to Decrypt the password if you’ve forgotten it) to open up the desktop session.

Make Sure you have your Private key (.pem file)

Remember back in Module 1 we created our public and private key pairs? Well at that stage you should have saved your private key pair .pem file (e.g. FirstKeyPair.pem) file. You’ll need this private key when configuring Jenkins later.

If you don’t have have this private key you can go back and create a new key pair. Much easier if you can find the one you created in Module 1 though.

Setup a new Security Group for Windows Client

For the Windows client machine it’s best to setup it’s own dedicated security group. We’ll need this as Jenkins connects to this machine with CIFS (Common Internet File System) and WinRM (Windows Remote Management). Both of these require access over specific protocols and ports.

To setup a new security group follow these steps

i. login to your AWS management console
ii. go to your AWS EC2 dashboard
iii. select ‘Security Groups’ in the left hand menu
iv. click on the ‘Create Security Group’ button
v. add the ‘Security Group Name’ – “Windows-client”
vi. add the description – “Windows client machine”
vii. leave the VPC field set to the default
viii. add the following rules then click ‘Create’

Security Group Rules

Inbound: Custom TCP Rule TCP 445
Inbound: Custom TCP Rule TCP 5985
Inbound: RDP TCP 3389
Outbound: All traffic All All

Part 1: Setup Our Windows Test Machine

Create a Windows 2012 Server Instance in AWS

Four things we need to create this Windows instance:

  1. Microsoft Windows Server 2012 R2 Base – ami-83a5bce2
  2. t2.micro
  3. Name: Windows-Client
  4. Security: default

Back to our EC2 dashboard then:

i. login to your AWS management console
Step 1: go to your AWS EC2 dashboard
Step 2: select the correct region
Step 3: click ‘instances’ in the side menu
Step 4: select the AMI listed above (Microsoft Windows Server 2012 R2 Base )
Step 5: select the Type listed above (t2.micro)
Step 6: accept all the ‘Instance Config’ defaults
Step 7: accept all the ‘Storage’ defaults
Step 8: [optional] add a tag for the Name if you like (e.g. Windows-Client)
Step 9: for the security group we’ll need to:
Step 10a: Select existing security groups
Step 10b: Check the ‘Windows-client’ security group AND
Step 10c: Check the ‘default’ security group
Step 11: Review and then ‘Launch’

Once the new instance is running you can use the EC2 console to put the instance into a “stopped” state. Don’t terminate it, just put it into a stopped state. We’ll start it up again using Jenkins later.

Part 2: Configuring Jenkins to Start A Windows Slave Machine

We’ve already configured Jenkins to run up one slave machine. This slave machine runs our Ubuntu Linux environment that hosts our Application Under Test (AUT) – Rocket Chat. Now we need to add to this environment and create a connection to the Windows slave machine that we’ll run our Selenium browser testing from.

First step is to configure this machine in Jenkins and make sure we can connect to it. We can complete all of this from the Manage Jenkins / Configuration page in the Jenkins Gui.

1. Open up Internet Exporer and type in this URL


2. In the Jenkins home page click on Manage Jenkins -> Configure System

3. At the bottom of the configuration page we need to create a new AMI. We already have our cloud settings configured so this time round we only need to add an additional AMI. So scroll to the bottom of the page and add an AMI

4. Enter the AMI details

Description: Windows-client
AMI ID: <see below>

We’re going to be using this slave machine to run a few different browsers and Selenium. We’ve already created instance and have this running with an AMI that gives us this type of server:

Microsoft Windows Server 2012 R2 Base 64-bit

The current AMI for this is…


However, this may change as AWS update their AMIs. The quickest way to check this and make sure you have the correct AMI is to

i. login to your AWS management console
ii. go to your AWS EC2 dashboard
iii. select the correct region
iv. click ‘instances’ in the side menu

Whilst you’re on this page it’s worth making a note of the ‘Availability Zone’ that your master Windows Jenkins machine is running in. We’ll need this in the next step. So note down this value…

Then continue with…

v. click the ‘Launch Instance’ button
vi. make sure the ‘Free tier only’ check box is checked
vii. look for the “Microsoft Windows Server 2012 R2 Base” AMI
viii. copy the AMI number (e.g. ami-83a5bce2)

You don’t actually need to launch an ‘instance’ here. You just need to make sure you have the AMI id that is associated with a Windows Server 2012 R2 Base 64 bit machine.

So at the end of this part you should have two pieces of information that we’ll need in a minute:

a. the ‘Availability Zone’ that your master Windows Jenkins machine is running in
b. the AMI id of a windows 64 bit 2012 R2 Base server that Amazon has available for you to create instances from.

Both of these peices of information will be needed in the next part.

5. Click on Check AMI to make sure AWS gives us access to use this AMI

6. Select the ‘T2Micro’ Instance Type (to make sure you stay on the AWS free tier)

7. Enter the Remote User, AMI Type and a few other values leaving everythying else as a default.

EBS Optimized: <un-checked>
Availability Zone: <get-from-aws-account> (*1)
Use Spot Intance: <un-checked>
Security group names: default, Windows-client (*2)
Remote FS root: c:\jenkins
Remote User: Administrator
AMI Type: windows
Windows Admin Password: <leave blank> (*3)
Use HTTPS: <un-checked>
Boot Delay: 3
Labels: SeleniumTestClient
Usage: Utilize this node as much as possible
Idle termination time: 30
Init script: <leave blank>

  *1 – set this to the same value as the Availabilty zone that your Windows Master Jenkins machine is running in

   *2 – we setup the Windows-client security group in the Prerequisites seciton of this module.

   *3 – we need to leave this blank for now. We’ll come back and update this in minute.


Once you’ve entered these values you should have something like this:

8. Enter the Advanced settings. You’ll need to click the ‘Advanced’ button to expose further settings that we need to update:

Most of these we can leave blank but you will need to update these values:

Stop/Disconnect on Idle Timeout: <checked>
Use private DNS: <checked>

The first setting here stops Jenkins ‘Terminating’ this windows instance if it’s been idle for 30 minutes. With this set if we hit the idle limit this instance will be ‘Stopped’ which preserves all our data.

The second setting for using the private DNS keeps the connection from our master Windows machine (running Jenkins) to our client Windows machine using DNS and hostnames that are private to our AWS Virtual Private Cloud. This just simplifies some of the security complexities for us.

At the bottom of the page you just need to click the ‘Save’ button

9. Start Windows Client Machine from Jenkins

In Jenkins we need to complete these actions to start the AWS Windows client machine instance:

i. go to the Jenkins home page
ii. go to ‘Manage Jenkins’ then ‘Manage Nodes’

On the nodes page we should be able to start our Windows client machine by clicking on ‘Provision via RocketChat Server’ button and selecting ‘Windows-client’

10. Check Windows Client Machine has started

On the Jenkins Nodes page you can click the ‘see log for more details’ link and see the node come up.

You should see a log message that confirms that the machine has been started…

>FINER: Node Windows-client (i-d9b15b01)(i-d9b15b01) is ready

However, you’ll also see that Jenkins is failing to connect and control this machine because we have this message repeating itself…

Waiting for WinRM to come up. Sleeping 10s.

At this stage we’ve started the Windows client machine from Jenkins. However, we’ve only been able to start it. We can’t yet connect to it from Jenkins. Automatically connecting to Windows machines is a little more complicated than our Linux equivalent (where we can just use SSH with our private key). Establishing the connection and control over this machine is what we’ll look at next.

Part 3: Completing the Windows Client Configuration

With the windows client machine started we still need login to it manually with an RDP session, setup some additional configuration manually and then update the Jenkins configuration. Once we’ve done this Jenkins will have complete control over this Windows client machine.

1. Connect Using RDP to the Windows Client machine

Going back to our AWS management console we need to start an RDP session on our Windows Client machine. Follow these steps to open the RDP connection:

i. login to your AWS management console
ii. go to your AWS EC2 dashboard
iii. select ‘Instances’ in the left hand menu
iv. click on the entry for the new Windows Server 2012 instance
v. right click on this instance and select ‘Connect’
vi. select ‘Get Password’ (you’ll need your .pem private key file again)
vii. Decypt the password and save it somewhere safe
viii. click ‘Download Remote Desktop File’

** Make sure you keep a copy of the password – we’ll need it again latert **

At this point you should have got to here…

2. Set Termination Behaviour

Whilst you’re in your AWS account now is probably a good time to set the termination behaviour for a couple of your instances. For instances where you don’t want to lose data it’s worth ‘Enabling Termination Protection’. This just stops you terminating these instances by accident.

i. select the instance
ii. right click on the instance and select ‘Instance Settings’
iii. then select ‘Change Termination Protection’ in the sub menu
iv. in the pop up dialogue box click the ‘Yes, Enable’ button

You’ll want to change this protection setting for both your Windows-master and Windows-client machines. Do NOT change it for your Unix-client machine (we need to terminate this on a regular basis).

3. Configure Windows client to allow CIFS and WinRM access

To allow Jenkins to install it’s Jenkins client on this machine and take control of it we need to update a few settings. We need to enable WinRM (Windows Remote Management) with the following:

i. Open a command prompt
ii. execute the WinRM commands below

WinRM Commands
winrm quickconfig
winrm set winrm/config/service/Auth @{Basic=“true”}
winrm set winrm/config/service @{AllowUnencrypted=“true”}
winrm set winrm/config/winrs @{MaxMemoryPerShellMB=“1024”}



That should give Jenkins the access it needs to connect to the machine. All that’s left now is to install Java so that Jenkins can install it’s client application. It’s this client application that gives Jenkins full control over the machine.

4. Install Java JDK on the Windows client

Jenkins needs Java installed (and Java in the path variable) in order to install and run it’s client/slave application. It’s this app that gives the Jenkins master machine full control over the client and enables it to run Jobs on the client. To install Java we’ll need to:

a. Open Internet Explorer and search for …

Download Java JDK

This should take you to a web page like this…

b. This will present you with this web page where you need to accept the licence agreement then select this package to save…

Note that you might have to change your Internet Explorer security settings here to allow you to download it.

Save this package to the desktop:

The run the installer (as Administrator):

Accept all the defaults as you follow through the install wizard.

c. Last thing is to just check that the install completed successfully and that we have Java in our path.

i. open a NEW command window
ii. type ‘java -version’
iii. type ‘echo %PATH’
iv. check for this in the path: C:\ProgramData\Oracle\Java\javapath

5. Update the Node Configuration in Jenkins

On the Jenkins master machine we just have to update a few configuration settings for the client now. So we’ll need to complete the following actions:

** NOTE: we’re back on our master Jenkins RDP session now! **

i. go to the Jenkins home page (http://localhost:8080)
ii. go to ‘Manage Jenkins’ then ‘Manage Nodes’

On the nodes page we should see our Windows client machine (with an ‘X’ against it as Jenkins has failed to connect with it). From here…

iii. Click the ‘Configure’ link in the side menu

On the Jenkins master machine weiv. Enter the Windows password

From step one above, we decrypted our password for this AWS instance. Hopefully you’ll still have this so that you can enter it here (if not go back to your AWS management console and get the password for this windows-client instance)

On the Jenkins master machine wev. click the save button at the bottom of the screen.

6. Establish Connection Between Jenkins master and Windows Client

At this point we should have everything we need for the master Jenkins machine to communicate with, and take control of, our windows client machine. From our Jenkins master machine follow these steps:

** NOTE: we’re still on our master Jenkins RDP session here! **

i. go to the Jenkins home page (http://localhost:8080)
ii. go to ‘Manage Jenkins’ then ‘Manage Nodes’

On the nodes page we should see our Windows client machine (with an ‘X’ against it as Jenkins has failed to connect with it). From here…

iii. click on the windows-client link
iv. click the ‘Relaunch slave agent’ button

At this point you should see the log file displayed automatically. After a minute or so the log file should start to show messages like this…

>FINER: Node Windows-client <instance-id> is ready
>Connecting to <ip-address> with WinRM as Administrator
>Waiting for WinRM to come up. Sleeping 10s.
>WinRM service responded.
>WinRM should now be ok on Windows-client
>slave.jar sent remotely. Bootstrapping it
>This is a Windows slave
>Slave successfully connected and online

At this point we have our Windows slave machine up and running. It’s running the Jenkins slave Jar too so the Jenkins master machine can control it and send jobs to it.

However, before we move on, let’s talk a little about what happens if you can’t get this connected.

Part 4: What to do when you Can’t Connect the Windows Slave

The connection between the Jenkins master machine and the Jenkins windows slave is established using Windows Remote Management (WinRm). This is installed and running by default when we create our Windows slave instance using the Windows Server 2012 R2 Base AMI.

However, sometimes you may find you run into problems connecting the two machines using WinRm. To see if the connection issue is down to WinRm configuration you can try this on the Jenkins Windows Master machine.

  1. In your AWS account find out what the ‘Internal IP address’ is of the “Client/Slave Windows Machine”. You’ll also need the Administrator password for this machine too.
  2. On the Jenkins master machine open a command terminal
  3. Run this command in the terminal window
winrm identify -r:http://<ip-address-of-client>:5985 -auth:basic -u:Administrator -p

What this does is try to connect to the client machine using WinRm (don’t forget to replace <ip-address-of-client> with the right client ip address). It’s basically simulating what Jenkins tries to do when it connects to the slave machine. Now, if you get this result, or any other connection error, you know you have an issue with your WinRM configuration…

>Message = The WinRM client cannot process the request.
>Unencrypted traffic is currently disabled in the client
>configuration. Change the client configuration and try
>the request again.

In which case it’s worth trying commands like this on the Jenkins Windows Master machine:

>winrm set winrm/config/client @{AllowUnencrypted=“true”}
>winrm set winrm/config/client/Auth @{Basic=“true”}

Then when you try this command again…

>winrm identify -r:http://<ip-address-of-client>:5985 -auth:basic -u:Administrator -p

You should get a response along these lines:

>ProtocolVersion =
>ProductVendor = Microsoft Corporation
>ProductVersion = OS: 6.3.9600 SP: 0.0 Stack: 3.0
>SecurityProfileName = …

This is telling us that the connection was successful. If this is successful then Jenkins should be okay making the WinRm connection now. So you can try going back to Jenkins on the nodes pages and “Relaunching the slave agent”.

WinRm is a complete topic in it’s own right. However, using the commands above may well help you at least identify where the connection issue lies.

If you need to dig deeper into this topic then there’s a lot of helpful information on this page:

Installation and Configuration for Windows Remote Management

Another issue you might run into is slow response times from the slaves. This is especially true when we’re running low tier AWS instances. If you’re seeing the connection succeed then drop out again, there are a couple of options.

First, you could increase the AWS teir that your instance is running on. Of course if you do this you may run up charges and costs as these are unlikely to be on the free teir.

Secondly, you could try turning off the response time check as a temporary fix.
Go to Manage Jenkins -> Manage Nodes -> Configure, and uncheck the “Response time” box. This should stop Jenkins dropping the slave just because it’s not responding fast enough.

Part 5: Installing the Software we need on the Windows Slave

We have the slave Jenkins machine running and under the control of our main Jenkins machine. Now we can deploy some of the software we need to this slave machine so that it can run our Selenium tests. Software packages we’ll need will include:

Python 3.5

1. Open an RDP session on the Windows 2012 slave machine

i. login to your AWS management console
ii. go to your AWS EC2 dashboard
iii. select ‘Instances’ in the left hand menu
iv. click on the entry for the new Windows Server 2012 instance
v. right click on this instance and select ‘Connect’
vi. click ‘Download Remote Desktop File’

2. On your Windows 2012 Server client machine download and install the following:

(Windows x86-64 executable installer)

I’m not going to take you step by step through this because it should be pretty straight forward. Once you have those three installed we can walk through the process of setting up Selenium and then writing our first Selenium scripts.

3. Check Python is installed correctly.

a. open a command prompt
b. type ‘py’ followed by return
c. you should see “Python 3.5.1 (v3.5.1:37a07cee5969…”
d. type ‘exit()’ to come out of the Python shell

If this didn’t work you’ll have to go back to step 2 above and try re-installing Python again.

Now all we need to do is add the Selenium Python package to our install.

4. Install the Selenium Python package

If Python is installed correctly you should find it installed here…


We need to change directory now and make sure we have ‘pip’ the Python Package and software management tool installed. We’ll need this to install Selenium. So double click on the ‘Scripts’ folder and look for the file/executable ‘pip’.

From a command prompt run this command:

C:\Users\Administrator\AppData\Local\Programs\Python\Python35\Scripts\pip.exe help

This should show you some help information to confirm that pip is installed correctly. From here we need to replace the ‘help’ command with the ‘install’ command to install the Selenium packages. From the command prompt run

C:\Users\Administrator\AppData\Local\Programs\Python\Python35\Scripts\pip.exe install -U selenium

So we have Python installed and the Selenium package. We have our browsers installed but we don’t have the Selenium drivers installed for those browsers. Well that’s not strictly true. FireFox comes installed with the Selenium driver but IE and Chrome don’t. So we’re missing these bits…

Next then we need to install the Selenium drivers for IE and Chrome.

5. Install the Chrome Selenium Driver

On our Windows 2012 Slave machine use a browser to search for the ‘Chrome selenium driver’. This should take you to a Url like this…

From here download the latest ChromeDriver, for example:

Extract the exe from the zip file and copy this file to the ‘C:\Windows’ folder.

The only reason we’re copying it to the ‘C:\Windows’ directory is because this directory is in the PATH environment variable already. Which means when we execute it the system will have no problem finding the exe for this driver. You can try this if you like by running this command:


This shows the Chrome driver running and waiting for a connection.

Next then the IE driver.

6. Install the IE Chrome Selenium Driver

Bit more tricky to find this one. However, a hunt on the web for ‘IE Selenium Driver’ should take you to a place like this…

Select the latest folder/version and download the latest IEDriverServer file, for example:

Again, extract the exe from the zip file and copy this file to the ‘C:\Windows’ folder.

One last thing on the IE setup. We need to disable protected mode for all zones:

Then close down IE and we should be ready to go.

Part 6: Create Our First Selenium Script

So at this point we have everything we need on this Windows 2012 slave machine. We just need to create a few Selenium tests and run them locally. Once we have them running locally we can start triggering their execution from the master Jenkins machine.

To begin with we’ll check Python and Selenium can drive all our browsers. We’ll great a short script to open each browser in turn.

1. Open notepad and enter the following script

from selenium import webdriver driver = webdriver.Firefox() driver.get("") driver.quit()

driver = webdriver.Ie() driver.get("") driver.quit()

driver = webdriver.Chrome() driver.get("") driver.quit()

Save this script to the desktop with the file name ’’

2. Open a command prompt and run this script


Which should be executed as follows:

Running this should open each browser in turn, go to the Google home page and then close each browser.

This just confirms for us that everything we need is setup correctly on the slave machine. Just need to check we can run this from our Jenkins master machine now.

3. Checking the script return codes

When Jenkins runs scripts on remote slave machines it determins the success or failure of the job/build based on the return code from the LAST script or command executed. If you run the script again on the Slave machine then type this at the command prompt:

echo %errorlevel%

In the example below where the script ran successfully we see the error code is ‘0’ which means success. If Jenkins sees this it’ll mark the Job/build as a pass (which is what we want).

Lets change the script slightly so that we make it fail. You can do this in notepad by changing this line…

driver = webdriver.Ie()


driver = webdriver.IE()

Note the change in case of ‘IE’. Save this and run the script again. This time we should see a failure

And when we look at the error code we see a ‘1’. This means the script failed. When Jenkins picks up on this it will mark the Job as failed. We going to keep this state whilst we configure Jenkins.

Part 7: Running the Selenium Script from the Jenkins Master Machine

Lets setup the Jenkins master machine to kick off this selenium script on the Slave machine. The Selenium scirpt is configured to fail at the moment so we should see this fail the job first time round. Then we’ll update the Selenium script to fix it and see the job pass. That’ll just confirm for us that we’ve configured all of this correctly.

1. Create a new Job/build task in Jenkins

** We’re back on the Master Jenkins Windows 2008 machine now **

2. Give the new job an item name and select Freestyle project

Item Name: RunSeleniumTests
Freestyle project: <selected>

3. Configure the job with the following settings:

Restrict where this project can be run: <checked>
Label Expression: SeleniumTestClient

This forces the job to run on our Windows 2012 client/slave machine where we’ve setup our Selenium tests.

At the bottom of the configuration page we need to setup a build step. Click the ‘Add build step’ button and select ‘Execute Windows batch command’. Then add the following command


Then click the ‘save’ button.

4. Run the job and check that it fails

Now we click on ‘Build Now’ in the side menu to kick off this job on our Windows slave machine:

The build should show up in the Build que but because we’ve set it to run on our client machine labeled ‘SeleniumTestClient’ it may take a while for this client to start or for Jenkins to re-establish communications with the client. Once it’s started though you should see this…

And then if you return to the home page (click back to dashboard) you should see this job/build fail.

Click the ‘Console output’ menu item next to the failed build number to check why this has failed. You should see something like this:

Which is what we expected. We had ‘IE’ instead of ‘Ie’ in our script so the script failed with this error message

AttributeError module ‘selenium.webdriver’ has no attribute ‘IE’

And returned an error code of ‘1’ that Jenkins interpreted to mean the job failed.

5. Update the Selenium script on the client machine changing this…

driver = webdriver.IE()

back to

driver = webdriver.Ie()

and save the ’’ file

6. Run the job again and check that it passes

Go back to the home page and click the ‘Bulid now’ icon

This time round we should see a blue button to indicate that the build passed.

Click on the ‘Console Output’ icon again and review the results from the Selenium script run. You should see something like this…

You can see from this that the error level was 0 (e.g. success) and Jenkins used this to mark the job/build as a pass.

Now all we have to do is write some selenium scripts to test our Rocket Chat application and get Jenkins to run these scripts once the build of Rocket Chat has completed.

Part 8: Checking we can Access Rocket Chat from our Windows Client

What we need to do now is run up our application under test, Rocket Chat. Now we already have that scripted in Jenkins and designed to run on a dedicated AWS Linux instance. So lets start out by running this up.

1. Kick off the ‘BuildRocketChatOnNode’ job in Jenkins

So click on the ‘build now’ icon for the Rocket Chat job.

This might take a few minutes to complete but you should see this in Jenkins:

2. Make sure your Windows Selenium test client/slave is running

If you check on your Jenkins dashboard and you see the Windows Client as offline like this…

Then you can bring it back online automatically by running this Jenkins job too

So we’ve started up both Jenkins slave machines automatically from our Jenkins master machine. From our ‘Build Executor Status’ panel we should be seeing something like this…

And of course in our AWS account we should see all three of our instances all up and running together for the first time:

Next we need to make sure our Windows client machine can see the application under test running on our Linux client machine.

3. Check the Windows Client can access the Rocket Chat application on the Linux client

So we’re going to look to establish this link (shown in red below)

Essentially this link is just a browser on the Windows client machine displaying the web page that is being generated by the Rocket Chat application running on our Linux client machine.

Now don’t forget that when these Linux clients run up they pick up a new public Ip address and hostname. So you’ll need to check your AWS account to find the Hots name of the Linux client:

Then taking this host name or Ip address we’ll open up a browser on the windows client machine. We need to bear in mind here that Rocket Chat runs on port 3000 so we’ll need to enter this Url in your browser on the Windows client machine:

So you should get something like this…

At this point we’re ready to write a few Selenium scripts on our Windows client machine that login to Rocket Chat.

Part 9: Writing Some Selenium Tests for Rocket Chat

Now we’re not going to get into how to develop good Selenium scripts in this module. Plenty of good resources on the web that already cover that. What we’ll do is write a login/logout script and run that for each browser type. This will give you enough to expand on as you develop your own environment.

We’re going to take an example directly from the Selenium Python Bindings documentation (found below) and modify this for our purposes:

1. Open notepad with the script and replace the contents with the following script (making sure you get the indentation correct):

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from import By
from import WebDriverWait
from import expected_conditions as EC 
import sys
import time

if (sys.argv[1] == "Firefox") : 
        driver = webdriver.Firefox()
elif ( sys.argv[1] == "Ie" ) :
        driver = webdriver.Ie()
elif ( sys.argv[1] == "Chrome" ) :
        driver = webdriver.Chrome()
        print ( "Unknown driver type" )


userName = "admin"
password = "tester123"
email = ""

assert "Rocket.Chat" in driver.title

elemUsername = driver.find_element_by_name("emailOrUsername")
elemPass = driver.find_element_by_name("pass")

elemSubmit = driver.find_element_by_class_name("button")

        accountNotCreated = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "toast-container")))


        #use existing Admin account

        elemUser = driver.find_element_by_class_name("data") 
        assert elemUser.text == userName


        elemLogout = driver.find_element_by_class_name("icon-logout")


        elemLogin = driver.find_element_by_xpath("/html/body/section/div/form/div[2]/button/span")
        assert elemLogin.text == 'LOGIN'


       #create a new Admin account
        elemRegister = driver.find_element_by_class_name("register")

        elemUsername = driver.find_element_by_name("name")
        elemEmail = driver.find_element_by_name("email")
        elemPass = driver.find_element_by_name("pass")
        elemConfPass = driver.find_element_by_name("confirm-pass")

        elemSubmit = driver.find_element_by_class_name("submit")

        elemConfirm = driver.find_element_by_class_name("button")


        # Deal with first login warning message
                button = driver.find_element_by_class_name("confirm")

                print("Warning box not found")

        elemLogin = driver.find_element_by_name("username")

        elemUser = driver.find_element_by_class_name("data")
        assert elemUser.text == userName


        elemLogout = driver.find_element_by_class_name("icon-logout")

        elemLogin = driver.find_element_by_xpath("/html/body/section/div/form/div[2]/button/span")
        assert elemLogin.text == 'LOGIN'


Save this file as:


With this in place we can run tests with a command like this… [driver_type] [http://aut_ip-address:port]

For example Chrome

We’ve updated this script a little from the example given on the Selenium Python Bindings web site. Most of what this script does is explained on that web site but the few changes we’ve made are detailed below. We can run the command with a call like this:

a. we’re passing in the driver type in the command line. So now we can control which browser we’re testing in. Later we’ll use Jenkins to control which browser we’re testing by getting Jenkins to set this driver type.

b. we’re now passing in the URL and port number in the command line. We’re passing this in as parameter number 2 on the command line (argv[2]). This is because down the line we’ll need Jenkins to pass this value in each time it runs the job. The host name for the server running the AUT will change so Jenkins will need to deal with this.

c. we’re using XPATH and Class Names to identify objects on the web page and then interact with those objects. We’d recommend using Firefox with the FirePath extension to identify objects in your applications. Makes it all much simpler.

d. we have conditional logic that looks to see if this is a NEW install of Rocket Chat. If it is then we need to create the account first before we do the login. If it’s NOT a new install then we use an existing Admin account and login without creating a new account.

e. the %errorcode% returned from running this script is set directly by the success or failure of the unit tests. If the unit test asserts fail it’s an %errorcode% of ‘1’ which tells Jenkins that the job has failed. If it’s an %errorcode% of ‘0’ this indicates a pass and Jenkins will report the job/build as a pass.

And that’s it for now on the Selenium script side of things. Just enough to test that the application is running, that we can create a new account and that we can login/logout.

Part 10: Running an RDP/RDC Session on our Master Jenkins Machine

The next challenge (and it’s kind of a big one) is to get our master Jenkins machine to open up an RDP/RDC session to our client windows machine that runs our Selenium scripts. We are going to run GUI tests on a jenkins windows slave with a remote desktop connection running on the Jenkins master machine.

We want this because the Selenium scripts are designed to run on a terminal with the browsers running. We could run some browser tests in a headless mode but this comes with it’s own set of issues and isn’t really a close enough representation of what the end user will be doing (in my opinion). So we need to get Jenkins to create an RDP/RDC terminal on the Jenkins master machine automatically.

This is fraught with issues. But there is a neat solution…even if it is a bit convoluted.

The key problem we need to overcome is that Jenkins is running on the master machine as a service. And with Windows 7 and later ‘running as a service’ means you can’t start applications (e.g. an RDP session) on that machine directly from that service. You run into all sorts of issues with User Account Control Setting issues.

The way round this then is to run a Jenkins slave process on the Master machine. This slave jenkins process is run from a user account. Having it run from a user account (using a batch file or something similar) means that this Jenkins slave process does have the right account privileges to run applications (e.g. RDP sessions). You then get the Jenkins master to call the Jenkins slave (which is on the same machine) and run up the RDP session. And once the RDP session is running we can then get Jenkins to run the Selenium scripts.

Here’s how we do this then…

1. Create the jenkins working directory

First create a jenkins directory on the master machine, so that you have this directory…


Which looks like this…

2. Download the Jenkins slave.jar on the Jenkins master machine

In a borwser on the master machine enter this Url


Once the file has downloaded open the download folder and copy the slave.jar file to the C:\jenkins directory. You can use a command like this if you want

copy C:\Users\Administrator\Downloads\slave.jar C:\jenkins

3. Configure this slave as a node in Jenkins

In Jenkins we need to setup this machine so that it is seen as a slave machine. To do this go to ‘Manage Jenkins’ and then ‘Manage Nodes’

From here we can click on ‘New Node’

We’ll call the node “SlaveOnMaster” (to indicate that we’re running a Jenkins slave on the Jenkins master machine) and select ‘Dumb Slave’

On the configuration page we’ll need to set a few fields

Name: SlaveOnMaster
Description: <blank>

# of executors: 1

Remote root directory: C:\jenkins
Labels: SlaveOnMaster
Usage: utilize this node as much as possible
Launch method: Launch slave via execution of command on the Master
Launch command: <blank>
Availability: Keep this slave on-line as much as possible

A few things to point out here. We have the label set to ‘SlaveOnMaster’ because we want to force specific jobs to run on this machine. We have left the launch command blank. We DON’T want this Jenkins master process to launch the slave Jenkins process. We want to do this manually (more on this in a minute). It’s lanuching this slave process outside of the Jenkins master process that’ll give us the privileges we need to automatically start an RDP session.

So this should give us this:

Once we save this we should have:

With the red cross indicating that this Slave isn’t connected. So next we need to connect it!

4. Run the Jenkins Salve.jar process

Now we need to run this Jenkins slave process and get it to connect to the Jenkins master process (both of which are running on the same machine).

We need to create a batch file called ‘jenkinsSlave.bat’, add the following command and save the batch file to the desktop:

cmd /k “C:\Program Files (x86)\Jenkins\jre\bin\java” -jar C:\jenkins\slave.jar -jnlpUrl http://localhost:8080/computer/SlaveOnMaster/slave-agent.jnlp

Make sure this is all on 1 line. It should look like this:

What this batch file does is lanuch the Jenkins slave process using the Java executable. This will then connect to the Jenkins master process.

To create the connection double click and run this batch file.

You should then see this in the Nodes dashboard in Jenkins:

Note: if you want you can create a shortcut for this .bat file then locate the Startup folder and save this batch file shortcut in the startup folder. This way it’ll automatically start when the server starts up.

5. Create the Jenkins Job to start the RDP terminal

Now Jenkins can talk to the Slave process we can get jenkins to lanuch the RDP session automatically. We just need to create the job

Go back to the Jenkins home page and select ‘New Item’

Call the new job ‘RunClientRDPSession’ and select ‘Freestyle project’

Only things we need to update on this now job are:

Restrict where this project can be run: SlaveOnMaster

So we should have…

And then select ‘Add build step’ and configure as follows:

cmd /c cmdkey /generic:TERMSRV/<windows-selenium-client> /user:Administrator /pass:<password-for-host>
mstsc /v:<windows-selenium-client> /w:1920 /h:1080 set BUILD_ID=dontKillMe

For now we’re going to hardcode the hostname of our client windows machine that will run our selenium tests. We’ll come back and add this as a parameter later.

The first line of this command sets up our master machine connection to the windows host so that it connects automatically with the Administorator account. You will need to specify the Admin password in this command.

The second line starts the RDP session with the window heigh and width parameters defined. And the last line just makes sure that Jenkins doesn’t kill the RDP session we’ve just started when the commands have completed.

Save the job and then we’re ready to check that we can automatically start an RDP session from Jenkins.

6. Run the RunClientRDPSession job

Go back to the Jenkins home page and run the RunClientRDPSession job.

You should see Jenkins open an RDP session on the windows Jenkins machine automatically.

7. Check the Selenium tests run in the RDP session

You should be able to run the selenium tests on the client machine now

And you should see those tests running in the RDP session that you’e just opened with Jenkins too.

Part 11: Closing the RDP/RDC Session on our Master Jenkins Machine

Bit simpler this bit. Once we’ve opened the RDP session and then run our Selenium tests we’ll want to close the RDP session. We’ll need to close it because if we run the process again we don’t want to end up with 2 RDP sessions open.

1. So at this stage you should be able to create a new project by clicking ‘New Item’ on the Jenkins home page.

2. Then use the Item name:

Item Name: CloseClientRDPSession

3.We don’t need to restrict where this project can be run as we want it to run on the master machine that has the RDP terminal session open. So all we need to set is the batch command to run.

4. Then add an ‘Execute Windows batch command’ build step and enter the following script:

cmd /c taskkill /IM mstsc.exe

So we’ll have…

5. Run this project (click the ‘build now’ icon) and just make sure it closes the current RDP terminal session that’s open.

With that in place it’s just a question of linking all the projects together.

Part 12: Running the AUT build followed by the Selenium Tests

We now have the 4 things we need in place to get this up and running fully. We have the Jenkins project that installs and starts the application under test. We have the project that opens the client RDP terminal session on the master machine. And we have the project that starts and runs the Selenium tests on the client. Lastly we have the project that closes the client RDP terminal session. All we need to do now is string them together.

First we’ll link the RDP session opening (RunClientRDPSession) to the Selenium test execution (RunSeleniumTests) and the RDP close session (CloseClientRDPSession). Then we’ll link in the application under test install at the start of that (BuildRocketChatOnNode).

1. Modidfy the RunSeleniumTest project by clicking on ‘Configure’ for the project

2. Update the RunSeleniumTest Build Trigger settings so that we have…

Build after other projects are build: <checked>
Projects to watch: RunClientRDPSession
Trigger only if build is stable: <set>

Like this:

This needs to be set so that the RunSeleniumTests project gets kicked off after other project is built. It only gets kicked off IF the RunClientRDPSession is stable. If there RDP session isn’t stable and doesn’t open no point in running the Selenium tests. So in this case we’re checking that RunClientRDPSession project has completed successfully.

3. Update the CloseClientRDPSession Build Trigger to:

Build after other projects are build: <checked>
Projects to watch: RunSeleniumTest
Trigger even if the build fails: <set>

Like this:

Note that we’ve set this to ‘Trigger even if the build fails’. We want the RDP terminal session to close even if the Selenium tests fail. That’s so the terminal always closes and is ready for the next Selenium test run.

Save these project updates then we can test that this works.

4. Test that the project dependencies work

First make sure you close the RDP session on the master machine. Then go back to the main Jenkins dashboard. Kick off the RunClientRDPSession project manually

Then what you should see of course is the RunClientRDPSession project execute followed directly by the RunClientRDPSession and then finally the CloseClientRDPSession project run. Our Jenkins dashboard should show the results as follows:

Now all we need to do is pull in the initial project that builds and installs the AUT (BuildRocketChatOnNode).

5. Add condition that starts Client RDP Session when ATU Build is complete

At this stage just one last thing to add to this chain of events. When we run the build/install of the AUT (the Jenkins RunClientRDPSession project) we need the completion of this job to trigger RDP session creation (RunClientRDPSession) and the selenium tests (RunSeleniumTests)

We need to update the RunClientRDPSession Build Trigger to

Build after other projects are build: <checked>
Projects to watch: RunClientRDPSession
Trigger only if build is stable: <set>

Like this:

Only problem we have now is that when ever we do a complete install of the AUT and Amazon AWS creates a new instance we’ll have a new Url for the AUT. So we need to pick the hostname and Url out of the BuildRocketChatOnNode project and pass that to the subsequent jobs.

6. Determin hostname/Url of AUT and pass to downstream projects

So the next bit gets interesting. The problem is that our ‘RunSeleniumTests’ job has the host name the AUT is running on hardcoded. For example:

C:\Users\Administrator\Desktop\ Ie
C:\Users\Administrator\Desktop\ Chrome
C:\Users\Administrator\Desktop\ Firefox

This is going to cause problems when we start the whole process from scratch and we run up a new AWS instance. That new instance will have a new hostname and our hard coded ‘RunSeleniumTests’ job won’t work. So we need to parameterise these commands. This requires a number of things in Jenkins

a. Jenkins 'BuildRocketChatOnNode' job to record it's AUTs public host name for the host it is run on
b. Jenkins master machine to know the AUTs public hostname
c. Jenkins 'RunSeleniumTests' job to pick up the hostname as a parameter

Probably many ways to approach this (for example using some of the Jenkins parameterization plugins). We’re going to approach it in the simplest way I know using the the ‘Environment Injector Plugin’, ‘Copy Artifact Plugin’ and the in-built ‘Archive Artifacts’ capability in Jenkins.

What these plugins allows us to do is create a file on the slave running the AUT that contains a variable with the public host name in. This file then gets pulled back to the master jenkins machine. This file is then copied from the ‘BuildRocketChatOnNode’ workspace to the ‘RunSeleniumTests’ workspace. When we come to run the ‘RunSeleniumTests’ job, this job reads in the file and uses the public host name as a parameter.

First we need to install the ‘Environment Injector Plugin’ and ‘Copy Artifact Plugin’.

i. go to the Jenkins home page
ii. go to ‘Manage Jenkins’ then ‘Manage Plugins’
iii. click the ‘Available’ tab and search for ‘Environment Injector Plugin’
iv. check the check box for this plugin
v. search for the ‘Copy Artifact Plugin’
vi. check the check box for this plugin
v. scroll to the bottom of the page and click the ‘Install without restart’ button
vi. wait for the success indication and then go back to the dashboard

From here we need to do three things. First, create the file that contains the public host name. Second, configure the ‘BuildRocketChatOnNode’ job to pull the file back from the slave to the master jenkins machine and keep it as an archived file. Third, configure the ‘RunSeleniumTests’ job to read the file and use the public host name value.

First then we need to update the ‘BuildRocketChatOnNode’ so that it creates a file with our public host name in. Go back to the Jenkins home page and configure the ‘BuildRocketChatOnNode’ job.

At the bottom of this page, in the Execute Shell Command section you need to add these two line at the bottom of the script:

> cd ~/workspace/BuildRocketChatOnNode/
> echo "PUBLIC_HOSTNAME=http://$(curl 2>/dev/null):3000" > publicHostname.txt

So it should look something like this…

All this command does is find out the public host name from one of the Amazon servers, concatenate this with the string ‘http://’ and the ‘:3000’ port number. It then puts this information into a file called ‘publicHostname.txt’. The contents of which will look something like this…


Next we need to get this ‘publicHostname.txt’ file back from the workspace area on the linux slave to the workspace on the Jenkins master machine and archive the file. To do this we just use the in built build step to ‘Archive the artifacts’ post build action.

To configure this configure the ‘BuildRocketChatOnNode’ job again…

At the bottom of the configuration screen add a ‘Post Build Action’ to ‘Archive the artifacts’

In the ‘Archive the artifacts’ section that is added enter the name of our file containing the Host Name of the AUT machine. This file name is ‘publicHostname.txt’

Save the job. What this will do is pull the publicHostname.txt file back from the AUT machine and save it on the Jenkins master machine in the project workspace. Then all we need to do is copy this file from ‘BuildRocketChatOnNode’ workspace to the ‘RunSeleniumTests’ workspace. We can do this by editing the ‘RunSeleniumTests’ job from the Jenkins home page:

And then adding a ‘Copy artifacts from another project’ build step…

Then complete the following fields:

Project name: BuildRocketChatOnNode
Which build: Latest successful build
Artifacts to copy: publicHostname.txt

Like so…

Now we have the file containing the host name available to this job. Next we need this job to process the file at run time. For this we need to ‘Inject environment variables’. So we’ll add another build step for ‘Inject environment variables’:

Add the following field data:

Properties File Path: publicHostname.txt

Like so…

When this job executes on the Windows slave machine it will set the environment variable …


All we need now is for the part of the job that runs the selenium tests to use this hostname. For that we just edit the existing ‘Windows Batch Command’ changing it from:

C:\Users\Administrator\Desktop\ Ie
C:\Users\Administrator\Desktop\ Chrome
C:\Users\Administrator\Desktop\ Firefox


C:\Users\Administrator\Desktop\ Ie %PUBLIC_HOSTNAME%
C:\Users\Administrator\Desktop\ Chrome %PUBLIC_HOSTNAME%
C:\Users\Administrator\Desktop\ Firefox %PUBLIC_HOSTNAME%

So it should look like this now…

C:\Users\Administrator\Desktop\ Ie %PUBLIC_HOSTNAME%
C:\Users\Administrator\Desktop\ Chrome %PUBLIC_HOSTNAME%
C:\Users\Administrator\Desktop\ Firefox %PUBLIC_HOSTNAME%

We now have 3 build steps in this Job. It’s really important that they run in the right order though. You’ll notice on the build steps that you can ‘grab’ the handle and change the order the steps. For example…

To get this working correctly you MUST put the build steps in this order…

  1. Copy artifacts from another project
  2. Inject environment variables
  3. Execute Windows batch command

In short you need the publicHostname.txt artifact available to the job before you can inject it. Once publicHostname.txt is available to the job the ‘Inject environement varialbes’ build step can read the file and make the PUBLIC_HOSTNAME environment variable available to the job. Then the windows batch command can use the %PUBLIC_HOSTNAME% variable when the commands are executed.

Save the changes you’ve made to the job and return to the Jenkins home page. Run the ‘BuildRocketChatOnNode’ job which will then trigger the ‘RunClientRDPSession’ job follwed by the ‘RunSeleniumTests’ job automtically.

With a bit of luck you should see the BuildRocketChatOnNode job create the publicHostname.txt file. That publicHostname.txt copied for use in the RunSeleniumTests and these test complete successfully.

You may find the first Selenium test against IE failed. This will probably be down to the trusted sites setup in IE. In which case you’ll want to define the domain as a trusted site so the ‘warning message’ doesn’t stop the execution. You can do this on these dialogue boxes:

All we need to do now is run this right from the start which Jenkins creating a new instances of the Linux machine (with a new public hostname).

7. Test the whole workflow

That’s everything configured. All we need to do now is test the whole process. To do this we’ll

i. Go to the Amazon EC2 dashboard
ii. Select ‘Instances’ in menu
iii. ‘Terminate’ the Linux Ubuntu client
iv. ‘Stop’ the Windows client

Make sure you get these the right way round. We don’t want to Terminate the Windows client

Now we can go to the Jenkins dashboard and kick of the full process. To do this click ‘Build Now’ for the ‘BuildrocketChatOnNode’ project.

This should create the Linux EC2 instance, build the AUT and install Rocket Chat. If that completes successfully then the next projects in the chain will start that run the RDP session and kick off the Selenium tests. Last but not least the RDP terminal session is closed once the Selenium tests are finished. Job done.


So we’ve configured Jenkins to start a slave Windows machine that will run our browser tests. We’ve install the software we needed to run our browser tests (e.g. a set of different browsers, Python and Selenium). We’ve created our first Selenium script and checked that this runs against our application under test.

With the automated tests in place we’ve then configured Jenkins to run these scripts auotmatically on our Windows client machine. We’ve worked out how to run RDP/RDC terminal sessions on our Master Jenkins machine so that these Selenium tests can run in a real environment (not just some headless mode). In order to do this we’ve gone through some quite complex configuration steps in this module. Most of the complexity needed to get round Windows User Account Control (UAC).

With all the steps/projects created in Jenkins we’ve then completed the simple process of chainning all our Jenkins projects together. And hopefully on running this for the first time we’ve seen everything run end to end. Right from starting our client AWS machines, building/installing our AUT and running our automated tests.

What we’ve completed in this module is the crux of building out our test automation framework. Yes it’s a little tricky in places but once running you have a setup that you can expand and develop further. You know all the key tricks for building this out to test your own appplications now. You just need to plugin your application, write your own Selenium tests and plug those tests into this framwork too.

Free Test Automation Framework Course
Learn the Practical steps to building an Automation Framework. Six modules, six emails (each email a short course in it's own right) covering Amazon AWS, Jenkins, Selenium, SoapUI, Git and JMeter.

We have the test environment in place (if you don’t check here) and we know enough to be dangerous with Amazon Web Services. Next step is probably the most important. We need to get Jenkins installed, configured and running. Jenkins has to be one the testers most useful tools. It’s not just handy for kicking off automated tests it’s indispensable for automating all sorts of processes you have to complete day in day out.

We’re going to cover 4 main areas here…

1. Jenkins Installation
2. Installing Jenkins Plugins
2. Jenkins Configuration
3. Deploying and Building the Application Under Test

By the time we’ve completed the module we will have Jenkins running on our Windows Master machine. The Jenkins configuration will be completed so that it will carry out six main actions…

i. Run up a new/clean EC2 Ubuntu instance (client)
ii. Install the Jenkins client on the Ubuntu client
iii. Download the AUT from GitHub to the Ubuntu client
iv. Build the AUT on the Ubuntu client machine
v. Install and start the AUT running on the client machine
vi. Run some simple post build checks

What You’ll Learn

In this module we’ll go through the 12 parts listed below looking at the practical aspects of setting up Jenkins. By the ended of the module you will understand the following concepts:



When we’re finished we’ll have this setup:

BTS Overview Module 2 New Page

Note that we’re building on top of the AWS infrastructure we built in the first module by adding Jenkins and a bit of configuration. Whilst we created a client Ubuntu machine in in the first module we’re not actually going to be using that machine. What will happen is that Jenkins will fire up a completely new instance of the Ubuntu machine automatically whenever it’s needed. This is an ideal use case for bringing up Amazon Instances as and when we need them. More on how we do this in a minute.

Why Use Jenkins

On the Jenkins web site they don’t talk about Continuous Integration or Dev Ops or use any other popular buzz words. They simply refer to Jenkins as the “Open Source Automation Server”. And that’s the key. Jenkins isn’t just an app for developers. It’s an automation tool for testers too.

I personally love Jenkins. I don’t think of it as an app for developers or build engineers. I don’t think of it as an app for continuous integration or for building software. Yes it does all of that. But! It’s just the best app for automating a load of the daily or weekly tasks that the test team have to perform. Whether that’s installing the application you want to test or setting up your test environment. It’s just the best way to save yourself loads of time and effort on repetitive tasks.


A few things we need to make sure we have setup correctly before we start.

Make Sure your Windows Master EC2 Instance is Running

First, if you don’t have your Windows master machine up and running yet then you’ll need to revisit Module 1 and take a look at the section on Running up the Windows Instance.

With the Windows 2008 master machine configured then we’ll need make sure the instance is running and that we have an RDP session open. You can complete these steps with

1. Open your AWS management console, and view your list of Instances (making sure you have the correct Region selected).


BTS module2 img2 AWS start instance



2. Once started open an RDP session on this Windows machine.


BTS module2 img3 AWS RDP Connect



Then enter the password to open up the desktop session (you may need your .pem private key file to decrypt the password if you’ve forgotten it) .

Create and Save Your Amazon ‘Access Key Id’

Second we need to get our Amazon ‘Access Key ID’. We’ll need this later as we’re configuring Jenkins to start our client/slave machines. For other systems or programs to connect to your AWS account they need an access key (like a password but for an application not a person). Jenkins needs this access key if it’s going to be allowed to start EC2 instances.

To get this Id follow these steps:

1. Login to your AWS account

2. Click on your user account drop down and select ‘Security Credentials’


BTS module2 img3 AWS RDP Connect

3. Expand the ‘Access Keys’ section and click on ‘Create New Access Key’




4. You can either download or show the access key. Either way you need to record these two parts of the key


Access Key Id
Secret Access Key

NOTE: once you’ve downloaded or viewed the secret access key you CAN NOT obtain it again. If you lose it you’ll have to create another access key (no big deal but it’s easier if you just look after it now).

Make Sure you have your Private key (.pem file)

Third, remember back in Module 1 we created our public and private key pairs? Well at that stage you should have saved your private key pair .pem file (e.g. FirstKeyPair.pem) file. You’ll need this private key when configuring Jenkins later.

If you don’t have have this private key you can go back and create a new key pair. Much easier if you can find the one you created in Module 1 though.

Terminate any Running Unix Client Instances

Lastly, we need to make sure we don’t have any Linux Ubuntu clients running. When Jenkins creates and starts new instances of the Linux clients it checks to see that no existing instances are running OR in a stopped state. So we need to TERMINATE any running instances. To do this follow these steps:

i. login to your AWS management console

ii. go to your AWS EC2 dashboard

iii. select the correct region

iv. click ‘instances’ in the side menu

v. in the list of instances select your Unix-client instance

vi. right click the Unix-client entry, select ‘Instance State’ followed by ‘Terminate’

vii. in the list of instances you should now see this:


What Next?

With our Amazon Access ID, Key pair private key, our Linux machine terminated, our Windows master machine running and a desktop session available we’re ready to start installing and configuring Jenkins.

Part 1: Installing Jenkins

At this point we’re ready to visit the web site and download the Windows install package. Follow these steps to complete the install on your Windows 2008 server master machine.

1. Open up Internet Explorer and type in this URL

2. On the home page for Jenkins, right hand side, you’ll see the Download Native packages for Windows section. Click this link.


3. Open the download folder and drill down into the Jenkins zip file. Extract the file in the zip file by dragging it to the desktop


4. Run the installer accepting all the default install options. Nothing very exciting seems to happen on completion of the install. Except that after a while a new tab should open in Internet Explorer with the following:


If this page doesn’t open automatically you can type in this Url:


This is the default location and port that Jenkins runs on. This’ll be just fine for what we need.

And that’s it. That simple. Jenkins installed and ready to go. Next step configuration and creating your first job.

A Quick Overview of Jenkins

Before we start configuring Jenkins lets take a look at some key features and concepts. As we’ve already mentioned Jenkins is an automation server that gives us the capability to automate many process within our development and test projects.

Part 2: The Main Jenkins Configuration Areas

We need to achieve three things with our configuration of Jenkins

  1. Start a slave machine: we’ll get Jenkins to automatically start our Linux Ubuntu machine when we’re ready to build and install the application under test on this Linux machine
  2. Download, Build and Install the application under test: once the client Linux machine is running we’ll download the source code for the AUT, build the AUT and install it on our Linux client machine.
  3. Run some Smoke Tests Against the AUT: once the AUT is installed and running we’ll run a couple of very simple tests to make sure that it is up and running.

Part 3: Jenkins’ Plugins

A fundamental concept within Jenkins is it’s plug-in module capability. There’s a plug in module to help you automate just about any task you need to automate. A full list of the modules can be found here..

These plugins are broken into a number of different types that deliver different types of capabilities:

Source Code Management: Whilst Jenkins supports source code tools like Subversion and CVS out of the box there are other source code tools you might want to work with. Source Code Management plugins support all sorts of other source code tools. We’ll be working with Git and GitHub so we’ll need to add a plugin for GitHub in a minute.

Build Triggers: When do you want, or need, to build and test the AUT? Well plugins for Build Triggers allow you to monitor external activities and trigger a build of your AUT when certain conditions are met. For example a developer may check in some code and you’ll want to trigger a build off that check in.

Cluster Management and Distributed build: Typically with lots of automation activities being carried out, and many builds to process, you’ll distribute these activities to other client machines. You won’t run everything on your main Jenkins server. Distributed build plugins allow you to run up, communicate and control other machines. We’ll be using an Amazon EC2 Plugin that runs up and controls our Amazon EC2 Linux Ubuntu slave machine.

Slave Launchers and controllers: Depending on the type of slave machines you run up you’ll want to find different ways of communicating with those slaves. You’ve already seen how we’ll be communicating between our Windows master machine and Ubuntu client machine with SSH. So we’ll be installing the SSH Slaves plugin so that our Windows master Jenkins machine can communicate and control the Linux Ubuntu slave machine over SSH.

Other Post Build Actions: Once the build is complete you may want to complete other actions like running Windows Power shell scripts or Unix shell scripts. Things like this can be setup with ‘Post Build Action’ plugins. We’ll be using Hudson Post build task’ plugin to run some install completion scripts.

There are many other categories of plugins, including Build notifiers, Artifact up-loaders and build reporting. There are hundreds of plugins so chances are any automation task you need to complete will be covered either by features native to Jenkins or a plugin. If it’s not you can always write your own plugin too (but that’s a topic for another day).

Part 4: Installing Plugins

Let’s install the plugins we need for this project. We’re going to need these:

Amazon EC2 Plugin

With this plugin we can use our Jenkins machine to run up an Amazon EC2 instance automatically. We’ll be running up a Linux Ubuntu instance that will then run a Jenkins client application. This Jenkins client application will then be responsible for downloading the AUT source code, building the AUT and running the AUT.

SSH Slaves Plugin

If you remember from our first Module we connected to our Linux Ubuntu client machine using SSH. This plugin gives Jenkins the capability to use our SSH private key to connect to the Amazon EC2 client machine. Once Jenkins has a connection to the client machine it can install the Jenkins client application on this machine and then control this machine.

GitHub Plugin

For this project our AUT is an application called Rocket.Chat. This project’s source is hosted on GitHub. With the Jenkins Github plugin we can connect to GitHub and download the source automatically.

Post Build Tasks

Once we’ve completed the AUT build and install we’ll want to run a few tasks to check that everything is okay. With this plugin we’ll be able to check the logs for successful install messages and run a few checks to make sure the AUT is running correctly.

How do we install and/or update these plugins? Well Jenkins makes is really easy. Just follow these steps:

1. Click ‘Manage Jenkins’ in the control panel

2. Click ‘Manage Plugins’ in the control panel


On the Plugin Manager screen you should see a list of plugins grouped by tabs that cover; Updates, Available, Installed and Advanced tabs. First we’ll need to update one plugin.

3. On the ‘Update’ tab find the ‘SSH Slaves plugin’, check the ‘install’ check box and click the ‘Download now and install after restart’ button.



5. Let the install and upgrade complete so that ‘Download Successfully’ is displayed. Click the ‘Go back to the top page’ link.


Update success

So that’s one plugin updated just another 3 to install now. To install these plugins..


  • Select the ‘Available’ tab and search for each of these plugins in turn, then checking the check box:Amazon EC2 pluginselect-plugins
    GitHub PluginHudson Post build task


Then click the ‘Download now and install after restart’ button again.

At this point Jenkins should start installing the plugins and all the dependency plugins. So you’ll see a lot more than just 3 plugins installed.

Once you see all of the ‘Downloaded Successfully’ messages you can go ahead and restart Jenkins.

6. Restart Jenkins just by clicking on the ‘Restart Jenkins when installation is complete…’ check box


At which point you should see this restart message:

With all our plugins installed we’re ready to start configuring Jenkins.

Part 5: Configuring Jenkins

Now for the interesting part. We need to configure Jenkins so that the tasks we need automating are setup to be run by Jenkins. There are four key parts to this configuration.

Firstly we need to set Jenkins up so that it is integrated with our Amazon AWS service. The ‘Amazon EC2 Plugin’ we’ve installed needs to have the ‘Amazon Access Key Id’ installed so that it has permissions to drive AWS.

Secondly, we need to configure Jenkins so that it will start our Linux Ubuntu machine on demand. When Jenkins starts that machine it needs to specify the Amazon AMI to use and install other software packages that we’ll need for our build/instll.

Thirdly, we’ll also need to setup an initialisation script for the Linux Ubuntu instances that we start via Jenkins. This script will be responsible for updating existing software packages on the Linux client and adding some additional software that isn’t included in the AMI we’ve used.

Lastly, we need to configure the ‘job’ that kicks of the build and install of our AUT. Jobs in Jenkins are just automated tasks that you configure Jenkins to run. For each Job you configure you can run that Job (e.g. build now), track the changes you’ve made to it and see all the run results for that job.

Hot tip: this setup is designed so that the client/slave machine that we install our application under test on gets shut down and everything deleted on a regular basis. This means any manual configuration you do, or data you add, on this slave machine gets deleted too. This is by design. By good design.

When you know your test server will get deleted and everything removed on a regular basis it FORCES you to make sure EVERYTHING is in the automation scripts. Every actions that you need to build the test environment is scripted. This means everything is documented and repeatable.

Part 6: Configuring Jenkins – Amazon AWS Integration

Back on the Jenkins home page you’ll need to:


1. click on the ‘Manage Jenkins’ link again



2. click the ‘Configure System’ option


There are a lot of options for configuring Jenkins. All we need to concern ourselves with is the ‘Cloud’ section (at the bottom of the page). The Cloud section is where we define our Amazon EC2 instance start up details.

3. Scroll to the bottom of the Jenkins ‘Configure’ page where you should find the ‘Cloud’ section. Then click ‘Add a new cloud’ and select the ‘Amazon EC2’ option

From here we can enter all of the details needed to run up our EC2 Ubuntu instance. Before we can complete this though we’ll need to get our Amazon ‘Access Key ID’ which you should have from going through the Prerequisites Section of this module.

4. Enter the following details for your Amazon EC2 instance

 Name: RocketChatServer
Access Key ID: <key created in prerequisite stage of module>
Secret Access Key: <key created in prerequisite stage of module>
Region: <select your region>*1
EC2 Key Pair’s Private Key: <copy in the contents of your .pem file>*2

*1 – make sure you select the same region that your Windows 2008 server machine is running in. Login to the AWS management console and check the region if you’re not sure. It’s critical that you select the right region.

*2 – in the Prerequisite section in this module we talk through getting your .pem private key file that you created in module 1. You’ll need to open this .pem file in a text editor and paste the contents in to the Jenkins ‘EC2 Key Pair’s Private Key’ text box.

NOTE: if you see this error message once you’ve pasted in your private key you can usually just ignore it:


Once all of these fields are configured you can click the ‘Test Connection’ button.

Once you’ve completed the test you should see the word ‘Success’ to the left of the ‘Test Connection’ button.


Now we’re connected to our AWS account we’re ready to start configuring the details for the Linux Ubuntu slave machine we need to start up.

Part 7: Configuring Jenkins – Starting the Linux Amazon Instance Slave


5. On the ‘AMI’s line click the ‘Add’ button

6. Enter the following details for your AMI instance

Description: RocketChat-Server
AMI ID:<see below>

So the AMI id is the Amazon Machine Image number that correlates with our Linux Ubuntu machine that we want to run up. So it’s the same Id as the Linux Ubuntu machine instance that we already have configured in our AWS account. We find this ID with …

i. login to your AWS management console

ii. go to your AWS EC2 dashboard

iii. select the correct region

iv. click ‘instances’ in the side menu

v. in the list of instances select your Unix-client instance

vi. in the instance details panel (bottom of page) look for this:

From here you can pick out the ‘ami’ which should be something like:


Then we just need to click on the ‘Check AMI’ button. This should give you confirmation that AWS can find the image you need. It should look like this:

  1. Complete the rest of the AMI details

Instance Type: t2.micro (IMPORTANT*)

EBS Optimized: Un-checked

Availability Zone: <leave blank>

Use Spot Instance: Un-checked

Security Group Names: default, Unix-AUT (see below)

Remote FS root:<leave blank>

Remote user: ubuntu

AMI Type: unix

Root command prefix:<leave blank>

Remote ssh port: 22

Labels: RCBuildDeployServer

Usage: Utilize this node as much as possible

Idle termination time: 30

Init script:<leave blank>

*-make sure you select ‘t2.micro’ here as this is the only Instance Type on the free tier. Select anything else and you’ll be charged for it!

The security groups you configure here will be determined by what you’ve setup in your AWS account. To find the right value:

i. login to your AWS management console

ii. go to your AWS EC2 dashboard

iii. make sure you have the correct region selected

iv. click ‘Security Groups’ in the side menu

v. in the list of security groups find your Unix-AUT group name

vi. find the ‘Group Name’. You’re looking for this:

This group will be specific to you. You may have used the same name as me but it’s best to check using the steps above. You’ll need to use both the Unix security group (Unix-AUT) and the default security group (default)

That should give us everything we need for this initial stage. Once you have the settings looking like this click the ‘Apply’ button.

Now we’re ready to check that Jenkins can run up an Amazon EC2 Linux Ubuntu instance for us automatically.

8. Return to the Jenkins home page (click ‘Jenkins’ link top right)

9. Click the ‘Build Executor Status’ link on the home page

This will take you to the ‘Nodes’ page. On this page you’ll have a list of nodes that can act as build machiens. You’ll already see your Windows 2008 Master machine listed. You won’t see a slave machine in the list yet because it’s not running yet. However, we can run the salve machine up from here.


10. Click the ‘Provision via RocketChatServer’ button and select the AMI we defined earlier


At this point you should see a message from Jenkins that ‘This node is being launched’

From here we can do two things:

  1. check the Jenkins log to make sure the node is launched correctly
  2. check our AWS account to see the running node

11. Check the Jenkins log by clicking ‘See log for more details’

At this point Jenkins is logging all the actions carried out as it brings up this Ubuntu instance. What you need to check for is this…

This just tells us that the instance was started and that Jenkins successfully logged in using our private key under the account ‘ubuntu’. You may see some warning messages but as long as the machine is running and the login worked we’re on the right track.

12. Check the AWS management console

On the AWS management console we shold be able to see our new instance running now.

i. go to your AWS EC2 dashboard

ii. make sure you’ve selected the right region

iii. click ‘Instances’ in the side menu

iv. in the list look for Unix-client (or similar)

v. select the Unix-client in the list and check it’s running

At this point you should see something like this…

We’ve successfully run up our Linux Ubuntu instance automatically from Jenkins. Great!

A few things worth mentioning…

Firstly, you may have noticed that one of the parameters we configured in Jenkins for this instance was:

 Idle termination time: 30

This means that if the instance hasn’t done anything for 30 minutest Jenkins will automatically shut the machine down for us. No need to do anything ourselves we can just leave it to Jenkins to shut the system down when it’s not needed.

Secondly, you may have noticed, in the Jenkins logs, that there were some error messages around the installation of Java. Something along the lines of:

WARNING: Failed to download Java

We’ll solve that in the next step when we get Jenkins to run some initialisation scripts on the server. We do need Java though as the Jenkins Java client needs to be installed on this Ubuntu machine in order for Jenkins to have full control over it.

Part 8: Configuring Jenkins – Instance Initialisation Script

13. Set Up the Instance Initialisation Script

The last thing we need to setup during the run up of this instance is the installation of a few additional software packages. These packages aren’t installed on the AMI we’ve started out with. To get round this Jenkins has a field in the EC2 configuration called “Init Script”

To update this we’ll need to:

  1. return ot the main Jenkins dashboard (click the ‘Jenkins’ link top left)
  2. click the ‘Manage Jenkins’ link
  3. click the ‘Configure System’ link
  4. scroll to the bottom of the page where we configured our Amazon EC2 setup
  5. the last section on the page should be the ‘Init script’ field

So we’ll need to add some Unix shell commands to this Init script field. The purpose of these commands is to:

a. update software already on the system

b. install Java (Jenkins needs Java to run it’s remote client app)

c. install npm (more on this later)

You don’t really have to understand this script (it’s just a unix shell script) but as long as you understand why it’s there that’s the important part. So copy the following into the Init Script field:

# Update existing packages and install Java
export JAVA="/usr/bin/java"
if [ ! -x "$JAVA" ]; then
    sudo apt-get update
    sleep 10
    sudo apt-get install -y openjdk-7-jre ca-certificates-java tzdata-java libcups2 libcups2 libjpeg8 icedtea-7-jre-jamvm openjdk-7-jre-headless openjdk-7-jdk git npm
    sleep 5
# Add Swap Space
if [ ! -f $SWAP ]; then
    sudo dd if=/dev/zero of=$SWAP bs=1M count=2K
    sudo chmod 600 $SWAP
    sudo mkswap $SWAP
    # add new swap to config and start using it
    echo "$SWAP none swap defaults 0 0" | sudo tee -a /etc/fstab
    sudo swapon -a

You should have this now:

14. Return to the Jenkins home page (click ‘Jenkins’ link top right)

15 .Click the ‘Build Executor Status’ link on the home page

16. Click the ‘Provision via RocketChatServer’ button and select the AMI we defined earlier

At this point you should see a message from Jenkins that ‘This node is being launched’. This time round when you click on the ‘See log for more details’ you should see a bit more going on. In summary the log file should show you …

i. the Instance launching (INFO: Launching instance: i-1ebaa1c7)

ii. the ssh connection (INFO: Connected via SSH)

iii. execution of the Init Script (INFO: Executing init script)

iv. updates to existing software packages

v. installation of Java (INFO: Verifying that java exists)

vi. launching the Jenkins slave (INFO: Launching slave agent)

And then the final entry in the log…

Slave successfully connected and online

17. Go back to the Nodes dashboard (click ‘Jenkins-> Nodes’ in the top menu bar)

In the node list we should now see our RocketChat-Server node up and running:

At which point we have a fully configured Jenkins slave machine, running all the software we need that’s ready for us to start installing our AUT on. Next we’ll look at how to configure Jenkins to get that AUT built and installed automatically.

Part 9: A Few Points about our Jenkins Slave Machine

At this stage we can use Jenkins to start up a slave test machine automatically. There are a few points worth mentioning about how we can control this slave machine now.

Once the slave is running Jenkins runs a ‘Slave Agent’ on this machine. It’s this agent that allows the slave to talk to the master Windows Jenkins machine. It’s this slave that gives Jenkins full control of jobs and actions that can be carried out on the slave.

Slave Idle Termination Time

For example in our configuration for the Amazon EC2 slave (Manage Jenkins -> configuration) we had a setting called ‘Idle termination time’

This tells Jenkins that if the slave is inactive for more than 30 minutes to ‘Terminate’ this slave Linux EC2 instance. Great for keeping control of our Amazon charges (don’t want to be spending money on machines that are standing idle). Not great if we have data on that machine that we need as we’ll lose it all when it’s terminated. For us though we’re not worried about losing data so this’ll be just fine.

Managing Slaves from the Jenkins Master Machine

Return to the Jenkins home page and click on the link for our slave.

Viewing details for our slave you’ll see a number options in the left hand menu. These options allows us to:

Delete Slave: select this and you’ll Terminate the Amazon EC2 instance for this slave.

Configure: from here you can view and modify some of the configuration parameters for this slave

Build History: details about builds we’ve run on this slave.

Load Statistics: stats on how hard this machine is having to work.

Script Console: feature allowing us to run Groovy script on the slave machine from our Jenkins master machine

Log: details about the running up and install activities that took place when we created this instance

System Information: full list of system properties

Disconnect: the ability to disconnect Jenkins from the slave but leave the instance running still (you can reconnect later if you need to)

Part 10: Setting Jenkins up to install the AUT

So the last bit to configure is the download, build and install of the application under test (AUT) on our slave machine. The AUT we’re going to use is a chat application called Rocket Chat.

Now it’s worth mentioning here that this is just an example app we’re using for this course. We’ll go through the steps to build and install this app but you DON’T need to understand the scripts needed to build and install Rocket Chat. These are specific to Rocket Chat.

In practice, when you’re testing your own applications, you’ll need to replace the Rocket Chat build and install scripts with your own scripts. This will mean talking to your developers or build engineers. Then adding the build/install steps for your app into Jenkins.

So from this you just need to understand how you configure Jenkins. You don’t need to understand the Rocket Chat build/install process. In real life you’ll have your own applications build/install process to add to Jenkins.

Jobs and Node Labels

Jenkins is all about configuring and running jobs. Those jobs are typically build jobs but they can include any task you need to automate. For example jobs that run tests and jobs that install applications. Anything you need to automate Jenkins can be configured to run as a job.

Jobs need to run on nodes or slaves. Jenkins has some pretty clever features for working out the best places to run jobs (e.g. on the master machine or a particular slave). We’re going to keep it simple though. We going to give our slave/node a label (e.g. RCBuildDeployServer). Then when we configure the Job we’ll say “the job must be run on a slave/node that has label RCBuildDeployServer”. This way we tell Jenkins not to run everything on the Master machine (which is the default) but to run up an instance of the slave that we need (if it’s not already running) and run the job there.

So first up we to make sure we’ve configured our node/slave label.

Node Labels

To give our Linux Ubuntu instances a lable we need to go back to our Jenkins coniguration page.

1. On the Jenkins home page click on ‘Manage Jenkins’

2. On the Manage Jenkins page click on ‘Configure system’

3. Scroll to the bottom of this page and find the ‘Amazon EC2’ section

4. In the ‘Labels’ field enter ‘RCBuildDeployServer’


5. Click the ‘Save’ button

Setup the Job

Now we can specify where we want to run a Job (on the slave/node labled ‘RCBuildDeployServer’) we can configure a new job

1. On the Jenkins home page click on ‘create new jobs’

2.  Enter a Job name (e.g. ‘BuildRocketChatOnNode’), select ‘Freestyle Project’ and click OK

3.  On the Job configuration page set these key values

Restrict where this project can be run: RCBuildDeployServer

Source Code Management: None

Build Triggers: Build periodically

No need to add a schedule here yet. We’ll kick our jobs off manually for now.

4.  Click on the ‘Add Build Step’ button and select ‘Execute Shell’

5. In the command text box we need to add our build script. Remember you don’t need to understand the detail here. Just that for our demo app these actions build and install the app. You’ll have a completely different set of actions for your application.Copy and paste all of the scripts below into the ‘Command’ field:

#Clean up if any left overs from last build
SCREEN_RUNNING=’/usr/bin/pgrep SCREEN’
ifSCREEN_RUNNING fi NODE_RUNNING=’/usr/bin/pgrep node’ ifNODE_RUNNING fi if [ -f ]; then rm -f fi INSTDIR=./Rocket.Chat-master if [ -dINSTDIR fi MONDIR=/home/ubuntu/db if [ -d $MONDIR ]; then rm -Rf /home/ubuntu/db fi pwd #Install packages we need for the build sudo apt-get install unzip curl | sh sudo npm install -g n sudo n 0.10.40


#Configure Mongo Database
if [ ! -d $MONDIR ]; then
mkdir /home/ubuntu/db


#Build and install Rocket Chat
export PUBLIC_HOSTNAME=”$(curl 2>/dev/null)”
unzip -o
cd ./Rocket.Chat-master
meteor build –server $PUBLIC_HOSTNAME –directory .
cd ./bundle/programs/server
npm install
cd ../..


#Make sure processes continue to run when Jenkins script completes
echo -e “\n”
ps -ef
export BUILD_ID=dontKillMe
echo -e “\n”


#Start mongo DB and Rocket Chat
screen -d -m /home/ubuntu/.meteor/packages/meteor-tool/.1.1.10.ki0ccv++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/mongodb/bin/mongod –bind_ip localhost –dbpath /home/ubuntu/db/ –smallfiles
sleep 10
export MONGO_URL=mongodb://localhost:27017/rocketchat
export ROOT_URL=http://localhost:3000
export PORT=3000
screen -d -m node /home/ubuntu/workspace/BuildRocketChatOnNode/Rocket.Chat-master/bundle/main.js
sleep 10


# list system details and what's running on the system before we complete script
    echo -e "\n"
    echo -e "\n"
    echo -e "\n"
    ps -ef

So at this stage we should have the Build Execute script entered as shown

Like I’ve said this little lot is particular to the Rocket Chat app we’re using for this example. When you come to implement this in the real world you’ll work with your developers to determin what goes in here.

We can then click on the ‘Save’ button

6. On the Job summary page click the ‘Build Now’ button

7. Jenkins will go through a few steps now. First it will run up a Linux Ubuntu EC2 slave machine. This is the slave machine Jenkins needs to buid and install the application on.

If you check your AWS dashboard you should see this instance initializing. It may take a while as it has to run through all the initialisation scripts we defined too.Then, once the slave is ready, we’ll see the build start.This is our first build, as indicated by the ‘#1’ tag. We can view the detail and progress on this build by clicking on the ‘down arrow’ next to the #1 and selecting ‘Console Output’


The console output will list all of the actions being carried out as part of this build and install.

This may take some time!

Once the build and install is complete you should something like this at the end of the log file

From here we can check if the application is actually running on our slave EC2 Linux Ubuntu instance.

7. On your Windows RDP session, open a new tab in Internet explorer. Enter the following address: http://localhost:3000

You should see the application running like this:

If you want to login and play with Rocket Chat you can click on the ‘Register a new account’ link (you don’t need to follow any email confirmation process – the first login can be created direct from the GUI).

If you go back to your Jenkins home page you should see the successful build reported as follows:

Accessing Rocket Chat from your Local Machine

If you want to access the Rocket Chat application from you desktop/laptop then follow these steps:

  1. in your EC2 management console select the Linux host
  2. find the ‘Public DNS’ value
  3. Construct the following Url: http://<public-dns-value>:3000
  4. Enter this Url in your browser

Note that this will only work if your security groups are configured correctly and Public DNS settings are configured. See module 1 of this course for more info.

All that remains now is to setup a few clean up actions and a few tests to make sure Jenkins reports the build/install as a success.

Part 11: Jenkins Post Build Actions and Smoke Tests

Once we have a setup that will build and install our application it’s worth configuring Jenkins to run a few checks and tests just to confirm that things completed without any issues.

We’ll use the Post-Build Actions feature to carry out two actions:

a) check the logs for successful install messages
b) use ‘wget’ to check that the Rocket Chat home page is displayed

We can set these up by following these steps:

1. go back to the Job configuration by selecting configure on the home page.


2. Scroll to the bottom of the configuration page to find the Post Build Actions section

3. Click the ‘Add post-build action’ button and select the ‘Post Build Task’ option

Check the Logs

We’ll configure a task to look for 2 log messages. Two log message that we know we should see during the install if it’s been a successful install.

4. In the Log text field enter this text (it’s a reg ex expression)

00:..:.. SCREEN -d -m node /home/ubuntu/workspace/BuildRocketChatOnNode/

Then click the ‘Add’ button and add a 2nd log message check

00:..:.. SCREEN -d -m /home/ubuntu/.meteor

These checks are specific to the Rocket Chat application install. Again when you build this for your own application you’ll have to replace these with checks specific to your build. For info though these messages are checking that the Rocket Chat processes were successfully started and running.

5. In the Script field enter this shell script text

#keep checking to see when Rocket Chat is up and running
for I in 1 2 3 4 5 6 7 8 9
sleep 60
# use wget to see if we can download index page
# if wget returns true we can download index page
# -T 3 => wait 3 seconds for response
# -t 1 => means only retry once
if (exec /usr/bin/wget -T 3 -t 1 http://localhost:3000)
echo “Rocket Chat now running ”
exit 0
echo “Waited $I minute(s) - Rocket Chat not running yet ”
echo “ Rocket Chat didn’t start ”
exit 1

Again this is specific to Rocket Chat. We’re just running this script on the Linux Ubuntu Rocket Chat server to see if Rocket Chat starts running. We’re using a Unix command called ‘wget’ which gets a web page, the Rocket Chat home page. If this command returns a 404 over 10 attempts then we fail the check. If it returns a http 200 response code then we pass the check.

6. Check the ‘Run script only if all previous steps were successful’ check box

It’s only worth running the shell script if the checks on the log file pass.

7. Check the ‘Escalate script execution status to job status’ check box

If these checks pass then mark the build in Jenkins as a pass. If any of these checks fail then mark the build as failed in Jenkins. We’ll see the reporting of these results in a minute.

Then click the ‘Save’ button

Part 12: Starting Subsequent Jenkins Builds

Now at this point, do you remember this setting in your Amazon Ec2 configuration…

 Idle termination time: 30

Well your Linux slave machine has probably been idle for 30 minutes now. As a result it’s probably shut itself down. Then again maybe you went through this really quickly and it’s still running. Either way, depending on the state of this slave machine Jenkins will do one of two things when we start a new build:

a. If it has shut itself down:

When we kick off a new build it will start from the beginning and create a new EC2 instance.

NOTE: if the node has been shutdown and then a new node run up for a subsequent build then the new node with have a NEW IP ADDRESS and HOSTNAME.

b. If it hasn’t shut itself down:

The slave is still running, so Jenkins just needs to start a new build and install using the existing slave machine

If you want to avoid these nodes shutting down you can always extend the Idle termination time to more than 30 minutes. Or set it to 0 so that it always stays running (mind your AWS spend if you do this though).

Start the next build

1. Return to the Jenkins home page

2. Click the Job name (just to the right of it) and select ‘Build Now’ from the drop down menu

Assuming you didn’t have a slave running the whole process will kick off again, starting with the running up of the Amazon EC2 instance. Then the build/install will take place. Then we’ll run our completing checks/tests.

Don’t forget it can take a few minutes for these Amazon EC2 instances to run up. It can also take up to 10 minutes to complete the full job including the build and install. This is because we’re using free tier instances within AWS. Please be patient.

If the build, install and tests all succeed then we’ll see this Blue lamp and the sun icon on the Jenkins home page:

If anything fails we’ll see a red lamp and the weather will start to turn cloudy:


Getting it badly wrong over a number of builds and the weather turns to Thunder and Lightning.


In short the lamp indicates the state of the last job. The weather is an indication of job stability. So an aggregation of results across the last few builds.


Building on our AWS environment we’ve configured Jenkins so that we can control a slave machine (an Ubuntu Linux machine) and build and install the application under test. We’ve also factored in a few small checks to make sure everything is up and running.

The key take away here is that Jenkins is not just for building software. It’s not even just for implementing continuous integration. Both of these are great goals. However, on it’s own it’s just a great tool for testers. It allows us to automate a lot of our repetitive tasks like running up test environments and installing the software we need to test.

Then as we add to this the capability to kick of automated tests we’re into making our test process far more efficient. And what we’ll move on to next, in module 3 of this course, is linking in automation tools like Selenium.

Free Test Automation Framework Course
Learn the Practical steps to building an Automation Framework. Six modules, six emails (each email a short course in it's own right) covering Amazon AWS, Jenkins, Selenium, SoapUI, Git and JMeter.

Our 6 module course on Building the Test Automation Framework starts with Amazon Web Services (AWS). We’re going to use AWS, and more specifically, the Elastic Compute Cloud (EC2) to build our test environment and automation system. It all starts with configuring and running up the virtual machines we need to run everything on.

If you haven’t read our introduction to Building the Test Automation Framework you can read more here:

An Introduction to

Building the Test Automation Framework

This first module is all about understanding Amazon Web Services (AWS) and Elastic Compute Cloud (EC2).


We’re going to cover six key areas:

  • First, creating an AWS account (you can skip this part if you already have one)
  • Secondly, configuring two security groups.
  • Thirdly, creating a security key pair.
  • Fourthly, we’ll run up two Virtual Machines (one Windows master and one Unix slave)
  • Fith, we’ll install Putty for our Secure Shell (so we can connect from the windows machine to the unix machine)
  • Lastly, Monitoring Usage

The last few parts we’ll look at checking your AWS usage and making sure you run your virtual servers down. We want to avoid you getting charged for using these Amazon virtual machines (more on this in a moment).

Why Use Amazon Web Services?

As a tester why would you want to use Amazon Web Services (AWS)? Well it’s fast to get setup, it’s scalable and it’s cost effective. Let’s address each of those points in turn.

So you need a new environment to test in. If don’t already have the hardware you’ll need to purchase the hardware, install the operating system, configure the network, etc, etc. If you have an AWS account all you need to do is pick a machine image and fire it up. No purchase, no ordering, no hardware configuration and no operating system install. It’s all there in minutes.

The AWS Compute side of things isn’t called Elastic Compute Cloud (EC2) for nothing. If you don’t need it you run it down. If you need more of it you scale it up. What you need is there, when you need it – on demand. No acquiring physical machines that are obsolete within a few years.

The whole concept is you use, and pay for, what you need when you need it. The traditional model is that you purchase a PC and pay the full cost up front. If you only use your PC for half the time (e.g. you don’t use it throughout the night) you don’t pay half price for it. You still pay full price. And then there’s all the hidden costs, like network infrastructure, power, racking, cooling, etc. This all adds up, but you never see it. Okay, you can rack up big costs fast with AWS if you’re not careful. Manage it well though and you’ll save money.

Yes this is a new way of working (well not even that new anymore), but it’s a smart way for testers to work.


You don’t need a lot to cover this or any of the other module in this series. However, you will need the following:

1. A windows desktop or laptop that runs RDP
You can check if you have RDP on your machine with:

  • press the Windows button + r
  • in the Run window type ‘mstsc’
  • this should bring up the Remote Desktop Connection window

2. A credit card
No, this course doesn’t cost you anything. However, to sign up for an AWS account, even if you stick to using the AWS free tier, you’ll need to provide credit card details. Sorry that’s not our requirement – it’s Amazon’s requirement.

We have designed this course so that you use the Amazon Free Tier. So even though you have to provide your credit card details you should not be charged. HOWEVER, a word of WARNING! It is YOUR responsibility to monitor your usage and keep track of your AWS resource usage. If you go above the free tier allocation Amazon WILL BILL YOU. If this happens it will be YOUR bill to pay. We can’t be held responsible for any charges you incur! Sorry.

What You’ll Learn

In this module, covered by the 11 parts listed below, we’ll go through the practical aspects of using AWS. By the end of the module you’ll understand the following concepts:


What will your setup look like when we’re finished?

Over the course of the 6 modules you’ll get to this….


For this module you’ll run up the Windows master machine (that we’ll run Jenkins on later) and one slave Unix machine (that we’ll use to build and install the application under test on). Don’t worry if you’re not familiar with Unix or you’re not familiar with Windows. We’ll cover all the steps you need. By the end of this module we’ll have created the following:


You’ll start out with a master/control machine, running Windows Server 2008 (yes I know that’s old but it’s less resource hungry and delivers everything we need). This machine will be running Jenkins which will control all the other machines. You’ll access this machine using RDP from your desktop or laptop.

Then you’ll add a Unix (Ubuntu) machine where we’ll build, install and test our Application Under Test (AUT). Normally you’ll have build and test on different machines but for this module one machine will do. The application we’ll be working with is Rocket Chat (see More on why we’ve chosen Rocket Chat back in the introduction post here.

Over time, and as you progress through these modules, we’ll be adding various other nodes/hosts to this network. They’ll include Selenium, JMeter and SoapUI nodes.

So we’re looking for a distributed network of nodes that will run within their own Amazon Virtual Private Cloud (VPC). We’ll make sure the machines have public DNS records so that we can access these machines from outside of the VPC. Within that VPC you’ll configure the security groups so that these machines only allow access for the services we need. Finally, we’ll connect both machines (Windows and Unix) by installing Putty (a Secure Shell client) on the Windows machine. Putty will allow us to SSH from the Windows machine to the Unix machine. If you’ve not come across Putty and SSH we’ll cover it all in just a moment.

For this module we’re only going as far as setting up the AWS virtual machines. We’ll be looking at installing Jenkins and setting up the AUT in the next module.

Let’s get started!

Part 1: Creating an AWS Account

We’re going to build our test system using Amazon Web Services. This allows us to spin up virtual machines in the cloud on demand. Once you’ve signed up for an AWS account you’ll have access to all of these services through the AWS dashboard:

AWS Dashboard

You’ll only need the EC2 service (along with the assoicated EBS storage, security and machines images) for now. So how do we create our AWS account?

If you already have an AWS account you can skip the following and jump to AWS Fundamentals.

Be warned though……. if you have an existing AWS account you may be out of your 12 month Free tier period. Or you may have nearly exghasted all of your free tier resources. So Amazon may charge you for running up the virtual servers covered here. It’s YOUR responsibility to monitor and, if necessary, pay for any usage above the free tier. If you don’t know how to monitor this we show at the bottom of this module.

Follow these steps to create your AWS account:

1. Click the ‘Sign In To The Console’ Button here (you can’t miss it!)…

2. On the “Create an AWS Account” page:

AWS Sign up

– enter your email
– select ‘I am a new user’
– click ‘Sign in’

3. On the “Login Credentials” page:

AWS sign up step 2

– complete all the fields
– click ‘Create account’

4. On the ‘Contact Information’ page:

AWS Sign up form

– select ‘Company Account’ or ‘Personal Account’
– complete all the fields (including a CORRECT telephone number*)
– Read the AWS Customer Agreement and confirm (no I didn’t read it … but you sould)
– click ‘Create account and Continue’

* – I don’t like giving out my telephone number either but it’s the only way to sign up for an AWS account as they use this to confirm your details.

5. Enter your ‘Payment Information’

AWS phone number

– review the Free Usage Tier info
– enter credit card details
– click ‘Continue’

6. Complete the ‘Identity Verification Step’

AWS Phone Number

– confirm the correct telephone number
– click ‘Call Me Now’
– complete steps 2 and 3

7. Select your ‘Support Plan’

AWS Select Your Free Plan

– just stick to the Basic (Free) option
– click ‘Continue’

8. Sign in to the console

At this point you should see the welcome screen and be able to…

– click the ‘Sign In to the Console’ button
– enter your email address
– select ‘I am a ternurning user…’
– enter password
– click ‘Sign in using our secure server’

And that’s it. At this point you should have an AWS account, you should be logged in and see the management console that lists all of the available services.

AWS Dashboard

Part 2: AWS Fundamentals

These fundamentals are all you’ll need to complete all of our modules in this series. It’s all you’ll need to get to productive with your testing on AWS. Trust me, despite all the different services and options, it’s easy to get started with AWS. You just need to grasp the following:

Services: Amazon provide a range of different services. A service can be thought of as the type of work a particular cloud resource provides. These services are grouped into categories like Compute, Storage, Database, Networking and others.

As an example, in the ‘Compute’ category, we have the EC2 (Elastic Compute Cloud) service along with other services like VPC (Virtual Private Cloud). In the ‘Storage’ category we have the EBS (Elastic Block Store) service along with other storage services like S3 (Simple Storage Service).

All of the services available are listed in the first page you see when you login to AWS:

AWS Management Console

For the purpose of these modules we’ll be focusing on EC2, EBS and VPC. Each of these are described in more detail below.

Zones and Regions: are used to reduce latency between the end users and the services that are provided. Amazon provisions these services in data centers located in different regions. For example services are provided from locations like US East (N. Virginia), EU (Ireland), Asia Pacific (Singapore), and many others. Note that not all services are provided in each region.

When you start services you’ll want to make sure that you have the correct regieon selected so that the service is started in your regeion. Also, and this is important, when you create VPCs, Security Groups and Key Pairs these are all linked to a region. Everything you need should be defined and created in the same regeion. So it’s best to select the region up front and stick with it.

Management console: the AWS management console gives you the ability to start, configure and manage the services you need. For example you can spin up an EC2 instance and from the management console start an RDP (remote desktop) session on that EC2 instance.

The key components of the management console are highlighted in this image:

AWS EC2 Management Console

EC2: is an abbreviation for Elastic Compute Cloud. Essentially EC2 is the service that delivers resizeable computing capacity. When you select EC2 in the management console you are given the capability to start, configure and manage virtual machines in Amazons cloud. These virtual machines are known as ‘Instances’.

AWS Management Console EC2 Option

EC2 Instances: from the management console you can run up virtual machine instances running all sorts of different operating systems, with different hardware platforms and with a range of different default software installed. Each virtual machine you run up is referred to as an ‘instance’. For example you can run up a virtual machine instance that has Windows 2008 Server, with 2 CPUs, 4 GB memory on a 64 bit architecture.

AWS Management Console EC2 Instance

EC2 Instance Types: Each ‘Instance’ you run up will be of a specific type. A ‘Type’ defines the the CPU, memory, storage and networking capability of the ‘Instance’. Types typically range from ‘T2.nano’ (low capacity on all fronts) to different families of ‘Large’. In the example above we have an instance Type of ‘t2.micro’ which has 1 vCPU and 1GiB of memeory.

You will find a list of all the different instance Types here…

Images and AMIs: each time you start a virtual machine you don’t want to have to install and configure the operating system and additional software from scratch. To save you building the instance from scratch Amazon give you the capability to start the machine with a predefined image already installed. These images are known as AMIs (Amazon Machine Images). AMIs define the operating system, architecture (32 bit or 64 bit), launch permissions and storage for the root device.

A full break down of the AMI components can be found here….

Elastic Block Store and Volumes: a virtual machine is no good without storage. Amazon provide a huge number of options here.

The most common data storage service is known as EBS (Elastic Block Store). EBS is a service that provides storage volumes that can be attached to running ‘Instances’, the data stored is persistent (e.g. data is retained across reboots, shutdowns, etc) and each EBS volume can persist independently from the life of the ‘Instance’ if need be.

Also available, and commonly used, is Instance Store storage. This storage service is automatically provisioned with some EC2 instances and AMIs. The defining feature is that the storage is physically attached to the host machine. You should be aware though that Instance Storage is NOT persistent. Data is not retained across shutdowns (stopping or terminating a machine).

For us, we’ll be using EBS. EBS because the data stored is persistent. Also, EBS is automatically provisioned on some AMI’s. The AMI’s we’ve selected for this course all have EBS storage (e.g. so you’ll have storage that is automatically setup and that will persist when you stop or terminate the machine).

You can see the storage type you have associated with your Instance here:

AWS Management Console EC2 Instance Type

And if you want to know more about AWS storage it’s all in here…

Network and Security: when you start creating instances from your Amazon management console you might notice that Amazon creates a Virtual Private Cloud (VPC) for you automatically. Every Instance you create is placed in your VPC automatically too. You will find you click on the console home icon then select VPC you’ll be taken to the VPC dashboard. If you didn’t notice then this is where you’ll see details for the VPC that your instances are placed in..

AWS Virtual Private Cloud

Pretty much all of the setting here you don’t need to worry about. AWS takes care of all of this for you. However, there’s one setting ‘DNS Resolution’ that you will need to update and set to ‘yes’. We’ll talk about how and why later but this makes all the Intances you create visible with public hostnames and Ip addresses outside of your VPC. More on this later.

Once Instances are run up you’ll see them associated with your VPC here…

AWS Virtual Private Cloud

This VPC is essentially the same as the networks you’d find for you physical PCs and laptops on in your office. Just be aware that by default your instances are NOT given public IP addresses and host names. Everything is defined by default with local VPC IP addresses and host names. Again we’ll look at configuring the VPC to provide public IP addresses and host names in a moment.

Security Groups: Built in to the whole AWS framework are some pretty clever security capabilities. Key to running our instances is the concept of Security Groups. Each security group contains a set of fire wall rules. Each time you start an instance you specify the security group you want to associate to the instance and the instance inherits these firewall rules.

AWS Security Groups

So for the example above when we run our AUT (Rocket Chat) on one of our instances the AUT needs the rules defined to allow access to the AUT services that will be running.

What we’ll do for your setup is define a couple of security groups that contain rules for each Instance that you want to setup. Then each time we start and instance that will be running either Rocket Chat or some other applications we’ll associate the required security group with the instance.

Key Pairs: AWS uses key pairs to encrypt and decrypt login details (e.g. passwords). These are the login details you’ll need to access your both your Windows and Unix Instances.

The concept here is that a public and (related) private key are created. Amazon encrypt the password with the public key. Amazon provide you with the encrypted password that no one can decrypt. That is unless you have the associated private key. Assuming you do have the private key can then decrypt the encrypted password.

So to log in to any Windows or Unix Instance you create you will need to create a public and private key pair first. Only once you have the key pair can Amazon encrypt the login details, and you decrypt the login details.

It’s slightly different for Windows and Unix Intsances here. For Windows instances it’s the login password that is encrypted and used to login. For Unix instances it’s the key pair that’s needed in conjunction with SSH (Secure Shell) to login (there are no passwords for Unix machines).

More about all of this in a moment. For now though just be aware that you can create your public and private keys in the EC2 dashboard in the management console here…

AWS Key Pairs

We’ll go into much more detail about this in a moment. Just remember that once you have the private key you MUST NOT lose it. You can ONLY download the private key you need ONCE!!!! Don’t lose it!!!!

So what’s next then?

Well we understand the fundamentals so lets start running up our instances and building our test system. First we’ll need to configure some security groups and setup a security key pair. Once this is complete we’ll be ready to run up our Windows 2008 instance and our Ubuntu Unix instance.

Part 3: Configuring Security Groups

First make sure you select the right region (select this from the management console top right). Security Groups are created FOR each region and can’t be used across regions.

Once you’ve selected the region we need to create 2 security groups. The first will be used for our Windows master machine. The second security group for our Unix machine that is running Rocket Chat (the application under test).

You can configure these two security groups by following these steps….

1. on the Management Console home page click the “EC2” icon
2. on the EC2 Dashboard page click on “Security Groups” menu item
3. on the security groups page click the ‘Create Secuirty Group’ button

The settings for both of these security groups you need to create are as follows:

Security group name: Windows-Master
Description: Windows Jenkins Master Instance
VPC: “Select your default VPC”
Inbound rules (click to enlarge this image):

AWS Security Group Example 1

What this will give you is a machine that lets you access Jenkins which is web based serving up HTTP and/or HTTPS traffic to your local browser. It will also all SSH (Putty) access so that you can create a terminal on the Unix instance you’ll be setting up. Finally RDP access is provided so that you can create an RDP session from your local PC or laptop.

Note that we’re selecting a ‘Source’ of ‘My IP’ so that only YOUR MACHINE will have an inbound connection to the instances we run up that use this security group. No other machines will have access (note that if you’re on a laptop and you move locations you may end up being blocked yourself). You can leave the source as for each rule but it’s less secure.

Security group name: Unix-AUT
Description: Unix Rocket Chat AUT Instance
VPC: “select your default VPC”
Inbound rules (click to enlarge this image):

AWS Security Group Example 2

What this will give you is access to the Rocket Chat AUT that serves up HTTP/HTTPS traffic to your local server. The Rocket Chat application also needs to provide and consume data from the the Mongo Database that is accessed on port 27017. It will allow SSH for terminal access from the Windows master machine. That’s all we need for now.

Just as a note you’ll see under security groups that by default you already have one security group created. This group will have a source id that refers to the same security group id (e.g. sg-xxxxx). It’s a circular reference if you like but just means that any instances in this security group can access any other instances in the same security group.

Once you’ve completed this you should have something like this…

AWS EC2 Instance Security Groups

Part 4: Creating a Security Key Pair

Again, make sure you have the right region selected (select this from the management console top right). Key pairs are create FOR each region and can’t be used across regions.

Now you need to create your security key pair. The private key you obtain from this process you’ll need to keep safe as we’ll need it later in the process. It’ll be needed to decrypt the windows password and to login to the Unix machine using SSH/Putty.

To create your key pair follow these steps:

1. on the EC2 Dashboard page click on “Key Pairs” menu item
2. on the key pairs page click the ‘Create Key Pair’ button
3. give the Key Pair a name (e.g. FirstKeyPair)
4. at this point you should be prompted to download the .pem file
5. click okay and save the .pem file somewhere safe

AWS EC2 Key pairs


You can open this .pem file with a text editor (e.g. notepad) if you like. You’ll see it’s just an RSA Private Key. Dont’ worry about this for now though. You just need to know that we’ll need this later when we want to get access to our instances.

Now we’re ready to start the fun stuff!

Part 5: Running up the Windows Instance (VM)

There’s four things we need to create our Windows Instance:

  1. AMI – Microsoft Windows Server 2008 R2 Base – ami-c5a7bea4
  2. Instance Type – t2.micro
  3. Security Group – Windows-Master (we created this earlier)
  4. Private Key – .pem file (from our key pair created earlier)

Next then, on the EC2 console/home page click “EC2 Dashboard”. You should see a resource list like this….

AWS Resource List

From here you click the ‘Launch Instance’ button. You should be able to follow the steps using the ‘four’ items of info listed above to create your first instance. Just accept all the defaults as you go through the steps.

Step 1: select the AMI listed above (2008 R2 Base – ami-c5a7bea4)
Step 2: select the Type listed above (t2.micro)
Step 3: accept all the ‘Instance Config’ defaults
Step 4: accept all the ‘Storage’ defaults
Step 5: [optional] add a tag for the Name if you like (e.g. Windows-Master)
Step 6: for the security group we’ll need to:
Step 6a: Select an existing security group
Step 6b: Check the ‘Windows-Master’ security group AND
Step 6c: Check the ‘default’ security group *
Step 7: Review and then ‘Launch’

At this point you’ll be asked to select a Key Pair. We’ll use the Key Pair we created earlier. The public key from this pair will be used to encrypt the windows password for the instance that is about to be created. If you don’t have the .pem file with the private key in you’ll never be able to decrypt the password and you’ll never gain access to your instance. You’ve to the private key safe right?!

Step 8a: Choose an existing key pair and select the ‘FirstKeyPair’
Step 8b: Acknowldege the warning and click ‘Launch Instances’

* Selecting both of these security groups will give you public access to the instance and give private (within your VPC) access to any other machines we create in your VPC.

Whilst this instance is coming up we’ll start creating our Unix instance.

If you’d like a more detailed account of how to launch a Windows Instance I’d recommend this:

Part 6: Running up the Ubuntu Unix Instance (VM)

There’s four things we need to create our Unix Instance

  1. AMI – Ubuntu Server 14.04 LTS (HVM), SSD Volume Type – ami-5189a661
  2. Instance Type – t2.micro
  3. Security Group – Unix-AUT (we created this earlier)
  4. Private Key – .pem file (from our key pair created earlier)

Next then, go back to the EC2 console/home page click “EC2 Dashboard”. From here you click the ‘Launch Instance’ button again. You should be able to follow the steps using the ‘four’ items of info listed above to create your second instance. Just accept all the defaults as you go through the steps.

Step 1: select the AMI listed above (Ubuntu Server 14.04 LTS*)
Step 2: select the Type listed above (t2.micro)
Step 3: accept all the ‘Instance Config’ defaults
Step 4: accept all the ‘Storage’ defaults*
Step 5: [optional] add a tag for the Name if you like (e.g. Unix-Client)
Step 6: for the security group we’ll need to:
Step 6a: Select an existing security group
Step 6b: Check the ‘Unix-AUT’ security group AND
Step 6c: Check the ‘default’ security group
Step 7: Review and then ‘Launch’

At this point you’ll be asked to select a Key Pair. We’ll use the Key Pair we created earlier. The public key from this pair will be used to encrypt the windows password for the instance that is about to be created. If you don’t have the .pem file with the private key in you’ll never be able to decrypt the password and you’ll never gain access to your instance. You’ve to the private key safe right?!

Step 8a: Choose an existing key pair and select the ‘FirstKeyPair’
Step 8b: Acknowldege the warning and click ‘Launch Instances’

* This instance does not come with EBS storage. It comes with SSD storage. That is we have storage that is not Persistant. When we ‘stop’ or ‘terminate’ this instance we’ll lose all our data. That’s fine for what we want to do but just be aware that nothing is retained on this instance.

If you’d like a more detailed account of how to launch a Linux Instance I’d recommend this:

If you go to the EC2 Dashboard now you should see something like this…

AWS EC2 Running Instances

And if you click on ‘2 Running Instances’ you should see a list of your instances like this….

AWS EC2 Running Instances 2

Now we’re ready to login to the windows master machine, install putty and then connect to the unix machine. At which point it’ll be job done. So not much left now.

Part 7: Connecting to the Windows Master Machine

Pretty straight forward to connect to the Windows machine. Just follow these steps:

Right click and select ‘Get Windows Password’

AWS EC2 Get Windows Password

Select the path to the ‘Private Key’ you saved earlier

AWS EC2 Decrytp Windows Password

Click the ‘Decrypt Password’ button

Write down the Windows Admin password and the Public DNS host name

IMPORTANT: If you don’t get a public DNS host name you’ll have to complete these steps and change this setting:

AWS EC2 Public DNS
  1. Click ‘Services’ (top menu bar)
  2. Select ‘VPC’ (in the drop down menu)
  3. Select ‘Your VPCs’ (in the side menu)
  4. Right click on your VPC entry
  5. Select ‘Edit DNS hostnames’ (in the context sensitive menu)
  6. Select the ‘Yes’ radio button and then ‘Save’
  7. Then return to ‘Services -> EC2’ and your list of running hosts

At this point you should have the IP address, Hostname of your Windows server, your user name (Administrator) and your password. You can right click on your ‘Windows Instance’ entry in the list and select ‘Connect’

AWS EC2 Windows connect

Open the ‘Remote Desktop File’. When you open the file with the default application ‘Remote Desktop Connection’…

AWS EC2 Windows connect with RDP

… you should then be able to connect using the credentials you have. With the RDP session established and access to the Windows server desktop we’re in a position to start installing Putty and connecting to our Unix machine.

Please note that you’ll need to connect using this method each time you need access. Don’t bother saving the RDP file as this will have a specific Public DNS record / Host name. When you restart this machine the Public DNS / Host name will change. This won’t be a problem if you always connect following the steps above because Amazon takes care of updating the RDP file with the correct IP address and host name each time.

Part 8: Installing Putty and SSH on the Windows Machine

Not sure if you’ve come across Putty yet but this is a great little (actually not that little if you consider how much has gone into this suite of tools) application for connecting via a secure shell from a windows machine to a unix machine. If you set it up correctly you can open a Shell session (command prompt) on the Unix machine at a click of a button without even entering a password. Here’s how:

  1. Open Internet Explorer within the RDP session on your Windows server
  2. Either search for Putty or enter this URL


  3. Download the Putty Installer: putty-0.66-installer.exe
  4. Run the installer selecting all the defaults

At this stage you should be able to see all the Putty tools in the Start menu. The tools we’ll be needing are…

  • PuTTYgen: allows us to convert our .pem key (see below)
  • Pagent: authentication agent that runs on the windows machine
  • PuTTY: Secure Shell client that connects to our unix machine

Part 9: Connecting to the Unix Client Machine

Three very straight forward steps to getting connected to our Unix machine. Very important we set it up correctly though. First, because it makes our life easier (quickly creating a terminal session on the unix machine) and secondly because later our Jenkins setup will connect automatically from this Windows machine to the Unix machine using SSH. SSH is a key component in our process of automating everything.

Step 1. Convert our .pem key
So the .pem private key Amazon gave us for connecting to our Windows and Unix machines isn’t supported by default by Putty. But it’s easy to convert it to the right format. Just….

  1. On your desktop/laptop copy your .pem file (e.g. FirstKeyPair.pem)
  2. On the windows server paste the .pem file to the desktop
  3. On the windows server start ‘PuTTYGen’
  4. In ‘PuttyGen’ select SSH-2 DSA
  5. Then click ‘Load’

    AWS EC2 Load pem file in puttyGen

  6. Load the .pem file (making sure you select ‘All Files (*.*)’
  7. AWS EC2 Load pem file in puttyGen

  8. Enter a passphrase and click the ‘Save private key’ button


  9. Save the new .ppk file to the desktop (call it FirstKeyPair.pkk if you like)
  10. This should leave you with the following icon on the desktop

    AWS EC2 Load pem file in puttyGen

Step 2. Run Putty Agent
Now we have our .pkk file that Putty can use we setup Putty Agent. Putty Agent runs in the background and holds our private key (.pkk file). When you make a connection to the Unix machine Putty take the key from the agent and uses it to establish a secure connection. Don’t forget that when AWS created our Unix instance it used the public key as part of the process for setting up the connection credentials. So when we try to connect using our private key it should work in conjunction with the other half of the key pair, the public part of the key pair. So lets setup Putty Agent:

  1. Double click the putty icon on the desktop
  2. Enter your Passphrase and click okay
  3. Check you have the putty agent running

So you should see Putty Agent running in the task tray. If you double click on the icon you should see that your private key is loaded.

AWS EC2 Load pem file in puttyGen

Now all we need to do is open the connection up to the unix machine.

Step 3. Start Putty and connect with SSH
Now we’ll start Putty and connect to our Unix machine. We’ll configure Putty correctly so that it’s easy to connect each time we need a terminal open on our Unix machine. Complete thesesteps:

  1. From the Start menu start Putty

    AWS EC2 Run Putty

  2. From your AWS Management console find and copy your Unix Private IP
  3. AWS EC2 Run Putty b

  4. Paste the IP Address into the Putty window
  5. Make sure SSH is selected
  6. Enter a saved session name (e.g. Unix-client)
  7. Click ‘Save’

    AWS EC2 Run Putty c

  8. Click ‘Connection -> Data’ in the side menu
  9. Enter ‘ubuntu’ for the Auto-login username

    AWS EC2 Run Putty d

  10. Click ‘Session’ in the side menu
  11. Click the ‘Save’ button again
  12. Click ‘Open’

At this point you should be prompted with a security alert. First time round it is valid to select ‘Yes’. And from here you should go straight into a SSH shell session on your Unix client machine with NO login required.

At this point you can just type ‘Exit’ and return in the shell window.

When ever you need a connection to this machine you can just carry out the followign action….

AWS EC2 Run Putty e

This will take you straight into the SSH shell prompt on your unix machine (no password required).

To be honest we don’t have much requirement to actually use the shell prompt for what we need. However, Jenkins does require SSH setup and configured to use it as a build and install node. If this is working we should be okay for the next stage.

The next stage then is installing Jenkins on the Windows machine and setting the Unix machine up as a Jenkins node. Before we jump to that though, a couple of small points…

Part 10: The Difference Between AWS Terminate and AWS Stop

You’ll notice in the management console that you have 2 options for bringing your servers down (both for Windows and Unix).

AWS EC2 Shutdown and Stop

Stop: when you stop an instance the instance is shutdown. If it has EBS storage (like our Windows server does) the data on this storage is maintained. If you have SSD storage (like our Unix server does) the data on this storage is NOT maintained. When an instance is shutdown you can restart the instance when you need it again. Things like intance ID, EBS storage, private DNS and private IPs are maintained and restored. Things like Public DNS and Public IPs may change (they will on our setup).

Note that when an instance is in a ‘Stopped’ state you are not charged for is use. However, any EBS storage that is maintained you will be charged for. If you don’t want to be charged then you will need to ‘Delete’ the volume OR Terminate the instance.

Terminate: If you terminate the instance everything is deleted. Terminate an instance if you no longer need it as you can NOT restart it or connect to it again. So only use this if you don’t need any of the data anymore. You can also use this option if you want to make absolutly sure you are not charged anymore. Any EBS storage you have when you terminate an instance is deleted too (so you won’t be charged for EBS storage after you’ve terminated either).

NOTE that during this course we…

– DO NOT want to terminate the Windows instance. We need to retain the data on this machine as it’ll be our master machine running Jenkins.

– We don’t really want to terminate our Unix instance either. However, on this machine no data will stored that we need to keep. So if you do terminate it that’s okay as we can just run up another instance with the same AMI.

– We recommend that you ‘STOP’ instances when you are not using them. This will mean that you are not charged for the instances. However, YOU WILL continue to be charged (if you go outside of your free tier allocation) for EBS storage that is retained.

For this reason we strongly advise that you monitor your AWS spend.

Part 11: How to Check Your AWS Spend

Within the AWS management console you’ll find a ‘Billing and Cost Management’ option:

AWS EC2 Cost Management

In here you’ll see what your current balance is (should remain at $0.00 if you stay within your free tier constraints). The most important section though is th e’Top Free Tier Services by Usage’ section.

AWS EC2 Cost Management Details

Monitor the stats here to see how close you are to going over your free tier allocation. If you’ve started with a clean AWS account for this course you shouldn’t go over your free tier allocation.


If you want to be absolutely sure that you don’t go over the free tier allocation then I would recommend reading this….



So we’ve started with a new AWS account. Then we went through the sign up process to create a new account. We’ve learnt about the AWS fundamentals covering Virtual Private Clouds, Elastic Cloud Computing and storage. From here we created our first Windows and Unix instances from Amazon Machine Images. To finish off we set up our servers so that we have RDP and SSH access. In short we started out with nothing and now we have our own cloud environment with virtual machines and storage.

In the next module we’ll be installing Jenkins and looking at the process of automating the build/install of our application under test. Lastly, if you’d like notifications about this course and the Pdf course notes please feel free to sign up below.

Free Test Automation Framework Course
Learn the Practical steps to building an Automation Framework. Six modules, six emails (each email a short course in it's own right) covering Amazon AWS, Jenkins, Selenium, SoapUI, Git and JMeter.

Building the Test Automation Framework

January 18th, 2016 by Bill Echlin

Welcome to Building the Test Automation Framework. A 6 part course that takes you step by step through the process of building a test automation framework. A test automation framework we’ll build completely from open source tools. Each of the 6 modules is a short course in it’s own right. Each module covering tools like; Amazon Web Services, Jenkins, Selenium, Git, SoapUI and JMeter.

When we’ve finished we’ll have a distributed test automation framework. A framework based on the principals of Continuous Integration and Devops. This is about building a framework at the system level that gives you the platform to automate much of what you need to do on a day to day basis.


To build this platform we have a series of 6 modules to get through. You’ll be learning about putting together the following Open Source and free components to build an automation rig:

Amazon Web Services: the platform we’ll use for building out our
windows and unix test automation environment.

Jenkins: the continuous integration tool we’ll use to build the
application under test and trigger all our automated test actions.

Selenium: the test automation tool we’ll use to run our GUI browser based automation tests.

SoapUI: the test automation tool we’ll use to run our API based automation tests.

Git: the source code tool we’ll be managing and retrieving the source for our application from.

JMeter: the load test tool we’ll use to asses the performance of the application under test.

At the end of all 6 modules YOU will have built this…

Test Automation Framework


You don’t need to know anything about any of these tools to follow these tutorials. Each of these 6 modules will provide the key details you need to understand how the tools work. The important point is that you’ll learn how to bring these tools together to create a working test automation framework .

All the machines will be hosted in the cloud with AWS. This makes it easy for you to replicate the system using exactly the same hardware and software we’ve used in these tutorials. Jenkins will act as the control tool. We’ll cover the build process for the application under test and focus on how Jenkins can control our machines in this test environment.

Selenium will be used to run a range of browser based tests on different platforms. Again the focus is not so much on how to use Selenium, more on how to pull Selenium into a fully featured test system. SoapUI will be trigger to run REST based API tests. JMeter kicked off to give us some feedback on application performance.

With Git we’ll be using an existing source code repository hosted on GitHub. GitHub contains the source for our application under test. To start out we’ll just pull down the source, build and then deploy the application. As things progress we’ll look at how to trigger builds, deploy and execute tests based on code check-ins.

What are we going to test?

We’ve chosen an open source web based chat application called Rocket.Chat. This is a cross platform, multi browser, iOS, Android and Windows chat application. A decent feature set enabling us to build out a full featured automation rig. More on the Rocket.Chat application here…

And that’s it. We’re starting out with nothing. By the end we’ll have an integrated test environment using many of the biggest open source test tools available. In the first module, up next, we’ll look at Amazon Web Services. By the end of the second tutorial you’ll have automated the Rocket.Chat install. Then with the third and forth modules you’ll have a range of GUI and API tests in place. In the fith module we’ll be triggering tests from source code check-ins. And, finally, in the last module we’ll be kicking off our performance tests with JMeter.

Hope you enjoy the journey.

Free Test Automation Framework Course
Learn the Practical steps to building an Automation Framework. Six modules, six emails (each email a short course in it's own right) covering Amazon AWS, Jenkins, Selenium, SoapUI, Git and JMeter.
1 2 3 17