Testing JASPIC implementations using Arquillian and full wars
From time to time new servers come out, e.g. since the last article JBoss has released JBoss EAP 6.1, Oracle has released GlassFish 4 and WebLogic 12.1.2 and even Geronimo has released a new version of its server; Geronimo 3.0.1.
I previously tested via a series of handcrafted web applications that I would deploy manually, do some tests with it and then observe its behavior. This initially worked fine, but it's a bit laborious and perhaps even brittle to retest again some time later. It also makes it harder for others to verify the tests. So automation was the key, but how to automatically deploy several wars to a variety of application servers?
There are a couple of options for starting & stopping a Java EE application server, plus deploying and undeploying an archive (web, ear, ...) to it:
- JSR-88
- Arquillian containers
- Eclipse WTP server runtimes
- Creating an ANT script or task for each server
JSR-88 is the universal Java EE API/SPI to do exactly this in a universal way. Every Java EE server should fully support it because it's in the spec. Except not every server does this. In fact, only 2 servers really seem to have implemented it: Geronimo and WebLogic. Because of this and even though the feature itself seems hugely useful, it has been removed from Java EE 7. A shame perhaps (especially since there's no immediately replacement as there was with JAX-RPC and Entity Beans), but not much to do about it now.
Arquillian containers seem be able to do the trick as well. They are available for a multitude of servers, but not all. Although many containers have been available for a few years now, they all have alpha or at most CR versions. Some of them can only deploy to a server but not start or stop it. There are some newer versions available on GitHub for some containers, but they seem to sit there for months on end or even years without being officially released. Since Arquillian is 99% build for Maven (theoretically it isn't, practically yes it is) not having those containers in Maven Central or another repository is a bit troublesome. Interesting though is that the newer JBoss servers contain Arquillian connectors natively.
Eclipse WTP server runtimes are available for pretty much every server imaginable and they don't have all those alpha version numbers that the Arquillian Containers have. They all have the logic embedded to start & stop a server and deploy & undeploy an archive. Unfortunately Eclipse WTP server runtimes are very strongly tied to UI actions. There doesn't even seem to be a way to programmatically control them from within Eclipse.
Creating an ANT script per server that simply scripts the shell commands that pretty much each server provides was the last option I considered. Initially I thought of it as crude workaround, but I remembered that the Mojarra test harness uses exactly this method. Unfortunately Mojarra only has a very few servers already covered (basically GlassFish and WebLogic, and some level of coverage for Tomcat).
To actually create several web applications (war archives) for testing I had a couple of options as well. Using Arquillian's ShrinkWrap to create those archives programmatically from within a single project would have been an option, but I wanted those web applications to still be actual separate applications that I could deploy manually and standalone as well. A multi-module Maven project seemed to be quite suited for this; you can build them all from a master pom.xml, but also build them separately. In Eclipse each module is recognized as a separate web application and can be deployed independently from the other modules.
The project layout in Eclipse looks as follows:
Except for "common" and the Maven build folder "target", all sub-directories represent web applications. If needed they can be deployed individually by WTP as the following image shows:
As I was using Maven now, Arquillian seemed to be a logical choice. One issue though is that Arquillian predominantly focusses on unit testing so-called micro archives, which are archives that are assembled at the start of a unit test. In this case I just wanted to deploy the full war and not fuss around with programmatically adding each file. Another issue was that Arquillian unit tests run during the test phase of Maven, which means the code is compiled but the war archive isn't created yet.
As it appeared Maven can also run tests after the artifact has been assembled. For this the plugin named maven-failsafe-plugin had to be used instead of the well known maven-surefire-plugin one, and test classes by convention have to end with "IT" instead of "Test". Arquillian luckily also provided a solution; it can create the war to be deployed from a zip file on the filesystem as well. In other words, if Arquillian runs via failsafe, it can import the war archive that Maven just created. This happens via code such as:
@Deployment(testable = false)
public static WebArchive createDeployment() {
return ShrinkWrap.create(ZipImporter.class, "mywar.war")
.importFrom(new File("target/mywar-1.0.war"))
.as(WebArchive.class);
}
The following challenge was configuring the containers. The most important piece of configuration is the location of the actual application server. GlassFish embedded is an exception here; it's downloaded as a dependency of the Arquillian container (meaning it thus ends up in your local .m2 directory) and the full server runs within the same JVM as the test classes. It's completely zero-config. JBoss AS 7.2/EAP 6.1 can theoretically run embedded as well, but you still have to point the container to an already installed JBoss instance and unfortunately if you do that it still doesn't work.
Configuration is maybe not 100% intuitive. The issue is that an Arquillian container is not some kind of Maven plug-in orso, but it's just a dependency on the classpath following the highlander rule: There Can Be Only One. You can switch between them using Maven profiles. However, you can't directly configure a dependency in Maven; it's just a bunch of jars on the classpath after all. So you have to use a file called arquillian.xml that you also put on the classpath, somewhere... Then there's the issue that containers in arquillian.xml are given names via a qualifier attribute. Unfortunately a Maven profile for a given container doesn't automatically select the corresponding container in arquillian.xml; you have to make the connection yourself by setting an environment variable called arquillian.launch. Unfortunately again, a Maven profile cannot directly set an environment variable, but it can set a Maven variable. The failsafe plugin can set environment variables, so we can use it to do the conversion. This looks as follows:
First define a profile for a container in the parent pom.xml:
Second let the failsafe plugin convert the Maven property to a system/environment property:arquillian-jbossas72-managed arquillian-jbossas72-managed
Finally define a container in arquillian.xml where its qualifier attribute corresponds to the value of the system property we just set ("arquillian-jbossas72-managed"):maven-failsafe-plugin ${arquillian.launch}
/opt/jboss72
Granted, it's a bit convoluted and more than a little verbose but it does do the trick. However, it's not perfect yet. arquillian.xml does not sit neatly near pom.xml, but has to be put into some source folder. Since the parent project doesn't have any source, it has to be put into some common sub-module that's included with all other sub-modules. For such an important configuration file this is less than ideal. Files like this should be directly in the root of the project or at most in a /config sub-directory of the root, and not buried away 10 directories deep.
After some fiddling it appeared the failsafe plug-in can load an external .properties file and luckily the newer versions of Arquillian can process placeholders in arquillian.xml. One particular Maven problem I encountered is that the combined parent/sub poms run in the context of the sub-module. So a reference in the parent pom to the project directory (${project.basedir}) will 'magically' point to the project directory of the sub-module when this sub-module is being build and tested. There's a trick to work around this.
In the parent project's pom.xml define a variable to point to parent's basedir as follows:
And in the pom.xml of every sub-module define the following:${project.basedir}
Now we can put a .properties file next to the pom.xml, e.g. a containers.properties:${project.basedir}/..
# JBoss AS 7.2 and JBoss EAP 6.1 JBOSS72_HOME = /opt/jboss-eap-6.1 #GlassFish 4.0 GLASSFISH4_HOME = /opt/glassfish4 # ...We then let the failsafe plugin load this .properties file:
And finally we put placeholders in the arquillian.xml file shown before:maven-failsafe-plugin ${main.basedir}/containers.properties
Again it's a bit more verbose as it could have been but also once again it does do the trick.${JBOSS72_HOME}
To actually execute tests against a deployed web application there's a variety of choices as well. There's HttpUnit, Selenium and Arquillian Drone. HttpUnit allows you to do requests to a URL and then programmatically inspect the results. It's kind of an embedded virtual browser. Selenium is basically a layer on top of HttpUnit with extra options for using real browsers like Firefox and Chrome. Arquillian Drone is again a layer on top of Selenium, but with the extra option to inject the base URL of the deployed archive into the test class. Maybe there are more differences, but it didn't looked into them. I picked Arquillian Drone here since I was already using Arquillian and the injected URL was really convenient (saving us the trouble of configuring for each server on which HTTP port it listens by default).
Some words about the actual application servers and how they performed with Arquillian; as already mentioned GlassFish 3 embedded was the easiest. Nothing to install and zero-config. GlassFish 4 embedded unfortunately failed when running JSP pages, but the managed variant was easy to setup. GlassFish 4 is download & unzip, then point Arquillian to the unzip location. JBoss is a tiny bit more troublesome. JBoss AS 7.2 couldn't be downloaded (you can build it yourself if needed) and JBoss EAP 6.1 requires you to create an account and accept a restrictive license. Otherwise it too is download & unzip, then point Arquillian to the unzip location.
WebLogic 12.1.2 was a bit more trouble. It too required the creation of an account to download, but the unzip version required a little more effort than just an unzip. You have to set a bunch of environment variables, run configure and setup scripts, then create domains and set username and passwords before you can do the simplest thing like deploying a simple .war. Luckily Arun Gupta got it all covered, but it's a mystery to me why the WebLogic guys can't default all of this stuff. GlassFish and Liberty also have a concept of domains resp. servers, but they both DO default it. Worse, when you create this "domain" thing, apparently WebLogic hardcodes the JAVA_HOME that happened to be set when this domain was created. When you subsequently start up WebLogic using Eclipse and have a completely different JDK set for WebLogic, it keeps running on the other JDK. Took me some time to figure this out and its quite unintuitive.
The WebLogic Arquillian container was problematic too. It's a so-called "remote container" which means it can only do deploy & undeploy, but not start & stop. This has to be done manually. The behavior of either the Arquillian container or WebLogic (or the combination of those two) was erratic too. A bunch of deploys would succeed and then suddenly they would start failing. Finally, the configuration of the Arquillian Container was more involved too. Where the other containers just needed the path where the server was installed, the Arquillian WebLogic container needs no less than 6(!) properties, ranging from a URL, to a path, another path, a server (besides "domains", WebLogic also has "servers"), etc.
So what about the results of the actual JASPIC testing? I haven't added all tests yet, so I'll leave that for a next article. The intermediate results though are that GlassFish 4 passes all tests, WebLogic doesn't pass all tests, and JBoss EAP still has a lot of problems but does pass a few more tests without needing patches. I created a new path for JBoss EAP that makes it pass even more tests.
The tests themselves can be found at Gitbub.
Arjan Tijms
Comments
Post a Comment