To begin with, there were a number of spec omissions in JASPIC 1.0 (Java EE 6). The biggest one was that in order to register a server authentication module (SAM) an application ID had to be provided. This ID could not be obtained in a portable way. The JASPIC 1.1 MR rectified this.
Other spec omissions concerned JASPIC being silent about what would need to happen with respect to HttpServletRequest#login and HttpServletRequest#logout, and with forward and includes done from a SAM. The JASPIC 1.1 MR rectified these omissions too.
With respect to the actual behaviour there were a large number of very serious problems. Most concerned the very basic stateless nature of JASPIC. A JASPIC SAM is like a Servlet Filter; it's called for every request to both public and protected resources, and doesn't automatically create a session when a caller is authenticated. What actually happened differed per server back then. Some only called the SAM for protected resources, some automatically created a session and never called the SAM again, etc.
Another class of problems concerned the life cycle. A SAM has two seemingly simple methods; "validateRequest" that has to be called before Filters and Servlets are invoked, and "secureResponse" that has to be called after. Especially this "after" was ill understood. Some servers called "validateRequest" and "secureResponse" both before the Filters right after each other, while others called "secureResponse" every time data was written to the response.
A specifically peculiar thing was that no server back then was able to wrap the request and response, even though the JASPIC spec clearly states that this is required. Accessing resources from a SAM, such as EJB beans or datasources via JNDI, or CDI beans via the bean manager was a hit or miss as well. Basically every server behaved differently there.
Finally there were big issues with interpreting how portable a SAM should exactly be, and whether the technology should "just be there", or whether some server specific configuration had to be done first. One vendor seemingly interpreted the JASPIC spec as a portable "authentication mechanism" (the artefact that interacts with the user, such as Servlet's FORM), that then delegated to a proprietary (server specific) "identity store" (the artefact that stores the user data and groups, such as LDAP or a database).
In response to this I created a series of tests, that were later donated to the Java EE 7 samples project. Subsequently I worked with all vendors and asked them to improve their JASPIC implementations. With the exception of Geronimo all vendors were very cooperative, so I'd like to take the opportunity here to give them all a big thanks for their hard work.
So after 3 years of creating tests and reporting issues, what's the current situation like? To find out I executed the JASPIC tests against the current crop of servers. The result is shown below:
As can be seen the situation has greatly improved. With the unfortunate exception of WebLogic 12.2.1 the basics now work everywhere. WebLogic 12.2.1 is perhaps a special case as it seems to be hit by a major bug where the most basic version of authentication doesn't work anymore, while it did work in the previous version 12.1.3. The fact that "testProtectedPageLoggedin" and "testPublicPageLoggedin" fail mean that actual authentication doesn't work properly. In this specific case it appears that when a caller authenticates with name "test" and gets the role "architect", then those are not available to the application. E.g. request#getUserPrincipal() still returns null and request#isUserInRole() returns false. This unfortunately means that for the moment until this bug is fixed JASPIC can not really be used on WebLogic at all.
Looking further at the results we see that the seemingly difficult to understand "secureResponse" method is now always called at the correct moment, and wrapping the request and response that once no server was able to do is now working well in all servers.
Forwards are now supported by all servers as are logouts. Includes are supported by most servers, only Liberty seems to have some issues with these. Curiously no server is able to include a resource that uses JSF. This is likely a JSF issue (as a JSF EG member and Mojarra committer this is something I probably have to fix myself ;))
Invoking resources has improved somewhat, but remains troublesome. Neither EJB beans nor CDI beans can be obtained and invoked on every server. EJB (specially those in the app scope such as java:comp, java:app, etc) work on JBoss EAP/WildFly, WebLogic and Liberty, but not on GlassFish and derivative Payara. CDI beans work in GlassFish, Payara and WildFly, but not in WebLogic and Liberty. WildFly is the one server where they both work.
The resources situation is still a spec issue as well and JASPIC 1.1 remains silent on whether this should work or not. The spec lead has clarified that even though the spec is silent on accessing EJB beans and other resources from the web component's JNDI namespaces, this is something that ought to work and GlassFish' current behaviour is just a bug. A next revision of the JASPIC spec should clarify this though. For the CDI beans no such clarification has been given, so vendors can't be asked to support this based on what the spec requires. However, accessing CDI from a SAM is very likely going to be a requirement coming from JSR 375 (Java EE security). So even though JASPIC doesn't mandate this now, it would be good if vendors already supported this in order to be prepared for Java EE 8.
Another case worth looking at is providing a custom principal from a SAM. This is a feature of JASPIC where a SAM can provide its own custom principal, e.g. org.example.MyPrincipal, which then has to be returned from request#getUserPrincipal(). This works on most servers except on GlassFish. It currently also doesn't work on WebLogic, but without further investigation it's hard to say whether it doesn't support this at all, or just because of the earlier failure of making the principal (custom or not) available.
Setting a response status code from a SAM (like e.g. a 404 - NOT FOUND) is something that is supported by all Servers, except for JBoss EAP/WildFly. This is currently the only unique failure for WildFly. Sort of, since it actually has already been fixed, but a build containing that fix has not yet been released.
From the outcome of the tests shown above it would seem JBoss EAP/WildFly clearly has the best JASPIC implementation, but there's one small but very important detail not shown in that table; the question whether JASPIC needs to be activated in a proprietary way. Unfortunately, JBoss EAP/WildFly indeeds needs such activation. If this activation would entail placing a special configuration file in the application archive it wouldn't be so bad, but JBoss EAP/WildFly actually requires the container to be modified before JASPIC can be used. This therefor means a SAM can not be deployed to a stock JBoss EAP/WildFly, which is very unfortunate indeed. There's a programmatic workaround available that doesn't require the container to be modified (see the activation link), but this is rather hacky and may break with every new release of JBoss EAP/WildFly.
The other server that needs server specific configuration is Liberty. Earlier versions of Liberty required all users and groups that a JASPIC SAM handles to be known by Liberty's proprietary user registry. An often downright impossible requirement in general and specifically for fully portable SAMs, and one that even violates the JASPIC spec. The current versions of Liberty have somewhat improved the situation by only requiring groups to be made known to Liberty. While still a very unfortunate requirement, it's at least possible to do this. Still, listing all the groups that an application uses in a proprietary file inside the container is a bit anti to one of the major use cases for which JASPIC is used; portable and application managed custom authentication. Instead of listing all the groups there's a workaround available where a NOOP user registry is installed and configured.
JBoss EAP 7/WildFly 10rc4 are almost perfect, if only JASPIC worked out of the box or could be activated from within the application archive using a configuration file. Payara 18.104.22.168 is another very good server for JASPIC. Here JASPIC works out of the box, but it suffers from a somewhat nasty bug that prevents it from using application scoped JNDI namespaces. GlassFish 4.1.1 is almost as good, but suffers from an extra bug that prevents it from using custom principals.
Liberty is quite good as well. It has slightly more bugs to fix than JBoss and Payara, but about the same as GlassFish. GlassFish can't use custom principals, Liberty can't do includes. Both can't obtain and invoke a specific bean type (for GlassFish this is EJB, for Liberty it's CDI). But above all Liberty suffers from its conflicting user registry requirement, although by far not as badly as before.
WebLogic 12.2.1 can at the moment not be recommended for JASPIC. It suffers from a severe bug that prohibits an application to use the authenticated identity, which is the core of what JASPIC does. Hopefully the WebLogic team is able to squash this particular bug soon.
All in all we've seen there's a steady and definite improvement going on for the various JASPIC implementations, but as can be seen there's still room left for improvement.