WebLogic Remote EJB Calls Causing Security Exceptions

I stumbled upon a strange behavior (SecurityException-s thrown) in WebLogic 12c (and probably older releases) when calling remote EJBs related to the usage of InitialContext. This article summarizes my findings.

While processing a request, the WebLogic application server manages a stack of users. The last user pushed on the stack is the user that is currently active - calls to further EJBs will automatically run as this user. When creating a new InitialContext with a principal, WebLogic pushes it on the stack and the principal provided becomes the active user. It remains active until the InitialContext is closed (or the request processing finishes). This behavior paves the way for very unpleasant situations that I will try to explain based on the following test setup.

We have a JAX-WS web-service packaged in an EJB3 application invoking local and remote EJB2 beans after looking them up in the respective JNDI trees. This is shown in the sequence diagram below:

1. A consumer invokes the web-service. We have secured the web-service using WebLogic security configuration which requires HTTP basic authentication and checks the user against the users available in its realm. If the authentication succeeds, the FacadeJaxWs is executed as the authenticated user. In this example I will call the user ws_user.
2. FacadeJaxWs uses JNDI (with URL, principal backend_user and their credentials) to lookup the remote home interface of the BackendEjb EJB2 bean (hosted in a different domain), create the EJB stub and invoke a method on it.
3. The BackendEjb processes the request. Depending on what exactly this EJB does we might run into problems as described later.
5. FacadeJaxWs uses JNDI (without URL and credentials) to lookup the remote home interface of a BackendEjb EJB2 bean, collocated on the same managed server in a different EAR. As in the Backend case it creates a remote stub and invokes a method on it
6. FrontendEjb processes the request. Depending on what exactly this EJB does, we might run into further problems as described later

Depending on when and if at all the FacadeJaxWs closes the initial context when invoking the BackendEjb, a security exception can be thrown in different places.

Variant 1 - FacadeJaxWs does not close the InitialContext at all

This seems to be the usual thing a developer does. This is depicted in the code below:

With this code we have laid a mine - if the processInFrontend method does not call any further EJBs, the code will work without problems. If, however, the processInFrontend method tries to invoke another (local) EJB2 using the standard IntialContext mechanism, we will get an exception like this:

java.lang.SecurityException: [Security:090398]Invalid Subject: principals=[backend_user]

As you can imagine, this is extremely unpleasant, as it will not cause any problems (the mine will lay dormant) until another developer comes in and extends the processInFrontend method to call another EJB. Then the mine will go off and will catch the developer by surprise - something that they have done many times (calling an EJB from another EJB using InitialContext lookup) suddenly does not work in that specific situation. With the WebLogic JNDI knowledge from above, there is a good explanation for that - line 13 pushes the backend_user on the user stack and it becomes the active user. Any new EJB invokations will run as this user which is not really known in this domain. Note that using @EJB to inject the FrontendEjbHome somehow worsens the situation, as the JNDI lookup is done much earlier (with the correct user) and we do not run into the problem in all cases.

Variant 2 - FacadeJaxWs closes the InitialContext before invoking the actual remote EJB method

This variant is depicted by the code below:

In this variant we try to close the InitialContext right after looking up the remote home interface of the EJB - we might have an utility class to lookup, narrow and return remote home interfaces and close the initial context right after that. Unfortunately, this will not work and will cause another security exception like this:

java.lang.SecurityException: [Security:090398]Invalid Subject: principals=[ws_user]

This exception is actually thrown by the Backend managed server when we try to create the remote stub using the remote home interface on line 16. This is a much better situation than the previous variant, as we will always run into the exception (note: it does not seem to happen in EJB2 code). The explanation also comes easy - closing the context on line 15, pops the backend_user from the stack and the active user again becomes the ws_user. This user, however, is not known in the backend domain and thus the exception is thrown.

Variant 3 - FacadeJaxWs closes the InitialContext in a finally block after invoking the remote EJB method

Finally we get to the correct usage of InitialContext - we close it after invoking the actual business method like this:

This time we call the business method right after creating the initial context - this allows for the method to run in the context of the backend_user. Only after we have finished our interactions with the backend EJB, do we close the context (on line 21), switching back to the ws_user and executing the FrontendEjb as a user that is known in its WebLogic domain.

Performance Pitfalls: Inheriting JEE Singletons

In a previous blog post I wrote about the fact that JEE singleton EJBs have a very restrictive default lock mode - each method requires a WRITE lock which effectively allows for at most one method of the singleton to run at the same time.

Today I stumbled upon another gotcha: inherited methods are not affected by the default lock type set on the subclass. Consider the following code:

abstract class Parent {
  public void methodA() {}
}

@Singleton
@Lock(LockType.READ)
class Subclass extends Parent {
  public void methodB() {}
}

We have an abstract base class which implements some common functionality across multiple singletons (represented by the methodA()method). It is implemented as a POJO without any JEE annotations. Then we have the actual singleton which inherits from the base class and defines a default locking type READ. In our case, the developers were actually expecting that all methods, including those inherited from the Parent class, would have a READ lock type. Unfortunately, that was not the case. According to the EJB 3.1 specification, the methods inherited from the Parent class have a lock type as defined in the Parent class - if not specified then it falls back to the default WRITE lock type (page 113):

A concurrency locking attribute specified on a superclass S appplies to the business methods defined by S. If a class-level concurrency attribute is not specified on S, it is equivalent to specification of Lock(WRITE) on S.

In the example above we observed exactly that behavior. As a result, a thread running a method from the Parent class was effectively blocking any other thread from running any other method on the same singleton. In our case the situation was exacerbated by methods blocking on a slow external system.

The solution for us was easy as we did not really require any synchronization - switching to bean managed synchronization fixed the problem:

@Singleton
@ConcurrencyManagement(ConcurrencyManagementType.BEAN)
class Subclass extends Parent {
  public void methodB() {}
}

This time again we would have saved us some work if we switched to bean managed synchronization right away. I hope that this blog may help you work around similar issues in your applications.

Performance Pitfalls: JEE Singletons

JEE singletons are session beans which are guaranteed to exist at most once per application/JVM combination. They were introduced with EJB 3.1 as a replacement for the "normal" java singletons in a JEE context.

The java pattern for singletons relies on a private static field holding the single instance of the class. The static field is accessible through a static method. The field is either eagerly initialized in a static block or lazily initialized upon first invocation of the method. In order to guarantee that there is only one single instance, the method is usually synchronized. In high concurrency scenario you might use double checked locking and not class level synchronization.

Due to the fact that static variables are stored in the context of a class, such singletons are only guaranteed to be unique within the same class-loader. If you have a JEE application consisting of multiple modules containing the same singleton class, you are in trouble. Each module can (and probably will) have its own class-loader and thus its own static instance of the singleton resulting in multiple instances within the same JVM.

One very important aspect of JEE singletons is the fact that all of their methods are per default synchronized! If not otherwise configured, each singleton gets one read-write lock (check out this for details on read-write locks) from the container. Each method acquires the WRITE lock upon invocation. This effectively means that you can only have only one method of the class being executed at any point in time. If that is what you want, you are fine.

In order to achieve the same behaviour as with a "normal" java singleton (no synchronization of the singleton's methods) you will need to  explicitly switch to bean managed concurrency using the @ConcurrencyManagement(ConcurrencyManagementType.BEAN) annotation. If you need the read-write lock semantics, e.g. you have methods that change the state of the singleton and others that only read it, you can set the default lock mode of all methods to READ (check out section 4.8.5 of the specification) and only annotate the state-changing methods with the WRITE lock mode.

If you just migrate from java to JEE singletons without considering synchronization, you risk getting contention on the singleton's instance methods which would result in poor performance.

Performance Pitfalls: hibernate query plan cache thrashing

Hibernate internally uses a cache that maps HQL statements (as strings) to query plans. The cache consists of a bounded map limited by default to 2048 elements (configurable). All HQL queries are loaded through this cache. In case of a miss, the entry is automatically added to the cache. This makes it very susceptible to thrashing - a scenario in which we constantly put new entries into the cache without ever reusing them and thus preventing the cache from bringing any performance gains (it even adds some cache management overhead). To make things worse, it is hard to detect this situation by chance - you have to explicitly profile the cache in order to notice that you have a problem there. I will say a few words on how this could be done later on.

So the cache thrashing results from new queries being generated at high rates. This can be caused by a multitude of issues. The two most common that I have seen are - bugs in hibernate which cause parameters to be rendered in the JPQL statement instead of being passed as parameters and the use of an "in" - clause. 

Due to some obscure bugs in hibernate, there are situations when parameters are not handled correctly and are rendered into the JPQL query (as an example check out HHH-6280). If you have a query that is affected by such defects and it is executed at high rates, it will thrash your query plan cache because each JPQL query generated is almost unique (containing IDs of your entities for example).

The second issue lays in the way that hibernate processes queries with an "in" clause (e.g. give me all person entities whose company id field is one of 1, 2, 10, 18). For each distinct number of parameters in the "in"-clause, hibernate will produce a different query - e.g. select x from Person x where x.company.id in (:id0_) for 1 parameter, select x from Person x where x.company.id in (:id0_, :id1_) for 2 parameters and so on. All these queries are considered different, as far as the query plan cache is concerned, resulting again in cache thrashing. You could probably work around this issue by writing a utility class to produce only certain number of parameters - e.g. 1, 10, 100, 200, 500, 1000. If you, for example, pass 22 parameters, it will return a list of 100 elements with the 22 parameters included in it and the remaining 78 parameters set to an impossible value (e.g. -1 for IDs used for foreign keys). I agree that this is an ugly hack but could get the job done. As a result you will only have at most 6 unique queries in your cache and thus reduce thrashing.

So how do you find out that you have the issue? You could write some additional code and expose metrics with the number of entries in the cache e.g. over JMX, tune logging and analyze the logs, etc. If you do not want to (or can not) modify the application, you could just dump the heap and run this OQL query against it (e.g. using mat):
SELECT l.query.toString() FROM INSTANCEOF org.hibernate.engine.query.spi.QueryPlanCache$HQLQueryPlanKey l It will output all queries currently located in any query plan cache on your heap. It should be pretty easy to spot whether you are affected by any of the aforementioned problems.

As far as the performance impact goes, it is hard to say as it depends on too many factors. I have seen a very trivial query causing 10-20 ms of overhead spent in creating a new HQL query plan. In general, if there is a cache somewhere, there must be a good reason for that - a miss is probably expensive so your should try to avoid misses as much as possible. Last but not least, your database will have to handle large amounts of unique SQL statements too - causing it to parse them and maybe create different execution plans for every one of them.

Performance Pitfalls - Hibernate Criteria API

The criteria API is a way to programmatically create dynamic JPA queries in a type safe manner. It is supposed to be used when the JPA query cannot be statically defined up front because the actual query depends on input provided by the user - e.g. a search form with many optional filter conditions that can be flexibly combined in any possible way.

In hibernate the criteria API is used to create a graph of hibernate objects (implementations of the org.hibernate.Criteria interface and related objects) representing the required query. This graph of objects is then internally transformed to a JPQL query (basically a string). This string is then parsed and mapped to a SQL query. It is interesting to notice, that hibernate does not map the criteria object graph directly to SQL but rather goes through an intermediate JPQL query string. This process is repeated every time a criteria API query is executed.

Obviously, this process requires a lot of CPU cycles each time a query is executed. In cases when such queries are run very often this might already be a prohibitive factor. Unfortunately, it gets even worse.

The even more concerning issue is part of the process of transforming JPQL queries to SQL. Hibernate tries to load some of the tokens in the JPQL query as classes to determine whether they are actually classes. This results in hibernate calling ClassLoader.loadClass() with parameters like "a0", "b1" etc.

Class-loaders are organized in hierarchies. In an enterprise application a WAR class-loader would have a parent EAR class-loader with a parent the application server class-loader and so on. The default behaviour of each class-loader, when looking for a class, is to first try to load the class from the parent class-loader. If this fails, it will try to load the class itself. As a result of this strategy, trying to load a non-existent class is pretty much the worst-case scenario for a class-loader - it has to look for the class in the root class-loader and in each class-loader down the hierarchy to finally determine that the class is not known and throw a ClassNotFoundException. This has two issues. First of all, normally class-loaders do not cache such cases. When a class is found, some class-loaders might remember that (cache it) so that subsequent calls are much faster. Missing classes are usually not cached. This means that looking for a non-existent class (even the same one) over and over again always hits this worst-case scenario causing each class-loader to go through all of its classes. The second issue lays in the way the some class-loaders (e.g. the WebLogic application server class-loader) which assume that loading classes does not happen that often and use a very coarse-grained locking - loading a class basically locks the class-loader through the whole search process.

Lets get back to our criteria API query. One enterprise application obviously has one single class-loader instance (and the same class-loaders on the way up in the hierarchy). Each criteria API query tries to get the lock on the same class-loader while looking for JPQL tokens. This results in all criteria API queries contending for the same locks - a high lock contention in the class-loader and sync times are what follows.

In this blog post we saw that one needs to be willing to pay a very high price to use use criteria API with hibernate. Depending on the performance requirements it might be okay for some applications. For others, however, that is absolutely prohibitive and any high-load process must not use criteria API. My advice as a performance engineer is to avoid criteria APIs as much as possible. In many cases it is possible to use a few named queries (which, by the way, go through the same JPQL to SQL process but only once upon start-up) or even revert to native SQL or other technologies.