Misleading Cancellation of a Future in @Asynchronous

Starting with JEE 6 and EJB 3.1 specification, Oracle introduced an @Asynchronous annotation. JEE 7 has not brought any changes in this matter, and in both these versions, an @Asynchronousmethod might return a Future object. In this blog post we will have a look at how misleading its cancellation is, and what the possible solutions to address the problems that arise are.

Introduction

As an example, we will be generating UUIDs. Let’s say, we would like to have a background thread that is constantly generating new UUIDs (up to some limit), so that whenever a client request arrives, a batch thereof can be returned instantaneously. We start the generation at the moment of application deployment and control the maximal amount of identifiers to be generated by using a BlockingQueue (which is also responsible for a thread-safe data distribution among multiple threads). When the limit of the queue is reached, a call to the blockingQueue.put(...) method blocks until some client fetches another batch of values. And this particular example is not that important. What is essential here is that this is a long-running operation which includes a blocking call. And since this is a long-running operation, we will have a problem when trying to undeploy the application, as Java will not kill the process on our behalf. We need to do it by ourselves. This point alone exposes the first problem (we will get there in a second), but even stopping such blocking @Asynchronous method on demand might be challenging.

Getting into details

The first problem is that when @Asynchronous method is running, application server is not executing  @PreDestroy enterprise bean lifecycle method. Therefore, the first attempt to cancel the thread from within the same bean that started the thread fell flat. You will find a full example here.

@Singleton
(...)
@Startup
public class UnstoppablePreDestroyGenerator {
    (...)
    private Future<Void> future;

    @Resource
    SessionContext sessionContext;

    @PostConstruct
    public void startGeneration() {
        future = sessionContext.getBusinessObject(UnstoppablePreDestroyGenerator.class).generate();
    }

    @Asynchronous
    public Future<Void> generate() {
        (...)
    }

    @PreDestroy
    public void failingStop() {
        LOG.info("!!! SURPRISE: you will never get here");
    }
}

As a remedy to this problem, we can extract the lifecycle logic to a separate class. Now, the @PreDestroy is executed successfully, and this brings us to the essence of this blog post, which is a misleading future.cancel(true) call. This call does not stop the thread. At least not in case of the @Asynchronous method presented in this generator.

@Singleton
@Startup
public class UnstoppableCancelGeneratorLifecycle {
    private static final Logger LOG = (...)
    private Future<Void> future;

    @Inject
    UnstoppableCancelGenerator generator;

    @PostConstruct
    public void startGeneration() {
        future = generator.generate();
    }

    @PreDestroy
    public void failingStop() {
        boolean cancellationResult = future.cancel(true);
        LOG.info("!!! SURPRISE: future was not cancelled; cancellationResult=" + cancellationResult);
    }
}

@Singleton
(...)
public class UnstoppableCancelGenerator {
    (...)
    private final BlockingQueue<String> ids = new LinkedBlockingQueue<>(BATCH_SIZE);

    @Asynchronous
    public Future<Void> generate() {
        try {
            while (!currentThread().isInterrupted()) {
                ids.put(UUID.randomUUID().toString());
            }
        } catch (InterruptedException ex) {
            // ignore
        }
        return new AsyncResult<>(null);
    }

    public Collection<String> getNextBatch() throws InterruptedException {
        (...)
    }
}

Solution #1: ManagedExecutorService

This is a go-to solution number 1, a ManagedExecutorService, introduced in JEE 7. A future.cancel(true) call works in this case as expected. The background thread is successfully stopped when our application is undeployed. It could be also stopped as easily on demand. What is more, apart from making the future.cancel(true) call working, this solution does not require extracting a separate lifecycle class. You will find a full example here.

@Singleton
(...)
@Startup
public class StoppableExecutorGenerator {
    (...)
    private final BlockingQueue<String> ids = new LinkedBlockingQueue<>(BATCH_SIZE);
    private Future<?> future;

    @Resource
    ManagedExecutorService managedExecutorService;

    @PostConstruct
    public void startGeneration() {
        future = managedExecutorService.submit(() -> generate());
    }

    private void generate() {
        try {
            while (!currentThread().isInterrupted()) {
                ids.put(UUID.randomUUID().toString());
            }
        } catch (InterruptedException ex) {
            // ignore
        }
    }

    public Collection<String> getNextBatch() throws InterruptedException {
        (...)
    }

    @PreDestroy
    public void stopGeneration() {
        future.cancel(true);
    }
}

But we are giving up on the whole @Asynchronous approach, abandoning it altogether. In case of @Asynchronous method call, a container starts a new transaction for us, by default. In our case (in all @Asynchronous-based examples) this was turned off on purpose. But your logic might be different, so this is definitely something to keep in mind. In our case, the blocking call might be blocking for a really long time, exceeding the default transaction timeout. Since we do not want to exceed this timeout, all @Asynchronous-based generators were marked with @TransactionManagement(TransactionManagementType.BEAN) annotation. But when relying on ManagedExecutorService we have no choice. We need to always manage transactions manually using a UserTransaction (like we have to when using BEAN transaction management for regular enterprise beans).

So let’s have a look at what options we have provided that we want to keep this @Asynchronous approach.

Solution #2: Tampering with thread logic

We stick to the @Asynchronous method, but the stopping code we write requires deep knowledge of the actual algorithm. Therefore this solution is a much more fragile one. A change in the algorithm might require a change in the stopping part. Note that in our case, the stopping part calls ids.poll(), instead of  ids.take(). The polling method is not blocking, and in case the next value is not available immediately, it just returns false. And this former choice is exactly what we need. In case of the latter, we could run again into the unstoppable case problem again.

Still, the UUID generation example is trivial, but this approach might not scale well for more complex scenarios. Plus, we still need the lifecycle class. And, we keep the bean-managed transaction demarcation. You will find a full example here.

@Singleton
@Startup
public class TamperingWithLogicGeneratorLifecycle {
    private final GeneratorGuts generatorGuts = new GeneratorGuts();

    @Inject
    TamperingWithLogicGenerator generator;

    @PostConstruct
    public void startGeneration() {
        generator.generate(generatorGuts);
    }

    @PreDestroy
    public void stopGeneration() {
        generatorGuts.stopGeneration();
    }
}

class GeneratorGuts {
    (...)
    private final BlockingQueue<String> ids = new LinkedBlockingQueue<>(BATCH_SIZE);
    private volatile boolean continueGeneration = true;

    BlockingQueue<String> getIds() {
        return ids;
    }

    boolean isContinueGeneration() {
        return continueGeneration;
    }

    void stopGeneration() {
        continueGeneration = false;
        ids.poll();
    }
}

@Singleton
(...)
public class TamperingWithLogicGenerator {
    private GeneratorGuts generatorGuts;

    @Asynchronous
    public void generate(GeneratorGuts generatorGuts) {
        this.generatorGuts = generatorGuts;
        try {
            while (generatorGuts.isContinueGeneration()) {
                generatorGuts.getIds().put(UUID.randomUUID().toString());
            }
        } catch (InterruptedException ex) {
            // ignore
        }
    }

    public Collection<String> getNextBatch() throws InterruptedException {
        (...)
    }
}

Solution #3: Non-blocking/semi-blocking code

We stick to the @Asynchronous method, and also future.cancel(true) call works fine. What we had to sacrifice however are some CPU cycles, as the generation is not a fully blocking solution now. Even worse, what we also introduced is some accidental complexity. The additional code (handling timeout in the offer(...) method) is here not because it is essential to the algorithm in question, but because we are trying to solve a technical problem of stopping a thread. Unfortunately, this is also the solution apparently favoured by Oracle, as according to the official Oracle documentation, all that future.cancel(true) does, is setting the wasCancelCalled flag to true. And yes, lifecycle class, it is here again, as well as a bean-managed transaction scope. You will find a full example here.

@Singleton
@Startup
public class NonblockingGeneratorLifecycle {
    private Future<Void> future;

    @Inject
    NonblockingGenerator generator;

    @PostConstruct
    public void startGeneration() {
        future = generator.generate();
    }

    @PreDestroy
    public void stopGeneration() {
        future.cancel(true);
    }
}

@Singleton
(...)
public class NonblockingGenerator {
    (...)
    private final BlockingQueue<String> ids = new LinkedBlockingQueue<>(BATCH_SIZE);

    @Resource
    SessionContext sessionContext;

    @Asynchronous
    public Future<Void> generate() {
        try {
            while (!sessionContext.wasCancelCalled()) {
                ids.offer(UUID.randomUUID().toString(), 500, MILLISECONDS);
            }
        } catch (InterruptedException ex) {
            // ignore
        }
        return new AsyncResult<>(null);
    }

    public Collection<String> getNextBatch() throws InterruptedException {
        (...)
    }
}

Solution #4: Interrupting thread at all costs (do not do this!)

This one is really mostly for fun. It requires stealing the thread once @Asynchronous method starts running, and then calling an interrupt() method directly on this stolen thread. This means, that @Asynchronous method remains, but there is no point in returning a Future object, as we will not need it to stop the thread. The hackish nature of this workaround disqualifies the solution as a production one. But it works in our example. You will find it here.

@Singleton
@Startup
public class InterruptingThreadGeneratorLifecycle {
    private final ThreadHolder threadHolder = new ThreadHolder();

    @Inject
    InterruptingThreadGenerator generator;

    @PostConstruct
    public void startGeneration() {
        generator.generate(threadHolder);
    }

    @PreDestroy
    public void stopGeneration() {
        threadHolder.interrupt();
    }
}

class ThreadHolder {
    private volatile Thread thread;

    void keep(Thread thread) {
        this.thread = thread;
    }

    void interrupt() {
        thread.interrupt();
    }
}

@Singleton
(...)
public class InterruptingThreadGenerator {
    (...)
    private final BlockingQueue<String> ids = new LinkedBlockingQueue<>(BATCH_SIZE);

    @Asynchronous
    public void generate(ThreadHolder threadHolder) {
        threadHolder.keep(Thread.currentThread());
        try {
            while (!currentThread().isInterrupted()) {
                ids.put(UUID.randomUUID().toString());
            }
        } catch (InterruptedException ex) {
            // ignore
        }
    }

    public Collection<String> getNextBatch() throws InterruptedException {
        (...)
    }
}

GitHub examples

You will find a full, executable source code to the post in this github project. All the examples were tested on WildFly 10.1.0. You will find there the two unstoppable cases for:

• failing @PreDestroy and
• failing future.cancel(true),

and four stoppable ones for:

• ManagedExecutorService solution,
• a tampering with logic solution,
• non-blocking/semi-blocking solution and
• interrupting thread at all costs kind-of-solution.

For both unstoppable generators, the @Startup annotation was commented out. Otherwise the whole project would not be undeployable by default. For each case, there is a boundary and a control package, following the BCE pattern. Boundary packages expose REST services that allow testing generators with GET calls. Control packages contain generators and additional classes when necessary. Below, there are the endpoints you can access to see results of running generators (just remember to uncomment the two @Startup annotations!):

• http://localhost:8080/misleading-asynchronous-future/resources/unstoppablePreDestroy
• http://localhost:8080/misleading-asynchronous-future/resources/unstoppableCancel
• http://localhost:8080/misleading-asynchronous-future/resources/executor
• http://localhost:8080/misleading-asynchronous-future/resources/tamperingWithLogic
• http://localhost:8080/misleading-asynchronous-future/resources/nonblocking
• http://localhost:8080/misleading-asynchronous-future/resources/interruptingThread

The project can be compiled with mvn clean install command and deployed on a WildFly 10.1.0 application server. What I used while playing with the examples was a Docker environment scripted here, running on Windows 10 Pro, Oracle VirtualBox and Docker Toolbox. To start the environment, run:

1. .\windowsDockerMachineUp.ps1 script to create a dedicated Docker machine in Oracle VirtualBox (in a PowerShell window with Administrator privileges), and
2. jjs .\up.js to create docker images and containers, and to compile and deploy the project.

The benefit of having a Docker environment is ease of recreating a fresh WildFly instance in case a .war package cannot be undeployed:

1. docker rm -f wildfly-configured,
2. jjs .\up.js.

The whole Docker scripts setup is based on my other GitHub Links project. For details of that Docker environment setup have a look here. Both these setups are heavily influenced by the Docklands project, by Adam Bien.

Conclusion

@Asynchronous annotation brought official threading support into JEE in version 6. Together with a Future object returned from the method, we could also fetch the async result once it was ready. But really, all this feature was meant to do was to be a replacement for a misused JMS/MDBs, a simple solution for some ad hoc, short-running tasks. We still had no control over the thread pool used for such asynchronous processing and also stopping such background activity was cumbersome.

For more sophisticated scenarios and especially in case blocking calls come into play, using ManagedExecutorService from JEE 7 offers much more flexibility. Not only does it allow to conveniently interrupt the thread, but (what is even more important) it also gives full control over thread pools used for processing. With this feature, we are able to assign each ManagedExecutorService to a separate thread pool, introducing bulkheads to isolate different concerns and protect from over-saturating the server with just one type of activity. We will see this feature in action in the next blog post, in which we will be playing a little with JEE performance monitoring.