Working with large messages using Apache Camel and ActiveMQ Artemis improved in upcoming Camel 2.21 release

Historically the Apache ActiveMQ message broker was originally created in a time where large messages was measured in MB and not in GB as you may do today.

This is not the case with the next generation broker Apache ActiveMQ Artemis (or just Artemis) which has much better support for large messages.

So its about time that the Camel team finally had some time to work on this to ensure Camel work well with Artemis and large messages. This work was committed this weekend and we provided an example to demonstrate this.

The example runs Camel with the following two small routes:

The first route just route files to a queue on the message broker named data. The 2nd route does the opposite, routes from the data queue to file.

Pay attention to the 2nd route as it has turned on Camel's stream caching. This ensures that Camel will deal with large streaming payloads in a manner where Camel can automatic spool big streams to temporary disk space to avoid taking up memory. The stream caching in Apache Camel is fully configurable and you can setup thresholds that are based on payload size, memory left in the JVM etc to trigger when to spool to disk. However the default settings are often sufficient.

Camel then uses the JMS component to integrate with the ActiveMQ Artemis broker which you setup as follows:

This is all standard configuration (you should consider setting up a connection pool as well).

The example requires to run a ActiveMQ Artemis message broker separately in a JVM, and then start the Camel JVM with a lower memory setting such as 128mb or 256mb etc which can be done via Maven:

  export MAVEN_OPTS="-Xmx256m"

And then you run Camel via Maven

  mvn camel:run

When the application runs, you can then copy big files to the target/inbox directory, which should then stream these big messages to the Artemis broker, and then back again to Camel which will then save this to the target/outbox directory.

For example I tired this by copying a 1.6gb docker VM file, and Camel will log the following:
INFO  Sending file disk.vmdk to Artemis
INFO  Finish sending file to Artemis
INFO  Received data from Artemis
INFO  Finish saving data from Artemis as file

And we can see the file is saved again, and its also the correct size of 1.6gb

$ ls -lh target/outbox/
total 3417600
-rw-r--r--  1 davsclaus  staff   1.6G Oct 22 14:39 disk.vmdk

I attached jconsole to the running Camel JVM and monitored the memory usage which is shown in the graph:

The graph shows that the heap memory peaked at around 130mb and that after GC its back down to around 50mb. The JVM is configured with a max of 256mb.

You can find detailed step by step instructions with the example how exactly to run the example, so you can try for yourself. The example is part of the upcoming Apache Camel 2.21 release, where the camel-jms component has been improved for supporting javax.jms.StreamMessage types and has special optimisation for ActiveMQ Artemis as demonstrated by this example.

PS: The example could be written in numerous ways, but instead of creating yet another Spring Boot based example we chose to just use plain XML. In the end Camel does not care, you can implement and use Camel anyhow you like.

Apache Camel 2.20 released - What's new

Apache Camel 2.20 has been released today and as usual I am tasked to write a blog about this great new release and what's the highlights.

The release has the following highlights.

1) Java 9 technical preview support

We have started our work to support Java 9 and this release is what we call technical preview. The source code builds and runs on Java 9 and we will continue to improve work for official support in the following release.

2) Improved startup time

We have found a few spots to optimise the startup time of Apache Camel so it starts 100 - 200 milli seconds faster.

3) Optimised core to reduce footprint

Many internal optimisations in the Camel routing engine, such as reducing thread contention when updating JMX statistics, reducing internal state objects to claim less memory, and reducing the number of allocated objects to reduce overhead on GC etc, and much more.

4) Improved Spring Boot support and preparing for Spring Boot 2

We have improved Camel running on Spring Boot in various ways.

We also worked to make Apache Camel more ready and compatible with the upcoming Spring Boot 2 and Spring Framework 5. Officially support for these is expected in Camel 2.21 release.

5) Improved Spring lifecycle

Starting and stoping the CamelContext when used with Spring framework (SpringCamelContext) was revised to ensure that the Camel context is started last - when all resources should be available, and stopped first - while all resources are still available.

6) JMS 2.0 support

The camel-jms component now supports JMS 2.0 APIs.

7) Faster Map implementation for message headers

If you include camel-headersmap component on the classpath, then Camel will auto detect it on startup and use a faster implementation of case-insenstive map (used by camel message headers).

8) Health-Check API

We have added experimental support for a new health-check API  (which we will continue to work on over the next couple of releases).  The health checks can be leveraged in in cloud environments to detect non healthy contexts.

9) Cluster API

Introduced an experimental Cluster SPI (which we will continue to work on over the next couple of releases) for high availability contexts, out of the box Camel supports: atomix, consul, file, kubernetes and zookeeper as underlying clustering technologies through the respective components.

10) RouteController API

Introduced an experimental Route Controller SPI (which we will continue to work on over the next couple of releases) aimed to provide more fine-grained control of routes, out of the box Camel provides the following implementations:

• SupervisingRouteController which delays startup of the routes after the camel context is properly started and attempt to restart routes that have not been starter successfully.
• ClusteredRouteController which leverages Cluster SPI to start routes only when the context is elected as leader.

11) More components

As usual there is a bunch of new components for example we have support for calling AWS lambda functions in the camel-aws component. There is also a new json validator component, and camel-master is used with the new Cluster API to do route leader election in a cluster. There is 13 new components and 3 new data formats. You can find more details in the Camel 2.20 release notes.

We will now start working on the next release 2.21 which is scheduled in start of 2018. We are trying to push for a bit quicker release cycle of these bigger Camel releases, so we can go from doing 2 to 3 releases per year. This allows people to quicker pickup new functionality and components etc.

Also we want to get a release out that officially support Java 9, Spring Boot 2 and all the usual great stuff we add to each release, and what the community contributes.

Apache Camel route coverage tooling on the way

Last weekend I found some time to hack on new tooling for doing Apache Camel route coverage reports. The intention is to provide APIs and functionality out of the box from Apache Camel that other tooling vendors can leverage in their tooling. For example to show route coverage in IDEA or Eclipse tooling, or to generate SonarType reports, etc.

I got as far to build a prototype that is capable of generating a report which you run via the camel-maven-plugin. Having a prototype built in the came-maven-plugin is a very good idea as its neutral and basically just plain Java. And make its possible for other vendors to look at how its implemented in the camel-maven-plugin and be inspired how to use this functionality in their tooling.

I wanted to work on the hardest bit first which is being able to parse your Java routes and correlate to which EIPs was covered or not. We do have parts of such a parse based on the endpoint validation tooling which already exists in the camel-maven-plugin, which I have previously blogged about. The parser still needs a little bit more work however I do think I got it pretty far over just one weekend of work. I have not begun adding support for XML yet, but this should be much easier to do than Java, and I anticipate no problems there.

I recorded a video demonstrating the tooling in action.

I forgot to show in the video recording, that you can change the Camel routes, such as inserting empty lines, adding methods to the class etc, and when you re-run the camel-maven-plugin it will re-parse the route and output the line numbers correctly.

Anyway enjoy the video, its about 12 minutes long, so go grab a cup of coffee or tea first.

The plan is to include this tooling in the next Apache Camel 2.21 release which is scheduled in January 2018.

The JIRA ticket about the tooling is CAMEL-8657

Feedback is much welcome, and as always we love contributions at Apache Camel, so you are welcome to help out. The current work is upstream on this github branch. The code will be merged to master branch later when Apache Camel 2.20.0 is officially released.

When I get some more time in the future I would like to add support for route coverage in the Apache Camel IDEA plugin. Eclipse users may want to look at the JBoss Fuse tooling which has support for Apache Camel, which could potentially also add support for route coverage as well.