Apache Camel 3.11 What's New

Apache Camel 3.11 has just been released.

This is a LTS release which will be supported for 1 year with regular patch and security releases.

This blog post first details the noteworthy changes since the last 3.10 release from last month.

For readers that are upgrading from the last 3.7 LTS release then we have added a summary section that highlights all the important new features and changes (3.7 to 3.11).

At first what did we do since the 3.10 release.

So what's in this release since 3.10

This release introduces a set of new features and noticeable improvements that we will cover in this blog post.

Kamelets

Kamelets is a higher level building blocks that we keep innovating and improve over the coming releases.

For Camel 3.11 we worked on making Kamelets universal across the various runtimes such as standalone, Karaf, Spring Boot, and Quarkus.

We added a new camel-kamelet-main component that is intended for developers to try out or develop custom Kamelets. This module runs standalone which is intentional as we want to ensure Kamelets are not tied to a specific runtime (or the cloud on Kubernetes) but are truly universal in any environment where you can use Camel.

You can find an example with camel-kamelet-main at https://github.com/apache/camel-examples/tree/main/examples/kamelet-main

The YAML DSL has improved error reporting when parsing to better report to Camel end users where the problem is.

Common Source timestamp

We added a `getSourceTimestamp` API on `Message` to get hold of the timestamp from the source of the message.

The idea is to have a common API across all the Camel components that has a timestamp of the event (such as JMS, Kafka, AWS, File/FTP etc).

Cloud component

The Camel AWS, Azure, and HuaweiCloud components have had various bug fixes and smaller improvements.

Quarkus

This release is the baseline for Quarkus 2 support which is to follow shortly after this release with a new Camel Quarkus release.

Spring Boot

We have upgraded to latest Spring Boot 2.5.1 release.

No OSGi code in main project

We had about six remaining Camel components which had some special OSGi Java source code.

The OSGi code has been ported over to the Camel Karaf project.

Better Java 16 support

Although Java 16 is not officially supported, we did improve a few Camel components to make them work with Java 16.

The official support is Java 11 (primary) and Java 8 (secondary).

New components

This release has a number of new components, data formats and languages:

• camel-huaweicloud-functiongraph - To call serverless functions on Huawei Cloud
• camel-huaweicloud-iam - To securely manage users on Huawei Cloud
• camel-kamelet-main - Main to run Kamelet standalone
• camel-resourceresolver-github - Resource resolver to load files from GitHub

Upgrading

Make sure to read the upgrade guide if you are upgrading from a previous Camel version.

Release Notes

You can find more information about this release in the release notes, with a list of JIRA tickets resolved in the release.

Summary of changes since the last 3.7 LTS release

It is 6 months since the last 3.7 LTS release, and here is a high level summary of the most significant changes we have done:

• Optimized core (faster startup and quicker routing engine)
• Modularized core (even smaller core)
• Reduced Object Allocations (lower memory footprint)  
• Reflection free (Native compilation friendly)
• Optimized toD EIP for messaging based components
• Better startup and shutdown logging
• Java Flight Recorder
• Routes loader (Java, XML, YAML, Groovy, JavaScript, and Kotlin)
• YAML DSL
• Kamelets
• 17 new components
• Support for Spring Boot 2.5 and Quarkus 2.0

There are many other great new features and improvements that you can find detailed in each of the Whats New blog posts:

• Camel 3.8 What's New
• Camel 3.9 What's New
• Camel 3.10 What's New

Webinar: Integrate systems in the age of Quarkus and Camel

A few days ago I presented a webinar where he covered all the latest innovations with Apache Camel with focus on Camel Quarkus, Camel K, and Kamelets. This trio is a powerful combination that takes Camel to another level, which allows non developers and IT professionals, to manage and bind systems together without any Camel knowledge.

Kamelets being the Apache Camel solution for an app store experience with integration software.

The webinar is online on youtube and the slides is here.

Apache Camel 3.9 - No more saw tooth JVM garbage collection

We continue our effort to optimize Apache Camel. This is blog post part 7 which covers are latest effort on dramatically reducing the object allocations caused by Camel while routing messages.

The good news is that we have overachieved and was able to reduce object allocations to ZERO!!! - so no more JVM memory usage graphs with saw tooth (note: in real world use-cases there will always be user data causing object allocations - but I wanted to have a click-bait blog title).

To help identify potential areas of improvement in the core Camel, we put together a small performance application, which has only a single route triggered by a timer every producing 1000 msg/sec. These messages are routed to 10 different log endpoints (logging turned off). This allows us to focus on the internals of Camel only and what code paths is executed and what objects are being allocated and in-use by the internal routing engine. There are no message data (body or headers), or network communication etc. 

Running the example (JVM heap size set to max 32mb) for 10 minutes profiled by JFR and browsed in JDK mission control we can see the dramatic difference. 

In Camel 3.8 597mb of objects is allocated by Camel in total.

And in Camel 3.9 that is ZERO.

How did we get to zero?

That is a long journey that started about a year ago, and we have gradually optimised Camel which I have blogged about in the 6 parts preceding this post.

All this work is like pealing an onion, layer after layer. As one layer has been optimised, then the profiler reveals another layer, and so on. This time we could identify 5 areas for improvements:

• consumers
• core EIP patterns
• internal routing processor
• error handler
• exchange and message

The consumers are the source of incoming messages into Apache Camel. And so that is a great place to start. It's the consumers that allocate a new exchange, populate the exchange with message data such as body and headers. 

After that it's the internal routing engine that routes the exchange via EIP patterns. And here we identified several spots where we could eliminate object allocations, or reduce allocations when some features are not in use etc. Error handling is one of the most complex part in the core Camel, and it uses objects to keep state in case of exceptions to handle redeliveries and whatnot. We were able to split the error handling into two tasks that operate either as a simplified or complex task. In the core EIP patterns we were able to optimize code that reduces object allocations.

The 5th area we optimized is the exchange object. EIPs and the Camel routing engine store state per exchange on the exchange instance itself as exchange properties. That data is stored in a Map which means for each entry both a key is allocated in the java.util.Map. We optimized this to use an internal object array where each key is hardcoded as an index entry in the array. That means read/write is very fast and simple as its just an array index. 

And then we ..... cheated ... instead of allocating new objects (via new constructor) we recycle existing objects from the previous exchange to the next. In other words we are using a sort of object pooling - this feature is called exchange pooling in Camel.

Exchange Pooling

The diagram above with ZERO object allocation is in fact with exchange pooling enabled. If exchange pooling is turned off (default), then the diagram should have been as below:

As you can see there is saw-tooth graph. However the total object allocation is gone down from 597mb to 492mb (18% reduction).

Awesome this is fantastic. And yes indeed it is. However when using anything there are both pros and cons, and so with object pooling. There is tiny tiny overhead of Camel to manage the object pools, and to "scrub" objects before they can be recused. That is a possibly a very very tiny CPU overhead compared to the JVM allocate and initialise new objects; instead of pool reuse. The biggest con is object leaks .. if objects are no returned back in the pool. Therefore you can turn on statistics which will report a WARN if a leak is detected when you stop Camel. The objects must be manually returned back into the pool, which we have coded in all the Camel components, and of course in core Camel. Now object leaks in this situation is not severe as you just have a situation as if there are no pooling, the JVM will create a new object - so the object allocations goes up, but its not severe like a database pool leaking TCP network connections.

Upcoming work

There are a few very complex EIP patterns and Camel component which does not yet support object pooling. We have this on the roadmap for Camel 3.10.

Camel 3.9 is planned for release in March 2021.

Apache Camel 3.8 and Java Flight Recorder

In the upcoming Apache Camel 3.8 release we have a new Camel component to integrate with Java Flight Recorder.

Camel is now capable of capturing "work steps" during startup that can be recorded with Java Flight Recorder. This can be used to better diagnose and find where your Camel applications may be slow to startup, for example due to a misbehaving component or custom user code.

The screenshot below shows a recording that has captured a Camel application that takes about 3 seconds to startup. Its a very tiny application so we expected it to be faster. 

If we sort the events by duration in the JDK mission control, we can see that there are 4 events that take over 2 seconds.

The sequence is a sequence of the following step (sub step):

Initializing context -> Initializing routes -> Creating route (route2) -> Creating Bean processor (bean1)

What we can see is that the step with the highest depth is "Creating Bean processor" which takes about 2 seconds. This is the culprit of the bottleneck.

If we check the Camel route for where bean1 is in use, its in route2 at:

        from("direct:slow")
            .to("log:slow?level=OFF")
            .bean(MyBean.class, "hello");

Here we can see the bean is using MyBean class, which we can then look at next:

    public MyBean() {

        // force slow startup

        try {

            LOG.warn("Forcing 2 sec delay to have slow startup");

            Thread.sleep(2000);

        } catch (Exception e) {

            // ignore

        }

    }

Ah okay here is the problem. The bean is sleeping for 2 seconds. Yes of course this is a made up example, but it does affect the recording and allow us to find it via the JDK mission control tool.

We also offer a logging recorder where you can "see" some of the same information as in JDK mission control. However when using JDK mission control, you have the entire JFR recording that also captures alot of JVM information about CPU and memory use and whatnot.

To use Java Flight Recorder with Camel, all you have to do is to add camel-jfr on the classpath. Then Camel will auto-detect this and enable it. You can configure the recorder with various options which will be documented as part of the common options.

But for quickly finding startup bottlenecks for Camel applications then the logging recorder is a good start. The screenshot below shows the logging output, and as you can see from the red square we have identified where the "2 second" problem is.

The logging recorder comes out of the box in camel-core, and you can just use it by configuring:

camel.main.startup-recorder = logging

If you are using Camel Main, Camel Quarkus etc. And for Spring Boot, you can enable it with

camel.springboot.startup-recorder = logging

You can also set a custom recorder, or one of the out of the box implementation via Java code:

camelContext.adapt(ExtendedCamelContext.class)
  .setStartupStepRecorder(...);

You can try this example (camel-example-flight-recorder) from the Camel Examples git repository. From command line you can run 

mvn camel:run

And Camel will automatic capture a JFR recording and save to disk. The output of the file is shown in the log, which you can then open from JDK mission control.