Best Practices for Middleware and Integration Architecture Modernization with Apache Camel

Yesterday I gave the following virtual talk at the Stockholm Meetup.

Best Practices for Middleware and Integration Architecture Modernization with Apache Camel
What are important considerations when modernizing middleware and moving towards serverless and/or cloud native integration architectures? How can we make the most of flexible technologies such as Camel K, Kafka, Quarkus and OpenShift. Claus is working as project lead on Apache Camel and has extensive experience from open source product development.
I thank the organizers Forefront Consulting for inviting me. Unfortunately there was a glitch with the talk yesterday. As I could not be in person then the talk was pre-recorded and was cut half way. So I promised to post a blog today and upload the talk to youtube.

The title and abstract of the talk was somewhat given to me, and so was the session length of 30 minutes. As I am so heavily invested in Apache Camel, then I focused the talk about Camel and its evolution over the last 10 years as introduction and then using the latest innovations from Camel K, Camel Quarkus and Camel Kafka Connectors as the meat of the talk, and with 3 demos.

The talk can be watched on youtube and the slides are here.


Apache Camel 3.1 - Fast loading of XML routes

A feature that was added to Camel 3.1 is the ability to load XML routes much faster. This is part of the overall work we are doing on making Camel much smaller and faster.

You may say ewww XML. But frankly there are many users of Camel that have built applications with XML for defining routes. In Camel 2.x then you would have to use Spring or OSGi Blueprint for XML routes which both are becoming heavy in modern cloud native world.

In Camel 3 we have a standalone mode for Camel called camel-main. We use camel-main as a common way to bootstrap and configure Camel for standalone, camel-k, camel-quarkus, and for most parts of camel-spring-boot as well. This ensures an unified and consistent developer experience across those runtimes.

Okay this is probably a topic for another blog post to dive into camel-main as a great runtime for quickly running ... just Camel.

So what I wanted to say in this blog post is that we have made it possible to loading XML routes much quicker and with a lot less overhead. In Camel 2.x, and for Spring XML and Blueprint XML they rely on JAXP and JAXB which ... are heavy.

So what we have done for Camel 3.1 is to source code generate a XML parser based on the Camel DSL. This means anything we do changes to the DSL then the parser is re-generated. The parser just uses standard Java so there are no additional 3rd party library dependencies.

For loading XML routes in Camel we now have 2 parsers in the following JARs

- camel-xml-jaxb   (traditional JAXB based as in Camel 2.x)
- camel-xml-io       (new fast and lightweight source code generated parsers)

The example camel-example-main-xml is setup to use the new parser. But you can try for yourself and switch to the jaxb parser by changing the JAR dependency.

Lets see some numbers (note this is just a quick test on my laptop to run this example with the 2 XML parsers).

camel-xml-jaxb: Loaded 1 (808 millis) additional Camel XML routes from: routes/*.xml
camel-xml-io: Loaded 1 (76 millis) additional Camel XML routes from: routes/*.xml

So the new parser is about 10 times faster (76 vs 808 millis).

By profiling the JVM we can see that there is a lot less classes loaded as well: 4734 vs 3892. And on top of that JAXB leaves more objects and classes around in the JVM that may or may not easily be garbage collected, and would also be using more cpu and memory during its parsing.

And then on GraalVM then the new parser would be much quicker as you can avoid having the entire JAXB and JAXP API and implementation on the classpath and for the GraalVM compiler to crunch and compile. And speaking of GraalVM then we are working on some great improvements in the upcoming Camel 3.2 that should help reduce the image size and compilation, and allow to do more dead code elimination and whatnot to make Camel even more awesome. That's yet another topic for another blog post, so stay tuned.

Apache Camel 3.2 - Reflection free configuration of Camel

At the Apache Camel project we are working towards the next upcoming Apache Camel 3.2.0 release, which is planned for next month.

One of the ares we have worked hard on in Camel 3 is to make it smaller and faster. And one aspect of this is also configuration management. You can fully configure Camel in many ways and according to the 12 factor principles, to keep configuration separated from the application. A popular way to configure is to use properties files (eg application.properties) or in Kubernetes you can configure from config maps or environment variables as well.

So we have gradually over Camel 3.0, 3.1 and now 3.2 made configuration faster. With the latest work we are now fully reflection free.

Camel is capable of reporting when reflection based configuration are being used. Which can be configured with:

# bean introspection to log reflection based configuration

We have prepared the camel-example-main-tiny to report this. The numbers for Camel 3.0, 3.1, and 3.2 are as follows:

Camel 3.0: BeanIntrospection invoked: 12 times
Camel 3.1: Stopping BeanIntrospection which was invoked: 11 times
Camel 3.2: Stopping BeanIntrospection which was invoked: 0 times

What this means is that you can fully configure all your Camel endpoints, components, routes, EIPs, data formats, languages, camel main, camel context, and whatnot, in declarative properties files etc and then at runtime all of this ends up invoking the actual setter methods on all these instances (ie just direct java method calls, no java.lang.reflect).

This is possible because we source code generate configurer classes based on what options are present. And these configurer classes are reflection free. There can be many options so it would be impossible to implement this by hand, see for example the kafka endpoint configurer.

And btw another feature coming in Camel 3.2 is that we made all of the components options available for configuration, before we didn't include nested configuration options. And if you do not like configuring in properties files, then we have type-safe component-dsl and endpoint-dsl as well.


Upcoming Webinar - What's new in Apache Camel 3

On March 3rd, at 3pm CET (Central European Timezone) I will co-host, together with Andrea Cosentino, a webinar session for 1 hour, where we cover all the great new features that are in the Apache Camel v3 release.

Andrea and I will cover in more details the high level goals of Apache Camel 3, and focus on the key elements about making Camel smaller, lighter, and faster for the cloud native world. So you will find details about what we have done internally to make this happen.

We also cover and introduce Camel K, Camel Quarkus and Camel Kafka Connector where we have 4 live demos ready for you. And finally we present the roadmap for the upcoming releases.

At the end we have Q&A session where we will assist and answer as many questions you may have.

The webinar is free to attend, but requires registration, as the webinar is run by professional media company.

More details and registration here.

PS: Yes we will cover details up till the latest Camel 3.1 release which is going to be released today ;)


Apache Camel 3.1 - More camel-core optimizations coming (Part 3)

I have previously blogged about the optimizations we are doing in the next Camel 3.1 release

Today I wanted to give a short update on the latest development we have done, as we are closing down on being ready to build and release Camel 3.1 as early as end of this week or the following.

Since part 2, we managed to find additional 10% reduction on object allocations during routing.

We have also continued the effort of configuring Camel via source code generated configurers that performs direct Java method calls vs using java bean reflections. Now all components, data formats, languages, and EIP patterns is complete. Only in more advanced use-cases where configuration is based on nested complex objects that are dynamically configured would be outside the scope of the source code configures and Camel fallback to use reflection.

We also found a way to optimize property placeholder resolution on EIPs to avoid using source code generated configurers which means that there are 200 classes less to load on the classpath, and about 90kb of memory is saved. This is great as these classes and memory were only used during bootstrap of Camel, and now they are all gone.

We also managed to further modulaize camel-core, so JAXB and XML routes are optional.
Even for XML routes (not Spring or Blueprint as they have their own DOM XML parser) we have created an alternative, fast and light-weight pull based parser. The camel-example-main-xml is using this and by comparing JAXB vs Camel XML then its 6x faster (approx 1500 millis vs 250) and loads 700 classes less than JAXB.

However for non XML users (eg using Java DSL) then JAXB can be avoided on the classpath at all, and you can have tiny Camel applications, such as camel-example-main-tiny with the following dependency tree (bold are Camel JARs; the example uses the bean and timer components)

[INFO] org.apache.camel.example:camel-example-main-tiny:jar:3.1.0-SNAPSHOT
[INFO] +- org.apache.camel:camel-main:jar:3.1.0-SNAPSHOT:compile
[INFO] |  +- org.apache.camel:camel-api:jar:3.1.0-SNAPSHOT:compile
[INFO] |  +- org.apache.camel:camel-base:jar:3.1.0-SNAPSHOT:compile
[INFO] |  +- org.apache.camel:camel-core-engine:jar:3.1.0-SNAPSHOT:compile
[INFO] |  +- org.apache.camel:camel-management-api:jar:3.1.0-SNAPSHOT:compile
[INFO] |  +- org.apache.camel:camel-support:jar:3.1.0-SNAPSHOT:compile
[INFO] |  \- org.apache.camel:camel-util:jar:3.1.0-SNAPSHOT:compile
[INFO] +- org.apache.camel:camel-bean:jar:3.1.0-SNAPSHOT:compile
[INFO] +- org.apache.camel:camel-timer:jar:3.1.0-SNAPSHOT:compile
[INFO] +- org.apache.logging.log4j:log4j-api:jar:2.13.0:compile
[INFO] +- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] \- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.30:compile

I ran this example with the profiler and configured it to use 10MB as max heap (-Xmx10M) and as the summary shows this can easily be done. About 5mb is used in the heap.

There has also been a few other minor improvements to avoid using Camel 2.x based type converter scanning by default. This reduces a scan on the classpath.

Okay its time to end this blog series and finish up the last bits so we can get Camel 3.1 released.


Apache Camel 3.1 - More camel-core optimizations coming (Part 2)

I have previously blogged about the optimizations we are doing in the next Camel 3.1 release (part 1).

Today I wanted to post a status update on the progress we have made since, about 4 weeks later.

We have focused on optimizing camel-core in three areas:

  • unnecessary object allocations
  • unnecessary method calls
  • improve performance
In other words we are making Camel create less objects, calling fewer methods, and improving the performance during routing.

To help identify these issues in camel-core we were using a simple Camel route:

from timer:foo
  to log:foo

And other times we focused on longer routes:

from timer:foo
  to log:foo1
  to log:foo2
  to log:foo3
  to log:fooN

Or the focus on the bean component:

from timer:foo
  to bean:foo

And so on. We also added an option to the timer component to not include metadata so the message dont contain any body, headers or exchange properties. This allowed us to focus on the pure routing engine and its overhead.

So all together this has helped identify many smaller points for improvements that collectively gains a great win.

tl:dr - Show me the numbers

Okay let's post some numbers first and then follow up with details what has been done.

Object Allocations - (5 minute sampling)
Camel 2.25     2.9 M objects created
Camel 3.0       55 M objects created
Camel 3.1      1.8 M objects created

Okay we have to admit that Camel 3.0 has an issue with excessive object allocations during routing. There are no memory leaks but it creates a lot of unnecessary objects. And I will get into details below why.

However what is interesting is the gain between Camel 2.25 and 3.1 (40% less objects created).

Method Calls - (5 minute sampling)

Camel 2.25     139 different Camel methods in use
Camel 3.0      167 different Camel methods in use
Camel 3.1       84 different Camel methods in use

The table above lists the number of methods from Camel that Camel calls during routing. The data does not include all the methods from the JDK. As we cannot optimize those, but we can optimize the Camel source code.

As you can see from the table we have improvement. Camel 3.1 uses less than half of 3.0, and 40% less than Camel 2.2.5.

Camel 3.0
Okay so Camel 3.0 has a problem with using too much memory. A big reason is the new reactive executor which now executes each step in the routing via event looping, by handing over tasks to a queue and having workers that execute the tasks. So this handoff now requires creating additional objects and storing tasks in queue etc.

Some of the biggest wins was to avoid creating TRACE logging message which unfortunately was always created regardless if TRACE logging level was enabled. Another big win was to avoid creating toString representation of the route processes with child elements. Instead Camel now only output the id of the process which is a fast operation and dont allocate new objects.

Another problem was new code that are using java.util.stream. This is both a blessing and a curse (mostly a curse for fast code). So by using plain for loops, if structures, and avoiding java.util.stream in the critical parts of core routing engine we reduces object allocations.

Camel 3 is also highly modularised, and for example in Camel 2.x we had all classes in the same classpath and could use instanceof checks. So in Camel 3 we had some code that performed poorly doing these kind of checks (java util streams again).

Another problem was the reactive executor which was using a LinkedList as its queue. So if you have tasks going into the queue and workers processing them in the same pace, so the queue is empty/drained, then LinkedList performs poorly as it allocates/deallocates the object constantly. By switching to a ArrayQueue which has a pre-allocated size of 16 then there is always room in the queue for tasks and no allocation/deallocation happens.

There are many more optimisations but those mentioned above where likely the biggest problems. Then a lot of smaller optimisations gained a lot combined.

Many smaller optimisations

The UUID generator of Camel is using a bit of string concat which costs. We have reduced the need for generating UUIDs in the message and unit of work so we only generate 1 per exchange.

The internal advices in the Camel routing engine (advice = before/after AOP). Some of these advices has state which they need to carry over from before to after, which means an object needs to be stored. Before we allocated an array for all advices even for those whom do not have state and thus storing a null. Now we only allocate the array with the exact number of advices that has state. (very small win, eg object[6] vs object[2] etc, but this happens per step in the Camel route, so it all adds up.). Another win was to avoid doing an AOP around UnitOfWork if it was not necessary from the internal routing processor. This avoids additional method calls and to allocate a callback object for the after task. As all of this happens for each step in the routing then its a good improvement. 

Some of the most used EIPs has been optimized. For example allows you to send the message to an endpoint using a different MEP (but this is rarely used). Now the EIP detects this and avoids creating a callback object for restoring the MEP. The pipeline EIP (eg when you do to -> to -> to) also has a little improvement to use an index counter instead of java.util.Iterator, as the latter allocates an extra object

Camel also has a StopWatch that used a java.util.Date to store the time. This was optimized to use a long value.

Another improvement is the event notification. We now pre-calculate if its in use and avoid calling it all together for events related to routing messages. BTW in Camel 3.0 the event notifier was refactored to use Java 8 Supplier's and many fancy APIs but all of that created a lot of overhead. In Camel 3.1 we have restored the notifier to be like before in Camel 2.x and with additional optimisations.

So let me end this blog by saying that .... awesome. Camel 3.1 will use less memory, execute faster by not calling as many methods (mind that we may have had to move some code which was required to be called but doing this in a different way to avoid calling too many methods).

One of the bigger changes in terms of touched source code was to switch from using an instance based logger in ServiceSupport (base class for many things in Camel), to use a static logger instance. This means that there will be less Logger objects created and it's also better practice. 

Better performance
Other improvements is that we have moved some of the internal state that Camel kept as exchange properties to fields on the Exchange directly. This avoids storing a key/value in the properties map, but we can use primitives like boolean, int etc. This also performs better as its faster to get a boolean via a getter than to lookup the value in a Map via a key. 

In fact in Camel 3.1 then during regular routing then Camel doesnt lookup any such state from exchange properties which means there is no method calls. There are still some state that are stored as exchange properties (some of those may be improved in the future, however most of these states are only used infrequently). What we have optimized is the state that are always checked and used during routing. 

Exchange getProperty(5 minute sampling)
Camel 2.25     572598   getPropety(String)
Camel 2.25     161502   getPropety(String, Object)
Camel 2.25     161502   getPropety(String, Object, Class)
Camel 2.25     141962   getPropeties()

Camel 3.0      574944   getProperty(String)
Camel 3.0      167904   getPropety(String, Object)
Camel 3.0      167904   getPropety(String, Object, Class)
Camel 3.0       91584   getPropeties()

Camel 3.1           0   getProperty(String)
Camel 3.1           0   getPropety(String, Object)
Camel 3.1           0   getPropety(String, Object, Class)
Camel 3.1           0   getPropeties()

As you can see Camel 2.25 and 3.0 lookup this state a lot. And in Camel 3.1 we have optimized this tremendously and there are no lookup at all - as said the state is stored on the Exchange as primitive types which the JDK can inline and execute really fast.

The screenshot below shows Camel 2.25 vs 3.1. (The screenshot for 3.1 is slightly outdated as it was from yesterday and we have optimised Camel since). See screenshot below:

Okay there are many other smaller optimizations and I am working on one currently as I write this blog. Okay let me end this blog, and save details for part 3.


Apache Camel 3.1 - More camel-core optimizations coming

Hope all is good and you had a safe entry into 2020. 

The Camel team is already busy working on the next Camel 3.1 version. One of the goals is to continue optimize camel-core, and this time we have had some time to look into finding some hot spots in the routing engine.

One of the aspects we have looked at is also the object allocations that occurs per message that Camel routes. The JVM itself is great at allocation objects and garbage collecting them when they are no longer in use. However there are room for improvements if you can identify a number of objects that is unnecessary per EIP in the route.

So today I found several of these by just running a basic Camel route that is


Which basically routes 1000 messages per second. And prints each message to the log.

One of the bigger culprits in object allocations turned out to be human logging for the reactive executor which logs at TRACE level. So by avoiding this we can reduce a great deal of allocations, and string building for logging messages.

Other aspects we have optimised is the to EIP (the most used EIP) which is now smarter in its startup to avoid creating caches that was not necessary. And this goes together with areas where we now lazy creates some features in Camel that were very rarely in use that would otherwise also setup and create some caches. 

We also identified as part of the Camel 3 work, then the LRUCache was not pre warmed up as early as before, which meant Camel would startup a bit slower than it otherwise are capable of. So by moving this warmup to an earlier phase then Camel can startup faster by doing concurrent work on startup until the LRUCache is warmed up (its caffeine cache that needs this).

The log component has also been optimised to reduce its object allocations which building the logging message.

So all together a great day and if we compare startup up a Camel 3.0.0 vs 3.1.0-SNAPSHOT with the Camel route as shown above, then we have an awesome reducing in object allocations per second (thanks to YourKit for profiler).

The profile says that in Camel 3.0.0 then Camel would roughly generate about 22.000 objects per second (routing 1000 messages). And that has been reduced to about 6.000 objects per second in Camel 3.1. That is fantastic, and is almost a 4x reduction.