Best Practices for Middleware and Integration Architecture Modernization with Apache Camel

Yesterday I gave the following virtual talk at the Stockholm Meetup.

Best Practices for Middleware and Integration Architecture Modernization with Apache Camel

What are important considerations when modernizing middleware and moving towards serverless and/or cloud native integration architectures? How can we make the most of flexible technologies such as Camel K, Kafka, Quarkus and OpenShift. Claus is working as project lead on Apache Camel and has extensive experience from open source product development.

I thank the organizers Forefront Consulting for inviting me. Unfortunately there was a glitch with the talk yesterday. As I could not be in person then the talk was pre-recorded and was cut half way. So I promised to post a blog today and upload the talk to youtube.

The title and abstract of the talk was somewhat given to me, and so was the session length of 30 minutes. As I am so heavily invested in Apache Camel, then I focused the talk about Camel and its evolution over the last 10 years as introduction and then using the latest innovations from Camel K, Camel Quarkus and Camel Kafka Connectors as the meat of the talk, and with 3 demos.

The talk can be watched on youtube and the slides are here.

Apache Camel 3.1 - Fast loading of XML routes

A feature that was added to Camel 3.1 is the ability to load XML routes much faster. This is part of the overall work we are doing on making Camel much smaller and faster.

You may say ewww XML. But frankly there are many users of Camel that have built applications with XML for defining routes. In Camel 2.x then you would have to use Spring or OSGi Blueprint for XML routes which both are becoming heavy in modern cloud native world.

In Camel 3 we have a standalone mode for Camel called camel-main. We use camel-main as a common way to bootstrap and configure Camel for standalone, camel-k, camel-quarkus, and for most parts of camel-spring-boot as well. This ensures an unified and consistent developer experience across those runtimes.

Okay this is probably a topic for another blog post to dive into camel-main as a great runtime for quickly running ... just Camel.

So what I wanted to say in this blog post is that we have made it possible to loading XML routes much quicker and with a lot less overhead. In Camel 2.x, and for Spring XML and Blueprint XML they rely on JAXP and JAXB which ... are heavy.

So what we have done for Camel 3.1 is to source code generate a XML parser based on the Camel DSL. This means anything we do changes to the DSL then the parser is re-generated. The parser just uses standard Java so there are no additional 3rd party library dependencies.

For loading XML routes in Camel we now have 2 parsers in the following JARs

- camel-xml-jaxb   (traditional JAXB based as in Camel 2.x)
- camel-xml-io       (new fast and lightweight source code generated parsers)

The example camel-example-main-xml is setup to use the new parser. But you can try for yourself and switch to the jaxb parser by changing the JAR dependency.

Lets see some numbers (note this is just a quick test on my laptop to run this example with the 2 XML parsers).

camel-xml-jaxb: Loaded 1 (808 millis) additional Camel XML routes from: routes/*.xml
camel-xml-io: Loaded 1 (76 millis) additional Camel XML routes from: routes/*.xml

So the new parser is about 10 times faster (76 vs 808 millis).

By profiling the JVM we can see that there is a lot less classes loaded as well: 4734 vs 3892. And on top of that JAXB leaves more objects and classes around in the JVM that may or may not easily be garbage collected, and would also be using more cpu and memory during its parsing.

And then on GraalVM then the new parser would be much quicker as you can avoid having the entire JAXB and JAXP API and implementation on the classpath and for the GraalVM compiler to crunch and compile. And speaking of GraalVM then we are working on some great improvements in the upcoming Camel 3.2 that should help reduce the image size and compilation, and allow to do more dead code elimination and whatnot to make Camel even more awesome. That's yet another topic for another blog post, so stay tuned.

Apache Camel 3.2 - Reflection free configuration of Camel

At the Apache Camel project we are working towards the next upcoming Apache Camel 3.2.0 release, which is planned for next month.

One of the ares we have worked hard on in Camel 3 is to make it smaller and faster. And one aspect of this is also configuration management. You can fully configure Camel in many ways and according to the 12 factor principles, to keep configuration separated from the application. A popular way to configure is to use properties files (eg application.properties) or in Kubernetes you can configure from config maps or environment variables as well.

So we have gradually over Camel 3.0, 3.1 and now 3.2 made configuration faster. With the latest work we are now fully reflection free.

Camel is capable of reporting when reflection based configuration are being used. Which can be configured with:

# bean introspection to log reflection based configuration
camel.main.beanIntrospectionExtendedStatistics=true
camel.main.beanIntrospectionLoggingLevel=INFO

We have prepared the camel-example-main-tiny to report this. The numbers for Camel 3.0, 3.1, and 3.2 are as follows:

Camel 3.0: BeanIntrospection invoked: 12 times
Camel 3.1: Stopping BeanIntrospection which was invoked: 11 times
Camel 3.2: Stopping BeanIntrospection which was invoked: 0 times

What this means is that you can fully configure all your Camel endpoints, components, routes, EIPs, data formats, languages, camel main, camel context, and whatnot, in declarative properties files etc and then at runtime all of this ends up invoking the actual setter methods on all these instances (ie just direct java method calls, no java.lang.reflect).

This is possible because we source code generate configurer classes based on what options are present. And these configurer classes are reflection free. There can be many options so it would be impossible to implement this by hand, see for example the kafka endpoint configurer.

And btw another feature coming in Camel 3.2 is that we made all of the components options available for configuration, before we didn't include nested configuration options. And if you do not like configuring in properties files, then we have type-safe component-dsl and endpoint-dsl as well.