2015-04-24

Apache Camel Eclipse Tooling for visual Data Transformation

So Keith Babo, who created the JBoss SwitchYard project, is now leading a team that works on data mapping. This effort is partly Apache Camel as they leverage the Camel Dozer component, and a slick UI that is part of the JBoss Tools for Apache Camel (formerly known as Fuse IDE).

Fuse Data Transformation Tooling as part of the JBoss Tools for Apache Camel

The source code is in the github repository at: https://github.com/fusesource/fuseide. Yes we have not yet moved that to another organization so it still says FuseSource.

During their development they release their work as milestones, and they just released milestone 4. The tooling is build regularly and can be installed using the following instructions.

For the busy people like me, its great they record videos from time to time to demonstrate this in action. So do like me, grab a cup of coffee and watch these 5-10 minute short videos.


And if you are interested then there is videos about the various mapping styles currently supported



2015-04-09

Getting started with JBoss Fuse - Where should I start?

Christina Lin, a Fuse evangelist, has created a great blog series of how to get started with JBoss Fuse.

She has done a great job of creating real life use-cases and detailing how to do this step by step, in both text and slides.


People who are new to JBoss Fuse is encouraged to take a look at this, as IMHO its easier to follow an use-case, than reading through then 100s of pages of reference documentation that is part of the product.

As Apache Camel is a corner stone of JBoss Fuse, then its often also even easier to get started by just learning some basic Camel first without thinking about Fuse / App Servers / OSGi / Karaf / Blueprint / Fabric / and many other concepts that JBoss Fuse brings to the table. Learn some Camel skills, which you can easily run on your computer locally, from within your Java editor. And then only thereafter take the leap and dive into the world of JBoss Fuse.

To get started with Apache Camel I point people to read this article written by Jonathan Anstey several years ago (don't worry the information is today 100% up to date, as Camel is table and dont throw you under the bus by frequent changes) which really captures well what Camel is and what it can do. For example the foundation of Camel which James Strachan laid out 8 years ago is still the road we travel today. If you are looking for other explanations what Camel is, then there is a good QA on stackoverflow.

Another great starting point to learn Apache Camel is chapter 1 of the Camel in Action book. And this is not the only Camel book, in fact there is 4 known books published.


2015-03-19

On this day 8 years ago Apache Camel was born

So its March 19th which is 8 years ago the first public commit of Apache Camel was pushed to ASF by James Strachan in 2007.

Using the lovely github browser its easy to have a flashback and take a look at what this first commit contains.

So James included APIs and concepts that end users of Apache Camel today would find familiar.

Lets start with the unit test of a Camel route, which is as shown below. The unit test includes three tests, one is a test of a simple route, that is just a from -> to as shown below:

RouteBuilder from the very first commit - sometimes we hit the nail from the first try

If you look at the code above, then that code would be 100% the same today. Its a route using the Java DSL using the same class name, RouteBuilder, where the routes are defined in the configure method. And this route is a simple from -> to using the seda endpoints.

The other two unit tests are using the content based router, aka choice in Camel.

Camel route using choice and predicates
These routes would almost be the same today, its only the predicate in the choice that has changed from:

headerEquals("foo", bar")

to a more rich DSL

header("foo").isEqualTo("bar")

If we keep looking in the first commit, we see that it includes APIs and concepts we use today

org.apache.camel.Endpoint
org.apache.camel.Exchange
org.apache.camel.Predicate
org.apache.camel.Processor
org.apache.camel.builder.RouteBuilder

It sure was a great start from the very first commit.

In summer 2007 Apache Camel 1.0 was released, which I had a flashback in 2012 and blogged about what I though was a great start for Camel.

And here 8 years later Apache Camel is as strong as ever.
Just take a look at the latest impressive Camel 2.15 release.


2015-03-15

Apache Camel Eclipse Tooling with Debugger

About 2 months ago Lars Heineman blogged about improved Apache Camel Eclipse tooling on the way as part of the JBoss Tool stack

In the upcoming release they have integrated the Camel debugger with the native Eclipse debugger, so you get the Eclipse debugging experience, when you use breakpoints, single step through the Camel routes. And you can of course see all the details of the Camel Exchange and Message on the way. And as well change the data on the fly.

A screenshot is shown below, which I have borrowed from Lars's blog.

Eclipse debugging your Apache Camel routes at runtime.
Lars also worked on adding support for editing Camel endpoints uri's using a properties panel, so each option is provided individually. That work was based on Apache Camel 2.14 that has some support for this.

But since we have improved this tremendously in Apache Camel 2.15, which I recently blogged about. And therefore Lars is currently working on upgrading to Camel 2.15 so the Eclipse Tooling becomes even better.

Now imaging that we take the properties panel based on Camel 2.14 and add all the extra information and documentation we have from Camel 2.15, which will allow Eclipse to present a similar enriched properties panel as hawtio can do.

Using Camel 2.14 showing an empty properties panel (no documentation, no information about defaults, required, limited enums, etc, consumer vs producer option etc.)

Eclipse Camel properties editor based on Camel 2.14

And below the enriched Camel 2.15 which has all of above information, show currently in hawtio.

Hawtio Camel properties viewer based on Camel 2.15

So imagine that the Eclipse properties panel will be able to include out of the box:

  • documentation
  • default values
  • enum types (eg choices to select among)
  • required vs optional
  • deprecated
  • simple and java type
  • option as part of uri path or query parameter
  • consumer only option
  • producer only option
  • custom category for the option (eg security, advanced, etc.)

And with Camel 2.15, we are able to do this for all the components - they all provide all this information.

Camel 2.15 also brings to the table, that it would allow the Eclipse tooling to dynamic generate the EIP palette, as Camel include information about all the EIPs and their options as well. So imagine the Eclipse tooling is able to adjust to which version of Camel you are currently using in the project. And yeah all the EIP options is now documented as well, which the tooling can present to you.

I am really exited about the possibilities that Camel 2.15 brings to the table in terms of tooling and also runtime experience we can enhance.

We also work on JBoss Forge commands that allows to add Apache Camel to existing projects, to dockerize and/or fabric8 enable the projects - that is something for another blog. But as part of this work, we are working on commands to add/edit Camel components/endpoints. So the idea would be they can show all the endpoints uris in your project, and present a nice properties editor for you to have "type safe" editing.

Going back to the title of this blog. Yeah great work Lars and the Eclipse team, we now have a great Apache Camel debugger. And its using the same Camel debugging API that hawtio uses also - no magic trick. In fact this week I was in talk with a company that has built their data integration platform on top of Apache Camel and also leverages its debugging api, to allow its developers and users to debug the deployed Camel routes, on the platform.


2015-03-11

In the year 2015 - Apache Camel 2.15.0 was released

So it will be known to mankind that in the first quarter of the year 2015, the awesome Apache Camel 2.15.0 was released.

Apache Camel 2.15.0 was released in 2015

Top 10 of great new features

I will try to do a top 10 of new functionality this release brings to the table.

#1 - Self documented

This release brings together, the steps we have taken over the last couple of released, in terms of being self documented. So what does this mean?

I have previous blogged about the work we have done


So what does this mean? It means that the same documentation Camel provides for all its:
  • Java and XML DSLs
  • Enterprise Integration Patterns
  • Camel Components
  • Camel Data formats
  • Camel languages
Is now included at both design and runtime. So for example if you design Camel routes in either Java or XML then the documentation is at your fingertips. For Java its documented as javadoc, and for XML that same documentation is now injected into the Camel XSD schemas, which allows IDE tooling to show you that documentation.

Here is a screenshot of IDEA editing a Camel route in XML and showing the documentation for the splitter EIP.

Camel XML now includes EIP documentation out of the box
At runtime your Camel applications and routes can present the documentation as well, but with the extra twist that the documentation can overlay with the runtime configuration in use. In other words we can now show what all those EIP options, component options, endpoint options, and whatnot means, and what their current values are, and what their default values are, whether they are required or options, which type they are, and if they are enum based, what options the can be, and much more.

All that comes together nicely if using the latest hawtio web console, which can present this nicely as the next screenshot shows

hawtio showing EIP endpoints at runtime with current values and documentation
To access the documentation at runtime, then Camel offers Java, JMX and Camel commands as the APIs. hawtio uses the JMX API.

#2 - Categorized components

All the Camel components have now been categorized, which means we can group the components, for example to know which Camel components there is for SQL, social, messaging, file, and so on. That is not all, we have also categorized all the EIPs, data formats as well.

And that information is available at both design and runtime, just as with the documentation. And this information can be reached using Java, JMX, and Camel commands. For example the Camel commands now has a set of Camel catalog commands that can be used, to find out which components we have for social, database, etc. The following screen shot shows the Camel commands in action using the new Camel Catalog component to do just that.

Camel commands to list Camel components by group

#3 - Camel Catalog for Tooling

There is a new standalone camel-catalog JAR which is intended for third party tool developers (and also Camel itself). The JAR contains a list of all Camel eips, components, data formats, languages, Camel XSD schemas, and the Maven archetypes. And also full documentation for all those as well. In other words the camel-catalog JAR is an offline JAR that has all the information we have talked about in #1 and #2. The JAR has Java and JMX API, as well .properties, .xsd, and .json schema files for all that. So its easy for tools to tap into that information.

Apache Camel uses the catalog itself, in the Camel commands as shown in the screenshot above.

As mentioned before there is a set of new Camel commands being developed for JBoss Forge, which is intended for making development with Apache Camel easier. You can say that this is an example of a third party developing Camel tooling. There is commands to make it easier to add new Camel components, configure endpoints, and whatnot. What makes JBoss Forge awesome is that it allows to run the same binary commands (forge addon) in any of its supported environments, which are

- command line (eg CLI)
- Eclipse
- Netbeans
- IDEA
- web

So for example we have a Camel command to add a new component that allows to filter by groups. The following screenshot shows which components we have that is about files.

Camel Forge commands running in Eclipse, as a wizard to add a Camel component to your project
And the same command used from web (work in progress)

Camel Forge command running in web browser, as a wizard to add a Camel component to your project

#4 - Reuse Camel commands

In previous releases of Apache Camel, the Camel commands was only for Karaf users. But in this release we have made these commands reuseable, in the camel-commads-core module. And there is an out of the box implementation for Karaf and Jolokia. 

And for third party there is JBoss Forge Camel commands as part of the fabric8 project. By using these commands you can manage Camel applications at runtime, for example to check the state of your routes, and so on.

The screenshot below shows how to use the camel-routes command from JBoss Forge. The same command can be run from Apache Karaf as well.

Camel commands connecting to a remote JVM using jolokia to check the route status
The Karaf commands only works in the same JVM as Karaf itself. But the jolokia commands allows to manage remote JVMs as well. In the screenshot above I have a local running JVM with Apache Tomcat and deployed the standard Camel Tomcat Servlet example. And from a JBoss Forge JVM I use the commands to connect to the Tomcat JVM and list all its running Camel routes. If you are familiar with hawtio then its the same principle, as hawito is able to do that same as well.


#5 - Camel Boot / Spring Boot

For the micro services trend we have Camel boot supported, though most noticeable by the camel camel-spring-boot module. Henryk Konsek did a great job implementing and documenting this module, so there is plenty of material there to run Camel and Spring happily together.

For Camel Boot we are improving on alternatives with CDI, Guice, and plain Java in the upcoming release. For CDI in particular there is works in a much improved and working camel-cdi, which is expected to be included in Camel 2.16.


#6 - Improved swagger 

The camel-swagger module now supports any kind of runtime (before it was tied to Spring). In the next release we will add support for hosting multiple Camel applications from the same camel-swagger module, so you can deploy it once, and use it for all your Camel apps in the same JVM.

We are also hoping the swagger team finish their highly anticipated pure Java implementation, so we can drop Scala and make camel-swagger more friendly to use / and much lighter without the Scala libraries.


#7 - Improved Rest DSL

We have improved and fixed some issues with the new Rest DSL that was introduced in Camel 2.14 based on community feedback. For example its easier to configure using custom jackson modules for json. We also made it easier to return custom error messages when routing has failed etc.

We will continue to do so as we have still some outstanding work. 


#8 - Routing Engine optimizations

We also optimized Camel itself in particular two areas:

  • Optimized the case insensitive headers, but dropping the need for an internal shadow map.
  • Optimized the type converter to avoid conversion attempts that was not needed, which in some cases reduced the factor by 2-4 x during routing.

#9 - More detailed information about inflight messages

Camel now tracks inflight messages with more details, which allows to pin point exactly where in the routes the inflight messages current is being processed at. This information can be accessed from Java and JMX. And there is also a Camel command to list inflight messages. Also hawtio has a web page that list all the inflight messages with all that information as well.

hawtio showing inflight exchanges for the selected route. Notice the message is currently at "delay1" in the route

And the Route JMX MBean now also include a JMX attribute that contains the oldest inflight duration, which makes it easier for monitor tooling to quickly identify which routes may have messages that may "take too long time".

Also during shutdown of Camel itself, in case of the graceful shutdown was not able to shutdown and complete all inflight exchanges, then Camel now logs which exchanges are still inflight, and where in the routes they currently are.

#10 - Configuring endpoints in XML with long attributes can now break lines

When using XML for configuring Camel routes, you may use endpoints that has many options you need to configure, which then may make the endpoint uri long and less human readable. What you can do now, is to use line breaks in the attribute as shown below

Configuring endpoints in XML can now use line breaks (you can have multiple options in the same line)
An alternative is also that the less known can be used to configure endpoints, which now support style as well, as shown:

Configuring endpoints using property style


#11 - And as always a bunch of new components

And as usual we have a bunch of new components. This time there is 16 new components, for example for the talk of the town, Docker. And for Spring 4.0.x users there is a working camel-spring40-test module for testing. As camel-spring-test requires Spring 4.1.x.




You can find more details in the Apache Camel 2.15.0 release notes. And make sure to read the bottom of the release notes that details what to consider when upgrading.

The release can be downloaded from maven central, and the Apache Camel website.

2015-02-02

Your Apache Camel applications now includes out of the box documentation

A couple of months back I blogged about the upcoming 2.15 release includes functionality to obtain fine grained information about each property configured on your endpoints in the blog titled - Apache Camel please explain me what these endpoint options mean.

We have continued on this path, and today we have extended this from endpoints to cover EIPs, data formats, and languages as well. In a nutshell almost all what you use and configure in your Camel applications now has the capability of including documentation out of the box at runtime. In addition all of those are associated with label(s) which we use as a way of categorizing them. So you can list all components for database, cloud, http, rest, messaging, etc etc.

As the EIPs is provided in the camel-core those all their options is fully documented, including details about their default values, if the option is required, whether an option is deprecated, and of course type information etc. The EIP itself is although currently only documented with a brief summary.

As Camel includes about 175 components then it will take some time to include documentation about each one of them. But we will setup in the build process of Apache Camel a report that outputs which components has missing documentation.

Okay a picture is worth a thousand words, so let me include a bit of eye candy of this in action using hawtio - the web console.

First an example of the EIP such as the aggregate EIP which has many options.

Showing properties of the Aggregate EIP in our Camel application,
including documentation out of the box 
Notice in the top there is checkboxes to:

  • show/hide documentation (the documentation is always provided as a tooltip)
  • show/hide default values (not in current screenshot, as just added)
  • show/hide unused values

And another screenshot showing the properties of the endpoints:

Showing properties of an endpoint in our Camel application,
including documentation out of the box
This page is similar to the EIP page. It will also include the same checkboxes to filter.
Also notice those grey badges, they represent the label(s) this endpoint has been associated with. As we can see this is an endpoint from the core, and its scheduling.

All this information can be accessed at both runtime and design time. At runtime there is Java and JMX API from camel-core. And at design time the information is included in a new camel-catalog JAR that has json schema files for each EIP, component and so on. Likewise these json schema files can be retrieve at runtime using the Java and JMX API from camel-core as well.

So in other words this allows both end users and tooling to access this information.

For more information, read my previous blog, which I refer to in the top of this blog post.

What's Next

We have yet to implement a maven plugin as part of the build process of Apache Camel, that injects the EIP documentation in the XSD schema for the Spring and Blueprint XML DSL. When this is in place, then any IDE tooling should be able to leverage this, so while you edit those XML files, then the tooling can assist you and present the documentation etc.


2015-01-20

Webinar - Integrating microservices with Apache Camel

Tomorrow on wednesday 21st of january, our fellow Camel rider, Christian Posta is giving a webinar about microservices and Apache Camel.
Microservices architectures are all the rage these days, but we have to balance hype with reality. Microservices make it harder to manage your deployments and makes for complex inter-service communications patterns. How do you balance the tradeoffs and focus on getting the most out of your investment in highly scalable, decoupled systems?
Join this webinar to learn how to use patterns from SOA to build out intelligent routing systems with Apache Camel, and centralized management, service discovery, versioning, and tooling support from Red Hat JBoss Fuse for managing complex integrations using a microservice approach.
Date: Wednesday, January 21, 2015
Time: 16:00 UTC | 11:00 am (New York) | 5:00 pm (Paris) | 9:30 pm (Mumbai)
Duration: 60 minutes

You can find more details and register for the webinar using the following link.