2008-12-01

Camel and file sorting

The file component to be used for file connectivity in Camel is a key component in the kind of integration works I am hustling with a clients. We are still doing a lot of file and messaging based on IBM platforms from large mainframes and other older platforms. So I am not riding on the Camel with the yellow jersey in the "peloton" (hint: Tour de France).

So my Camel is up for a maintenance upgrade now that we are working on Camel 2.0. A part of this upgrade is being able to support ease of use for pluggable file filtering and sorting.

The sorting comes in two flavors:
- comparator
- textual expressions

Sorting by comparator
We leverage the java.util.Comparator to allow end users to configure their own implementation to use, or use what we have on the shelves out-of-the-box for the regular sorting by name, modified date, size etc provided in org.apache.camel.component.file.DefaultFileSorter.

We can configure the comparator in the file endpoint using the sorterRef option:
from("file://inbox/?sorterRef=myFileSorter").to("bean:processInbox");

... where myFileSorter is lookup up in the registry, typically Spring application context defined as a spring bean:

<bean id="myFileSorter" class="com.mycompany.MyFileSorter"/>


Sorting by expressions
This is where we did go the extra mile and implemented textual configuration of sorting. You just add a sortBy option to the file endpoint URI configuration where you express what and how to sort.

A plain example would be:
from("file://inbox/?sortBy=file:name").to("bean:processInbox");

We can specify reverse if we wanted to sort in reverse order
from("file://inbox/?sortBy=reverse:file:name").to("bean:processInbox");

And the ignore case is also supported so you can process files regardless of the case
from("file://inbox/?sortBy=ignoreCase:file:name").to("bean:processInbox");

And on top of that we have grouped sorting so you can append a 2nd sort by, where the groups are separated with semi colon. So if we want to sort first by file extension and then by name we do:
from("file://inbox/?sortBy=file:name.ext;file:name").to("bean:processInbox");

As the expression is the file language you got its full power as well so if you want to sort by modified timestamp but it should only consider the date part, we can use the yyyyMMdd pattern and express this as:
from("file://inbox/?sortBy=date:file:yyyyMMdd").to("bean:processInbox");

Currently this is only implemented on the file component, but next up is to implement this for the FTP component as well.

Check out the file component and the file language to get an idea how this new sortBy option can be used.

12 comments:

satyam said...

Hi,
I have some files(i.e xml files) in a folder. Can u guide me how can i sort that files based on time stamp and also i have to pick the latest file.
Thanks in advance,
satyam

Claus Ibsen said...

Yeah check out this page the section - using sortBy
http://camel.apache.org/file2.html

You do something like
from("file://somefolder?sortBy=file:modified")

And you can reverse the sorting by using reverse
from("file://somefolder?sortBy=reverse:file:modified")

satyam said...

Hi Claus,

thanks for ur reply....

Vinu Raj said...

sorterRef=myFileSorter is not working in the vesion (2.13.1) I am using. Instead sorter=#myFileSorter is working fine.

Nagamani mandava said...

In my case I need to transfer two types of files, .csv file and its corresponding .tigger file.

correspondig Trigger file should only get transferred only once .csv files are loaded to destination.

please help me with solution to this problem.

Claus Ibsen said...

See about done file name on the file documentation at: http://camel.apache.org/file2

Nagamani mandava said...

Thanks. That works.

But if there is a single route which is transferring both the csv and trigger files. How to implement logic such that first csv files should get transferred , once csv files sre transferred don efile should be created automatically at target location and then trigger file should be transferred

Nagamani mandava said...

If I write a condition on done file csv files also dont get transferred till done file is created in target folder.

In my case .csv file should act like done file to transfer .trigger file

Nagamani mandava said...

How to sort more than 2 files based on some condition using sorterRef.

Like how do i send files to the Comparator class from camel

Nagamani mandava said...

How do i use sorterRef to sort more files. how do i pass files to comparotor class from camel

colors2007 said...

How can i sort all the files available in directory and pickup first file in soreted list in one poll.

then second poll does the similar operation. So that files should pickup in first come first reach to the folder.

lets say files present in folder are
file2.txt
file1.txt
file3.txt

first poll should sort above 3 and pick up only file1.txt removing remaining from the inprogressing repository leaving them in the source directory itself.

How to achieve this?

Claus Ibsen said...

Use maxMessagersPerPoll=1 and set eagerMaxMessagesPerPoll=false