Dynamic FTP Client using Apache Camel and Spring

I was recently asked to develop an FTP client that could transmit files to various FTP servers as a part of a delivery system in a Java enterprise application. The requirements dictated a flexible implementation:

  • Three different FTP protocols should be supported, namely FTP, FTPS and SFTP
  • It should be possible to transmit different files to several different servers
  • The files to be sent were generated in runtime, they had different file names and were stored in different directories

Basically, I should implement the following interface:

FtpSender.java

where the FtpProperties was another interface:

FtpProperties.java

Rather than implementing the entire FTP client from scratch, I investigated the capabilities of existing frameworks. Most solutions recommend using an existing FTP framework such as Apache Commons / Jakarta Commons Net for FTP and FTPS and then wrap it in a SSH layer like Jsch / Java Secure Channel for SFTP. However, soon I discovered Apache Camel, “a powerful open source integration framework based on known Enterprise Integration Patterns with powerful Bean Integration” (their words). They support a various range of components, anything from XQuery and Atom to XSLT and SQL, just to name a few. To my luck, they also support all three FTP protocols that I needed.

Fundamentals

What does the Camel FTP API look like? In its basic form it is a little more than a one-liner:

FtpRouteBuilder.java

So what is going on here? The RouteBuilder is one of Camel’s core classes that defines the Camel specific DSL. In this example, a Route from the localDirectory to a remoteDirectory over FTP with the provided credentials has been created. As can be seen, it is easy to create and configure different endpoints using String URIs. It should also be mentioned that it is possible to fetch files from the remote FTP server by swapping the URIs in the to and from methods.

A few more lines are needed to activate the route:

A CamelContext is created for managing the route. The previously described FtpRouteBuilder is instantiated and added to the camelContext which is subsequently started. As a consequence, any file that is placed in the “localDirectory” folder will be automatically transferred to the “remoteDirectory” of the FTP server while the program is running.

Adding Configurability

The requirement of supporting different FTP protocols is solved by changing the URI from ftp:// to ftps:// or sftp:// respectively. It is just as easy to fulfill the second requirement of multiple server support by changing the host, user, port and other parameters of the URI. Likewise, additional settings may be configured by adding more options if needed, see the Camel FTP documentation.

Adding Dynamism

The last challenge was to add the dynamic support of selecting source files and destination servers during runtime. The easiest solution is to use another of Camel’s base classes, the ProducerTemplate. Again, the solution is a one-liner:

The sendBodyAndHeader() method name reveals a glimpse of Camel’s underlying messaging architecture. The parameters used are the destination URI with additional options, the message body, i.e. the File to be sent, a message header parameter that identifies the last parameter as the name of the file it will have on the remote server once it has been transferred.

Spring Integration

With some refactoring and a little Spring magic we have all the bits and pieces needed to implement the previously defined FtpSender interface:

FtpSenderImpl.java

Using the Camel namespace, the required plumbing work is delegated to the Spring application config:

spring-config.xml

Conclusions

Camel and Spring provide a simple, yet configurable, way of implementing a dynamic FTP client.

Acknowledgments

Thanks to Claus Ibsen and other members of the Apache Camel Forums for providing valuable and rapid support.

Glossary

  • FTP File Transfer Protocol
  • FTPS FTP Secure is an extension to FTP that adds support for the TLS, Transport Layer Security, and the SSL, Secure Sockets Layer, cryptographic protocols
  • SFTP SSH File Transfer Protocol, i.e. FTP over the Secure Shell protocol

References

Camel API
Camel Components
Camel FTP
Camel ProducerTemplate
Camel Spring
Camel ProducerTemplate and Spring
Camel Enterprise Integration Patterns
Camel Forums

21 Comments

  1. This is a great and well written blog entry. I love how you explain the problem and shows piece by piece how to build the solution.

    I have added a link to this blog entry from the Camel articles collection at
    https://cwiki.apache.org/confluence/display/CAMEL/Articles

  2. I always use Apache for my home server and Camel is a great addition to any kind of setup.

  3. Mattias Severson

    @Claus: Thank you very much!

  4. Hi Mattias

    I found your article extremely useful and in fact using your input I also convinced my client to use FTP2 library of apache Camel as enterprise wide FTP solution.

    I have a small problem, I need a “working-example” of FTPS end-to-end to prototype quickly to the client. I would greatly appreciate if you could help me out on this. I struggled with the existing samples on the internet. I need an end-to-end java code sample that performs an FTP(S)/SFTP.

    thank you
    Regards
    Nagesh

  5. Mattias Severson

    @Nagesh: Thank you.

    Most bits and pieces that you need are provided. The simplest possible approach would be to copy and paste:

    • FtpSender.java
    • FtpProperties.java
    • FtpSenderImpl.java
    • spring-config.xml

    Things that have been left out deliberately:

    • An implementation of the FtpProperties interface (a no-brainer)
    • A spring application, including a class that gets the FtpSender bean injected (i.e. the rest of the application business logic)

    Consider refactoring FtpProperties and / or the createFtpUri() method in the FtpSenderImpl to suit your needs. The FTP component have more URI options that you may find useful.

  6. Niranjan S

    Hi Mattias

    This is a great and well written blog entry to explore dynamically ftping files.

    I was also recently asked to develop an FTP client that could transmit files to various FTP servers as a part of a delivery system. I discovered “Apache Camel” is the best suited framework for integration. I started learning Camel from few days ago.

    I just followed you as explained above. The only different is that I don’t use Spring. I created the camel context in a java class (in a main method) and started the context (context.start()). Then I set the FtpProperties and called ftpSender.sendFile(ftpProperties, file, producer).

    Plz note the producer is context.createProducerTemplate().

    The main java class is as follows
    ————————————————————————-
    public static void main(String[] args) {
    CamelContext context = new DefaultCamelContext();
    try {
    context.start();
    File file = new File(“message1.xml”);
    FtpProperties ftpProperties = new FtpPropertiesImpl(“ftp”, “dfs”, “******”, “dev.e-xxxxx.com”, “images”, true);
    FtpSender ftpSender = new FtpSenderImpl();
    ProducerTemplate producer = context.createProducerTemplate();
    // Thread.sleep(10000);
    // context.stop();
    ftpSender.sendFile(ftpProperties, file, producer);
    } catch (Exception e) {
    System.out.println(“Error while Starting Camel ” + e);
    }
    }
    ————————————————————————-

    When I ran the above program I got the following error.

    Dec 19, 2010 5:07:20 PM org.apache.camel.component.file.remote.RemoteFileProducer connectIfNecessary
    INFO: Connected and logged in to: Endpoint[ftp://dfs@dev.e-xxxxx.com/images?passiveMode=true&password=******]
    Dec 19, 2010 5:07:21 PM org.apache.camel.component.file.remote.RemoteFileProducer handleFailedWrite
    WARNING: Writing file failed with: Cannot store file: images/message1.xml
    Error while Starting Camel org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: message1.xml]

    But I can able to ftp files when I use static router (Created at the coding time) as follows
    ———————————————————————-
    CamelContext context = new DefaultCamelContext();
    context.addRoutes(new RouteBuilder() {
    public void configure() {
    from(“file:D:/Niranjan/Books/Camel/camelinaction-source/chapter1/file-copy/data/inbox?noop=true”)
    .to(“ftp://dfs@dev.e-xxxxx.com/images?password=*******&passiveMode=true”);
    }
    });
    context.start();
    Thread.sleep(10000);
    context.stop();
    ——————————————————————–
    Am I missing that Spring suppose to do by default??
    Any help would be appreciated.

    • Mattias Severson

      @Niranjan: It depends on your use-case. The straight forward method using Spring is to create a static configuration like you suggest (compare with the FtpRouteBuilder implementation above). There are several ways in which Spring can instantiate the route builder bean from the Spring config.

      The reason why I did not use a static configuration was that my requirements was not static. On the contrary, in my case file names, FTP protocols and FTP servers were all parameters that changed for each transmission.

      Just a quick look at your code, it seems like the program attempts to send a file and then terminates, although the file may not have been transmitted. Likewise, you should not stop the camel context until the program stops, or at least not before the file has been transferred.

  7. Niranjan

    Thanks Mattias,

    My use case is also same as yours. I need to set ftp parameters at runtime, so your code example perfectly meet my use-case.

    I just change my Main program and set the thread.sleep to 1 minute. The file message1.xml is very small file (has only 20 chars). Then I commented the line context.stop(), so that I’m not stopping the camel context .

    Another change I did was I directly created ftp endpoint uri.

    Plz review my new code below

    public class TestICSControlSystem {

    public static void main(String[] args) {

    CamelContext context = new DefaultCamelContext();
    try {
    context.start();
    System.out.println(“*******CamelContext STARTED*******”);
    File file = new File(“message1.xml”);
    FtpProperties ftpProperties = new FtpPropertiesImpl((“ftp”, “dfs”, “******”, “dev.e-xxxxx.com”, “images”, true);

    ProducerTemplate producer = context.createProducerTemplate();
    producer.sendBodyAndHeader(“ftp://dfs@dev.e-xxxx.com/images?password=******&passiveMode=true”, file, Exchange.FILE_NAME, file.getName());

    Thread.sleep(60000); //Sleep the thread for a minute
    // context.stop();
    } catch (Exception e) {
    System.out.println(“Error while Starting Camel ” + e);
    }
    }
    }

    Still I got the same error

    *******CamelContext STARTED*******
    Dec 20, 2010 12:31:28 AM org.apache.camel.component.file.remote.RemoteFileProducer connectIfNecessary
    INFO: Connected and logged in to: Endpoint[ftp://dfs@dev.e-xxxx.com/images?passiveMode=true&password=******]
    Dec 20, 2010 12:31:29 AM org.apache.camel.component.file.remote.RemoteFileProducer handleFailedWrite
    WARNING: Writing file failed with: Cannot store file: images/message1.xml
    Error while Starting Camel org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: message1.xml]

    Can you run this program with your ftp server details? Just eager to know what am I missing. (Only you need to have a file “message1.xml” in same directory in which this program runs)

    Just one more quick question.
    After transfering the file, How do I rename that file? any thought on this? I need to rename, because there is file picker program in my remote machine pick this file.

    Thanks again for your help

    Regards,
    Niranjan

  8. Niranjan S

    Hi Mattias,

    I found out the mistake. The file “message1.xml” is not in the correct path. Now it working fine.

    Just one more quick question.
    After transffering the file, How do I rename that file? any thought on this? I need to rename, because there is file picker program in my remote machine pick this file.

    Thanks again for your help

    Regards,
    Niranjan

  9. venkat

    Hi Mattias Severson ,

    Nice Article. Could you please share the example code?

    Thanks,
    Venkat

  10. Mattias Severson

    @venkat: Thank you.

    The solution that I developed was part of a customer project and therefore there is no project source code that I can share. The example in the blog post has been simplified compared to the production code to keep it less verbose, but all relevant parts are there.

    Please also read my previous comment.

  11. Fedx

    Thank you for the example. Appreciate your effort in writing this entry.

    “A spring application, including a class that gets the FtpSender bean injected ”

    How exactly would you write this? with your suggested spring config, I get this in the log:

    INFO [main] (DefaultCamelContext.java1310) – Apache Camel 2.8.1 (CamelContext: camelContext) is starting
    INFO [main] (DefaultCamelContext.java1327) – Total 0 routes, of which 0 is started.
    INFO [main] (DefaultCamelContext.java1328) – Apache Camel 2.8.1 (CamelContext: camelContext) started in 0.219 seconds

    And the program ends..!

  12. Mattias Severson

    @Fedx: I used @Autowired to inject the FtpSender bean. The FTP client was part of a web application, so the web server was running the Spring application.

    It has been a while since I last used Camel, and I have not read recent release notes. Maybe you can find any clues there? The project that this example was based on used Camel version 2.3.0.

  13. Rajendra

    Hi.. Can we use similar technique if we have to read from an user ftp (configurable parameters) and store to local (fixed) directory?

    How will the producertemplate configuration would look like in this case?

    Any help would be much appreciated!

  14. Mattias Severson

    @Ragendra: I have not looked into details, but the ProducerTemplate javadoc states that:

    Important: Read the javadoc of each method carefully to ensure the behavior of the method is understood. Some methods is for InOnly, others for InOut MEP.

    (MEP being the Message Exchange Pattern).
    Another option might be to implement a ConsumerTemplate.

  15. milan

    i am able to download file from ftp but if the file is excel file then it is being corrupted….so , is there any solution to stop the crashing of file.

    • Mattias Severson

      It has been a while since I last used Camel, so I am not completely up to date. You can try change the charset option of the File Component. I used Camel 2.3.0 when I wrote the blog post, and it seems like several other options have been added since then. Likewise, several options have been added to the FTP components, perhaps you can find something useful there? Another option that you can try is to add an Error Handler that may provide some clues. Lastly, you can post a question on the Camel User Forum, maybe someone else has encountered problems with Excel files.

Trackbacks for this post

  1. Tweets that mention Dynamic FTP Client using Apache Camel and Spring — Jayway Team Blog -- Topsy.com
  2. Dynamic FTP Client using Apache Camel and Spring — Jayway Team Blog | ftp
  3. Dynamic FTP Client using Apache Camel and Spring — Jayway Team Blog « apache

Leave a Reply