20

I would like to monitor all of the files in a given directory for changes, ie an updated timestamp. This use case seems natural for Camel using the file component, but I can't seem to find a way to configure this behavior.

A uri like:

file:/some/directory

will consume the files in the provided directory but will delete them.

A uri like:

file:/some/directory?noop=true

consumes each file once when it is added or when the route is started.

It's surprising that there isn't an option along the lines of

consumeOnChange=true

Is there a straightforward way to monitor file changes and not delete the file after consuming?

5 Answers 5

26

You can do this by setting up the idempotentKey to tell Camel how a file is considered changed. For example if the file size changes, or its timestamp changes etc.

See more details at the Camel file documentation at: https://camel.apache.org/components/latest/file-component.html

See the section Avoiding reading the same file more than once (idempotent consumer). And read about idempotent and idempotentKey.

So something alike

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:size}")

Or

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:modified}")

You can read here about the various ${file:xxx} tokens you can use: http://camel.apache.org/file-language.html

7
  • Thanks for taking the time to respond, Claus. I really appreciate your work on Camel. I just tried this for 20 minutes before my next meeting and I can't seem to get it working. I tried with size and modified to no avail. I now understand that noop turns idempotent true by default which makes sense. I'll keep looking at the file tokens when I have a bit more time, but I expected it to start picking up modified files when I added ${file:name}-${file:modified}. Is there more to it? Thanks again!
    – Janx
    Commented Nov 20, 2013 at 18:53
  • Ah just remembered I fixed a bug recently with idempotentKey, though reported against FTP. Its CAMEL-6936. So will be fixed in upcoming releases. Sorry for that bug. Commented Nov 20, 2013 at 20:20
  • no worries - so a url like the second one you posted should work on a future release? Are SNAPSHOT artifacts published anywhere?
    – Janx
    Commented Nov 20, 2013 at 20:26
  • Hmm - I tried the latest 2.12.2-SNAPSHOT and 2.13-SNAPSHOT. No luck. To be clear, my url now looks like " ?noop=true&idempotentKey=${file:name}-${file:modified} Maybe it's not in the builds yet?
    – Janx
    Commented Nov 20, 2013 at 22:28
  • Ah thanks the first fix from CAMEL-6936 didn't cover it all. I have fixed this again. So any future SNAPSHOT updates should include the fix. Commented Nov 21, 2013 at 8:11
2

Setting noop to true will result in Camel setting idempotent=true as well, despite the fact that idempotent is false by default.

Simplest solution to monitor files would be:

.from("file:path?noop=true&idempotent=false&delay=60s")

This will monitor changes to all files in the given directory every one minute.

This can be found in the Camel documentation at: http://camel.apache.org/file2.html.

1

I don't think Camel supports that specific feature but with the existent options you can come up with a similar solution of monitoring a directory.

What you need to do is set a small delay value to check the directory and maintain a repository of the already read files. Depending on how you configure the repository (by size, by filename, by a mix of them...) this solution would be able to provide you information about news files and modified files. As a caveat it would be consuming the files in the directory very often.

Maybe you could use other solutions different from Camel like Apache Commons VFS2 (I wrote a explanation about how to use it for this scenario: WatchService locks some files?

1
  • Well said. This is the pragmatic answer :-) In my case (a Camel FTP client shipping files as soon as they're made available) I added a JMS queue where the file producer posts a message that wakes up the FTP client. Commented Jan 17, 2017 at 15:39
1

I faced the same problem i.e. wanted to copy updated files also (along with new files). Below is my configuration,

public static void main(String[] a) throws Exception {

    CamelContext cc = new DefaultCamelContext();

    cc.addRoutes(createRouteBuilder());

    cc.start();

    Thread.sleep(10 * 60 * 1000);

    cc.stop();
}


protected static RouteBuilder createRouteBuilder() {
    return new RouteBuilder() {
        public void configure() {
            from("file://D:/Production"
                    + "?idempotent=true"
                    + "&idempotentKey=${file:name}-${file:size}"
                    + "&include=.*.log"
                    + "&noop=true"
                    + "&readLock=changed")

            .to("file://D:/LogRepository");
        }
    };
}

My testing steps:

  1. Run the program and it copies few .log files from D:/Production to D:/LogRepository and then continues to poll D:/Production directory
  2. I opened a already copied log say A.log from D:/Production (since noop=true nothing is moved) and edited it with some editor tool. This doubled the file size and save it.

At this point I think Camel is supposed to copy that particular file again since its size is modified and in my route definition I used "idempotent=true&idempotentKey=${file:name}-${file:size}&readLock=changed". But camel ignores the file. When I use TRACE for logging it says "Skipping as file is already in progress...", but I did not find any lock file in D:/Production directory when I editted and saved the file.

I also checked that camel still ignores the file if I replace A.log (with same name but bigger size) in D:/Production directory from outside.

But I found, everything is working as expected if I remove noop=true option.

Am I missing something?

2
  • 1
    Did you ever try posting this as a separate question? Commented Feb 19, 2016 at 9:18
  • Is there any flag to trigger the route when from a watched folder a file is deleted instead of modified?
    – Gagan
    Commented Jul 3, 2019 at 15:41
1

If you want monitor file changes in camel, use file-watch component.

Example -> RECURSIVE WATCH ALL EVENTS (FILE CREATION, FILE DELETION, FILE MODIFICATION):

from("file-watch://some-directory")
.log("File event: ${header.CamelFileEventType} occurred on file ${header.CamelFileName} at ${header.CamelFileLastModified}");

You can see the complete documentation here: Camel file-watch component

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.