4

I am writing a watchman command with watchman-make and I'm at a loss when trying to access exactly what was changed in the directory. I want to run my upload.py script and inside the script I would like to access filenames of newly created files in /var/spool/cups-pdf/ANONYMOUS . so far I have

$ watchman-make -p '/var/spool/cups-pdf/ANONYMOUS' -—run 'python /home/pi/upload.py'

I'd like to add another argument to python upload.py so I can have an exact filepath to the newly created file so that I can send the new file over to my database in upload.py,

I've been looking at the docs of watchman and the closest thing I can think to use is a trigger object. Please help!

2 Answers 2

3

Solution with watchman-wait:

Assuming project layout like this:

/posts/_SUBDIR_WITH_POST_NAME_/index.md
/Scripts/convert.sh

And the shell script like this:

#!/bin/bash
# File: convert.sh

SrcDirPath=$(cd "$(dirname "$0")/../"; pwd)
cd "$SrcDirPath"

echo "Converting: $SrcDirPath/$1"

Then we can launch watchman-wait like this:

watchman-wait . --max-events 0 -p 'posts/**/*.md' | while read line; do ./Scripts/convert.sh $line; done

When we changing file /posts/_SUBDIR_WITH_POST_NAME_/index.md the output will be like this:

...
Converting: /Users/.../Angular/dartweb_quickstart/posts/swift-on-android-building-toolchain/index.md
Converting: /Users/.../Angular/dartweb_quickstart/posts/swift-on-android-building-toolchain/index.md
...
0

watchman-make is intended to be used together with tools that will perform a follow-up query of their own to discover what they want to do as a next step. For example, running the make tool will cause make to stat the various deps to bring things up to date.

That means that your upload.py script needs to know how to do this for itself if you want to use it with watchman.

You have a couple of options, depending on how sophisticated you want things to be:

Use pywatchman to issue an ad-hoc query

If you want to be able to run upload.py whenever you want and have it figure out the right thing (just like make would do) then you can have it ask watchman directly. You can have upload.py use pywatchman (the python watchman client) to do this. pywatchman will get installed if the the watchman configure script thinks you have a working python installation. You can also pip install pywatchman. Once you have it available and in your PYTHONPATH:

import pywatchman

client = pywatchman.client()
client.query('watch-project', os.getcwd())
result = client.query('query', os.getcwd(), {
   "since": "n:pi_upload",
   "fields": ["name"]})
print(result["files"])

This snippet uses the since generator with a named cursor to discover the list of files that changed since the last query was issued using that same named cursor. Watchman will remember the associated clock value for you, so you don't need to complicate your script with state tracking. We're using the name pi_upload for the cursor; the name needs to be unique among the watchman clients that might use named cursors, so naming it after your tool is a good idea to avoid potential conflict.

This is probably the most direct way to extract the information you need without requiring that you make more invasive changes to your upload script.

Use pywatchman to initiate a long running subscription

This approach will transform your upload.py script so that it knows how to directly subscribe to watchman, so instead of using watchman-make you'd just directly run upload.py and it would keep running and performing the uploads. This is a bit more invasive and is a bit too much code to try and paste in here. If you're interested in this approach then I'd suggest that you take the code behind watchman-wait as a starting point. You can find it here:

https://github.com/facebook/watchman/blob/master/python/bin/watchman-wait

The key piece of this that you might want to modify is this line:

https://github.com/facebook/watchman/blob/master/python/bin/watchman-wait#L169

which is where it receives the list of files.

Why not triggers?

You could use triggers for this, but we're steering folks away from triggers because they are hard to manage. A trigger will run in the background and have its output go to the watchman log file. It can be difficult to tell if it is running, or to stop it running.

The interface is closer to the unix model and allows you to feed a list of files on stdin.

Speaking of unix, what about watchman-wait?

We also have a command that emits the list of changed files as they change. You could potentially stream the output from watchman-wait in your upload.py. This would make it have some similarities with the subscription approach but do so without directly using the pywatchman client.

1
  • Thanks for the detailed response! I ended up making another directory called .archived, and in my script I check the contents of the watched directory, then upload them to my database and then move the contents to the archived dir so that next time I list the contents only new files go through the process. Works like a charm so far! Oct 25, 2017 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.