13

Is there a way to get a program I can run via the command line that would do a checksum of a remote file? For instance get a checksum of https://stackoverflow.com/opensearch.xml

I want to be able get an update of when a new rss/xml entry is available. I was thinking I could do a checksum of a file every once in a while and if it is different then there must be an update. I'm looking to write a shell script that checks new rss/xml data.

5 Answers 5

16

A quick way to do this with curl is to pipe the output to sha1sum as follows:

curl -s http://stackoverflow.com/opensearch.xml|sha1sum
3
  • 4
    I add -L too... curl -sL URL | sha256sum (macOS) or curl -sL URL | shasum -a 256 ...
    – vigo
    Commented May 13, 2021 at 19:21
  • curl -sL is silent and follows redirects. On a Mac I replace sha256sum with gsha256sum.
    – Pro Backup
    Commented Jun 20, 2021 at 17:01
  • Having -L is crucial when getting files from sites like github. They mask the real URL and redirect to the actual resource location Commented Sep 22, 2023 at 15:06
6

In order to make a checksum on the file, you'll have to download it first. Instead of this, use If-Modified-Since in your request headers, and server will respond with 304 not modified header and without content, if the file is not changed, or with the content of the file, if it was changed. You may be interested also in checking for ETag support on the server.

If downloading the file is not a problem, you can use md5_file to get md5 checksum of the file

5

curl

curl has an '-z' option:

   -z/--time-cond <date expression>|<file>
          (HTTP/FTP) Request a file that has been modified later 
          than the given time and date, or one that has been modified before
          that  time.  The  <date expression> can be all sorts of date strings
          or if it doesn't match any internal ones, it is taken as a filename
          and tries to get the modification date (mtime) from <file> instead.
          See the curl_getdate(3) man pages for date expression details.

So what you can do is:

$ curl http://stackoverflow.com/opensearch.xml -z opensearch.xml -o opensearch.xml

This will do actual download if remote file is younger than the local one (local file may absent - in this case it will be downloaded). Which seems to be exactly what you need...

wget

wget also has an option to track timestamps - -N

When running Wget with -N, with or without -r or -p, the decision as to whether
or not to download a newer copy of a file depends on the local and remote
timestamp and size of the file.

-N, --timestamping               Turn on time-stamping.

So in case wget one can use:

$ wget -N http://stackoverflow.com/opensearch.xml
1
  • 1
    WARNING: timestamping does nothing in combination with -O. See the manual for details
    – BozoJoe
    Commented Oct 16, 2020 at 21:28
1

You can try this under your bash:

wget <http://your file link>

md5sum <your file name>
5
  • thanks. I know that. I would like to avoid downloading the file if I could I guess. Just constantly checking for a new update would be nice...so the second there is an update I know. Isn't that what rss readers do?
    – rich
    Commented Jul 23, 2011 at 16:34
  • So when there is a update, where do you want to display the msg? in the logfile? or inside the current bash?
    – Kit Ho
    Commented Jul 23, 2011 at 16:36
  • current bash. What want to do is first time get the whole file and check for conditions (look for certain words) and then future updates just check the new stuff. I would like to check an unlimited number of urls. Basically a rss reader but then I want to check the data and do something with the data if it meets a certain condition. But of course I don't want to get in trouble with the domains for checking every second.
    – rich
    Commented Jul 23, 2011 at 16:42
  • so, without the polling mechanism, how does a normal rss reader work? :) What i can offer to you is using infinite for loop and checking the file every sec....
    – Kit Ho
    Commented Jul 23, 2011 at 16:50
  • Not sure how a rss reader works :). If I check the file every second I have a feeling domains won't be too happy with me. I was planning on doing a for loop but I want to make it with the same rules as a rss reader. I'm sure there are some sort of rules for how often they check and the like...right?
    – rich
    Commented Jul 23, 2011 at 16:55
0

You should first examine the HTTP headers to see if the server itself is willing to tell you when the file is from; it's considered bad form to fetch the entire file if you don't need to.

Otherwise, you'll need to use something like wget or curl to fetch the file, so I really hope you don't plan to be working with anything large.

2
  • is this how rss readers work? How do they know there is a change to the xml file and notify me? Do they constantly check HTTP headers? I like this idea...any site you can link to that would be a good example of doing this via the commandline?
    – rich
    Commented Jul 23, 2011 at 16:32
  • yes, if you read the documentation for curl you'll see it has various options for fetching just HTTP headers.
    – Olipro
    Commented Jul 23, 2011 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.