5

I'm working on writing a function in Clojure that will process a file character by character. I know that Java's BufferedReader class has the read() method that reads one character, but I'm new to Clojure and not sure how to use it. Currently, I'm just trying to do the file line-by-line, and then print each character.

(defn process_file [file_path]
(with-open [reader (BufferedReader. (FileReader. file_path))]
    (let [seq (line-seq reader)]
        (doseq [item seq]
            (let [words (split item #"\s")]
                (println words))))))

Given a file with this text input:

International donations are gratefully accepted, but we cannot make any statements concerning tax treatment of donations received from outside the United States. U.S. laws alone swamp our small staff.

My output looks like this:

[International donations are gratefully accepted, but we cannot make]
[any statements concerning tax treatment of donations received from]
[outside the United States.  U.S. laws alone swamp our small staff.]

Though I would expect it to look like:

["international" "donations" "are" .... ]

So my question is, how can I convert the function above to read character by character? Or even, how to make it work as I expect it to? Also, any tips for making my Clojure code better would be greatly appreciated.

1
  • What does the output look like if you use prn instead of println? println doesn't print quotes, so you may actually be getting the results you're expecting (e.g. ["International" "donations" "are"....
    – deterb
    Jul 26, 2012 at 15:39

3 Answers 3

5
(with-open [reader (clojure.java.io/reader "path/to/file")] ...

I prefer this way to get a reader in clojure. And, by character by character, do you mean in file access level, like read, which allow you control how many bytes to read?

Edit

As @deterb pointed out, let's check the source code of line-seq

(defn line-seq
  "Returns the lines of text from rdr as a lazy sequence of strings.
   rdr must implement java.io.BufferedReader."
  {:added "1.0"
   :static true}
  [^java.io.BufferedReader rdr]
  (when-let [line (.readLine rdr)]
    (cons line (lazy-seq (line-seq rdr)))))

I faked a char-seq

 (defn char-seq 
   [^java.io.Reader rdr]
   (let [chr (.read rdr)]
     (if (>= chr 0)
     (cons chr (lazy-seq (char-seq rdr))))))

I know this char-seq reads all chars into memory[1], but I think it shows that you can directly call .read on BufferedReader. So, you can write your code like this:

(let [chr (.read rdr)]
  (if (>= chr 0)
    ;do your work here
  ))

How do you think?

[1] According to @dimagog's comment, char-seq not read all char into memory thanks to lazy-seq

4
  • Yes, at a file-access level just like read. Once I have a 'reader' variable, what is the best approach to calling it's 'read()' method? Jul 26, 2012 at 15:03
  • 1
    Try looking at how line-seq does it - just type (source line-seq) into the REPL.
    – deterb
    Jul 26, 2012 at 15:35
  • 1
    char-seq does not read all chars in memory thanks to lazy-seq call. Also change (cons chr ... to (cons (char chr) ... because .read returns int. And I think (but not sure) that type hint could be relaxed to just java.io.Reader.
    – Dimagog
    Jul 26, 2012 at 23:11
  • This will actually create a sequence of Integers. The decoding is not happening. How to inject a decoding phase there?
    – jgomo3
    Dec 5, 2020 at 21:24
3

I'm not familiar with Java or the read() method, so I won't be able to help you out with implementing it.

One first thought is maybe to simplify by using slurp, which will return a string of the text of the entire file with just (slurp filename). However, this would get the whole file, which maybe you don't want.

Once you have a string of the entire file text, you can process any string character by character by simply treating it as though it were a sequence of characters. For example:

=> (doseq [c "abcd"]
     (prntln c))
a
b
c
d
=> nil

Or:

=> (remove #{\c} "abcd")
=> (\a \b \d)

You could use map or reduce or any sort of sequence manipulating function. Note that after manipulating it like a sequence, it will now return as a sequence, but you could easily wrap the outer part in (reduce str ...) to return it back to a string at the end--explicitly:

=> (reduce str (remove #{\c} "abcd"))
=> "abd"

As for your problem with your specific code, I think the problem lies with what words is: a vector of strings. When you print each words you are printing a vector. If at the end you replaced the line (println words) with (doseq [w words] (println w))), then it should work great.

Also, based on what you say you want your output to look like (a vector of all the different words in the file), you wouldn't want to only do (println w) at the base of your expression, because this will print values and return nil. You would simply want w. Also, you would want to replace your doseqs with fors--again, to avoid return nil.

Also, on improving your code, it looks generally great to me, but--and this is going with all the first change I suggest above (but not the others, because I don't want to draw it all out explicitly)--you could shorten it with a fun little trick:

(doseq [item seq]
        (let [words (split item #"\s")]
            (doseq [w words]
              (println w))))

;//Could be rewritten as...

(doseq [item s
        :let [words (split item #"\s")]
        w words]
  (println w))
1
  • Wow, thank you for the detailed response. Unfortunately using slurp won't work for me since it's possible that the file I'm going to be taking in will be too large to fit in to memory. I didn't realize that I could take a vector and make it a sequence like that. Thanks again. Jul 26, 2012 at 15:02
1

You're pretty close - keep in mind that Strings are a sequence. (concat "abc" "def") results in the sequence (\a \b \c \d \e \f).

mapcat is another really useful function for this - it will lazily concatenate the results of applying the mapping fn to the sequence. This means that mapcating the result of converting all of the line strings to a seq will be the lazy sequence of characters you're after.

I did this as (mapcat seq (line-seq reader)).

For other advice:

  • For creating the reader, I would recommend using the clojure.java.io/reader function instead of directly creating the classes.
  • Consider breaking apart the reading the file and the processing (in this case printing) of the strings from each other. While it is important to keep the full file parsing inside the withopen clause, being able to test the actual processing code outside of the file reading code is quite useful.
  • When navigating multiple (potentially nested) sequences consider using for. for does a nice job handling nested for loop type cases.

    (take 100 (for [line (repeat "abc") char (seq line)] (prn char)))

  • Use prn for debugging output. It gives you real output, as compared to user output (which hides certain details which users don't normally care about).

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.