On Linux, I have a directory with lots of files. Some of them have non-ASCII characters, but they are all valid UTF-8. One program has a bug that prevents it working with non-ASCII filenames, and I have to find out how many are affected. I was going to do this with find and then do a grep to print the non-ASCII characters, and then do a wc -l to find the number. It doesn't have to be grep; I can use any standard Unix regular expression, like Perl, sed, AWK, etc.

However, is there a regular expression for 'any character that's not an ASCII character'?

share|improve this question
2  
is perl installed? – Paul Creasey Jan 23 '10 at 17:50
1  
Paul, yes I can use perl – Rory Jan 23 '10 at 18:13
up vote 216 down vote accepted

This will match a single non-ASCII character:

[^\x00-\x7F]

This is a valid PCRE (Perl-Compatible Regular Expression).

You can also use the POSIX shorthands:

  • [[:ascii:]] - matches a single ASCII char
  • [^[:ascii:]] - matches a single non-ASCII char

[^[:print:]] will probably suffice for you.**

share|improve this answer
    
don't you mean [~\x20-\x7f] – adrianm Jan 23 '10 at 19:34
2  
@adrianm: No, ^ is valid in PCRE. – Alix Axel Jan 23 '10 at 20:05
10  
That's exactly right. However you have to use pcregrep, not standard grep. [^[:print:]] won't work if your terminal is set up in UTF8. – Rory Jan 24 '10 at 12:24
1  
What is the final /(pcre)?grep/ command then? – Xeoncross Jul 12 '14 at 18:04
    
@Rory, why :print: won't work in a UTF8 terminal? This works for me in pry in a UTF8 terminal: 27.chr =~ /[^[:print:]]/ – akostadinov Nov 10 '14 at 18:58

[^\x00-\x7F] and [^[:ascii:]] miss some control bytes so strings can be the better option sometimes. For example cat test.torrent | perl -pe 's/[^[:ascii:]]+/\n/g' will do odd things to your terminal, where as strings test.torrent will behave.

share|improve this answer

This turned out to be very flexible and extensible. $field =~ s/[^\x00-\x7F]//g ; # thus all non ASCII or specific items in question could be cleaned. Very nice either in selection or pre-processing of items that will eventually become hash keys.

share|improve this answer

You can use this regex:

[^\w \xC0-\xFF]

Case ask, the options is Multiline.

share|improve this answer

No, [^\x20-\x7E] is not ASCII.

This is real ASCII:

 [^\x00-\x7F]

Otherwise, it will trim out newlines and other special characters that are part of the ASCII table!

share|improve this answer

You don't really need a regex.

printf "%s\n" *[!\ -~]*

This will show file names with control characters in their names, too, but I consider that a feature.

If you don't have any matching files, the glob will expand to nothing.

share|improve this answer
1  
Oddly, this doesn't work correctly in Bash. – tripleee May 28 '12 at 12:07

You could also to check this page: Unicode Regular Expressions, as it contains some useful Unicode characters classes, like:

\p{Control}: an ASCII 0x00..0x1F or Latin-1 0x80..0x9F control character.
share|improve this answer
    
standard grep does not support those. – eis Jun 5 '17 at 15:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.