26

I'm trying to use this function in a C program that needs to be able to compile in Linux and Windows. At first I tried using strtok_r, but then when I compiled on windows, it complained about the function not existing and said it would assume it's an extern function, but then failed. I then used strtok_s and it compiled! Then I tried on Linux but now it's complaining that there is an "undefined reference to 'strtok_s'".

Is one a windows only function and the other a linux function??? What can I do to make it compile on both?

10
  • 3
    strtok_r() is POSIX; strtok_s() is Windows. Use plain strtok() which is available in all implementations (supporting C89).
    – pmg
    Commented Jan 26, 2012 at 16:37
  • 4
    The problem is I need to strtok a string while strtok-ing another string, which is why I need this one since it saves state.
    – petranaya
    Commented Jan 26, 2012 at 16:39
  • 3
    @pmg - unless you need thread or buffer overrun safety Commented Jan 26, 2012 at 16:40
  • 1
    @MartinBeckett Wait what? Where does it save the state? There are only two arguments, one is NULL (on the second token), the other is what character you want to find. I want to be able to strtok a string, find something in that string and strtok that one, then go back to the original string and continue to strtok it.
    – petranaya
    Commented Jan 26, 2012 at 16:46
  • 7
    This is an ancient question by now, but I felt the need to point out that strtok_s is C11, not just Windows.
    – remmy
    Commented Jan 25, 2015 at 15:33

6 Answers 6

44

strtok_s is simply the Windows version of strtok_r which is standard everywhere else.

One (common I would think) way to make a program portable when it comes to functions like strtok_s/strtok_r is to use the preprocessor:

#if defined(_WIN32) || defined(_WIN64)
/* We are on Windows */
# define strtok_r strtok_s
#endif

As the prototypes and functionality is the same, you can now use only strtok_r.

4
  • This would probably break if compiling on cygwin which reports itself as windows but has posix interfaces like strtok_r already defined. Using something like #ifndef HAVE_STRTOK_R (and detecting it in the build scripts) would be better. Commented Jan 26, 2012 at 16:38
  • 6
    @R..: cygwin does not in fact define _WIN32 (or _WIN64) for precisely this reason -- its commonly used to identify code that is non-unix like. Cygwin defines __CYGWIN__ and __unix__ instead
    – Chris Dodd
    Commented Apr 23, 2013 at 22:17
  • 3
    It may be good to know that there is an other strtok_s in C11 with a completely different signature: en.cppreference.com/w/c/string/byte/strtok Commented Jan 6, 2017 at 14:37
  • 2
    @ChrisDodd There are still problems on MinGW. MinGW predefines _WIN32 but it supports strtok_r. I think it is better to check the _MSC_VER macro.
    – Victor
    Commented Feb 15, 2019 at 3:01
11

Both of these functions are really ugly, unintuitive idioms for parsing strings, and usually fail to meet your particular application's requirements in subtle ways. Even moreso for the plain strtok in standard C. Just throw them out and write your own code to iterate over the char array and break it up as needed. strchr, strspn, and strcspn can be helpful in doing this or you can just work from scratch on the array.

5
  • 2
    Can you explain a little more about why these functions are bad? It seems more intuitive to me than going through character by character and searching for things.
    – petranaya
    Commented Jan 26, 2012 at 16:52
  • 4
    (1) the separator char is overwritten/lost, so unless you only have one possible separator, you're losing information. (2) multiple consecutive separators are treated as one, which makes sense if your separator is whitespace but not if it's a comma, for instance.. Commented Jan 26, 2012 at 17:01
  • 5
    Please note that surely in some situations, the strtok semantics are just what you want and the only issue is the statefulness (which strtok_r and strtok_s fix). I just find that every time I tried to use them, there's at least one aspect of my parsing needs they don't support, and it ends up being easier to just write the code directly. Commented Jan 26, 2012 at 17:03
  • 9
    -1: For parsing a string that's been read into a temporary buffer, where you want to throw away the delimeters and you only want the strings valid until you're done parsing, strtok is precisely what you want and strtok_r/strtok_s is much better as it is reentrant.
    – Chris Dodd
    Commented Apr 23, 2013 at 22:19
  • "write your own code" -- see also How do I split a string without strtok? and String separator, similar to strtok and glibc/string/strtok_r.c
    – milahu
    Commented Feb 28, 2024 at 11:58
11

I don't have enough reputation to comment on other answers, so I'll have to provide my own.

1) To address this statement:

"strtok_s is a buffer overrun safe version of strtok on Windows. The standard strtok on windows is thread safe..."

This is not true. strtok_s is the thread safe version for the MSVC compiler. strtok is not thread safe!

2) To address this statement:

"This would probably break if compiling on Cygwin which reports itself as windows but has POSIX interfaces like strtok_r already defined."

Again, not true. The difference is which compiler you use. When using Microsoft's Visual C++ compiler, MSVC, the function is strtok_s. Another compiler, such as the GNU Compiler Collection, GCC, may use a different standard library implementation such as strtok_r. Think compiler, not target platform, when identifying which function to use.

In my opinion, Joachim Pileborg's answer is the best one on this page. However, it needs a small edit:

#if defined(_WIN32) /* || defined(_WIN64) */
#define strtok_r strtok_s
#endif

Both _WIN32 and _WIN64 are predefined macros provided by the MSVC compiler. _WIN64 is defined when compiling a 64 bit target. _WIN32 is defined for both 32 and 64 bit targets. This is a compromise that Microsoft made for backwards compatibility. _WIN32 was created to specify the Win32 API. Now you should consider _WIN32 to specify Windows API -- it is not specific to a 32 bit target.

2
  • 3
    You are wrong about the first point: strtok is thread-safe on Windows, since two distinct threads can safely use that function within the same process, even if strtok have other problems as explained in another answer by @JameyKirby.
    – AntoineL
    Commented Feb 24, 2014 at 20:10
  • MinGW predefines _WIN32 but it supports strtok_r, so it is better to check the _MSC_VER macro.
    – Victor
    Commented Feb 15, 2019 at 3:02
7

Just to clarify. strtok is thread safe in Windows. strtok uses a TLS variable to maintain the last pointer for each thread. However, you can not use strtok to interleave access to more than one token string per thread. strtok_r and strtok_s both address this interleaving problem by allowing the user to maintain the context via the third parameter. Hope this helps.

1
  • Although not helpful for the OP, who is looking for a portable solution, this was very helpful for me to find out why my legacy code crashes on Linux but not on Windows. As I thought I knew how strtok behaves, I didn't care to read the Windows strtok documentation thoroughly...
    – Matthias
    Commented Jul 6, 2021 at 5:10
5

strtok_r is a thread safe version of strtok on POSIX systems

strtok_s is a buffer overrun safe version of strtok on Windows. The standard strtok on windows is thread safe, so strtok_s should be.

5
  • strtok_r is part of POSIX and has been for years (decades?) Commented Jan 26, 2012 at 16:36
  • I still think the answer isn't terribly useful since it just covers what OP (mostly) already knew, not how to solve the problem. Commented Jan 26, 2012 at 16:45
  • @R - the solution is you can use strok_r on Posix and strtok_s on Windows, Joachim's solution describes how. OR you can program windows using only the Posix subsystem Commented Jan 26, 2012 at 16:46
  • @MartinBeckett it's C11 Annex K standard, not just Windows :) Commented Dec 20, 2017 at 6:13
  • 3
    @AlexanderRiccio Microsoft strtok_s and C11 strtok_s are completely different! Microsoft strtok_s has only 3 parameters, while C11 strtok_s has 4 parameters. The prototype of Microsoft strtok_s is char* strtok_s(char* str, const char* delimiters, char** context); while the prototype of C11 strtok_s is char *strtok_s(char *restrict str, rsize_t *restrict strmax, const char *restrict delim, char **restrict ptr);.
    – Victor
    Commented Feb 15, 2019 at 3:11
1

MinGW also predefines _WIN32 but it supports strtok_r, so I don't think it is a good idea to check the _WIN32 macro. It is better to check the _MSC_VER macro, which is the macro for Microsoft Visual Studio.

#ifdef _MSC_VER
#define strtok_r strtok_s
#endif

WARNING: Microsoft strtok_s and C11 strtok_s are completely different! Microsoft strtok_s has only 3 parameters, while C11 strtok_s has 4 parameters, so it can be a compatible issue in the future.

The prototype of Microsoft strtok_s is

char* strtok_s(char* str, const char* delimiters, char** context);

The prototype of C11 strtok_s is

char *strtok_s(char *restrict str, rsize_t *restrict strmax, const char *restrict delim, char **restrict ptr);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.