Anyway, I find out in the project info that if n<100, then O(n^2) is
more efficient, but if n>=100, then O(n*log2n) is more efficient.
Let us start by clarifying what is Big O
notation in the current context. From (source) one can read:
Big O notation is a mathematical notation that describes the limiting
behavior of a function when the argument tends towards a particular
value or infinity. (..) In computer science, big O notation is used to classify algorithms
according to how their run time or space requirements grow as the
input size grows.
Big O
notation does not represent a function but rather a set of functions with a certain asymptotic upper-bound; as one can read from source:
Big O notation characterizes functions according to their growth
rates: different functions with the same growth rate may be
represented using the same O
notation.
Informally, in computer-science time-complexity and space-complexity theories, one can think of the Big O
notation as a categorization of algorithms with a certain worst-case scenario concerning time and space, respectively. For instance, O(n)
:
An algorithm is said to take linear time/space, or O(n) time/space, if its time/space complexity is O(n). Informally, this means that the running time/space increases at most linearly with the size of the input (source).
and O(n log n)
as:
An algorithm is said to run in quasilinear time/space if T(n) = O(n log^k n) for some positive constant k; linearithmic time/space is the case k = 1 (source).
Mathematically speaking the statement
Which is better: O(n log n) or O(n^2)
is not accurate, since as mentioned before Big O
notation represents a set of functions. Hence, more accurate would have been "does O(n log n)
contains O(n^2)
". Nonetheless, typically such relaxed phrasing is normally used to quantify (for the worst-case scenario) how a set of algorithms behaves compared with another set of algorithms regarding the increase of their input sizes. To compare two classes of algorithms (e.g., O(n log n)
and O(n^2)
) instead of
Anyway, I find out in the project info that if n<100, then O(n^2) is
more efficient, but if n>=100, then O(n*log2n) is more efficient.
you should analyze how both classes of algorithms behaves with the increase of their input size (i.e., n) for the worse-case scenario; analyzing n
when it tends to the infinity

As @cem rightly point it out, in the image "big-O
denote one of the asymptotically least upper-bounds of the plotted functions, and does not refer to the sets O(f(n))
"
As you can see in the image after a certain input, O(n log n)
(green line) grows slower than O(n^2)
(orange line). That is why (for the worst-case) O(n log n)
is more desirable than O(n^2)
because one can increase the input size, and the growth rate will increase slower with the former than with the latter.