9

What is the difference between Tokenization and Segmentation in NLP. I searched about them but I didn't really find any differences .

1 Answer 1

4

Short answer: All tokenization is segmentation, but not all segmentation is tokenization.

Long Answer:
While segmentation is a more generic concept of splitting the input text, tokenization is a type of segmentation and it is carried out based on a well defined criteria.
For example - in a hypothetical scenario if all your input sentences are compound sentences of two sub-sentences, then splitting them into two independent sentences can be termed as segmentation (but not tokenization).
Tokenization is a form of segmentation which is performed on the basis of a semantic criteria or using a token dictionary - e.g. a word or sub-word tokenization, mainly with an intention of assigning them token ids for downstream processing.

2
  • could you please give me a real world example for more clarification Commented Nov 20, 2021 at 20:15
  • 4
    Breaking your text corpus into sentences is segmentation, but not tokenization. Using sub-words of a sentence for generating token ids as input to a transformer model is tokenization (hence also segmentation)
    – jdsurya
    Commented Nov 20, 2021 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.