322
questions
-1
votes
1
answer
37
views
Automatic Word Boundary Detection for German
I want to rephrase that: I need a corpus of German words so that I can check if a segment is a word. My solution so far is to take the string, check if it's in the dictionary and if not, delete the ...
0
votes
0
answers
38
views
Can log2 be substituted with ln in logDice association measure in R?
I have a question about logDice association measure for collocational analysis.
This is the formula for logDice:
logDice = 14 + log2(2.w1w2/w1+w2)
Where:
w1w2 = the frequency of the word x and y
w1 = ...
0
votes
0
answers
20
views
Using numeric column in dataframe within formula in R
I am a newbie and trying to do a transformation from one unit to another using the numbers that make up a column in a dataframe that I'm using.
The formula is [(26.81x)/(1960+x)]-0.53.
I've tried a ...
1
vote
0
answers
66
views
Query Wikidata via SPARQL to get specific word etymology from Wiktionary
I'm trying to get the specific word etymology from Wikidata.
For example, this query to get the word "exact" in Wikidata but I wasn't able to get the etymology part for this word.
SELECT ...
-1
votes
1
answer
650
views
What does "assign A to B" mean?
If I say "assign A to B", does it mean (a) A ← B or
(b) B ← A?
In other words, is it (a) A or (b) B that is being modified?
(a) makes sense because A has responsibility over B, so A is ...
0
votes
1
answer
71
views
Problems with reproducing the training of the spaCy pipeline
I'm trying to reproduce the training of one of the spaCy pipeline for Italian language: it_core_news_sm.
This pipeline is trained on 2 datasets:
UD_Italian-ISDT for the conllu tasks
WikiNer for NET ...
1
vote
0
answers
117
views
In NLTK, how to generate a sample of sentences from PCFG, respecting the probabilities
NLTK has a generate method which enumerates sentences for a given CFG. It also has a PCFG class for probabilistic context-free grammars. Is it possible generate a sample of sentences with respect to ...
0
votes
1
answer
212
views
Weighted Distance Matrix for QWERTZ Keyboard for Levenshtein Distance Algorithm
I have a weight Matrix for a Levenshtein Distance Algorithm which looks like this
int[,] weights = new int[6, 6]
{
{ 0, 1, 2, 1, 1, 2 },
{ 1, 0, 1, 2, 1, 2 },
{ 2, 1, 0,...
0
votes
1
answer
334
views
How to develop a corpus(corpus analysis)
I am goingt to build a linguistic corpus, but i don't understand which technologies should i use for it. Is it true, that for developing a courpus for any language i necessarily have to use IMS Corpus ...
-1
votes
1
answer
66
views
How do I study linguistic features of NLP libraries like spacy/NLTK in-depth? [closed]
My Objective
I am trying to do Natural Language Processing(NLU to be precise) with spaCy. But the more I dive into this wonderful library, the more I realize that I need to understand some linguistics ...
0
votes
1
answer
96
views
Tool for detecting differences between text passages from two different groups
I have text data from two different groups. In total I have around 4000 text passages with around 300 words.
I am searching for a tool that allows me to analyze the difference between these two groups....
0
votes
0
answers
26
views
R - readtext and list of .xml files
I'm trying to create a corpus and a vcorpus with a bulk of .xml files, for quantitative linguistics
With txt files I usually write
library(tm)
library(stopwords)
library(magrittr)
library(dplyr)
...
2
votes
0
answers
56
views
How can I determine if a word is a part of an english word or is a portmanteau (a word created by combining parts of valid English words)?
I am trying to create a validator that takes in words and tries to determine if the word is one of the following:
It is a valid English word
It is a part of an English word
It is an abbreviation
It ...
0
votes
0
answers
42
views
Reducing time for minimum edit distance in python?
I am trying to create a list with the edit distances between each word in a set of documents, ranging from 10k-42k words. If my idea of edit distance is correct, I would end up with a distance for ...
1
vote
0
answers
156
views
Customization of Wav2Vec2CTCTokenizer with rules
my goal is to fine-tune an ASR model, WavLM, that relies on the pretrained tokenizer Wav2Vec2CTCTokenizer.
I want to fine-tune this ASR model with another language and to perform the tokenization ...
1
vote
0
answers
289
views
spaCy custom tokenizer to separate word with underscore and also to include the whole word
After referring to the link: How to tokenize word with hyphen in Spacy
I got to know how to tokenize by separating words containing hyphen/underscore but my requirement is to tokenize by separating it ...
1
vote
1
answer
1k
views
How to do the post hoc test in the linear mixed model if I have three predictors (two factor variables and one numeric variable)
I'm using a linear mixed effects model to analyze the reaction time of learners of English as a second language. I have two factor variables - grammaticality (grammatical v.s. ungrammatical) and ...
2
votes
2
answers
200
views
Can I use a vector as a regex pattern parameter in R?
I want to search a phonetic dictionary (tsv with two columns, one for words, another for phonetic transcription: IPA) for certain consonant clusters according to the type combination (e.g. fricative+...
1
vote
0
answers
54
views
How do I recognize (spaceless) English text?
[Note. This question has mild spoilers, but no answers, for a crossword puzzle in the May 2022 issue of The MagPie.]
In the puzzle, we are given a 52-character string. It is encrypted using Playfair,...
1
vote
1
answer
61
views
How to implement the theory of copy movement in Prolog?
Using this input:
parse(Parse, [what,did,thomas,eat], [])
I want to produce this output:
sbarq(
whnp(
wp(what)
),
sq(
vbd(did),
np(nnp(thomas)),
vp(
vb(eat),
whnp(
...
1
vote
1
answer
478
views
How to turn an adjective into a noun in python (pymorphy2)?
I have a word in Russian (which is actually where the difficulty arises). It's an adjective. And I need to make it in a noun form.
I found an interesting library that can parse words, inflect and ...
0
votes
1
answer
40
views
Algorithm to rank the simplicity of a random name
I have been looking for a name for a new project. I want the name to have available domains and social media handles. For months, all those I can think of are taken.
So I generated a list of names ...
-2
votes
1
answer
188
views
Tips for writing an algorithm for paraphrasing sentences(machine learning)
I am doing a project at the university and I need to train an algorithm to rephrase sentences, what can you advise for implementation? Is it possible to use a translator to translate into another ...
1
vote
2
answers
731
views
Relationship between vocab size and complexity
I have 2 corpuses, if one has a larger vocabulary size than the other, does it mean its language is more complex?
Apart from complexity of the language, what else can effect the size of the vocabulary ...
0
votes
1
answer
248
views
Draw an FSA that recognizes: (A∗ | AB+). (The bar outscopes the other operators, so its equal to: (A∗) | (AB+).) Use as few states possible
I've attached what I have. My problem is that I don't know if its correct and if I've even used the fewest states possible to answer this question. Really appreciate any help on what I currently did ...
0
votes
0
answers
52
views
Generating pronounceable digrams in Python
I am generating all two letter (digram) combinations of the standard alphabet
import string
import itertools
UPPERCASE_ALPHABET = string.ascii_uppercase
TWO_LETTER_PERMS = itertools.permutations(...
0
votes
1
answer
138
views
Chinese Whispers for NLP How to implement for my corpus file
I'm new to Python, I want to make this code to implement my corpus from (.csv)files Can anybody help me how to implement it like in the picture below.
from chinese_whispers import __version__ as ...
1
vote
2
answers
461
views
Sentence tokenization w/o relying on punctuations and capitalizations
Is there a possible approach for extracting sentences from paragraphs / sentence tokenization for paragraphs that doesn't have any punctuations and/or all lowercased? We have a specific need for being ...
0
votes
1
answer
73
views
Can Mandarin pronunciation be coded by ARPABET phone set?
The code doesn't need to be exactly correct. A pronunciation in English can be used to represent a similar pronunciation in Chinese, e.g. /ʈ͡ʂ/ can use "CH" to represent.
1
vote
2
answers
639
views
Python - How to loop through each index position in a list?
Given a list [[["source1"], ["target1"], ["alignment1"]], ["source2"], ["target2"], ["alignment2"]], ...] , I want to extract the words in ...
2
votes
0
answers
49
views
Entropy Calculation as a Linguist Feature
I was going through a paper on Sarcasm Detection which proposed a Hybrid Attention Based LSTM. The Model included some linguistic features to be extracted first, which included Entropy of a word in a ...
0
votes
1
answer
265
views
What kind of encoding is this html encoding?
I am doing a project that involves searching words in the Arabic script on Wiktionary, and when I do a GET request on certain word pages, I get something like this for example:
title="\xd8\xb1\...
0
votes
0
answers
56
views
Identifying phrases which contrast two corpora
I would like to identify compound phrases in one corpus (e.g. (w_1, w_2) in Corpus 1) which not only appear significantly more often than their constituents (e.g. (w_1),(w_2) in Corpus 1) within the ...
0
votes
1
answer
400
views
Convert lower case letter 'ß' to upper case letter 'ẞ' in PHP [duplicate]
I'm trying to convert the lower case letter 'ß' to the upper case pendant. I already tried using the built-in functions strtoupper and mb_strtoupper.
The following happened when I use strtoupper:
echo ...
1
vote
0
answers
198
views
Google Natural Language Syntax Analysis
I'm not sure this is a coding issue/question.
I'm using Google's NLP to analyze the syntax of some sentences and I'm seeing some inconsistencies with Plural vs Singular designation. Perhaps I'm doing ...
1
vote
0
answers
88
views
NLTK.sem.logic cannot infer logical types of functions given types of argument and result, why?
I am working with NLTK.logic.sem to handle the logical structure of sentences. I also use NLTK to handle inferring the logical types of expressions. The NLTK book has an example of when it cannot ...
1
vote
2
answers
941
views
Find a word in a list of words that has minimum distance with a given word
let say given a word list ['windows','hello','python','world','software','desk'] and an input word 'widow', how to (quickly) find the word from the word list that has the minimum edit distance with ...
0
votes
1
answer
63
views
Removing from a text sequences of items from a list
I'm conducting research on code-switching. I have a collection of bilingual Polish text messages with English code-switches (say, corpus A) as well as an English dictionary (also a list, corpus B). I ...
3
votes
3
answers
844
views
PROLOG - Get a list of all the rules an entity verifies
I'm formalizing linguistic data into predicates and entities and doing some reasoning in prolog. Imagine I begin with:
breathe(X) :- snore(X).
sleep(X) :- snore(X).
rest(X) :- sleep(X).
live(X) :- ...
1
vote
3
answers
96
views
Negative lookbehind in RegEx: Matching multiple POS-tags at once
I am still fairly new to regex, so I would appreciate any help.
I am trying to use regular expressions to find specific grammatical patterns in a text corpus that was part-of-speech-tagged using the ...
0
votes
0
answers
128
views
NLTK - Access Brown corpus as one file
I have recently started working with NLTK, and I have downloaded the Brown corpus to do analyze its data. However, I see that the corpus is split into around 500 files.
I need to run a script on the ...
1
vote
1
answer
215
views
How to determine structure of answer for a WH-question
Consider a wh-question such as "Who closed the door?". Personally, I can determine that an answer will look like "NP closed the door.", where NP would be a noun phrase. Another ...
0
votes
1
answer
727
views
Why mono sounds are preferred over stereo sounds when it comes to analyse acoustic parameters like intensity, RMS amplitude, fundamental frequency etc
In phonetic analyses, while using software like Praat, I have seen people's liking mono sounds over stereo sounds. Even if the recordings were done using a stereo microphone, the audio files are often ...
4
votes
1
answer
2k
views
Distance between strings by similarity of sound
Is the a quantitative descriptor of similarity between two words based on how they sound/are pronounced, analogous to Levenshtein distance?
I know soundex gives same id to similar sounding words, but ...
0
votes
2
answers
309
views
Finding the rhyming words in a corpus with R, regex
I have the following corpus:
corpus_rhyme <- c("helter-skelter", "lovey-dovey", "riff-raff", "hunter-gatherer",
"day-to-day", &...
0
votes
2
answers
71
views
Tartar language and UTF-8
I'm working on a project that includes transforming latin symbols into the corresponding cyrillic ones. I'm talking about Tartar language which is used by one of many nations in Russia. I tried find ...
1
vote
0
answers
30
views
What is the more natural parsing, the one that leads to the preferred reading of the sentence
I have those rules:
and those two possible parse trees:
I am asked for the next question:
What is the more natural parsing, the one that leads to the preferred reading of the sentence?
Can anyone ...
0
votes
1
answer
25
views
Get rows from MySQL table that have a column in common but slightly diffrent value [duplicate]
Might not even be a programming question, but the problem I'm having is I have a big MySQL table with products from different sources (stores) and in many cases it's the same product with different ...
3
votes
1
answer
367
views
Python regex selection of verbs with present perfect
In a given string, I am trying to catch verbs that are in present pefect tense. I do that by using the following regular expression in python:
import re
sentence = "The Batman has never shown his ...
0
votes
1
answer
67
views
To remove non- pair entries in a dataframe?
A column 'Consonant' in my dataframe 'df' has two levels with numbers, 250 for 'Geminate and 214 for 'Singleton'. Because Geminates are more in number, how can I remove the extra entries for Geminates,...