Questions tagged [text-processing]
Mechanizing the creation or manipulation of electronic text.
1,968
questions
-3
votes
0
answers
27
views
how to extract line items from unstructured invoice pdf and make datafarme [closed]
I want to extract line items from invoice pdf, items name and description may be multiline.
here is my extracted text
text = """
Resistor 0 25 1 OHM RESISTOR Edgewound 11.48 287.00
EWR2-...
-3
votes
1
answer
90
views
Extract N lines with no duplicate strings from either of the two first columns
I have a huge CSV file with pairs of unique combinations. I need to extract N random lines of this file, but I need that the strings in col 1&2 to be unique, so that the list of unique strings in ...
-1
votes
0
answers
10
views
How to process texts on PC screen in real time?
I want to process the texts on PC screen in real time by following the mouse pointer when I have multiple monitor setup or the primary screen when I have only one monitor. So, I had the idea that I ...
1
vote
1
answer
40
views
Using a Word Counter in Python is understating results
As a complete preface, I am a beginner and learning. But, here's the sample schema of my products review table.
Record_ID
Product_ID
Review Comment
1234
89847457
I love this product it was shipped ...
0
votes
0
answers
42
views
How to Add Product ID Search Functionality to Python Word Counter and Display Associated Words?
I have a very large list of Movie Reviews. This is my schema sample...it's about 4000 rows, so there's a lot going on.
MovieID
FreeformComment
1234
This was a great movie it made me happy
5678
This ...
-2
votes
1
answer
41
views
How to merge some elements in a python list based on some features
This is a list, each elements consists of two strings and a "/t" between.We can call the string on the left "label" and the part on the right "text".
continued In the ...
-2
votes
1
answer
50
views
Sed copy first string and prepend it to line [duplicate]
I have a file:
Text1 text2 50
Text3 text4 60
I would like to make the following using sed command:
Text1 Text1 text2 50
Text3 Text3 text4 60
I need to copy the first string and add it to the line.
-2
votes
2
answers
106
views
How can I find out the number of tabs at the beginning of each line in a text file?
I have a text file, where each line may start with a number of tabs, including no tabs. For example, the first line starts with no tab, the second line with 1 tab, and the third line with 2 tabs:
...
1
vote
2
answers
103
views
Athena/Trino/Presto code to parse text file with custom line separator
I want to parse some log files in plain text file with records spanning across multiple lines using Trino/Presto code. My data looks like this: each record has multiple lines and each line has a ...
1
vote
1
answer
100
views
SAM alignment: Extract specific region in the query sequence and its enclosing parts in the CIGAR string
I need to perform the local alignment of a given region of a DNA sequence for which a global alignment was made, and update the corresponding part of the global CIGAR string.
The steps would be as ...
0
votes
1
answer
57
views
How to reliably intercept (and cancel) all the input in contenteditable?
I need to catch all the input in my contenteditable element and process the input in the code. Basically, what I need is to prevent input event from changing data in element and to know what data was ...
1
vote
1
answer
109
views
Text processing with Python
I need to extract 1,500,000 out of 25,000,000 records and group them.
The groups and the UUIDs of the records to extract are defined in a separate file (200MB) with the following format:
>Cluster 0
...
-1
votes
1
answer
28
views
Processing through files in Python
Good morning fellow coders. I have a question regarding the use of finding specific lines in a file using Python. One method is to use if line.startswith(word) or to use if not line.startswith(word) ...
1
vote
1
answer
71
views
How to split an array of string nodes at given index?
I probably need to use other data structure, but I'm stuck with this solution for now. Will appreciate any advice on this.
For now I have this data structure:
const data = [
{
id: 'node-1',...
0
votes
1
answer
29
views
How to 'normalize' an array of objects?
I am not sure if the term 'normalization' is correct in my case, so I would appreciate if you point me at the right one. I found this term in DOM node method called normalize, which is doing kinda the ...
2
votes
3
answers
224
views
AWS Textract OCR reading PDF as single line instead of preserving line breaks
Hi I am new to AWS Textract.
I am using Amazon Textract to extract text from a PDF file. However, the output is not preserving the line breaks from the original PDF.
For example, in the PDF there are ...
0
votes
1
answer
47
views
Conversion of data to perform logistic regression
I have a dataset which contains the data of cricket matches from 2010 to 2023. I have attached a sample of the data for reference:
Team_1
Team_2
Ground
Matchdate
Bangladesh
Srilanka
Mirpur
2010-01-...
3
votes
3
answers
44
views
Counting string occurrences in second column which corresponds to first columns of a file [closed]
I have this Input text file:
CD196_RS15035 normal alleles
CD196_RS15035 normal alleles
CD196_RS15035 truncation in the allele
CD196_RS15035 truncation in the allele
...
1
vote
0
answers
74
views
How to handle ligature issue while using pdf text
I need to capture some text from some PDFs. I use PymuPDF to do this. But facing ligature issue while writing those selected text inside a text file.
I use the following code snippet to read the PDF
...
-2
votes
1
answer
55
views
How to compress multiple lines into single/fewer lines using regular expression or wildcard character?
I want to compress multiple lines of string having some common elements into single or fewer lines by replacing the changing values with wildcard character.
Example
Input:
Lorem ipsum dolor sit amet_0
...
1
vote
1
answer
91
views
Can't find model 'en_core_web_lg'. It doesn't seem to be a Python package or a valid path to a data directory. Eventhough they are in same directory
I am trying different text processing model. I am trying to use spacy and it's model en_core_web_lg.
import spacy
import spacy.language
from spacy_langdetect import LanguageDetector
from spacy....
0
votes
3
answers
88
views
Replacing field values in a fixed-position file using awk
The ask here is to replace position 9-12 with a blank, for example, in all fixed-sized records.
I used
cat file | awk '{sub(substr($0,9,12)," ",$0);print}'
Works, unless there is a blank ...
2
votes
1
answer
68
views
using awk to identify disease for patients
I have a tab seperated text file A.tsv of icd code matrix which the columns are patient id and icd codes, rows are observations for each patients. NA indicate the patient are not diagnosed as the icd ...
0
votes
0
answers
45
views
Easy way shortened text in EditText
I'm looking for customizable shortcut text in Android Java. I tried this:
@Override
public void onCreate(Bundle savedInstanceState) {
String strxd = "Hello World XD";
int xdindex = ...
0
votes
0
answers
40
views
Extracting phrases and words with Apache Tika
I'm facing an issue with the extracting texts and phrases using Apache Tika.
I made a unit test where it seems to be missing some expected elements and has additional unexpected elements in the actual ...
0
votes
0
answers
22
views
Looking for a tool to search acronyms and find their alternative funny meanings
I'm searching for a tool or library that can help me search for acronyms in general and find their alternative funny meanings.
For example, the tool should be able to recognize "BBC" as an ...
-1
votes
1
answer
72
views
How can I use grep in Linux to find lines with multiple matching conditions in a text file?
I'm working on a Linux system and need assistance in finding specific lines within a large text file using the grep command.
The criteria for these lines are:
Inclusion of either 'term1' or 'term2'.
...
0
votes
0
answers
26
views
Tools for finding words and idioms that can be humorously misconstrued
I am looking for tools or libraries that can help me identify words and idioms that can be intentionally misconstrued to mean something else, resulting in humor. Here's an example from a video clip ...
0
votes
1
answer
59
views
How to recognise the word as a single word in OCR that end of the line with a hyphen
I am using ORCmyPDF. How can I recognise end-of-the-line words that break with the hype? And remember them as one word.
This will increase my search result with that words.
It will make my PDF ...
0
votes
2
answers
54
views
Remove an item from Python list if the item's dictionary key has some value
I have a list of dictionaries.
for i, d in enumerate(test_list):
print(i, d)
0 {'role': None, 'content': 'Closing the gap between actual and potential crop yields', 'bounding_regions': [{'...
0
votes
1
answer
37
views
Processing dataframe using multiprocess/multithread
I have a large pandas dataframe with one column "sentence" that contains text (each entry might be around 100 words and there are around 200000 entries). I would like to make a dictionary ...
0
votes
0
answers
39
views
Stripping Headers/Footer from Plain Text Email Content
Has anyone used/recommend any services/products/tools that have been successful at stripping header and footer content and chain replies from plain text e-mail content?
I have several years of e-email ...
0
votes
0
answers
66
views
Hastening URL decoding process in R
I am working in R and I am trying to make the function below computationally faster. I am trying to decode UTF-8 encoded URLs. Currently, we look for the characters in the raw URL, x, that have ...
-1
votes
2
answers
248
views
Automatic generation of dots in Notepad++ when pressing space
I am encountering an issue in Notepad++ where dots are automatically generated when I press the spacebar. This behavior is unexpected and causes inconvenience while working with text.
To provide some ...
0
votes
0
answers
5
views
Efficient method for extracting and categorizing frequent terms from unformatted string column in SQL table
I have a string column in a SQL table that contains job titles. This field is filled in by the user and therefore has no formatting.
To obtain a typology of users, I'd like to extract the terms that ...
0
votes
0
answers
20
views
'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128) : BookNLP
I am trying to run BookNLP model on a txt file in Colab, but I keep getting an error.
the code:
input_file="alice.txt"
output_directory="alice"
book_id="alice"
booknlp....
1
vote
1
answer
176
views
DJango: Textarea/Form get, process and show
Good evening, i'm currently working on a small DJango project, and am currently facing an issue:
The whole of the project is to provide a textarea to the user:
1- let the user write down the text that ...
1
vote
1
answer
123
views
How to split text into sentences and create a new dataframe with one sentence per row?
I have a dataframe df that has 3 columns containing speechdata: filename, president, text.
I split the text data into sentences using:
# Split 'text' column into sentences and create a new 'sentences' ...
0
votes
1
answer
102
views
How to combine multiple .txt files into a single dataframe?
I am trying to import 250 .txt files that are in one directory as one grouped data frame so that the dataframe consists of two columns with one column taking on the .txt file name and one column ...
0
votes
2
answers
62
views
Replacing a string unless it contains a given substring with sed
I have a file which contains many links of the form <a href="foo">. These need to be given a .html extension, and I have the following command to do so:
sed -i 's/\(href="\)\([^&...
2
votes
3
answers
84
views
Want to change prefix characters, then add suffixed characters in changed string
I want to create a poor-man's replace for HTML-izing a string text.
A sample text is:
THIS IS A TEST.@@01 THIS IS ANOTHER TEST.@@02
What I have been using (which works for both Mac and LINUX) is this:...
0
votes
1
answer
130
views
Standardize the text in the pandas column with some common string
I have following DataFrame df
id1 id2 text_column
key1 220 ABC corp
key1 220 ABC Pvt Ltd
key2 300 PQR Ltd
key2 300 PQR
key2 300 PQR something else
key2 ...
2
votes
1
answer
115
views
how to get scikit-learn's tokenizer to use punctuation to separate tokens?
I have the following code using scikit-learn to count ngram frequencies:
c = ["data. format", "data are format hello world"]
vectorizer = CountVectorizer(ngram_range=(1,2))
X = ...
-2
votes
1
answer
123
views
Removing html code from text data in Spark
i'm working with Stack Exchange data dumps and I want to clean body of a question from code which occurs really often.
I tried using BeautifulSoup but it leaves the text which occurs inside the code ...
-1
votes
1
answer
80
views
How to keep blank lines while reading text file Python
Reading in a generic text file with blank lines, i.e. no tabs, no spaces along some lines. I want to use these blank lines as a delimiter to process the files in the program, but I cannot stop them ...
0
votes
1
answer
50
views
How to split words in a paragraph when there is a <a> tag with href attribute within the <p> tag
I am working on a web scraping project and I want to split the words in a paragraph when there is an <a> tag with href attribute within the <p> tag. I'm new to scraping and programming in ...
0
votes
0
answers
107
views
How to properly split streaming text into paragraphs?
I am using okhttp subscribing to a SSE( https://en.wikipedia.org/wiki/Server-sent_events) stream, the onEvent callback is use to receive new data from upstream and passed by data param as String
It ...
-3
votes
1
answer
102
views
Python : remove duplicate character
string input:
Tem1 = 'Hhelloo ookkee'
I want to make output like
Tem1 = 'helo oke'
I have try this link form stackoverflow
(Python: Best Way to remove duplicate character from string)
I've tried ...
1
vote
1
answer
42
views
Pull a value from a text file to use as a comparitor
I need the pull the temperature from a text file, and then use the value as a comparitor to run one or the other python script. the data looks like this:
time=14:10
temp=22.4 <----- I need this ...
0
votes
3
answers
42
views
find values that occurs the least in 2 columns file
I've a dynamically generated txt file of this structure -> Code[space]Link
This is a simple
15210 https://test1.com
4 https://test2.com
1020 https://testc.com
152 https://testz.com
152 https://...