Questions tagged [data-scrubbing]

The process of detecting and correcting (or removing) corrupt or inaccurate records from a data set

data-scrubbing
Filter by
Sorted by
Tagged with
0 votes
0 answers
27 views

Spacy using all available CPU when running via Docker

I'm using scrubadub_spacy to clean text documents. When processing a document in a Docker environment built on amazonlinux, it maxes out the 8 cores on my MacBook and in Amazon Fargate. I'm confused ...
Dan's user avatar
  • 1
0 votes
0 answers
10 views

Dataload Error - Insert failed, Can Upsert work?

I was doing an insert dataload yesterday and some of the records failed. I still need to insert about 1300 ids. Is it possible to do an upsert now for the same file? I dont want to sort the ones that ...
user134185's user avatar
0 votes
1 answer
26 views

Unnesting an R dataframe from a JSON string in .txt file

I am new to using data in JSON strings. I have quite a similar issue to @Christopher here (Parsing JSON arrays from a .txt file in R - several large files) with the US Energy Information ...
Davidmac's user avatar
0 votes
0 answers
79 views

How can you use R to clean an excel(Xls) data source?

I have an excel file that has this shape I need to format this data so that it looks like this In the second image the second column is just the month written as a number. and the concatenation of ...
Aitor Parrilla's user avatar
1 vote
1 answer
208 views

Pandas calculating time deltas from index

I have a months time series data that I am trying calculate total hours, minutes, seconds in the dataset as well as for a unique Boolean column for when the column is True or a 1. And for some reason ...
bbartling's user avatar
  • 3,370
0 votes
1 answer
402 views

Fill missing values in dataframe in R

I have the following problem: I have a dataframe with several columns. (See below) I am trying to fill in missing values. Concretely, I only want to fill in values when I have a datapoint before and ...
Max 's user avatar
  • 45
-3 votes
1 answer
163 views

Convert Entire Dataframe to a column and insert into a table

Currently my data frame looks something like this A B C D 1 2 2 3 4 5 5 6 I want some thing like this Insert_date Bulk_load 09-09-2022 [[A, B,C,D], [1,2,2,3], [4,5,5,6]] ...
razs's user avatar
  • 1
-1 votes
2 answers
62 views

How do I get all the links from multiple web pages in python?

enter image description here from bs4 import BeautifulSoup from urllib.request import Request, urlopen #import re req = Request("https://www.indiegogo.com/individuals/23489031") html_page =...
Navreet Saini's user avatar
1 vote
1 answer
168 views

How to use BeautifulSoup to find specific class elements on a web page

Goal: To perform a web search that looks up a business and from the results, looks for either a "Permanently Closed" text or "Open" with hours or basically anything BUT "...
spareTimeCoder's user avatar
1 vote
0 answers
133 views

How to extract table and text from docx?

I am working on extracting text and tables from Docx files using pydocx library. I have to extract text and tables separately from the doc file that is creating issue of linking tabular data with text ...
Hamza Shaikh's user avatar
0 votes
2 answers
76 views

Python/ Beautiful Soup Data Displaying Issue [duplicate]

I am trying to pull some data from a website. Once I checked the data that I pulled with beuatifulsoup (using print(soup) in the code below) does not seem very well. It is different than once I check ...
marista's user avatar
  • 35
0 votes
1 answer
382 views

Find if integer exists within a list of ranges

Given an array N of 1,000,000 unique integers ranging from 0 to 1,999,999. What is the fastest way to filter out integers that do not exist within any range inside of M - where M is a fixed group of ...
fastmath's user avatar
0 votes
3 answers
674 views

Fastest String Filtering Algorithm

I have 5,000,000 unordered strings formatted this way (Name.Name.Day-Month-Year 24hrTime): "John.Howard.12-11-2020 13:14" "Diane.Barry.29-07-2020 20:50" "Joseph.Ferns.08-05-...
fastmath's user avatar
0 votes
1 answer
2k views

ceph pg repair doesnt start right away

Every now and then i get a single pg inconsistency error on my cluster. As suggested by the docs I run ceph pg repair pg.id and the command gives "instructing pg x on osd y to repair" seems ...
Nyquillus's user avatar
  • 189
3 votes
5 answers
14k views

Ceph PGs not deep scrubbed in time keep increasing

I've noticed this about 4 days ago and dont know what to do right now. The problem is as follows: I have a 6 node 3 monitor ceph cluster with 84 osds, 72x7200rpm spin disks and 12xnvme ssds for ...
Nyquillus's user avatar
  • 189
0 votes
1 answer
231 views

Python pandas if column value is list then create new column(s) with individual list value

I'm using pandas to create a dataframe from a SaaS REST API json response and hitting a minor blocker to cleanse the data for visualization and analysis. I need to tweak the python script by adding a ...
Wave Chappelle's user avatar
2 votes
1 answer
3k views

How to read data from a PDF using SAS Program

Problem Statement: I am unable to read data from a PDF file using SAS. What worked well: I am able to download the PDF from the website and save it. Not working (Need Help): I am not able to read the ...
anil kumar's user avatar
1 vote
3 answers
1k views

How to scrape data from a web page using SAS

Problem statement: I am required to get the data from web and put it into SAS dataset by using SAS Program. Worked well: I am able to fetch the contents of the target web page by SAS. Not working (...
anil kumar's user avatar
0 votes
0 answers
144 views

How to get scrape specific URL from multiple URL in Webpage Java

I am doing data scraping for the first time. My assignment is to get specific URL from webpage where there are multiple links (help, click here etc). How can I get specific url and ignore random links?...
Alam Mahmood's user avatar
2 votes
3 answers
940 views

How to extract user ratings from a movie dataset

This screenshot is the sample of the merged movielens dataset, I have two questions: If I want to extract only user 191 movieid, title, genres and ratings alone, how will I do this? How can I list ...
Adeolu Temitope Olofintuyi's user avatar
-2 votes
3 answers
967 views

Can anyone suggest data science project [closed]

I just completed data science topic i want to fo a mini project in data science can anyone suggest me a project . I searched a lot in google but i not found any projects can anyonr suggest a project
Bala Praveena's user avatar
0 votes
2 answers
163 views

Clean data to be imported into Neo4J database

I am a Neo4j and data analytics noob here. I am looking for programmatic way to format data that I collect from Active Directory to have it prepared to be imported into Neo4j. Right now, I am using ...
POSH Geek's user avatar
  • 174
1 vote
3 answers
203 views

How to scrub date of birth in java in a random way which results in same random number generated when i feed the same original date of birth

I am trying to do data scrubbing, where I am trying to scrub date of birth field, but I want it to be consistent in a way, that the same random number or date of birth be generated for the same input ...
Shaun's user avatar
  • 11
0 votes
1 answer
44 views

Importing data from website with subscription

The website I import data from is now subscription-based, I have a subscription but HTML import function doesn't pull data. =IMPORTHTML("https://www.footballoutsiders.com/premium/defense-vs-receivers?...
Ricky's user avatar
  • 11
2 votes
1 answer
67 views

How to slice a part of a string in DF when you don't know exact position?

I'm struggling with slicing. I thought that generally it's easy and I understand it but when it comes to the below situation my ideas don't work. Situation: In one of my columns in DF I want to ...
QbS's user avatar
  • 454
0 votes
1 answer
23 views

Data Scrub off Interactive Map - Cal Fire related

http://calfire-forestry.maps.arcgis.com/apps/webappviewer/index.html?id=5306cc8cf38c4252830a38d467d33728&extent=-13547810.5486%2C4824920.1673%2C-13518764.4778%2C4841526.1117%2C102100 how can i ...
James's user avatar
  • 69
0 votes
0 answers
512 views

ZFS without ECC, how checksum would work?

assuming I have read a lot about ZFS with/without ECC, there are quite a few opinions online.. I have still doubts that I could not clarify myself reading the available documents. Suppose I have two ...
Tiutto's user avatar
  • 179
1 vote
2 answers
125 views

Shiny App R - Scrubbing and Error

I'm building a Shiny App in R and I'm trying to scrub off the web information about the user's selected Pokemon, but I keep running into the problem of 'Error: SLL certificate problem' when trying to ...
Donald Johnson's user avatar
1 vote
1 answer
1k views

Group duplicate columns and sum the corresponding column values using pandas [duplicate]

I am preprocessing apache server log data. I have 3 columns ID, TIME, and BYTES. Example: ID     TIME     BYTES 1     13:00     10 2   &...
user avatar
-2 votes
1 answer
92 views

Form submit with multiple redirection

I'm trying to fetch data from a website where once you submit the form it redirects to a loading page which is set to be automatically redirected to the final results page. The issue is that the ...
Mehran's user avatar
  • 185
0 votes
2 answers
7k views

How to scrub data in Excel, specifically removing extreme outliers that are outside of a given range?

I have thousands of cells of data that are output from a model, with my results formatted as follows: cell ID is column header, each row is a timestep, and each cell's results over hundreds of ...
cr0's user avatar
  • 73
2 votes
1 answer
3k views

Pandas: How can I convert 'timestamp' values in my dataframe column from object/str to timestamp?

My timestamp looks like below in the dataframe of my column but it is in 'object'. I want to convert this into 'timestamp'. How can I convert all values such in my dataframe column into timestamp? 0 ...
jubins's user avatar
  • 317
-3 votes
1 answer
2k views

SQL Server Iterating through a table and scrubbing sensitive data

I have a table that contains the names of tables and fields that need to be scrubbed. I need to loop through this table and scrub the field name in each table. I would also like any suggestions on a ...
Joe's user avatar
  • 1
0 votes
1 answer
88 views

Extracting specific patient IDs from email subject line

I'm looking to extract patient IDs from an email subject line. I'm working with two data frames: One has output from an SQL database (contains email subject line) and the other has patient information ...
j.reyes's user avatar
0 votes
2 answers
63 views

R editing dataframe based on column value

Suppose I have a table of around 1M rows with the following format: id paid_2000 paid_2001 paid_2002 paid_2003 censor_yr 1 10 20 10 20 2001 2 15 25 ...
George's user avatar
  • 55
0 votes
1 answer
808 views

phantomjs always return page not found on a particular website

im trying to get sport matches resul from this website http://www.oddsportal.com but any address from this website that i try i end up with page not found while it opens in my own browser here ...
max's user avatar
  • 3,706
1 vote
0 answers
75 views

Changing values of numbers in CodeMirror by using scrubbing addon

I'm having trouble with CodeMirror. I'm trying to add in live number scrubbing, similar to Brett Victor's example, and Khan Academy's capability, but I am not having too much luck. I can't post ...
TorranceY's user avatar
0 votes
1 answer
269 views

Data Scrubbing In EXCEL

This might be an easy one for all of the computer wiz out here! I would like to extract a string of characters in a EXCEL text box. The dataset is kind of messy, there is space, characters, and other ...
Kevin Liu's user avatar
2 votes
2 answers
316 views

Anonymize names in paragraph variable by matching and replacement

I am analyzing a school's student report card database. My dataset consists of around 3000 records structured similarly to the example below. Each observation is one teacher's assessment of one ...
Anders Swanson's user avatar
1 vote
1 answer
419 views

SQL Update on Azure ML Not Working?

I'm trying to clean some data in Azure ML. I have an Apply SQL Transform block with the following code in it: UPDATE t1 SET CreditScore = -1 WHERE CreditScore>900; It is a numeric column. ...
W. Christopher Moses's user avatar
0 votes
1 answer
2k views

Cassandra 3 Repair never finishes

We have a cluster with 6 nodes in datacenters (3 nodes each). We are starting a repair on one node and shortly afterwords we can find something like this in the logs: ERROR [Repair#1:1] 2016-05-31 01:...
Mike Petersen's user avatar
-1 votes
2 answers
170 views

Perl "scrub" characters while parsing

I'm parsing through a file - first thing I do is concatenate the first three fields and prepend them to each record. Then I want to scrub the data of any colons, single quotes, double quotes or ...
BigRedEO's user avatar
  • 837
0 votes
1 answer
114 views

Pivot a table 4 rows at a time

I'm trying to figure out how to pivot a table in SQL Server in the following way: Every time the value is "PEAK", I want it to start on a new row. Essentially it would look like this. col1 col2 ...
SQLSuperHero's user avatar
0 votes
1 answer
67 views

Javascript pull data from website (member site)

I haven't written any code for this yet as I am trying to figure out how to do it first. I have a site at work that I have to log into with a password. Its a site that we use to manage our lanes (I ...
Robert E DelValle's user avatar
42 votes
3 answers
67k views

What is python's equivalent of R's NA?

What is python's equivalent of R's NA? To be more specific: R has NaN, NA, NULL, Inf and -Inf. NA is generally used when there is missing data. What is python's equivalent? How libraries such as ...
power's user avatar
  • 1,730
0 votes
2 answers
75 views

What do I do to understand an online database to get query results into a spreadsheet?

So very simply, I'm looking at a web query for this database of doctors: http://www.aahivm.org/ReferralLink/exec/frmAdvSearch.aspx When I search for, say, HIV Primary Care doctors, by state, I get a ...
Christopher Edwards's user avatar
1 vote
1 answer
1k views

How to load dynamically generated webpage?

I am trying to load the webpage, http://www.artstation.com/artist/nicotine, so I can scrub the page, unfortunately the page seems to be generated via code so the tags that I am looking for aren't ...
Chris L's user avatar
  • 103
3 votes
2 answers
2k views

How to display preview thumbnail while scrubbing the video.

I am trying to display the preview thumbnail when user move his finger over video scrubber. The only solution I m finding is to extract thumbnails using some 3rd party tool and save it to server or ...
Umair's user avatar
  • 1,236
2 votes
3 answers
502 views

Need Better Algorithm to Scrub SQL Server Table with Java

I need to scrub an SQL Server table on a regular basis, but my solution is taking ridiculously long (about 12 minutes for 73,000 records). My table has 4 fields: id1 id2 val1 val2 For every group ...
Palladium's user avatar
1 vote
0 answers
34 views

Alternatives to Coding Downloader Programs

At my job we regularly need to grab data from external sources, whether that be via ftp, sftp, e-mail scraping, web services, or web scraping. The formats vary from screen scraping/parsing, to CSV, ...
Jon Gaudette's user avatar