Questions tagged [data-mining]
Data mining is the process of analyzing large amounts of data in order to find patterns and commonalities.
3,083
questions
-1
votes
0
answers
16
views
Want to Scrap google scholar to extract meta data related to genome editing in plants. How can I achieve this?
For academic purpose, I want to Scrap publications from Google Scholar related to the topic of "genome editing in plants" to extract specific metadata from the scrapped publications, such as:...
-1
votes
0
answers
13
views
How to perform backpropagation in feedforward multi-layer networks?
My given neural network is in the picture (https://i.stack.imgur.com/gg8E4.png). Each hidden neuron uses a sigmoid activation function σ(t) = 1/(1 + exp(−t)). The loss is the mean square error and the ...
-3
votes
0
answers
17
views
Searching for Cancer-Related Healthcare Datasets for AI Model Training [closed]
I'm in the process of developing an Artificial Intelligence (AI) model focused on healthcare, specifically cancer diagnosis, treatment, and patient outcomes. I am seeking datasets that can provide ...
-1
votes
2
answers
40
views
Python as_strided method, how does it work?
I am trying to do some anomaly detection in one of my assignments and I am trying to create sliding windows, but I want to make them so that one window does not overlap with the other. For example
...
0
votes
0
answers
15
views
Why is this .csv file not woking in Weka?
I am working on a data mining project using Python sci kit and Weka. The dataset I am using is text-file and I have converted it to CSV using Python (with Pandas library). Unfortunately when I try to ...
0
votes
0
answers
22
views
Decision Tree data as category
i want to draw a tree based on this data set:
Source Port,Destination Port,NAT Source Port,NAT Destination Port,Bytes,Bytes Sent,Bytes Received,Packets,Elapsed Time (sec),pkts_sent,pkts_received,Class
...
0
votes
0
answers
9
views
How to get K-means unsupervised learning algorithm on orange3?
I installed orange3 using miniconda. I can not find K-Means unsupervised learning algorithm. where as other unsupervised learning algorithms are there.
I tried installing the educational add-on but K-...
2
votes
2
answers
102
views
I am trying to create an accumulator function for an vector of sets. But I keep getting strange results
Basically, I have a vector of sets. I'd like to merge the sets based on a criteria (you can see this criteria in the if statement). More than two sets may be merged. Here is an example data set:
[{:...
0
votes
1
answer
22
views
R Code Warning: number of items to replace is not a multiple of replacement length
From the Iris dataset, I have created the following confusion matrix:
In attempting to perform multiclass classification using the one vs many strategy, I have come up with the following R code:
mtrx ...
-1
votes
1
answer
67
views
Converting Python dataframe manipulation code to R
I have a dataframe manipulation in Python that I am trying to translate into R. However, I'm running into issues with my R translation.
col_names = China_2.columns
provinces = col_names[0]
days = ...
0
votes
0
answers
15
views
How to combine non-sequence features in BiLSTM for prediction?
I'm new to LSTM. Now I have a task about destination prediction of online car-hailing, each order is the basic unit of the dataset, and the following are the features:
feature_1: GPS trajectories. ...
0
votes
0
answers
8
views
Comparing topological structure of similar data sets
Are there any methods within Topological Data Analysis that would allow me to check for the existence of similar topological substractures within both point clouds? Example (a bit artifficial): I have ...
0
votes
0
answers
18
views
Process mining dependency matrix issue
I've been playing around with various tools to try and understand / generate process mining dependency matrices.
This is my smalldataset :
patient handling handling_id registration_type time ...
0
votes
1
answer
72
views
ValueError: Target size (torch.Size([555, 3])) must be the same as input size (torch.Size([2, 3, 320, 320]))
def train_loop(dataloader, model, loss_fn, optimizer, device):
size = len(dataloader.dataset)
model.train()
for batch, (X, y) in enumerate(dataloader):
# Move data to the same ...
0
votes
0
answers
37
views
Request failed due to timeout issue on comtradeapicall
Here's the issue: I have code for a Comtrade call. I'm iterating for each month in the year and splitting 5 HS Codes for each API call. Here are the details of my Python code:
import comtradeapicall
...
0
votes
0
answers
17
views
How to calculate incrementality of new SKU in portfolio?
I'm struggling to find a data science concept/approach that would help me answer a super simple business question: what is incremental sales from the big newness the company just launched?
Context:
...
1
vote
0
answers
91
views
How can I get natural language data from Instagram? Should I use the official API or is it better to use data scraping methods?
I am working on a data analytics project to research the rise of toxic masculinity on Instagram amongst Persian language users. I had a straightforward method to get data from Twitter and was able to ...
-3
votes
1
answer
37
views
How to set the equations to implement a Kalman filter for predicting the foot traffic of a restaurant?
I want to implement a Kalman filter to predict the actual daily number of dine-in customers in a restaurant with the two sets of time series data below.
Daily total number of people entering the ...
0
votes
0
answers
60
views
Should different association rule mining algorithms return different results
I have a dataset of air quality values. For the research that I am writing, I am comparing the Apriori, ECLAT and FP-growth algorithm against eachother. For the Apriori and FP-growth algorithms I have ...
0
votes
1
answer
19
views
Scrapy Problem - Whan I ran scrapy file, there is no output
`
import scrapy
class AudibleSpider(scrapy.Spider):
name = 'audible'
allowed_domains = ['www.audible.com']
def start_requests(self):
yield scrapy.Request(
url='...
-1
votes
1
answer
16
views
BeautifulSoup methods
I had install BeautifulSoup
but when I use any function in it I get the same error below
AttributeError: ResultSet object has no attribute 'find'. You're probably treating a list of elements like a ...
0
votes
0
answers
24
views
Automating Support Calculation for Efficient FP-Tree Based Database Compression
I'm currently working on compressing a transactional database using the FP-tree algorithm. While adjusting the support threshold, I've encountered a trade-off: setting a very low support value results ...
1
vote
0
answers
52
views
Compression of Database using FP trees
I have a dataset of around 200 MB, containing only integers. My goal is to use FP-trees to compress this dataset and reduce the final output file size. This is for academic purposes so I can't use ...
0
votes
0
answers
28
views
Logging and Scraping with Selenium
I tried to Log in into website tanfonline.com . In order to scrape data with selenium library in python, I need to log in. However, the server Identified me as a robot so I tried to use ...
0
votes
0
answers
316
views
What's the difference between Quantile Plot and Q-Q Plot?
I am trying to understand the theory behind the Q-Q Plots. So far, I can tell that Q-Q plots are used to plot one univariate quantile distribution against the corresponding quantile values of another ...
0
votes
0
answers
58
views
crawler break down, DUPEFILTER record 20000000+ uniqure url but jl record only 700000+ lines
I run a crawler with command "scrapy crawl any" in the dos window on my work computer with 16G ram and i5 12400 cpu.
Many times it broke down after the computer run slower and slower.
...
0
votes
0
answers
444
views
Bypassing human verification with selenium library
as part of a project, I entered site [Taylor & Francis online] ("https://www.tandfonline.com/loi/nvpp20") using selenium library in python
to bypass human verification .
I tried this ...
0
votes
0
answers
10
views
Data Mining small concept:Implicit and Explicit Data
Is there any criteria for identifying if data is implicit or explicit in data mining
I know data must be explicit and non-trivial for data mining but how do you identify if data is implicit or ...
0
votes
1
answer
34
views
cookies and verification problem in scraping
After running this code as part of a project, in order to enter login screen :
import pandas as pd
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver....
0
votes
2
answers
55
views
Clicking a link with selenium library
I have a project scraping data from tandfonline which I have institutional access to.
To get the data from each article I need to click the issue button each time with selenium library, which has a ...
0
votes
0
answers
53
views
Count repeating intervals in a plot [duplicate]
In the image, you can see five repetitive peak sections. I want to write an algorithm that counts this number, but I am not sure how.
I tried to draw a regression curve using seaborn.lmplot, but it ...
0
votes
0
answers
8
views
how can I the get data of the posting users in Instagram from Facebook GraphAPI?
Facebook in the GraphApi send post details without user information. but I need to users data for my analytics production.
in GraphApi docs write you cant get posting users media object.GraphApi ...
0
votes
0
answers
34
views
Perform K-means Clustering in RapidMiner - Dealing with Categorical Variables
I have a table customer_segmentation where each entry corresponds to a unique customer, and the structure of the table is as follows:
customer_id: (integer) Unique key to identify each customer
gender:...
0
votes
1
answer
131
views
Problems extracting text from pdfs in PYTHON
I'm doing a text extraction from some pdfs and return in the terminal what is the extracted text using python. However, there are some pdfs that are returning me instead of the text, special ...
0
votes
0
answers
38
views
How to standardize data with different dilution factors and minimum values for each dilution factor?
I have been having trouble running statistical analysis on a dataset from a mouse experiment. The dataset is derived from 3 different dilution factors (1:10, 1:20, 1:40) with each having different ...
0
votes
0
answers
28
views
why classification trees doesn't give a single split?
I want to come up with a decision tree with a single split. But with the following code, the node is not split.
dt<-rpart(data$y ~ . , data = data,
control = rpart.control(minsplit=1,...
0
votes
0
answers
23
views
how to fix id key not found in pentaho spoon
im trying to create a fact table from my dimension tables using foreign keys and primary keys
but when im trying to execute it gives me that id key couldnt be found
id key coudlnt be found
but i m ...
0
votes
1
answer
262
views
Error when running tensorflow install package
I'm trying to install tensorflow and i don't know how to fix this error:
Installing collected packages: tensorflow-intel, tensorflow
ERROR: Could not install packages due to an OSError: [Errno 2] No ...
0
votes
0
answers
15
views
Thresholding Neural Network outputs
So I currently have this graph dataset, which have two classes, and I intend to do a binary classificaiton problem. The following is the data’s information
from torch_geometric.datasets import ...
0
votes
0
answers
223
views
Installing SQL Server 2019 Analysis Services on a SQL Server 2022
I noticed that in SQL Server 2022, SSAS' Data Mining is discontinued as per this page: https://learn.microsoft.com/en-us/analysis-services/data-mining/data-mining-wizard-analysis-services-data-mining?...
0
votes
0
answers
99
views
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs
`here when i run the code , i face with the type error like :
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according ...
0
votes
0
answers
17
views
How to do association rule mining with different characteristics of one variable?
I am working on a task where I have data about patients with the variables reportid (date of treatment), type (T for Treatment, S for Symptoms, C for conditions) and a variable names with the assigned ...
0
votes
0
answers
28
views
How to select weeks with no precipitation recorded in R? [duplicate]
I conducted several experiments that lasted for approximately one week each. Precipitation data was recorded at 15-minute intervals, and my data looks like this
I want to select only those weeks in ...
0
votes
0
answers
52
views
Association Rules with Orange3 using table of orders/bills - How to convert into: cloumns is Items_name, Row is OrderID for associate?
I have a table of orders where we have many of orders, each order is an item. And in excel, you have 2 columns: OrderID and ItemName. So if u have 4 items in one order, you will have 4 row have the ...
1
vote
1
answer
62
views
LSTM Model not improve on sentiment analysis, what im doing wrong?
I have sentiment data that contains 3 labels (positive, negative, neutral)
and i have 3233 row data, already tested on naive bayes and svm model, my data got 90 % accuracy on naive bayes, and 92 % ...
1
vote
1
answer
42
views
How to select a certain period based on a condition in R?
I have conducted several experiments that lasted for about a week. For each week, I'd like keep periods/rows when precipitation began to the end of the week.
I know how to discard dry weeks, but I'm ...
0
votes
0
answers
24
views
Orange multiple choice survey import
is there any way to import answers to multiple choice questions of a survey other way than binary? The answers are purely categorical.
Right now the chosen answers of multiple choice question are in ...
-1
votes
1
answer
30
views
Oracle / SQL - summary table with grouped data
I have following raw data table:
Account
Owner
Strategy
POSITION_FLAG
50001
CompanyA
Equity
1
50002
CompanyA
Equity
0
50002
CompanyB
Fixed Income
1
Now I want to produce summary table:
X
Equity
...
0
votes
0
answers
19
views
In an imbalance classification problem, is it fair to balance the test data by removing some negative examples?
I'm working on an imblance classification problem, regarding the evaluation on the test set I had the following questions
Is it a fair practice to use balanced test set for evaluation ?
To create ...
0
votes
0
answers
15
views
Is PCA required for dimensionality reduction in data mining tasks?
I wonder if it is really necessary to use PCA to reduce the dimensionality of input variables in actual data mining tasks.
For interpretability reasons, shouldn't adding regularization terms be a ...