Questions tagged [data-science-experience]
IBM Data Science Experience is an interactive, collaborative, cloud-based environment where data scientists can use multiple tools to activate their insights.
262
questions
0
votes
1
answer
13
views
Target leak in Customer Churn Model
I am working on a customer churn model and I have a few doubts to confirm if there is a target leak or not due to certain features. Please, find the details below:
Customer base: Live as of Dec'22
...
0
votes
0
answers
41
views
VIF calculation difference
I have found two method to calculate VIF but through both method my end result VIF value is very different. Why?
first method -
X is independent variables.
def comput_vif(considered_features):
vif ...
-1
votes
1
answer
39
views
Trying to run a markdown code on my notebook and having some issues what do i do?
i am trying to use markdown code to import an img on my jupyter notebook when i press shift enter instead of displaying the img itthis is an example will shift to a new cell and u will see the code ...
0
votes
1
answer
183
views
I want to improve the efficiency of cosine similarity calculation to make it faster
I have a numpy array of size (96341,1000). I want to find the cosine similarity of this array. The machine I'm working on is 8 vCPU 32 GB. This is my initial code. And I want this function to run ...
0
votes
1
answer
313
views
how can merge multiple part file into single file in databricks
i am trying to merge multiple part file into single file. In staging folder, it itterating the all files, schema is same. part file we are converting .Tab files. Files are generating based on ...
1
vote
0
answers
61
views
ModuleNotFoundError: No module named 'sklearn.ensemble._bagging'
ModuleNotFoundError: No module named 'sklearn.ensemble._bagging'
Which version is suitable of scikit learn for the above error?
I am facing this issue when I am using the python 3.7 version. And I ...
0
votes
0
answers
613
views
Getting error when running deepseed in dolly training with exits with return code = -9
Describe the bug
I started running deepseed config on EleutherAI/pythia-2.8b model, I ran into error exits with run code= -9. After splitting and preprocessing the dataset i am getting [ERROR] [launch....
1
vote
0
answers
51
views
How to make a visualization that shows every seconds data?
I get data in my job which contains datetime, Temperature, Current(mA), Brake release. This data is of a crane motor. In one day there are about 80k-100k rows in excel generated by the tags.
I have to ...
0
votes
0
answers
29
views
AOC CURVE Issue
I have written AOC_Curve:
X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=.20,random_state=1)
# AUC and ROC for the training data
from sklearn.model_selection import *
from sklearn....
0
votes
0
answers
29
views
AttributeError: 'CountVectorizer' object has no attribute 'fit_transfrom'
1 from sklearn.feature_your textextraction.text import CountVectorizer
2 cv = CountVectorizer()
----> 3 X = cv.fit_transfrom(df['transformed_text']).toarray()
no error in this line
0
votes
2
answers
56
views
How to take a sum (in denominator) for calculating group by weighted average in a dataframe?
I have a data frame that looks like this.
import pandas as pd
import numpy as np
data = [
['A',1,2,3,4],
['A',5,6,7,8],
['A',9,10,11,12],
['B',13,14,15,16],
['B',17,18,19,20],
['B',21,22,...
0
votes
4
answers
1k
views
How to calculate percentage change with zero in pandas?
I want to calculate the percentage change for the following data frame.
import pandas as pd
df = pd.DataFrame({'team': ['A', 'A', 'A', 'B', 'B', 'B', 'C', 'C'],
'points': [12, 0, ...
0
votes
1
answer
33
views
How can we assign new variables after each for loop iteration in python?
Although it might be easy but i am not able to get a hang of it..
I want to assign the result to a new variable every time the for loop iteration occurs. I don't wish to do with initializing the list ...
0
votes
1
answer
62
views
How to check different rows values of a column within the same group and return a specific value?
I have the following code that generates the two columns.
import pandas as pd
data = {'Group': ['1', '1', '1', '1', '1', '1',
'2', '2', '2', '2', '2', '2',
'3', '...
1
vote
0
answers
177
views
How to do k-fold cross validation and use the model to predict unseen data?
I have a model whose training accuracy is 95-100 % and I believe there is overfitting. So, I want to avoid overfitting in my model. One way to avoid overfitting is to do k-fold cross-validation. So, ...
1
vote
2
answers
475
views
How to remove \n and empty string in a column in a dataframe?
I have a data frame and one column consists of list value. I have attached the picture in excel format and data frame as well.
column
"[
""Hello""
]"
"[
""...
0
votes
0
answers
415
views
How to match labels for train and test in machine learning using python?
The title may not be clear but I will try to explain my problem as clearly as possible. I have dummy data i.e.
data = {'month': ['2022-01-01', '2022-02-01', '2022-03-01', '2022-01-01', '2022-02-01', '...
0
votes
2
answers
250
views
How can we perform group by single column without aggregation in pandas?
I have a couple of questions about the group by function.
1. I would like to group by pandas data frame by single column without aggregation.
2. After group by, I would like to split the dataset into ...
0
votes
1
answer
390
views
Finding the root of an equation using binary search python
help me here, estimating the root of the equation.
Here is a code:
def binary_search(f,domain, MAX = 1000):
start, end = domain
if start >= end:
raise ValueError("Domain is ...
2
votes
1
answer
3k
views
how to perform box-cox transformation to single column in python
I am trying to apply a box-cox transformation to a single column but I am unable to do that. Can somebody help me with this issue?
from sklearn.datasets import fetch_california_housing
import pandas ...
0
votes
2
answers
380
views
how to convert multiple rows into single column for a time series data?
I have time series data or and one group has multiple rows. So, I need to transform that multiple rows into a single row for a group in python.
I have this dummy dataset.
data = [
['A','test1',1,2,3,...
0
votes
1
answer
200
views
How to fill NA values with applying condition?
I am trying to replace null values with 0 by applying certain conditions. Here is the code to generate the dataset.
data = {'month': ['2022-01-01', '2022-02-01', '2022-03-01', '2022-01-01', '2022-02-...
0
votes
1
answer
267
views
Predict a data using Pickle file
I have pre build machine learning model (saved as pickle file) to predict classification.
My question is when I use new dataset to predict using Pickle file do I need do all preprocessing steps (like ...
0
votes
0
answers
199
views
how can we interpret Precision, Precision@K, ROC curve and precision-recall AUC curve
I am performing a logistic regression and performing probabilistic modeling. When I go through the definition of this ** Precision, Precision@K, ROC curve, and precision-recall AUC curve** performance ...
1
vote
1
answer
55
views
draw objects on the center of scatter plots
i got this bunch of code :
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
from matplotlib import animation
%matplotlib inline
plt.ion()
fig = plt.figure()
ax = fig....
1
vote
1
answer
951
views
How to convert daily data into weekly or monthly in python with categorical and numerical column?
I have a daily dataset that has a categorical and numerical column. So, I want to change the daily dataset to the monthly dataset. How can I do that using python? For example, if I have a dataset ...
0
votes
1
answer
389
views
How to plot moving average by groupby in python?
I have a question related to plotting the moving average by doing group by. I have taken the dataset from Kaggle https://www.kaggle.com/code/kp4920/s-p-500-stock-data-time-series-analysis/comments. I ...
0
votes
0
answers
51
views
How to write a double summation in Python with multiple variables , I am a new programmer in the data science field
enter image description here
I am a new programmer in the Data field and need help to write in Python
-3
votes
2
answers
693
views
How to remove a single number from input number in python
Here are few example input & outputs to understand the question
example 1
input = 555
output should be = 55
example 2
input = 5455
output should be = 545
example 3
input = 6555
output ...
0
votes
0
answers
143
views
a problem to Convert XML File dataset to JSON in python
I have been trying to convert a dataset file in XML format to JSON. I tried many solutions, I always got the same error
ExpatError: not well-formed (invalid token): line 67667, column 103
I tried ...
0
votes
1
answer
164
views
Python: how to install dsx_core_utils package
I'm trying to adapt the code snippets presented on the following page:
https://www.ibm.com/docs/en/watson-studio-local/1.2.1?topic=data-write-sources
However, I don't understand where to get the ...
0
votes
2
answers
122
views
LF Framework to solve Data Science Issue
i'm looking for a framework that is able to solve the following Data Science issue:
I have several Teachers that can work for X amount of hours a week and have several subject that they can teach.
...
1
vote
1
answer
40
views
How to save the rows in different columns based on the conditions?
I want to select rows based on the index like 0 to 30 index & save them to a data frame and again pick the index from 30 to 60 & save them into the same data frame but different column and I ...
0
votes
0
answers
58
views
How would I connect to a WinDSX server with .Net so I can query the database?
I need to be able to query a WinDSX server from .Net. This is the first time I have worked with a DSX server and I can't seem to find anything that really explains how to do this. Is there some type ...
0
votes
1
answer
406
views
Web scraper giving random values
At work I was tasked with doing a market analysis of bricks. I chose some competitors and made web scrapers to collect their prices.
It works for most brick types, however on some it will change the ...
-1
votes
1
answer
599
views
Can Training dataset and testing data set be seperate instead of split [closed]
Can we have a separate data set for both Training and Testing.
I'm working on a project to pick up effective test case
AS part of this, i analyse the bug database and come up with triggers which have ...
0
votes
2
answers
181
views
How to split year and month column in python?
How can we split the data as mention below?
Sample Data:
EmployeeId City join_month Year
0 001 Mumbai 1 2018
1 001 Bangalore 3 2018
2 002 ...
0
votes
0
answers
59
views
My pandas code is not working, in the tutorial the same code worked without any error
(df.set_index('STNAME').groupby(level=0)['CENSUS2010POP']
.agg({'sum': np.sum, 'avg': np.average}))
I am getting the following error while using the above code:
nested renamer is not supported
0
votes
1
answer
988
views
How to initialise value for LPVARIABLE.matrix in PULP?
I am trying to set initial values for the below created decision variable, but when I try to do am getting an error?
dec_var = LpVariable.matrix("dec_var",(range(10),range(10)),0,1,cat = '...
-1
votes
1
answer
744
views
How to fit data with square wave in Python?
I have the following data:
Experimental data and ideal fitting function
I have been trying to fit it with a sequence of square pulses which I define as:
def f(x, x0, a, b, c):
sq = a * np....
0
votes
1
answer
134
views
Adding empty columns to python DataFrame
I am trying to add couple of empty columns to a python Dataframe , the columns to be added are in the form of list, how could I do it
list=[4,5,6]
data1={'77':[1.2,2.2,3.5],'18':[4.5,6.5,9.9],'89':[8....
0
votes
2
answers
58
views
Issues in creating jobs of size XL in Watson Machine Learning (WML)
I have an issue when trying to create jobs for Decision Optimization when using size XL in Watson Machine Learning (WML). The first job for the day I have no issues what so ever to create. But the ...
0
votes
1
answer
116
views
Apriori algorithm expert is needed
I have a dataset with 3.3M rows and 8k unique products.
I wanted to apply apriori algorithm to find association rules and connections between products.
Well, I did it before on a much smaller database ...
0
votes
1
answer
67
views
How can I impute missing value with the help of tree based model like random forest
In my datasets, I have one variable which contains 30% missing value.
I am trying to use tree based model but not getting clear picture how to implement it.
data['X'].value_counts()
OUTPUT-----
? ...
0
votes
1
answer
58
views
Read simple csv with PySpark
probably a silly issue, but I don't get it. I'm working on a Jupyter Notebook with Python3.6, Spark 2.4, hosted by IBM Watson Studio.
I have a simple csv file:
num,label
0,0
1,0
2,0
3,0
And to read ...
1
vote
0
answers
926
views
Unable to install sndfile with pip
I need to install sndfile. I'm on a Jupyter notebook, Python interpreter on IBM Watson studio.
I already installed the library libsndfile with !conda install -c conda-forge libsndfile.
When I call pip ...
0
votes
1
answer
122
views
using TQDM in Watson Studio Notebooks
I'm using Pandas, and I would like to use the TQDM progress bar in the notebook.
After loading TQDM:
from tqdm.auto import tqdm
tqdm.pandas()
and applying a function to the Pandas Dataframe:
new_df = ...
1
vote
1
answer
344
views
SQL Temporary Table Issue
I've created a temporary table DETAILS and follow the same syntax of creating and inserting in it. But I have not received any result set However, the CREATE and INSERT statements ran successfully and ...
1
vote
2
answers
17k
views
pickle data was truncated
i created a corpus file then stored in a pickle file.
my messages file is a collection of different news articles dataframe.
from nltk.corpus import stopwords
from nltk.stem.porter import ...
0
votes
1
answer
352
views
getting 'StructField' object has no attribute '_get_object_id' on BinaryClassificationMetrics
I was trying to get the binary classification report on pyspark and I ran into this error
StructField' object has no attribute '_get_object_id'
Here is my code
%%spark
from pyspark.mllib....