Questions tagged [data-science-experience]

IBM Data Science Experience is an interactive, collaborative, cloud-based environment where data scientists can use multiple tools to activate their insights.

Filter by
Sorted by
Tagged with
0 votes
1 answer
13 views

Target leak in Customer Churn Model

I am working on a customer churn model and I have a few doubts to confirm if there is a target leak or not due to certain features. Please, find the details below: Customer base: Live as of Dec'22 ...
Hitakshi Shah's user avatar
0 votes
0 answers
41 views

VIF calculation difference

I have found two method to calculate VIF but through both method my end result VIF value is very different. Why? first method - X is independent variables. def comput_vif(considered_features): vif ...
tez's user avatar
  • 9
-1 votes
1 answer
39 views

Trying to run a markdown code on my notebook and having some issues what do i do?

i am trying to use markdown code to import an img on my jupyter notebook when i press shift enter instead of displaying the img itthis is an example will shift to a new cell and u will see the code ...
Abdussamad's user avatar
0 votes
1 answer
183 views

I want to improve the efficiency of cosine similarity calculation to make it faster

I have a numpy array of size (96341,1000). I want to find the cosine similarity of this array. The machine I'm working on is 8 vCPU 32 GB. This is my initial code. And I want this function to run ...
Nared Fuengverojsakul's user avatar
0 votes
1 answer
313 views

how can merge multiple part file into single file in databricks

i am trying to merge multiple part file into single file. In staging folder, it itterating the all files, schema is same. part file we are converting .Tab files. Files are generating based on ...
KIRAN KUMAR's user avatar
1 vote
0 answers
61 views

ModuleNotFoundError: No module named 'sklearn.ensemble._bagging'

ModuleNotFoundError: No module named 'sklearn.ensemble._bagging' Which version is suitable of scikit learn for the above error? I am facing this issue when I am using the python 3.7 version. And I ...
Aditya rai's user avatar
0 votes
0 answers
613 views

Getting error when running deepseed in dolly training with exits with return code = -9

Describe the bug I started running deepseed config on EleutherAI/pythia-2.8b model, I ran into error exits with run code= -9. After splitting and preprocessing the dataset i am getting [ERROR] [launch....
Dinesh's user avatar
  • 9
1 vote
0 answers
51 views

How to make a visualization that shows every seconds data?

I get data in my job which contains datetime, Temperature, Current(mA), Brake release. This data is of a crane motor. In one day there are about 80k-100k rows in excel generated by the tags. I have to ...
Mohammad Zeeshan Ali's user avatar
0 votes
0 answers
29 views

AOC CURVE Issue

I have written AOC_Curve: X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=.20,random_state=1) # AUC and ROC for the training data from sklearn.model_selection import * from sklearn....
Rohit Kulkarni's user avatar
0 votes
0 answers
29 views

AttributeError: 'CountVectorizer' object has no attribute 'fit_transfrom'

1 from sklearn.feature_your textextraction.text import CountVectorizer 2 cv = CountVectorizer() ----> 3 X = cv.fit_transfrom(df['transformed_text']).toarray() no error in this line
67_Thorat Manish's user avatar
0 votes
2 answers
56 views

How to take a sum (in denominator) for calculating group by weighted average in a dataframe?

I have a data frame that looks like this. import pandas as pd import numpy as np data = [ ['A',1,2,3,4], ['A',5,6,7,8], ['A',9,10,11,12], ['B',13,14,15,16], ['B',17,18,19,20], ['B',21,22,...
Bad Coder's user avatar
  • 175
0 votes
4 answers
1k views

How to calculate percentage change with zero in pandas?

I want to calculate the percentage change for the following data frame. import pandas as pd df = pd.DataFrame({'team': ['A', 'A', 'A', 'B', 'B', 'B', 'C', 'C'], 'points': [12, 0, ...
Bad Coder's user avatar
  • 175
0 votes
1 answer
33 views

How can we assign new variables after each for loop iteration in python?

Although it might be easy but i am not able to get a hang of it.. I want to assign the result to a new variable every time the for loop iteration occurs. I don't wish to do with initializing the list ...
Ravi_DataScientist's user avatar
0 votes
1 answer
62 views

How to check different rows values of a column within the same group and return a specific value?

I have the following code that generates the two columns. import pandas as pd data = {'Group': ['1', '1', '1', '1', '1', '1', '2', '2', '2', '2', '2', '2', '3', '...
Bad Coder's user avatar
  • 175
1 vote
0 answers
177 views

How to do k-fold cross validation and use the model to predict unseen data?

I have a model whose training accuracy is 95-100 % and I believe there is overfitting. So, I want to avoid overfitting in my model. One way to avoid overfitting is to do k-fold cross-validation. So, ...
Bad Coder's user avatar
  • 175
1 vote
2 answers
475 views

How to remove \n and empty string in a column in a dataframe?

I have a data frame and one column consists of list value. I have attached the picture in excel format and data frame as well. column "[ ""Hello"" ]" "[ ""...
Bad Coder's user avatar
  • 175
0 votes
0 answers
415 views

How to match labels for train and test in machine learning using python?

The title may not be clear but I will try to explain my problem as clearly as possible. I have dummy data i.e. data = {'month': ['2022-01-01', '2022-02-01', '2022-03-01', '2022-01-01', '2022-02-01', '...
Bad Coder's user avatar
  • 175
0 votes
2 answers
250 views

How can we perform group by single column without aggregation in pandas?

I have a couple of questions about the group by function. 1. I would like to group by pandas data frame by single column without aggregation. 2. After group by, I would like to split the dataset into ...
Bad Coder's user avatar
  • 175
0 votes
1 answer
390 views

Finding the root of an equation using binary search python

help me here, estimating the root of the equation. Here is a code: def binary_search(f,domain, MAX = 1000): start, end = domain if start >= end: raise ValueError("Domain is ...
Dushu's user avatar
  • 31
2 votes
1 answer
3k views

how to perform box-cox transformation to single column in python

I am trying to apply a box-cox transformation to a single column but I am unable to do that. Can somebody help me with this issue? from sklearn.datasets import fetch_california_housing import pandas ...
Bad Coder's user avatar
  • 175
0 votes
2 answers
380 views

how to convert multiple rows into single column for a time series data?

I have time series data or and one group has multiple rows. So, I need to transform that multiple rows into a single row for a group in python. I have this dummy dataset. data = [ ['A','test1',1,2,3,...
Bad Coder's user avatar
  • 175
0 votes
1 answer
200 views

How to fill NA values with applying condition?

I am trying to replace null values with 0 by applying certain conditions. Here is the code to generate the dataset. data = {'month': ['2022-01-01', '2022-02-01', '2022-03-01', '2022-01-01', '2022-02-...
Bad Coder's user avatar
  • 175
0 votes
1 answer
267 views

Predict a data using Pickle file

I have pre build machine learning model (saved as pickle file) to predict classification. My question is when I use new dataset to predict using Pickle file do I need do all preprocessing steps (like ...
Sandumi's user avatar
  • 13
0 votes
0 answers
199 views

how can we interpret Precision, Precision@K, ROC curve and precision-recall AUC curve

I am performing a logistic regression and performing probabilistic modeling. When I go through the definition of this ** Precision, Precision@K, ROC curve, and precision-recall AUC curve** performance ...
Bad Coder's user avatar
  • 175
1 vote
1 answer
55 views

draw objects on the center of scatter plots

i got this bunch of code : import matplotlib.pyplot as plt from matplotlib.animation import FuncAnimation from matplotlib import animation %matplotlib inline plt.ion() fig = plt.figure() ax = fig....
Daniel's user avatar
  • 169
1 vote
1 answer
951 views

How to convert daily data into weekly or monthly in python with categorical and numerical column?

I have a daily dataset that has a categorical and numerical column. So, I want to change the daily dataset to the monthly dataset. How can I do that using python? For example, if I have a dataset ...
Bad Coder's user avatar
  • 175
0 votes
1 answer
389 views

How to plot moving average by groupby in python?

I have a question related to plotting the moving average by doing group by. I have taken the dataset from Kaggle https://www.kaggle.com/code/kp4920/s-p-500-stock-data-time-series-analysis/comments. I ...
Bad Coder's user avatar
  • 175
0 votes
0 answers
51 views

How to write a double summation in Python with multiple variables , I am a new programmer in the data science field

enter image description here I am a new programmer in the Data field and need help to write in Python
amjaeee's user avatar
-3 votes
2 answers
693 views

How to remove a single number from input number in python

Here are few example input & outputs to understand the question example 1 input = 555 output should be = 55 example 2 input = 5455 output should be = 545 example 3 input = 6555 output ...
Akhil Raj n's user avatar
0 votes
0 answers
143 views

a problem to Convert XML File dataset to JSON in python

I have been trying to convert a dataset file in XML format to JSON. I tried many solutions, I always got the same error ExpatError: not well-formed (invalid token): line 67667, column 103 I tried ...
ahad's user avatar
  • 11
0 votes
1 answer
164 views

Python: how to install dsx_core_utils package

I'm trying to adapt the code snippets presented on the following page: https://www.ibm.com/docs/en/watson-studio-local/1.2.1?topic=data-write-sources However, I don't understand where to get the ...
TheDude's user avatar
  • 1,235
0 votes
2 answers
122 views

LF Framework to solve Data Science Issue

i'm looking for a framework that is able to solve the following Data Science issue: I have several Teachers that can work for X amount of hours a week and have several subject that they can teach. ...
Specta's user avatar
  • 37
1 vote
1 answer
40 views

How to save the rows in different columns based on the conditions?

I want to select rows based on the index like 0 to 30 index & save them to a data frame and again pick the index from 30 to 60 & save them into the same data frame but different column and I ...
Amit Kumar's user avatar
0 votes
0 answers
58 views

How would I connect to a WinDSX server with .Net so I can query the database?

I need to be able to query a WinDSX server from .Net. This is the first time I have worked with a DSX server and I can't seem to find anything that really explains how to do this. Is there some type ...
Chris's user avatar
  • 838
0 votes
1 answer
406 views

Web scraper giving random values

At work I was tasked with doing a market analysis of bricks. I chose some competitors and made web scrapers to collect their prices. It works for most brick types, however on some it will change the ...
Jacob Wilson's user avatar
-1 votes
1 answer
599 views

Can Training dataset and testing data set be seperate instead of split [closed]

Can we have a separate data set for both Training and Testing. I'm working on a project to pick up effective test case AS part of this, i analyse the bug database and come up with triggers which have ...
Sai's user avatar
  • 77
0 votes
2 answers
181 views

How to split year and month column in python?

How can we split the data as mention below? Sample Data: EmployeeId City join_month Year 0 001 Mumbai 1 2018 1 001 Bangalore 3 2018 2 002 ...
manoj kumar's user avatar
0 votes
0 answers
59 views

My pandas code is not working, in the tutorial the same code worked without any error

(df.set_index('STNAME').groupby(level=0)['CENSUS2010POP'] .agg({'sum': np.sum, 'avg': np.average})) I am getting the following error while using the above code: nested renamer is not supported
rohit rasam's user avatar
0 votes
1 answer
988 views

How to initialise value for LPVARIABLE.matrix in PULP?

I am trying to set initial values for the below created decision variable, but when I try to do am getting an error? dec_var = LpVariable.matrix("dec_var",(range(10),range(10)),0,1,cat = '...
Sarath Kumar's user avatar
-1 votes
1 answer
744 views

How to fit data with square wave in Python?

I have the following data: Experimental data and ideal fitting function I have been trying to fit it with a sequence of square pulses which I define as: def f(x, x0, a, b, c): sq = a * np....
luca gravina's user avatar
0 votes
1 answer
134 views

Adding empty columns to python DataFrame

I am trying to add couple of empty columns to a python Dataframe , the columns to be added are in the form of list, how could I do it list=[4,5,6] data1={'77':[1.2,2.2,3.5],'18':[4.5,6.5,9.9],'89':[8....
SupportVector's user avatar
0 votes
2 answers
58 views

Issues in creating jobs of size XL in Watson Machine Learning (WML)

I have an issue when trying to create jobs for Decision Optimization when using size XL in Watson Machine Learning (WML). The first job for the day I have no issues what so ever to create. But the ...
Oskar Danielsson's user avatar
0 votes
1 answer
116 views

Apriori algorithm expert is needed

I have a dataset with 3.3M rows and 8k unique products. I wanted to apply apriori algorithm to find association rules and connections between products. Well, I did it before on a much smaller database ...
Yoav Barzilai's user avatar
0 votes
1 answer
67 views

How can I impute missing value with the help of tree based model like random forest

In my datasets, I have one variable which contains 30% missing value. I am trying to use tree based model but not getting clear picture how to implement it. data['X'].value_counts() OUTPUT----- ? ...
user2986845's user avatar
0 votes
1 answer
58 views

Read simple csv with PySpark

probably a silly issue, but I don't get it. I'm working on a Jupyter Notebook with Python3.6, Spark 2.4, hosted by IBM Watson Studio. I have a simple csv file: num,label 0,0 1,0 2,0 3,0 And to read ...
Vincenzo Lavorini's user avatar
1 vote
0 answers
926 views

Unable to install sndfile with pip

I need to install sndfile. I'm on a Jupyter notebook, Python interpreter on IBM Watson studio. I already installed the library libsndfile with !conda install -c conda-forge libsndfile. When I call pip ...
Vincenzo Lavorini's user avatar
0 votes
1 answer
122 views

using TQDM in Watson Studio Notebooks

I'm using Pandas, and I would like to use the TQDM progress bar in the notebook. After loading TQDM: from tqdm.auto import tqdm tqdm.pandas() and applying a function to the Pandas Dataframe: new_df = ...
Vincenzo Lavorini's user avatar
1 vote
1 answer
344 views

SQL Temporary Table Issue

I've created a temporary table DETAILS and follow the same syntax of creating and inserting in it. But I have not received any result set However, the CREATE and INSERT statements ran successfully and ...
Ali's user avatar
  • 35
1 vote
2 answers
17k views

pickle data was truncated

i created a corpus file then stored in a pickle file. my messages file is a collection of different news articles dataframe. from nltk.corpus import stopwords from nltk.stem.porter import ...
omkar patil's user avatar
0 votes
1 answer
352 views

getting 'StructField' object has no attribute '_get_object_id' on BinaryClassificationMetrics

I was trying to get the binary classification report on pyspark and I ran into this error StructField' object has no attribute '_get_object_id' Here is my code %%spark from pyspark.mllib....
Hamed Niakan's user avatar

1
2 3 4 5 6