Questions tagged [pandas]

Pandas is a Python library for data manipulation and analysis, e.g. dataframes, multidimensional time series and cross-sectional datasets commonly found in statistics, experimental science results, econometrics, or finance. Pandas is one of the main data science libraries in Python.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

why is my predicted LSTM model shifted left as well as the accuracy 0

Hey all i have seen some other posts with shifted LSTM models being shifted however looking into them I didn't find a solution.Is something wrong with how im slicing the data in the Sequential_Input ...
Steven's user avatar
  • 155
0 votes
1 answer
34 views

Apply function to values in a dataframe column

I have a function that calls a mapping API and returns longs and lats given unstructured address data. This works, and I can pass address = "12 & 14 CHIN BEE AVENUE,, SINGAPORE 619937" ...
opperman.eric's user avatar
2 votes
1 answer
50 views

Save a Pandas DataFrame to a CSV file without adding extra double quotes

I want to save a Pandas dataframe to a CSV file in such a way that no additional double quotes or any other characters are added to these formulas. Here is my attempt: import pandas as pd data = { ...
Javide's user avatar
  • 2,507
0 votes
1 answer
19 views

Exception in Pytorch multi-output regression

I'm quite new to both Python and Pytorch and I'm attempting to create a multi-output regression neural network with 4 inputs and 2 outputs, all numerical. Here is my code: import pandas as pd #For ...
JPilot2023's user avatar
1 vote
1 answer
22 views

Issue with removing dates, symbol, exchange and volume from dataset

I coded something to retrieve historical price data from an app called TradingView This is what the code looks like: from pandas import set_option from tvDatafeed import TvDatafeed, Interval tv = ...
FutureSightTrader's user avatar
0 votes
0 answers
28 views

Resource Management with Python Temporary Files and Zipfiles

I needed to create a subroutine that loops through a collection of outlook messages, opens the attachments and extracts any tabular data in a zip folder to a pandas data frame. To grab the tabular ...
Possdawgers's user avatar
1 vote
1 answer
58 views

Plot Histogram of the sum of values for each month [duplicate]

Plot Histogram for confirmed COVID 19 Cases Dataset where date needs to be on x-axis vs confirmed covid 19 cases on y-axis I want to plot month i.e.,FEB 2020,MAR 2020 on x-axis vs confirmed covid 19 ...
HumanRob231's user avatar
2 votes
2 answers
74 views

TypeError: get_loc() got an unexpected keyword argument 'method'

In pandas 2 with the following code: for time in tfo_dates: dt=pd.to_datetime(time) indx_time.append(df.index.get_loc(dt,method='nearest')) I get this error: TypeError: get_loc() got an ...
roudan's user avatar
  • 3,160
0 votes
1 answer
48 views

Determining logic based on a subset of data

I have the following data: data = {'Account': [1, 1, 1, 2, 2, 3, 4, 4], 'Customer': [1, 2, 3, 2, 4, 5, 6, 1], 'Key': [1, 0, 1, 0, 1, 1, 1, 1]} df = pd.DataFrame(data) Account ...
Konrad Lubelski's user avatar
1 vote
1 answer
29 views

how to get the count of eventTime with the same timestamp from a column which has list of events in a dataframe

Below is an example of my dataframe. i have events in a list within a column of a dataframe. I have to get the count of events where the eventDateTime is duplicated. how can I do it? i am not able to ...
sman's user avatar
  • 83
-1 votes
1 answer
46 views

Defining a Python Function [duplicate]

I need help to define a function that gives me a specific result (Column C in the pic attached) Column C contains the following formula: =IF(AND(OR(A3=A4,A3=A2),B3="Tenure"),"Rejected&...
blackice7's user avatar
0 votes
0 answers
42 views

Adding Contact Numbers to Dataframe by Weight

This function is intended to add phone numbers to a "Phone" column in the "data" dataframe by weight, using account number as the primary key. Weights (descending): home phone ...
Spencer's user avatar
0 votes
1 answer
50 views

Python - Parent-Child Hierarchy Recursive Structure

I am trying to form a parent to child relationship combination using data available in my pandas dataframe. Data Example: Parent Child A B A C B D B C C D Expected chain results: A|B A|C A|B|D ...
Balaji Srinivasan's user avatar
0 votes
0 answers
21 views

how do you create an excel workbook and write to different sheet using Pandas [duplicate]

I need to write 4 data frames to an excel spreadsheet. When I execute the code, I should be able to create a new excel sheet and write each data frames to different sheet. I tried this: from openpyxl ...
user1471980's user avatar
  • 10.2k
-3 votes
1 answer
33 views

Creating Multiple Dataframes based on starting characters in one column and matching to Dictionary

Let’s say I have a dictionary area_codes = { “New Hampshire”:[‘602’] “Vermont”:[‘803’] } And a data frame that looks like name job title phone number Bob doctor 602-123-4567 Joe pharmacist 803-123-...
datagolfer's user avatar
0 votes
2 answers
26 views

How to update multiple columns of a Pandas dataframe at once [duplicate]

Given a Pandas dataframe with columns A, B and C, I want to update A and B as the result of applying a function over C. This function has one parameter and two return values. import pandas as pd # ...
Samuel O.D.'s user avatar
1 vote
3 answers
56 views

Trying to change the first character of every row in a pandas series

I have a series list1: 1 listId 95 LaBc.defGabc 97 Kasd.defGabc 99 aaSd.defGabc 101 Basd.defGabc 103 Lasd.defGabc 105 Lasd.defGabc I want to lower() only the first character ...
Mitch's user avatar
  • 532
2 votes
2 answers
71 views

Plotting NaN values as gaps in Scatter Plot

I have a figure with six subplot control charts from a Pandas dataframe, currently plotted by index and I'd like to plot them against a datetime value. The raw data (from an XML file) contains blank ...
ChemEnger's user avatar
  • 151
0 votes
0 answers
6 views

Scoped_session has no attribute cursor: Flask-SQLAlchemy session in pd.read_sql

I am switching from the standalone SQLAlchemy package in my Flask app to Flask-SQLAlchemy. Everything went smoothly except for one problem. I cannot use the database session provided by Flask-...
RogerKint's user avatar
  • 484
0 votes
1 answer
38 views

How to replace NaN values with the median in the dataset

I have a dataset with missing values, I determined the best way to approach this problem is to find the median of the row and replace the NaN values with the median value. However, the code runs but ...
Puj's user avatar
  • 13
1 vote
1 answer
42 views

Python pandas DataFrame: Transfer data from dictionary to a right cell

Can you please tell me how to transfer data from a dictionary to a table using cross-matching? For example, there is a dictionary where each project has its own set of components and there is a table ...
macder's user avatar
  • 163
0 votes
2 answers
32 views

Unable to parse string for Int64 in pd read_csv

It doesn't seem as if Pandas 2.0.0 properly accounts for thousands=',' when parsing Int64 objects: import io pd.read_csv(io.StringIO('''a\n22,922'''), sep='\t', dtype={'a': 'Int64'}, thousands=',') ...
ifly6's user avatar
  • 5,015
-1 votes
1 answer
34 views

Pandas get_dummies altering shape

I want to one-hot encode the categorical features of my Pandas dataframe. Previously, the values stored variables of shape (60,). See code below: ohe_features = ["Gender", "Married"...
user19251203's user avatar
0 votes
1 answer
15 views

displaying snowflake data as drop downs in a table in streamlit

I am trying to build a streamlit app that connects to a snowflake table, fetches data and displays as editable table in streamlit and writes back the edited info back to snowflake, I am able to get ...
bobby1985's user avatar
0 votes
1 answer
42 views

Segmenting column on 4 quantiles

I want to create a column where it would tell us in which quantile it is. Currently I'm doing a groupby on country, industry and bucket and want to segment them into 4 separate segments on another ...
hjun's user avatar
  • 73
0 votes
0 answers
26 views

Rearrange dataframe to add new columns based on value of one column [duplicate]

I would like to have the column ds_id (contains 4 unique numbers incl. ) merged into the columns c1, c2 (c3, c4… etc,.) like the below desired output. There will not be any need for aggregation as it'...
donnyjeremiah's user avatar
2 votes
2 answers
36 views

Transposing columns to rows Pandas

I have a situation like following: ID Old value.Carbon Old value.Dioxide New value.Carbon New Value.Dioxide 123 34.89 13.45 56.66 11.11 456 12.13 55.66 66.88 12.33 My output that i want should be:(...
Andrew Pike 's user avatar
0 votes
1 answer
45 views

Resample and interpolate timeseries in Python

I have csv-file with wind direction (wd) and speed (ws): datetime wd ws 06.02.2023 00:55 297 3.2 06.02.2023 01:55 296 2.7 06.02.2023 02:55 299 3.0 06.02.2023 03:55 302 3.5 I would like to ...
fred's user avatar
  • 93
0 votes
0 answers
13 views

How to track loop progress when looping over a dataframe groupby object? [duplicate]

I have a pandas.core.groupby.generic.DataFrameGroupBy object and I need to loop over the groups. The dataset is quite big so I also want to track the progress of the loop. For normal DataFrame I do ...
confused_certainties's user avatar
0 votes
0 answers
44 views

How to create a dataframe column where each value on a given row depends on the values of subsecant rows?

So I'm working with time series data, forex data to be more precise. They are in the form of <DATE> <OPEN> <HIGH> <LOW> <CLOSE> 2023.01.03 1.20782 ...
Kyete Franck's user avatar
0 votes
0 answers
35 views

How to find approximate solution to equation?

I have a dataframe with thousands of rows, the first row is listed here: Here is an equation: FV_dirty in the equation is 'closing_price', C_i = C = coupon_rate * face_value * ...
1670511081's user avatar
0 votes
1 answer
60 views

How do I get the max timestamp in each row with NaT values?

First of all I am working with Databricks 11.3 with Python 3.9.5 so that might make a difference as I've come to realize in the past. For each row I need to get the latest timestamp but as soon as I ...
Dave4048's user avatar
  • 173
0 votes
1 answer
40 views

To_sql fails to update mysql table

I am trying to insert a dataframe to a table in mysql: sql = """ select MakeID, MakeName brand FROM Makes m """ sql_d = pd.read_sql(sql, conn) mycursor = mydb.cursor() ...
Thuy Nguyen's user avatar
0 votes
0 answers
23 views

Pythonic way of aggregating but without grouping [duplicate]

I have a df as follows, where I wish to find of Sum of orders and Number of unique order sizes, but I don't want to compress the df: list_of_lists = [['11','Berlin',2],['11','Berlin',2],['11','Berlin',...
cph_sto's user avatar
  • 7,234
0 votes
2 answers
46 views

Check if part of the string exists in different dataframe and get this part only

My DF_1 looks like below: id x 1 eu continent hamburg 2 asia singapore 3 austrlia hedland I have a second DF_2 that looks like below: name germany hamburg singapore china west australia ...
Tmiskiewicz's user avatar
0 votes
3 answers
32 views

Filter pandas DataFrame using a column of np arrays

I have a pandas dataframe with a column containing arrays. I want to filter my df based on values of the column with arrays. For example, for the df subject ...
jaweria's user avatar
  • 63
4 votes
1 answer
30 views

Converting a pandas dataframe to a pytorch tensor

I am trying to convert a pandas dataframe read from a CSV file to a pytorch tensor, but am getting a type error. I tried doing this: df = pandas.DataFrame({"spam": [1, 2, 3, 4], "...
Yoogi Kovendhan's user avatar
0 votes
0 answers
18 views

Passing a dictionary to replace with Pandas [duplicate]

I have the following dataframe: 0 1 2 0 2022-09-01T01:30:00+00:00 2543.207 -365.0 1 2022-09-01T02:00:00+00:00 2479.386 -365.0 2 2022-09-...
I hate coding's user avatar
1 vote
1 answer
49 views

Using explode() as Aggregation Function [duplicate]

How can I explode duplicate index rows in a pd.pivot_table()? import pandas as pd df = pd.DataFrame({ "group": [1,2,2,3,1,2,3], "panel": [1,1,1,1,2,2,2], "value&...
ascripter's user avatar
  • 5,706
1 vote
1 answer
30 views

How to create new column with new names that has value from other column and keep values for the previous columns? [duplicate]

I have the following dataframe_1 I would like to transform: Time Not_paid Paid 0 morning 30 10 1 afternoon 60 20 2 night 90 30 I would like to use the ...
codelifevcd's user avatar
0 votes
1 answer
44 views

Use "keras code" (Tensorflow) and the result is "loss: nan"

requirements.txt absl-py==2.0.0 astunparse==1.6.3 cachetools==5.3.1 certifi==2023.7.22 charset-normalizer==3.2.0 flatbuffers==23.5.26 gast==0.4.0 google-auth==2.23.0 google-auth-oauthlib==1.0.0 google-...
pozuhtuhv's user avatar
-1 votes
2 answers
37 views

24 Hour time range in pandas DataFrame Column

The Input file I have received is: I have to transform above dataframe, and I want to have a column which has 'time interval' like 00:00 - 01:00 for every day(24 hour), I want to know if there is any ...
AMIT BISHT's user avatar
0 votes
0 answers
34 views

Pandas, iterating through a Dataframe is not writing values correctly

I'm breaking my head with this problem. I'm iterating a Dataframe thru iterrows and updating values with df.at[], but when I add an IF-ELSE statement, it seems not to work. This is the code: def ...
Marcelo Miquel's user avatar
-2 votes
2 answers
54 views

Create Rows+Columns from 2 lists in pandas DataFrame [closed]

I have 2 lists with different numbers of elements in the each list. And I want to create csv file with values from these lists like: projects = ['Project_1', 'Project_2', 'Project_3', ..., 'Project_N']...
macder's user avatar
  • 163
1 vote
1 answer
29 views

itertuples indices chosen by column name

While working with dataframes with 40ish columns I'd like to choose the column by their name and not their number. Is it possible? Given these two dataframes: Custom ID value cdf 4 abc 3 Nodo_Cod ...
alexm's user avatar
  • 11
0 votes
0 answers
16 views

Hourly data out of 15 mins interval dataset [duplicate]

I am looking for help in python code. I have data with 15 mins values for 1 year. I just want to extract the values at hourly interval. No need to apply mean or any function. Just take the data in a ...
Jawairia's user avatar
  • 295
1 vote
1 answer
33 views

openpyxl - Get the reference a cell in another Excel workbook

I am working with an excel workbook that contains references to other workbooks, e.g., example1.xlsx has cells like: =B2 + 'C:path\to\file\[example2.xlsx]Sheet1'!$B$1 But when I read it with openpyxl: ...
Brayan Muñoz's user avatar
0 votes
0 answers
36 views

How can we use Scipy Optimize to Maximize Revenue Columns in a Dataframe

I have a dataframe that looks like this. from scipy.optimize import minimize import numpy as np import pandas as pd data = [{'Month': '2020-01-01', 'Stadium':3000, 'Casino':3000, 'Airport':3000, 'Max'...
ASH's user avatar
  • 20.7k
1 vote
0 answers
45 views

Pandas: already use pd.concat but still trigger "PerformanceWarning: Dataframe is highly fragmented"

I met a strange behavior in python pandas. To prevent the "Dataframe is highly fragmented" warning, I already use pd.concat. But after that, when I add one more column in the concatted ...
user16865829's user avatar
0 votes
1 answer
32 views

Split pandas dataframe by rows indices if row indices are recurrent

I have a huge CSV file with data from different experiment trials. Whenever the data for a new trial begins, the indices will restart at 0. So my pandas dataframe then would look something like this: ...
Sa Ata's user avatar
  • 3

1
2 3 4 5
5668