Questions tagged [dataframe]

A data frame is a 2D tabular data structure. Usually, it contains data where rows are observations and columns are variables and are allowed to be of different types (as distinct from an array or matrix). While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
1 vote
2 answers
28 views

Python - How to dynamically create dataframes with a for loop

My code currently looks like this: df_1 = portfolio_all[0].rename(columns={'Close': 'Close_1'} ) df_2 = portfolio_all[1].rename(columns={'Close': 'Close_2'} ) df_3 = portfolio_all[2].rename(columns={'...
adrCoder's user avatar
  • 3,167
0 votes
1 answer
31 views

Python - merge 4 or more Dataframes

I've read the following thread: How to merge multiple dataframes data_frames = [df1, df2, df3] df_merged = reduce(lambda left,right: pd.merge(left,right,on=['DATE'], ...
adrCoder's user avatar
  • 3,167
-3 votes
0 answers
50 views

Using Data Frame vs List [closed]

In Python which is more efficient in terms of speed and storage? my_list = [1, 2, 3, 4, 5] data = {'Column_Name': [6, 7, 8, 9, 10]} df = pd.DataFrame(data) Which is better? in terms of storage and ...
Tim's user avatar
  • 1
0 votes
1 answer
44 views

Having an issue converting pandas dataframe 'object' column into a float

I am trying to convert 2 columns in a pandas dataframe into float values as they are currently listed as 'object'. When I try to convert using .astype() the one column is then placed with value 0 ...
just_trying_'s user avatar
0 votes
0 answers
11 views

How to locate all rows of a pandas data frame where 'date' is in a particular set? [duplicate]

I have a set of dates in an array called 'signals'. I also have a pandas dataframe, with each row containing a different date's numbers. I want to pull all the 'Adj Close' values from a particular set ...
ak231472's user avatar
0 votes
0 answers
8 views

Does folium allow a html div box that updates its value based on coordinates on the map?

im new to using folium library so pardon my ignorance So what i am trying to achieve here is a div box that shows the speed of the vehicle travelling along the lat and lon provided.The current problem ...
M Tee's user avatar
  • 33
0 votes
0 answers
13 views

Copy of a slice from a DataFrame [duplicate]

I wan to execute the following code in Jupiter: cframe = frame[frame.a.notnull()] cframe['os'] = np.where(cframe['a'].str.contains('Windows'),'Win', 'Not Win') and get this error: use this instead : ...
Mostafa Hosseini's user avatar
0 votes
0 answers
27 views

How to determine the chunk limit based on the contents of the file I want to read in chunks?

I want to work with a huge csv file, but I get OOM errors when trying to read it into a dataframe in one go. The solution would be to read it in chunks, but the issue is that I want to calculate ...
Zegher V's user avatar
  • 123
-1 votes
0 answers
38 views

How to group values by category

How I can group the table by category (percentage should be calculated per group. I need to calculate the % of the category from the total number that got into the TOP 15 (746 markets) and show it on ...
unifly's user avatar
  • 1
-1 votes
1 answer
33 views

How to get proper value instead of "NA_character_" in pandas dataframe while calling R function from Python?

I'm calling a r-function from python script to apply smote on a dummy dataset. Here the majority class is 0(90%) and minority class is 1(10%). While calling r function directly giving me proper output ...
user22's user avatar
  • 120
0 votes
1 answer
22 views

Python Column in Percentage

I need someone's help with a data analysis issue. I am analysing a dataframe with data about sleep. It has 5 columns as follows: data frame Enough - Do you think that you get enough sleep? Hours - On ...
Paulo Henrique's user avatar
1 vote
0 answers
26 views

How to sample from a PySpark dataframe where each row has a probability value?

I have a Spark dataframe of 4.000.000 rows, and two columns, sample_id and probability, where the probability value was computed bu normalizing a weight between 0 and 1. I have to sample a subset of ...
Sara Munafò's user avatar
0 votes
1 answer
52 views

Converting list of dictionary to data frame and merging to orginal dataframe

I have a column in the data frame with a list of dictionary. import requests url = "https://data.maldita.es/ukrainefacts" resp=requests.get(url) data=resp.json() df=pd.DataFrame.from_records(...
Alexandra's user avatar
0 votes
2 answers
44 views

i can't use string replacing function in a data frame in python

i want to clean my data in a column of dataframe and omit 'k' character through it, but the function doesn't work . the string is : '$111K-$181K (Glassdoor est.)' when i use the function above i can't ...
iman maneshi's user avatar
0 votes
2 answers
52 views

Add a common string prefix to elements in DataFrame column to get pre-defined string length

I have an example DataFrame: df = sample_id data_sample 1 0.9 12 0.4 178 0.8 300 0.2 I'm trying to make a new column, sample_id_full ...
MAtennis9's user avatar
  • 127
0 votes
1 answer
23 views

'Workbook' object has no attribute 'write'

I plan to save the average data of the same month (a separate sheet in the Excel file for each month) in the last sheet and in the row corresponding to the average data after each time the following ...
m.mahdi.sangtarash's user avatar
-1 votes
0 answers
16 views

both the models give the same accuracy

I wanted to compare accuracy of SVM and Naive Bayes but they keep giving the same training accuracy of 1.0, testing is different. I am using a very small dataset of 200 true and false sentences. I ...
Grace's user avatar
  • 3
-1 votes
0 answers
43 views

What is the best way to convert data in python multidimensional lists from JSON to Excel table?

I have this JSON output from Python which includes lists in lists. I want to be able to look at it in an Excel table. Is there a command for such a conversion? [[[[0.0, 0.0, 0.0, 0.0, 1000.0], [0.0, 0....
Sherwin's user avatar
  • 31
2 votes
2 answers
40 views

How to create column of ratio for different groups

I need to calculate the ratio of basketball players shots attempts, for two different teams. Each row is one attempt. So, in the first row below, the player of team "D", shot and he made the ...
DR15's user avatar
  • 669
-2 votes
1 answer
50 views

Get rows whose status changed to `[4,5,6,7,8]` => COMPLETED states [closed]

Sample Data: created_at updated_at id event_id status_id 2022-11-18 9:46:21 2022-11-18 9:46:21 1 76 1 2022-11-18 9:46:22 2022-11-18 9:46:22 2 78 4 2022-11-18 9:46:22 2022-11-18 9:56:22 3 78 5 ...
anotherCoder's user avatar
0 votes
0 answers
35 views

Calculating the timestamps in a pandas dataset with duplicate values

I have a dataset containting IP, URL ID and timestamp columns. The IP column contains various IPs and each IP has one or multiple URL IDs as targets. Multiple IPs can share the same URL as the target ...
Niloofar Nazemi's user avatar
0 votes
3 answers
60 views

Pandas DataFrame: Replace a column by a list with a new column name

I have a Pandas DataFrame and want to replace a column with a new one. The column should have a new name and a list gives its content. I can do this in two different ways. See the code below. import ...
hbaromega's user avatar
  • 2,316
0 votes
0 answers
24 views

Create Pandas Dataframe from List of Dictionaries with missing keys [duplicate]

say for example: d1={'a':1, 'b':2, 'c':3} d2={'b':5, 'd':10} d3={'e':7, 'a':8} and I would like to create a df which can look like this: intended outcome I know one way to do it is to update the ...
Austin's user avatar
  • 9
0 votes
0 answers
37 views

Unable to Read From Zipped XLS files using Python zipfile module

I have a subroutine that takes a collection of outlook messages, saves the attached zip files as temporary files and reads the tabular data to pandas data frames. It seems to work fine for csv files ...
Possdawgers's user avatar
0 votes
0 answers
29 views

Logic in PySpark Dataframe to allocate a value over N rows given intervals

This is my first post here and I would need your support to achieve this result. I have a dataframe which contains values like : original df What i want to achieve is something like this: final df Do ...
aR7icx's user avatar
  • 1
1 vote
1 answer
48 views

Literal match of strings in dataframe to other dataframe with multiple match options

I have a dataframe (df) with values in a column 'country' that I wish to standardize, using another dataframe called 'country_codes'. A value from df can match against any item from 'country_codes', ...
Benjamin Allen's user avatar
0 votes
2 answers
41 views

Comparing 2 Pandas dataframes row by row and performing a calculation on each row

I have created 2 Pandas dataframes and I need to compare every row in DF1 to every row in DF2 if the rows match based on 2 column variables. Basically I need every row in DF2 to be compared to row 1 ...
Eslifkin's user avatar
1 vote
5 answers
63 views

Replace with NA in the whole dataframe using wildcards

I have a dataframe df like this Col1 Col2 Col3 1 2 5 6 x1 9 8 x3 7 5 3 x4 I want to replace all values starting with X by NA. The result should be Col1 Col2 Col3 1 2 5 6 NA 9 8 NA 7 5 3 NA ...
Bertrand G's user avatar
0 votes
1 answer
16 views

Exploding a StructType column in PySpark

Is there a way to explode a Struct column in a Spark DataFrame like you would explode an Array column? Meaning to take each element of the Struct (a key-value pair) value and create a separate row for ...
Filip Megiesan's user avatar
0 votes
2 answers
38 views

converting a list object to a dataframe with two columns

I have a list object "list" containing multiple lists with this structure below. All objects in the list have two letters like this: [[1]] [1] "A" "C" [[2]] [1] "A&...
jacbin76's user avatar
2 votes
1 answer
53 views

Python merge two dataframes (fuzzy matching, some columns exactly match one does not)

I have two dataframes: df1 district place year votes candidate 1 1 2000 25 bond james 1 1 2000 30 smith john peter 1 1 2000 10 caprio leonardo di 1 1 ...
MG Fern's user avatar
  • 119
1 vote
1 answer
39 views

Aggregate dataframe by dimension pairs for heatmap

I have a table with rows being individual visits from users to an online shop. There are multiple columns that are attributes of the visit (could be booleans or categorical with more than 2 possible ...
fredibeni's user avatar
0 votes
1 answer
21 views

Subset dredge with variable names, inside a data frame (MuMIn)

I have fitted models, and run dredge() on them. I have created a data frame that stores variable names used in the model, and the dredge object themselves (inside a list). By using rowwise(), I was ...
M. Riera's user avatar
  • 103
0 votes
2 answers
18 views

R: Replacing multiple values across multiple dataframes

Is it possible to replace values across multiple dataframes in R? I understand that one can replace values in a specified column in one dataframe, as so; df$y[df$y == 200] <- "" However, ...
Lorcan Kelly's user avatar
1 vote
3 answers
60 views

Pandas DataFrame: Filter out "flat lines" (values, that do not change for n rows)

I want to filter out data points who's values are equal for at least n rows. Specifically, detecting equal values for at least n rows and filter out all the consecutive equal data too. Here an example ...
theuema's user avatar
  • 107
1 vote
2 answers
28 views

rbind 2 data frames by the elements in 2 columns, avoiding nested loops

I have two data frames like the following: df1 <- data.frame(Marker1=c('+','+','+','-','-'), Marker2=c('+','+','+','+','-'), Marker3=c('+','-','+','-','+'), Sample=c(1,1,2,3,3), ...
DaniCee's user avatar
  • 2,455
0 votes
0 answers
51 views

How to include brackets as part of a string in a linear regression model formula?

I am currently looping through column headings in data frame imported from excel that contains brackets. Example of what the data frame looks like: df<-data.frame("ID"=c(233,233,243,243), ...
JLit98's user avatar
  • 17
0 votes
1 answer
56 views

String is truncated when I read data using Pandas library

I have a dataframe in .csv format. I have a column which has some cells that have data in a string which is very long (upto >5000 characters). When I read the data from that cell, I noticed that ...
Willy's user avatar
  • 1
0 votes
1 answer
37 views

Basic calculation with pandas data frame [duplicate]

I have dfA, 1 2 5 2 3 6 3 3 8 and dfB, 2 4 1 How to multiply first element in B with all elements in first column from A, second element from B with all elements in second column from A ...
user16454053's user avatar
1 vote
4 answers
61 views

Use grepl() to match two elements of a string with separators with two column values in a dataframe simultaneously

I have a dataframe as shown below and the desired output is shown below. df <- data.frame( col1 = c("abc_1_102", "abc_1_103", "xyz_1_104") ) selection <- data....
vp_050's user avatar
  • 619
0 votes
0 answers
54 views

Expanding lists as columns within an existing dataframe

I'll preface this by saying I'm relatively new to R and I'm what you might call a "brute force" learner -- I don't study the language, learn it, then put it into practice. Rather, I start ...
Simms34's user avatar
0 votes
0 answers
21 views

How to save varchar2 column in oracle using PySpark?

Since this bugfix I'm unable to save data as varchar, only clob. The problem is that oracle does not allow index on cblobs. I need to save ROWIDs in varchar2 columns to manage MLOG tables. My save ...
Barenko's user avatar
  • 23
0 votes
2 answers
52 views

How to read an excel file with rows as columns and columns as rows with Pandas in Python?

I have the following problem: I want to read an Excel file that was changed in its original form, using columns as rows and rows as columns (in this way the size of the excel file is reduced a lot, ...
Leober Ramos's user avatar
0 votes
0 answers
50 views

Epoch Datetime Conversion

I have this raw Data. An excerpt of the raw data was: data2 = """ |--------------------------------------------------| | Stored images: | |-----------------------------...
Nene Tetteh Wayo Akwetteh's user avatar
1 vote
2 answers
39 views

Exporting color coded data from R to Excel in a wide format

library(tidyr) # Create a data frame in long format long_data <- data.frame( digits = rep(1:5, each = 3), category = rep(c("A", "B", "C"), times = 5), value = c(...
Adnan Hajizada's user avatar
3 votes
3 answers
38 views

Splitting a column with delimiter and place a value in the right column

I have a data frame with a column that potentially can be filled with 3 options (a,b, and/or c) with a comma delimiter. import pandas as pd df = pd.DataFrame({'col1':['a,b,c', 'b', 'a,c', 'b,c', 'a,b'...
Yun Tae Hwang's user avatar
0 votes
1 answer
19 views

R - Loop into n columns of x dataframes and plot each variables in the same graph

I have been looking at other similar posts but cannot figure it out. I have a large number of dataframes, each having a large number of columns n. It's important to note that each dataframe has the ...
FanCou's user avatar
  • 15
0 votes
1 answer
25 views

Calculating new column based on multiple conditions in R

I have a data frame with the following sample format: ID Date Pressure_ft Pressure_psi 3 09-09 65 NA 4 09-09 NA ...
Jack Wassik's user avatar
0 votes
2 answers
42 views

Apply strsplit and ifelse condition accross multiple columns in R

I have a dataframe as shown below and the output from this code is shown below. Essentially, I want to read values from columns which can be of two formats as shown in col1 and col2. If the format is ...
vp_050's user avatar
  • 619
0 votes
1 answer
31 views

How do I collect all those rows/records from a dataframe which didn't pass this filter condition. any inbuilt methods or else?

I am still learning Scala/Spark. I am working on filtering of the dataframe based on some conditions or values. df.filter() - works well but I also want to store all those rows/records in another ...
Glarixon's user avatar

1
2 3 4 5
2882