Questions tagged [dataframe]

A data frame is a 2D tabular data structure. Usually, it contains data where rows are observations and columns are variables and are allowed to be of different types (as distinct from an array or matrix). While "data frame" or "dataframe" is the term used for this concept in several languages (R, Apache Spark, deedle, Maple, the pandas library in Python and the DataFrames library in Julia), "table" is the term used in MATLAB and SQL.

Filter by
Sorted by
Tagged with
1 vote
4 answers
36 views

Use grepl() to match two elements of a string with separators with two column values in a dataframe simultaneously

I have a dataframe as shown below and the desired output is shown below. df <- data.frame( col1 = c("abc_1_102", "abc_1_103", "xyz_1_104") ) selection <- data....
vp_050's user avatar
  • 617
0 votes
0 answers
22 views

Expanding lists as columns within an existing dataframe

I'll preface this by saying I'm relatively new to R and I'm what you might call a "brute force" learner -- I don't study the language, learn it, then put it into practice. Rather, I start ...
Simms34's user avatar
0 votes
0 answers
11 views

How to save varchar2 column in oracle using PySpark?

Since this bugfix I'm unable to save data as varchar, only clob. The problem is that oracle does not allow index on cblobs. I need to save ROWIDs in varchar2 columns to manage MLOG tables. My save ...
Barenko's user avatar
  • 23
0 votes
1 answer
28 views

How to read an excel file with rows as columns and columns as rows with Pandas in Python?

I have the following problem: I want to read an Excel file that was changed in its original form, using columns as rows and rows as columns (in this way the size of the excel file is reduced a lot, ...
Leober Ramos's user avatar
0 votes
0 answers
23 views

PANDAS EPOCH DATETIME CONVERSION

I have this dataframe in pandas ID Date Size Status 0 398 2022-02-14 04:01:43 2517210 1 1 398 2022-02-14 04:01:34 2517210 0 2 398 2022-02-14 04:01:26 ...
Nene Tetteh Wayo Akwetteh's user avatar
1 vote
2 answers
32 views

Exporting color coded data from R to Excel in a wide format

library(tidyr) # Create a data frame in long format long_data <- data.frame( digits = rep(1:5, each = 3), category = rep(c("A", "B", "C"), times = 5), value = c(...
Adnan Hajizada's user avatar
3 votes
3 answers
33 views

Splitting a column with delimiter and place a value in the right column

I have a data frame with a column that potentially can be filled with 3 options (a,b, and/or c) with a comma delimiter. import pandas as pd df = pd.DataFrame({'col1':['a,b,c', 'b', 'a,c', 'b,c', 'a,b'...
Yun Tae Hwang's user avatar
0 votes
1 answer
16 views

R - Loop into n columns of x dataframes and plot each variables in the same graph

I have been looking at other similar posts but cannot figure it out. I have a large number of dataframes, each having a large number of columns n. It's important to note that each dataframe has the ...
FanCou's user avatar
  • 15
0 votes
1 answer
22 views

Calculating new column based on multiple conditions in R

I have a data frame with the following sample format: ID Date Pressure_ft Pressure_psi 3 09-09 65 NA 4 09-09 NA ...
Jack Wassik's user avatar
0 votes
2 answers
37 views

Apply strsplit and ifelse condition accross multiple columns in R

I have a dataframe as shown below and the output from this code is shown below. Essentially, I want to read values from columns which can be of two formats as shown in col1 and col2. If the format is ...
vp_050's user avatar
  • 617
0 votes
1 answer
20 views

How do I collect all those rows/records from a dataframe which didn't pass this filter condition. any inbuilt methods or else?

I am still learning Scala/Spark. I am working on filtering of the dataframe based on some conditions or values. df.filter() - works well but I also want to store all those rows/records in another ...
Glarixon's user avatar
0 votes
1 answer
22 views

group_by multiple variables and calculating percentage within each subgroup in R

below is my data data <-tribble( ~fruit, ~transaction, ~season, ~n, "a", "sell", "s", 14, "a", "sell", "f&...
Mathica's user avatar
  • 1,241
0 votes
0 answers
31 views

How do I fill different values in the same column in pandas? [duplicate]

I have the following DataFrame: import pandas as pd d = {' ID': [1,2,3,4,5,6], 'Gender': ['Male', 'Male', 'Male', 'Female', 'Female','Female'], 'Weight': [150, 200, np.nan, 110, 130, np....
ABBY NYAKARA's user avatar
0 votes
0 answers
27 views

Filter data frame rows with character string in R

I have a large DF compiled from a list with 4 columns. Please see an example table below: Date Depth Pressure Level 2023-05-01 65 NA DRY 2023-06-...
Jack Wassik's user avatar
0 votes
2 answers
31 views

remove rows from dataframe using for loop

I have a dataframe (~1M rows) in which I would like to remove observations based on a condition. Some example data is below irm id state.h state.j 201403 113310E126 1 1 ...
jvalenti's user avatar
  • 608
0 votes
1 answer
24 views

How to Detect Missing String values for specific groups in a Long-Format Dataset in R?

I'm working with a long-format dataset in R where I have multiple groups and a column of strings denoting season (fall 2020, winter 2021, summer 2021, fall 2021, etc.). I expect each group to have ...
Christie's user avatar
0 votes
0 answers
23 views

sum of all previous rows in grouped data frame [duplicate]

In a grouped data.frame, I want to add a new column that sums up all previous values from another column. Here is an example data.frame: grouped_distances <- data.frame ( group = c (rep (&...
yenats's user avatar
  • 531
0 votes
0 answers
35 views

Use cut function with a list instead of a vector

I have a df with 3 columns, Subject, RT1 and RT2. Here is an oversimplified df. Note that RT1 and RT2 are not actually related. df<-data.frame(Subject = c(rep(1,15), rep(2,15)), RT1=rep(seq(100,...
L_ROTOLO's user avatar
-1 votes
1 answer
28 views

how do you groupby a field that contains a string

Facing a issue in using groupby function for data having column of strings , unable to use mean for groupby in my data its shows error as :Could not convert "string" to numeric tried going ...
Parag Gundal's user avatar
0 votes
0 answers
7 views

use Column values of Dataframe to fecth data from List

I want to get the weekday from the list passing index into the already defined list, but index value will come from a dataframe column. Issue is in last line where it is showing list indices must be ...
Mrinmoy's user avatar
1 vote
0 answers
12 views

Empty data frame is inserting the data - PySpark Left Anti

I have data frames when joined does not provide me the results. But when inserted, the data is getting inserted from first data frame though the join does not provide me the results. df1 = spark.read \...
Jim Macaulay's user avatar
  • 4,748
0 votes
0 answers
19 views

pyspark agg select from pivot column only if column exists

trying to select column based on pivoted data. Sometime it doesn't exists in a specific environment. How can we handle the select column if it doesn't exists in the pyspark dataFrame ? this is what I ...
NNM's user avatar
  • 368
0 votes
1 answer
24 views

Inserting value to pandas DataFrame to specific cell without deleting other values in column [duplicate]

I have a DataFrame df with roughly this format: alpha result 0.5 NaN 1 0.89 1.5 NaN I want to insert a new value where alpha == 0.5 in the result-column. When I try that, however, the existing ...
clusty_dusty's user avatar
0 votes
0 answers
14 views

look up for a parent record in the same table based on the list of boundary applied on another column

I have two dataframes one with a key and class family, another df with key and parent key and the list of class families(consider this list as a boundary). we need to find the parent that goes outside ...
emma's user avatar
  • 43
0 votes
0 answers
23 views

Why does the size of an R dataframe seem to depend upon its dimensions?

Why does the size of an R dataframe seem to depend upon its dimensions ? > z <- matrix(runif(10000)) > data.frame(matrix(z, nrow=200) ) %>% object.size() 86608 bytes > data.frame(matrix(...
Sam's user avatar
  • 127
1 vote
1 answer
17 views

R Find the earliest date of a group and create a factor variable Y/N to indicate if it is the first date of the group

I have a dataframe that looks like that: Issuer.Name Issue.Date name1 01/12/2021 name2 05/04/2022 name2 21/10/2021 name3 08/09/2020 name4 30/08/2023 name4 12/05/2021 name4 ...
Bertrand G's user avatar
1 vote
1 answer
21 views

dropna is filling all the column with None rather than just the NaN values

Working with a DataFrame about places to start and places to go, kind of a Uber app. It is given to me a DataFrame with 2 columns that have lots of NaN values: Some of them are NaN and other are ...
Manuel Veiga's user avatar
-6 votes
2 answers
49 views

Transform data by grouping column value then getting all its grouped data using Panda DataFrame

I got this structure of data from csv file. string | group | datetime some_string apple somedatime some_string apple somedatime some_string apple somedatime some_string ...
Marc Anthony B's user avatar
0 votes
0 answers
25 views

Encoding Error With pandas read_CSV file context

I was trying import 15 CSV files using pandas and and glob I have to create one combine data frame which is used to manipulated my data but I was importing the data that time I getting the error and ...
Manav Shah's user avatar
0 votes
0 answers
35 views

How can I align a merged dataset?

I have two datasets. One with supplies, other with sales. They have different dates and time. Suppplies Year Month Day Hour Item 0 2023 05 17 10 8 ...
Olsen_I's user avatar
1 vote
2 answers
46 views

Pandas merge data frames as of certain index

I have two pandas data frames: x_axis: index date 1 01-01-2023 2 02-01-2023 3 03-01-2023 4 04-01-2023 5 05-01-2023 6 06-01-2023 7 07-01-2023 8 08-01-2023 9 09-01-2023 df: index snap_date ...
Paul1911's user avatar
  • 173
1 vote
1 answer
51 views

Pandas groupby use while combining and inserting information on a column based on another column conditions

I have a dataframe like below: new_df = pd.DataFrame([('231', '2122', '1', 'some_text', 'agent_text_1', 'cust text_1'), ('231', '2682', '2', 'some_text', 'agent_text_2', 'cust ...
Bilal Sedef's user avatar
0 votes
1 answer
24 views

Array Index Out of Range Error When Accessing RSI Values in Backtrader

I'm working on a Python project using the Backtrader library to calculate the Relative Strength Index (RSI) for a financial dataset. However, I'm encountering an "array index out of range" ...
appancrp's user avatar
-2 votes
0 answers
39 views

Plotting a continuous 40 year Histogram data [duplicate]

I am trying to plot daily recorded data over forty years(14k+ data size). For some reasons, my histogram is faulty and all over the place(irregular and some datapoints not starting from base). I am ...
Sarthak Rawat's user avatar
0 votes
1 answer
15 views

Is there a way to get the average over X minutes while also averaging based on groups in R?

I'm trying to get the average temperature over a certain amount of time (ex. 30 min) based on the groupings in the code below group_by(Type, Rainfall, datetime). I'm thinking this is something I'll ...
Sien Smania's user avatar
1 vote
3 answers
54 views

How to select columns from a dataframe whose names are contained in another series?

I have a series A that looks like: data = {'Animal':['a.Bear', 'b.Elephant', '123.Giraffe', 'Kangaroo']} A = pd.DataFrame(data) Animal 0 a.Bear 1 b.Elephant 2 123.Giraffe 3 Kangaroo And ...
Mitch's user avatar
  • 544
0 votes
0 answers
15 views

Pandas_profiling error (I tried ydata_=profiling too)

I am trying to make a profile report on a data i imported to pandas. but it give me a pydantic error import pandas_profiling profile_report = df.profile_report() i also tried methods like from ...
Unreal Capacity's user avatar
0 votes
0 answers
13 views

Pyspark, How to Write Dataframe to text file using UFT-8_BOM encoding [closed]

I want to write draframe to text file, this file need to be save on Azure ADLS2 with encoding UFT-8_BOM, Please let me know how to proceed. Something like below code: df1.coalesce(1).write.format(&...
Gouri Mahapatra's user avatar
0 votes
0 answers
29 views

How can I parse out information I need out of column one with no delimiters in the html I'm scraping from

I'm scraping data from this website to get the name of the beer, the brewery, the alcohol percentage, and average rating. The issue I'm running into is the beer name, brewery, and alcohol rating are ...
Natphipps's user avatar
0 votes
2 answers
38 views

Select all df cols that have only two specific values, then drop them?

I've got a huge df (70k cols, all string format) that I'm trying to do some feature reduction. a noticeable amount of the cols have records = "skip" or NaN. .all the other cols have a range ...
surfer349's user avatar
  • 107
0 votes
0 answers
28 views

How to calculate IRR for each investment in a dataframe?

If I've created a scrubbed Pandas dataframe containing investments, dates and cash flows, anyone have tips for calculating the IRR for each investment according to the cash flows in the dataframe? I'...
afkkek's user avatar
  • 1
1 vote
2 answers
48 views

generate a pattern dataframe using R

I wish to make a dataframe as below. and I wish to have 3 parameter x,y,z, x is number of "a" (number of "b" = column number, at first column, number of "b" is 1), y is ...
user3047732's user avatar
2 votes
1 answer
27 views

PySpark: Groupby within groups and display sum in separate fields based on certain values

I have a dataframe which contains Company name, EmpId, Bonus & Salary. COMPANY EMPID BONUS SALARY APPLE 1234 No 5 APPLE 1235 No 7 GOOGLE 6786 Yes 6 GOOGLE 6787 No 5 GOOGLE 6788 No 6 TARGET ...
sanju's user avatar
  • 31
0 votes
0 answers
10 views

How to merge data frames

Hi I'm a beginner with R and trying to merge two data frames under specific conditions. I have a data frame called 'carabidae_data'. It has a column 'DATE' with dates, many of the same date values ...
user22647165's user avatar
1 vote
2 answers
34 views

r: how to mutate a new category based inputs from group_by() with multiple and different information?

Let's say I have: > df RECNUM diag.letter diag.number C_DIAG 1 A S 6 DS060 2 A T 15 DT151 3 A S 6 DS061A 4 B ...
cmirian's user avatar
  • 2,620
1 vote
2 answers
52 views

How do I forward fill on specific rows only in Pandas?

I have the following df: col1 col2 col3 0 a 1.0 1.0 1 b NaN NaN 2 c 5.0 3.0 3 d 5.0 NaN I want to do forward fill the rows only where col2 = NaN so the result ...
spg719's user avatar
  • 63
0 votes
3 answers
44 views

How to rename pandas column names by splitting with space

I have a dataframe like this: index col A col B col C index1 3 1 2 index2 1 4 9 index3 5 1 2 index4 8 2 2 index5 2 1 6 ...
Yash's user avatar
  • 287
-1 votes
1 answer
20 views

How to apply function to multiple rows [duplicate]

I have a data frame which consists of a column named "Time". Id like to add another column named "Time_Diff", which displays the difference between the value in the next "Time&...
Ido's user avatar
  • 1
1 vote
1 answer
27 views

scale values in one column based on separate column

I'm looking to scale values out of 100% based on the sample. My df contains the col Project that I'd like to group the col n by to normalize those values of n on a scale to 100%. library("dplyr&...
Geomicro's user avatar
  • 327
0 votes
3 answers
56 views

add total row count in dataframe without impacting column values

My data is as below Sec issues Gaps AAA 22 5 BBB 33 2 CCC 44 1 Desired result Sec issues Gaps AAA 22 5 BBB 33 2 CCC 44 1 TOTAL 99 8 I try below code df.append(df.sum(numeric_only=True), ...
user3044949's user avatar

1
2 3 4 5
2882