Questions tagged [partitioning]

Partitioning is a performance strategy whereby you divide possibly very large groups of data into some number of smaller groups of data.

Filter by
Sorted by
Tagged with
-2 votes
0 answers
22 views

How to fix my HDD that thinks that it has 20GB when it really has 128GB [closed]

My dad screwed up my HDD while trying to install Ubuntu and windows on separate partitions so now it says it has 20.8GB of space on the disk and it really has MUCH MORE but I don't know exactly how ...
batchBoy3's user avatar
-2 votes
0 answers
30 views

Sharing Xcode and Project repo between two volumes on one Machine [closed]

I'm developing on a macOS application and currently on OS Ventura 13.6 .. Now I need to integrate and test it on macOS Sonoma so I'll partition the drive and add a new volume on my MacBook to install ...
Omar Taha's user avatar
0 votes
0 answers
7 views

Difference between static partitioning hypervisors and separation kernel / microkernel?

I am trying to get an understanding of virtualization approaches. This paper describes static partitioning hypervisors. In this survey paper the category separation kernels and microkernels is ...
Finn 2808's user avatar
0 votes
2 answers
34 views

Best method for updating SQL Server partition function

I have several large SQL Server tables where the current partition functions are not extended beyond this month, and the procedure that is supposed to split the partition function into new date ranges ...
user8675309's user avatar
0 votes
1 answer
28 views

Add id for a series of events and that breaks once the same event appears

I have a dataset that looks like contains columns like id and event. What I would like to add is desired_id which gets incremented every times where is value impression in the column event. Could you ...
Norah Jones's user avatar
1 vote
1 answer
45 views

How to find the partition metadata of an extent in Azure Data Explorer

See this for background: Azure Data Explorer: How do Partitioning Policy and Merge Policy work? I have created a table with a Partitioning policy based on both a string and timestamp column. The ...
karl 's user avatar
  • 15
0 votes
1 answer
18 views

Milvus Data access control [closed]

I am using milvusDB and looking for providing subscription model, and facing providing efficient way to provide data access control. I am planning to establish a collection in Milvus that will ...
Gagan Mehta's user avatar
0 votes
1 answer
18 views

Apply partitioning and change clustering for replicated data base to BQ with DataFusion

I replicate data from MySQL data base to BigQuery using DataFusion. My original table in MySQL is not partitioned but I want it to be partitioned by a column when it is replicated to BQ. Additionally ...
Marcel Mars's user avatar
0 votes
0 answers
12 views

Trying to dual boot windows 11 with Pop1_Os, what is correct or best disk partition for this?

Taking a course on Linux and learning how to code , but I have many issues with playing league of legends in my free time. So I decided to dual boot. I have 2 nvme m.2 sticks (each 500gb) and a 240 ...
Abdalla Zubaidi's user avatar
0 votes
0 answers
27 views

@BeforeStep defined in ItemProcessor and ItemWitter is not getting called with spring batch partitioning

We were moving our spring batch implementations to use partitions. After making changes now the @BeforeStep is not getting called which is inside our processor and writter. Sample code here, @Bean ...
Shreyas Holla P's user avatar
1 vote
1 answer
68 views

Number Partitioning: non-recursive algorithm with determinate loops only

I am trying to generate the number partition set of a non-negative integer. There are several solutions, also on this site, like in Number of ways to partition a number in Python: def P(n): # ...
J. Ianni's user avatar
-1 votes
0 answers
53 views

Will be slow if I query with the 2nd part of a composite clustered primary key in a partitioned table?

I have a big partitioned table (close to a billion rows), which has a clustered primary key on the partition column and an identity id key (dateSubmitted Date, id long) and want to query it using only ...
DimitrisK's user avatar
-1 votes
1 answer
41 views

Best practices for primary keys in partitioned fact tables in a data warehouse with PostgreSQL

I'm seeking advice on best practices regarding the implementation of partitioned tables in a data warehouse when using PostgreSQL. My current understanding is that partitioning a fact table by a ...
kirill fedorov's user avatar
-2 votes
0 answers
18 views

Token authentication problem with remote connection area

I am using Cisco AnyConnect to access the VPN, and after accessing the VPN I access the remote desktop by providing username and password. This remote server has Windows installed. Yesterday I ...
Sakurai's user avatar
0 votes
0 answers
27 views

Syntax template for SAS partitioning of variance

For class I’m asked to see analyze the variance individually- with the presence of other variables AND WITHOUT the presence of variables. What is the code to partition variance (I know I will input my ...
Ava Wolski's user avatar
0 votes
1 answer
43 views

Partitioning of data in adx table

I was going through the microsoft docs of partition policy here. There it is written that By default, extents are partitioned by time of creation (ingestion), and in most cases there's no need to set ...
Saurav Likhar's user avatar
-1 votes
0 answers
41 views

Huge slowdown of basic queries when using a partitioned table

I have partitioned table "Transactions" with partition by mouth (Partition_Key = 'YYYYMM'). I see a strong performance degradation if i use basic queries to select one value of indexes ...
Petr Novák's user avatar
1 vote
1 answer
39 views

Spark Partition: Do I need to use the partition column in the query in order to take advantage of the performance partition?

I have a table in Spark partition by a column year_month (eg. 2020-01, 2020-02), but all the queries are done using a different column from with the year_month is derived date (e.g. 2020-01-30, 2020-...
Rednaxel's user avatar
  • 958
0 votes
1 answer
18 views

RxJava: how to extract blocks form a source observable?

I have [1,2,3,2,4,5,0,2,1,3,4,5,2,2,1,4,5] What I want as output is [[1,2,3,2,4,5],[1,3,4,5],[1,4,5]] This means I want to extract blocks from the input sequence with some start marker, here 1, and ...
Juergen's user avatar
  • 3,541
0 votes
0 answers
65 views

Discrepancies in PySpark DataFrame Results When Using Window Functions and Filters

When I do certain types of transformations on a dataframe that involve window functions with filters I am getting the wrong results. Here is a minimal example of the results I get with my code: from ...
Dani's user avatar
  • 514
0 votes
0 answers
32 views

How does Databricks ingestion time clustering work for read queries?

I was reading this document recommanding using the default ingestion time clustering rather than partition tables that contains less than a terabyte of data. But it's not convincing enough for me for ...
jli's user avatar
  • 83
1 vote
1 answer
43 views

It's normal that my data loading in postgresql with partitions takes too long?

I have a database that has to have approximately 20,000 records per day. From 2022 to 2024. I have created 6 partitions for each quarter of the years 2022 and 2023. In total there are 14 million ...
Emanuel Burgos's user avatar
1 vote
1 answer
69 views

How can I use hibernate validation with a partitioned table in Postgres?

I created a table with the following code: CREATE TABLE my_table( id BIGSERIAL, PRIMARY KEY(id) ) PARTITION BY RANGE (id); CREATE TABLE my_table_2023_10 PARTITION OF my_table FOR VALUES FROM ...
QKQL's user avatar
  • 35
1 vote
1 answer
53 views

InvalidInputException (status: 400): Partition index not supported for the given partition key type

I am trying to create an index on a partitioned column in AWS Glue for Athena. I keep getting an error: Unable to complete request to create index. InvalidInputException (status: 400): Partition ...
Forest Wilt's user avatar
0 votes
0 answers
29 views

Which is the best way of duplicating a table partitioned with hasing in PostgreSQL?

I have a partitioned table (using hashing) in a database hosted on PostgreSQL (it's an RDS on AWS). The table is more or less like this: CREATE TABLE snapshot ( url TEXT NOT ...
filmac's user avatar
  • 187
0 votes
2 answers
67 views

Use SQL to get session duration between two events

I've got a dataset (in postgres specifically) that I need to calculate the 'user session time' for. I'm fairly positive I need to use partitioning to answer the question, but I'm not really sure how ...
Greg's user avatar
  • 49
1 vote
1 answer
68 views

Randomly create a vector within specific requirements

I want R to create a vector for me that has length 15, each of the 15 values being in the range 0-100 and together adding up to 100. So an example outcome could be: `[12, 25, 2, 0, 6, 17, 2, 4, 1, 8, ...
LOS_2023's user avatar
0 votes
1 answer
52 views

SQL using Case When statement with MAX and Partition Over in Where Clause

I have this query using a case when expression inside a MAX() aggregate function to obtain the maximum value of the case when statement. However I need to filter also off that maximum value in the ...
js468718's user avatar
0 votes
1 answer
49 views

SQL Server Partitioned tables and full text indexing

Using SQL Server 2016+ I have a products table that requires partioning. It currently has a primary Identity key which is used as part of a full text index search. We need to partition the table a a ...
Matthew Baker's user avatar
0 votes
0 answers
59 views

How to iterate through all partitions in a PyArrow FileSystemDataset for aggregations

I have a parquet dataset partitioned by year, and I am trying to iterate through each year partition to do aggregations on each one, instead of loading the entire dataset into memory at once. The ...
Stephen Nicholson's user avatar
0 votes
0 answers
14 views

Hazelcast - DurableExecutorService behaviour whilst repartitioning

I'm looking for clarity on the behaviour of the DurableExecutorService during the repartitioning of a cluster. Imagine we have submitted a runnable to be executed on the key owner of key1. Whilst this ...
RobT's user avatar
  • 1
0 votes
1 answer
52 views

Performance impact when querying without partition key on a partitioned table

Assuming I partition a table of large size (say 6TB) by month. We are serving very high traffic of around 300 RPS. Sample schema Create table mydata( id bigint not null, date_created not null, ...
Harshita Mishra's user avatar
0 votes
0 answers
62 views

How do I determine whether or not postgres table partitioning is helpful?

I'm currently trying to determine whether or not postgres table partitioning is beneficial. Background: I have a table with 40 GB and around 200 million rows. The copy, merge, and regenerate jobs in ...
SPG's user avatar
  • 31
0 votes
0 answers
47 views

DynamoDB Table Design for User/Groups

I've been reading up on DynamoDB table design. I'm still not 100% sure on the best approach for this. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-modeling-nosql-B.html https://...
JonWillis's user avatar
  • 3,148
1 vote
1 answer
33 views

Creating Session ID Buckets

I am trying to create a session_id column that groups a bunch of sessions into buckets based on if there has been a 30 minute gap... ID Session_TM 001 00:15:12 001 00:17:17 001 01:00:00 001 01:55:...
TheJammyHamster's user avatar
0 votes
1 answer
31 views

How to speed up a writing and reading speed from data mart via splitting it into multiple parts in data warehouse and use it in OBIEE?

In our datawarehouse, we have a huge and one of the most used and important table, that stores account data(their ids, saldo corresponding to operational day, etc). On a daily basis, approximately 90 ...
Sherzodbek's user avatar
0 votes
1 answer
25 views

GCP BigQuery wont access a single partition

I'm experiencing a problem with a GBQ query which goes through all the data in a table, even though the table is partitioned. The partitioning is set to DAY on the datetime field. Currently the query ...
chocho.boss's user avatar
1 vote
2 answers
40 views

How to set partition size for date field in Databricks

I have timestamp field source_timestamp which is used a lot in the queries to the delta table. I would like to partition by this column with size of the partition = 1 month. How to achieve that? If I ...
archjkeee's user avatar
0 votes
0 answers
17 views

partion on date column + X days

Trying to drop duplicate transactions by account and date, that may have ocurred on the same date or within 10 days of the original transaction date. Also need to drop some test transaction data. Have ...
Rich's user avatar
  • 27
0 votes
1 answer
28 views

Postgresql index / partition optimization

We have an important table "client" in a postgresql 13 database, which has around 5Million rows. This table is linked to another table company with 2000 rows. We have 1 company with 2 ...
Cojy's user avatar
  • 50
0 votes
0 answers
12 views

Greedy number partitioning to minimize the maximum

I have a list l with 4 * n numbers. I have another array a with n numbers. All of them are integers. I want to partition l into n subsets: l1,l2,...ln. I add the first number of a to l1, second number ...
user1691278's user avatar
  • 1,813
0 votes
0 answers
62 views

Rebuilding Multiple Partitions of a Table with Clustered Column Store Index

I created a Table with partitions (on the date column) of a table with a clustered column store index. I am able to run Index Rebuilds on each partition as needed, which is definitely faster than ...
Vid's user avatar
  • 1
0 votes
1 answer
53 views

Partition 0CALDAY (yyyymmdd) column to year, month and day in a copy activity of Azure Data Factory

I am using SAP HANA connector to copy the data from SAP HANA to Azure data lake. In my query there is a column called 0CALDAY which is a string data type and has row values like 20230119. I would like ...
Lav Mehta's user avatar
1 vote
1 answer
42 views

MariaDB partitioning with char + date columns for PRIMARY KEY

We would like to partition an existing table of 20m rows by month but the documentation says, before starting partitioning, the table needs to have a composite PRIMARY KEY by the id (AUTO_INCREMENT) ...
mmac-rio89's user avatar
0 votes
0 answers
28 views

mysql partition with no primary key

I'm studying partitions, and I'm wondering what's different about the two situations. CREATE TABLE users( user_id INT NOT NULL, birthYear DATETIME NOT NULL, PRIMARY KEY (user_id, ...
Anjaehyun's user avatar
-3 votes
1 answer
172 views

Why partitioning key must be part of unique/primary key in MySQL?

I know from documentation and from lots of questions on stack overflow that whenever I am partitioning a table, I require a partition key and that partition key must be included in unique/primary key. ...
Avinash Pawar's user avatar
0 votes
0 answers
78 views

Oracle: partition a table with more than 1B records

Database: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 Issue: in production we have a table with more than 1B records that we are not able to query since the table is not partitioned/...
Mathew Linton's user avatar
0 votes
0 answers
52 views

Partitioning existing tables using PostgreSQL 13

I'm using PostgreSQL 13.8 and i have a lot of big tables (with a lot of records) and queries are spending too much. I though it's a good idea to partition some tables to speed them up. But i'm totally ...
Matias Wajnman's user avatar
0 votes
0 answers
30 views

Cassandra partition with modulo-based timestamp

I will be capturing an event stream as records in a Cassandra table. I might expect perhaps 1000 events every second, resulting in a 1000 rows per second. In terms of partitioning, my presumption is ...
Brent Arias's user avatar
  • 29.4k
0 votes
1 answer
19 views

redshift sql row selection based on min date

I need to extract all rows from a large data set with 30+ columnns based on the min time date, account can repeat across days and execution dates, execution id is distinct and can be thought of as a ...
Rich's user avatar
  • 27

1
2 3 4 5
64