Questions tagged [partitioning]
Partitioning is a performance strategy whereby you divide possibly very large groups of data into some number of smaller groups of data.
3,162
questions
-2
votes
0
answers
22
views
How to fix my HDD that thinks that it has 20GB when it really has 128GB [closed]
My dad screwed up my HDD while trying to install Ubuntu and windows on separate partitions so now it says it has 20.8GB of space on the disk and it really has MUCH MORE but I don't know exactly how ...
-2
votes
0
answers
30
views
Sharing Xcode and Project repo between two volumes on one Machine [closed]
I'm developing on a macOS application and currently on OS Ventura 13.6 .. Now I need to integrate and test it on macOS Sonoma so I'll partition the drive and add a new volume on my MacBook to install ...
0
votes
0
answers
7
views
Difference between static partitioning hypervisors and separation kernel / microkernel?
I am trying to get an understanding of virtualization approaches.
This paper describes static partitioning hypervisors.
In this survey paper the category separation kernels and microkernels is ...
0
votes
2
answers
34
views
Best method for updating SQL Server partition function
I have several large SQL Server tables where the current partition functions are not extended beyond this month, and the procedure that is supposed to split the partition function into new date ranges ...
0
votes
1
answer
28
views
Add id for a series of events and that breaks once the same event appears
I have a dataset that looks like contains columns like id and event. What I would like to add is desired_id which gets incremented every times where is value impression in the column event. Could you ...
1
vote
1
answer
45
views
How to find the partition metadata of an extent in Azure Data Explorer
See this for background: Azure Data Explorer: How do Partitioning Policy and Merge Policy work?
I have created a table with a Partitioning policy based on both a string and timestamp column. The ...
0
votes
1
answer
18
views
Milvus Data access control [closed]
I am using milvusDB and looking for providing subscription model, and facing providing efficient way to provide data access control.
I am planning to establish a collection in Milvus that will ...
0
votes
1
answer
18
views
Apply partitioning and change clustering for replicated data base to BQ with DataFusion
I replicate data from MySQL data base to BigQuery using DataFusion.
My original table in MySQL is not partitioned but I want it to be partitioned by a column when it is replicated to BQ.
Additionally ...
0
votes
0
answers
12
views
Trying to dual boot windows 11 with Pop1_Os, what is correct or best disk partition for this?
Taking a course on Linux and learning how to code , but I have many issues with playing league of legends in my free time. So I decided to dual boot.
I have 2 nvme m.2 sticks (each 500gb) and a 240 ...
0
votes
0
answers
27
views
@BeforeStep defined in ItemProcessor and ItemWitter is not getting called with spring batch partitioning
We were moving our spring batch implementations to use partitions. After making changes now the @BeforeStep is not getting called which is inside our processor and writter.
Sample code here,
@Bean
...
1
vote
1
answer
68
views
Number Partitioning: non-recursive algorithm with determinate loops only
I am trying to generate the number partition set of a non-negative integer. There are several solutions, also on this site, like in Number of ways to partition a number in Python:
def P(n):
# ...
-1
votes
0
answers
53
views
Will be slow if I query with the 2nd part of a composite clustered primary key in a partitioned table?
I have a big partitioned table (close to a billion rows), which has a clustered primary key on the partition column and an identity id key (dateSubmitted Date, id long) and want to query it using only ...
-1
votes
1
answer
41
views
Best practices for primary keys in partitioned fact tables in a data warehouse with PostgreSQL
I'm seeking advice on best practices regarding the implementation of partitioned tables in a data warehouse when using PostgreSQL. My current understanding is that partitioning a fact table by a ...
-2
votes
0
answers
18
views
Token authentication problem with remote connection area
I am using Cisco AnyConnect to access the VPN, and after accessing the VPN I access the remote desktop by providing username and password. This remote server has Windows installed.
Yesterday I ...
0
votes
0
answers
27
views
Syntax template for SAS partitioning of variance
For class I’m asked to see analyze the variance individually- with the presence of other variables AND WITHOUT the presence of variables.
What is the code to partition variance (I know I will input my ...
0
votes
1
answer
43
views
Partitioning of data in adx table
I was going through the microsoft docs of partition policy here. There it is written that By default, extents are partitioned by time of creation (ingestion), and in most cases there's no need to set ...
-1
votes
0
answers
41
views
Huge slowdown of basic queries when using a partitioned table
I have partitioned table "Transactions" with partition by mouth (Partition_Key = 'YYYYMM').
I see a strong performance degradation if i use basic queries to select one value of indexes ...
1
vote
1
answer
39
views
Spark Partition: Do I need to use the partition column in the query in order to take advantage of the performance partition?
I have a table in Spark partition by a column year_month (eg. 2020-01, 2020-02), but all the queries are done using a different column from with the year_month is derived date (e.g. 2020-01-30, 2020-...
0
votes
1
answer
18
views
RxJava: how to extract blocks form a source observable?
I have
[1,2,3,2,4,5,0,2,1,3,4,5,2,2,1,4,5]
What I want as output is
[[1,2,3,2,4,5],[1,3,4,5],[1,4,5]]
This means I want to extract blocks from the input sequence with some start marker, here 1, and ...
0
votes
0
answers
65
views
Discrepancies in PySpark DataFrame Results When Using Window Functions and Filters
When I do certain types of transformations on a dataframe that involve window functions with filters I am getting the wrong results. Here is a minimal example of the results I get with my code:
from ...
0
votes
0
answers
32
views
How does Databricks ingestion time clustering work for read queries?
I was reading this document recommanding using the default ingestion time clustering rather than partition tables that contains less than a terabyte of data. But it's not convincing enough for me for ...
1
vote
1
answer
43
views
It's normal that my data loading in postgresql with partitions takes too long?
I have a database that has to have approximately 20,000 records per day.
From 2022 to 2024.
I have created 6 partitions for each quarter of the years 2022 and 2023.
In total there are 14 million ...
1
vote
1
answer
69
views
How can I use hibernate validation with a partitioned table in Postgres?
I created a table with the following code:
CREATE TABLE my_table(
id BIGSERIAL,
PRIMARY KEY(id)
) PARTITION BY RANGE (id);
CREATE TABLE my_table_2023_10 PARTITION OF my_table FOR VALUES FROM ...
1
vote
1
answer
53
views
InvalidInputException (status: 400): Partition index not supported for the given partition key type
I am trying to create an index on a partitioned column in AWS Glue for Athena. I keep getting an error:
Unable to complete request to create index.
InvalidInputException (status: 400): Partition ...
0
votes
0
answers
29
views
Which is the best way of duplicating a table partitioned with hasing in PostgreSQL?
I have a partitioned table (using hashing) in a database hosted on PostgreSQL (it's an RDS on AWS).
The table is more or less like this:
CREATE TABLE snapshot
(
url TEXT NOT ...
0
votes
2
answers
67
views
Use SQL to get session duration between two events
I've got a dataset (in postgres specifically) that I need to calculate the 'user session time' for. I'm fairly positive I need to use partitioning to answer the question, but I'm not really sure how ...
1
vote
1
answer
68
views
Randomly create a vector within specific requirements
I want R to create a vector for me that has length 15, each of the 15 values being in the range 0-100 and together adding up to 100. So an example outcome could be:
`[12, 25, 2, 0, 6, 17, 2, 4, 1, 8, ...
0
votes
1
answer
52
views
SQL using Case When statement with MAX and Partition Over in Where Clause
I have this query using a case when expression inside a MAX() aggregate function to obtain the maximum value of the case when statement.
However I need to filter also off that maximum value in the ...
0
votes
1
answer
49
views
SQL Server Partitioned tables and full text indexing
Using SQL Server 2016+
I have a products table that requires partioning. It currently has a primary Identity key which is used as part of a full text index search. We need to partition the table a a ...
0
votes
0
answers
59
views
How to iterate through all partitions in a PyArrow FileSystemDataset for aggregations
I have a parquet dataset partitioned by year, and I am trying to iterate through each year partition to do aggregations on each one, instead of loading the entire dataset into memory at once.
The ...
0
votes
0
answers
14
views
Hazelcast - DurableExecutorService behaviour whilst repartitioning
I'm looking for clarity on the behaviour of the DurableExecutorService during the repartitioning of a cluster.
Imagine we have submitted a runnable to be executed on the key owner of key1.
Whilst this ...
0
votes
1
answer
52
views
Performance impact when querying without partition key on a partitioned table
Assuming I partition a table of large size (say 6TB) by month. We are serving very high traffic of around 300 RPS.
Sample schema
Create table mydata(
id bigint not null,
date_created not null,
...
0
votes
0
answers
62
views
How do I determine whether or not postgres table partitioning is helpful?
I'm currently trying to determine whether or not postgres table partitioning is beneficial.
Background: I have a table with 40 GB and around 200 million rows. The copy, merge, and regenerate jobs in ...
0
votes
0
answers
47
views
DynamoDB Table Design for User/Groups
I've been reading up on DynamoDB table design. I'm still not 100% sure on the best approach for this.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-modeling-nosql-B.html
https://...
1
vote
1
answer
33
views
Creating Session ID Buckets
I am trying to create a session_id column that groups a bunch of sessions into buckets based on if there has been a 30 minute gap...
ID
Session_TM
001
00:15:12
001
00:17:17
001
01:00:00
001
01:55:...
0
votes
1
answer
31
views
How to speed up a writing and reading speed from data mart via splitting it into multiple parts in data warehouse and use it in OBIEE?
In our datawarehouse, we have a huge and one of the most used and important table, that stores account data(their ids, saldo corresponding to operational day, etc). On a daily basis, approximately 90 ...
0
votes
1
answer
25
views
GCP BigQuery wont access a single partition
I'm experiencing a problem with a GBQ query which goes through all the data in a table, even though the table is partitioned.
The partitioning is set to DAY on the datetime field.
Currently the query ...
1
vote
2
answers
40
views
How to set partition size for date field in Databricks
I have timestamp field source_timestamp which is used a lot in the queries to the delta table.
I would like to partition by this column with size of the partition = 1 month.
How to achieve that?
If I ...
0
votes
0
answers
17
views
partion on date column + X days
Trying to drop duplicate transactions by account and date, that may have ocurred on the same date or within 10 days of the original transaction date. Also need to drop some test transaction data. Have ...
0
votes
1
answer
28
views
Postgresql index / partition optimization
We have an important table "client" in a postgresql 13 database, which has around 5Million rows. This table is linked to another table company with 2000 rows. We have 1 company with 2 ...
0
votes
0
answers
12
views
Greedy number partitioning to minimize the maximum
I have a list l with 4 * n numbers. I have another array a with n numbers. All of them are integers. I want to partition l into n subsets: l1,l2,...ln. I add the first number of a to l1, second number ...
0
votes
0
answers
62
views
Rebuilding Multiple Partitions of a Table with Clustered Column Store Index
I created a Table with partitions (on the date column) of a table with a clustered column store index.
I am able to run Index Rebuilds on each partition as needed, which is definitely faster than ...
0
votes
1
answer
53
views
Partition 0CALDAY (yyyymmdd) column to year, month and day in a copy activity of Azure Data Factory
I am using SAP HANA connector to copy the data from SAP HANA to Azure data lake. In my query there is a column called 0CALDAY which is a string data type and has row values like 20230119. I would like ...
1
vote
1
answer
42
views
MariaDB partitioning with char + date columns for PRIMARY KEY
We would like to partition an existing table of 20m rows by month but the documentation says, before starting partitioning, the table needs to have a composite PRIMARY KEY by the id (AUTO_INCREMENT) ...
0
votes
0
answers
28
views
mysql partition with no primary key
I'm studying partitions, and I'm wondering what's different about the two situations.
CREATE TABLE users(
user_id INT NOT NULL,
birthYear DATETIME NOT NULL,
PRIMARY KEY (user_id, ...
-3
votes
1
answer
172
views
Why partitioning key must be part of unique/primary key in MySQL?
I know from documentation and from lots of questions on stack overflow that whenever I am partitioning a table, I require a partition key and that partition key must be included in unique/primary key. ...
0
votes
0
answers
78
views
Oracle: partition a table with more than 1B records
Database: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0
Issue: in production we have a table with more than 1B records that we are not able to query since the table is not partitioned/...
0
votes
0
answers
52
views
Partitioning existing tables using PostgreSQL 13
I'm using PostgreSQL 13.8 and i have a lot of big tables (with a lot of records) and queries are spending too much. I though it's a good idea to partition some tables to speed them up. But i'm totally ...
0
votes
0
answers
30
views
Cassandra partition with modulo-based timestamp
I will be capturing an event stream as records in a Cassandra table. I might expect perhaps 1000 events every second, resulting in a 1000 rows per second.
In terms of partitioning, my presumption is ...
0
votes
1
answer
19
views
redshift sql row selection based on min date
I need to extract all rows from a large data set with 30+ columnns based on the min time date, account can repeat across days and execution dates, execution id is distinct and can be thought of as a ...