39,927 questions
Advice
1
vote
0
replies
51
views
How to decide number of partitions using repartition vs coalesce in Apache Spark for optimization?
How to decide I am trying to understand how to properly use repartition and coalesce in Apache Spark, especially for performance optimization.
From my understanding:
repartition can increase or ...
Advice
0
votes
1
replies
38
views
Sorting Multi Variable Data into Efficient Groups
I have an excel spreadsheet of results for the analysis of beams in a building.
For each beam I have 6 different loads for 6 different situations. These loads are not linearly related, sometime a beam ...
- 1
Advice
0
votes
1
replies
50
views
DDD Aggregate Roots: How to avoid loading unused child entities without tightly coupling to the ORM?
I am implementing Domain-Driven Design (DDD) and facing a problem: balancing database performance with clean domain modeling.
I want to stop loading child entities that I don't need when getting an ...
- 13
1
vote
0
answers
114
views
Optimizing Exact Filtered Pagination Count (COUNT(*)) on MySQL with Dynamic JSON Filters in Spring Boot API
Problem Statement:
We are running a Java/Spring GET/POST getAll pagination API on top of MySQL in AWS RDS and need help optimizing exact filtered counts
Current state:
The main dataset is courses with ...
- 544
Best practices
0
votes
4
replies
46
views
What is the most efficient way to maintain a real-time COUNT(DISTINCT column) on a large PostgreSQL partitioned table?
I have a PostgreSQL table foo partitioned by date (monthly partitions).
The table currently contains ~13 million rows and continues to grow.
This query runs any time someones opens the dashboard of ...
- 31
2
votes
1
answer
59
views
PyTorch ValueError: optimizer got an empty parameter list when building a Logistic Regression Model
I tried making a logistic regression model using nn.Module
class LogisticRegressionModel(nn.Module):
def __init__(self, input_dim= None) -> None:
super().__init__()
if input_dim ...
Advice
1
vote
8
replies
115
views
Portably disabling FMA for a single function
For robust ray–triangle intersection, it turns out that the following must be exactly true in floating-point math:
a*b - c*d = -(d*c - b*a)
There is no problem if this is computed as two ...
- 15.6k
1
vote
2
answers
112
views
Is there a numpy (or scipy, or whatever) vectorized function that can sample from an array of probabilities?
I need a function that behave like np.random.choice, but that takes the argument "p" to be a 2-dimension array, of which each row is a probability. The function should return a sample from ...
- 13
1
vote
1
answer
37
views
AMPLpy - battery optimization problem - How do I program LP? Not quite an LP
I’m trying to maximize the profit of a battery that can charge r_i or discharge q_i, with wholesale electricity prices P_elec,i, within a n=24 hour period. (eta is just the efficiency of the battery, ...
4
votes
0
answers
59
views
cudaMemcpyAsync (P2P D2D) serializes with kernel execution
Hi all — I’m debugging an unexpected ordering/progress issue with peer-to-peer device copies and I’d like help understanding what CUDA/driver/runtime behavior could explain it.
Setup
Single node, 2 ...
- 755
Best practices
1
vote
1
replies
68
views
Does Unity's Mouse.current do a Find or cache the value?
It's a fairly common bit of knowledge that you want to cache the current camera instead of using Camera.main because it does a Find function every time under the hood, which is expensive (this might ...
- 1,153
7
votes
1
answer
217
views
Compute sin(x)-x efficiently to double precision accuracy on range |x| <= pi
There is a trig related function missing from math.h, namely x-sin(x). I am trying to implement it accurate to full double precision for |x| <= pi. Minimum ripple polynomial or rational ...
- 3,865
1
vote
1
answer
222
views
Can arbitrary precision integer increment in Brainfuck be done in O(1) code size?
I was fiddling with the Brainfuck esolang over the past few days, and tried to implement an increment operation on an N-byte-wide integer in big-endian format. Note that I am imposing a structural ...
- 19
4
votes
1
answer
120
views
Speed up univariate logistic regression using IRLS on large number of subsampled samples
I'm trying to perform logistic regression on samples randomly subsampled from a huge dataset for binary classification. I implemented logistic regression using iterative reweighted least squares (IRLS)...
- 648
1
vote
1
answer
86
views
Why is this type-bound procedure causing a slow-down?
I am working on some simulation code in Fortran. Part of that are potential cells that induce velocity.
Here is the definition for the cell type:
type CELL
doubleprecision :: xmin,ymin,...
- 107