[S5R3/R4] How to check Performance when Testing new Apps

I just found this in another thread and think it might be suitable to be answered here:

No, there isn't.

Actually I wasn't aware of how many time and effort people here had spent on it. I read about a "Ready Reckoner" in some beta test thread and was satisfied that apparently people found a tool that would make it easier for them to judge Apps. Apart from that I was perfectly busy with the threads I'm subscribed to (which usually are the Beta App threads), and I'm probably the only one in the "office" that has a noticeable time to spend on reading the forums. Sure I wanted to read through all the related threads when I had a bit more time - it's just that I hadn't. I still have read only a small fraction of the posts.

Wow, what excitement and geniality I missed!

To add my two cents here's some very, very technical background (which you probably already largely figured out), and a proposal at the end:

The "Hierarchical Search" we are using since S5R2 consists of two steps that are performed alternating for each "template": A "F-Statistic" (or short "Fstat") step that is similar (but not equal to) an DFT, and a "Hough-transform based coincidence analysis". Previous runs consisted only of the Fstat step, and a coincidence analysis was part of the post-processing. In the Hierarchical Search we analyze data from two detectors with a Fstat method, then basically look for similarities in the results, as a gravitational wave should show up in the signals of both instruments, while local noise and disturbances shouldn't. This "coincidence analysis" is currently done in the Hough part.

When you point a profiler such as Intel's VTune or Apple's Shark to current Apps, you should see the pattern of alternating calls to the functions ComputeFStatFreqBand and ComputeFstatHoughMap (and subfunctions; in recent Apps replaced by optimized functions with the prefix "Local") that implement these two steps. Currently about 2/3 of a task's run-time is spent in the Fstat analysis, and 1/3 in the Hough part. The whole machinery around these two functions doesn't matter much for the computation time.

The time needed for the Fstat part is constant for every template, i.e. independent of the actual sky position and frequency. This isn't true for the Hough step. The size of a fundamental data structure in the Hough part depends on the declination of the current sky position (actually the (abs() or sqr() - I don't remember from the top of my head) sine of the angle between the sky position and the ecliptic plane), and so does the time of every computation that involves it (This is very much simplified, but if you neglect everything that averages out over a task even in S5R3, it boils down to this).

This effect averages out in an "all-sky" task, which we had in all previous analysis runs up to and including S5R2. This is the reason why we didn't see this run-time variation in S5R2 although we already used the Hierarchical Search there, and also why the credits are right on average in S5R3.

In order to get reasonable and equal run-times for each task (though the amount of data to be analyzed is constantly growing from run to run) we found it necessary to split up not only the frequency range, but also the sky positions between the tasks in S5R3. This splitting is actually done in the application:

Having had quite some trouble in early runs with calculating the grid of sky positions to look at in the Application (from numerical differences between architectures and compilers) we are distributing files with lists of pre-calculated sky positions, the "sky-grid" files. The granularity of this grid depends on the frequency; there is a new sky-grid file for every 10Hz. The sky positions in such a file start at one pole, followed by all sky positions next to it in order of right ascension, then followed by the next-nearest circle of sky positions and so forth, until the other pole is reached.

If you look at the command-line of the App (e.g. in client_state.xml) you'll see that it gets passed options named "numSkyPartitions" and "partitionIndex", originally set by the workunit generator for this particular workunit. The App splits the skygrid file in "numSkyPartitions" portions and calculates the sky positions of the portion number "partitionIndex". (In hindsight it looks more clever to me to take every "numSkyPartitions"th skypoint of the file starting with the "partitionIndex"th, which effectively would still cover the whole sky with every task, looking more interesting on the graphics and averaging run times.)

So the current run-time is the sum of a part that is determined only by the number of templates in each task (and can be derived from the currently assigned credit), and a part that varies with the (average) declination of the sky positions in the sky partition assigned to the task.

The trouble is that the ratio between these two is a little bit different of course for every App (version), but also for every machine, and finally also depends on the frequency (the higher the frequency, the larger the Fstat part). The Fstat part is FLOPs bound, it doesn't read or write much data, and depends on the speed of the FPU (or whatever SIMD unit the current App is using). The Hough part is largely limited by memory bandwidth, which is somewhat orthogonal to FP speed. The first App optimizations to the "Kernel loop" and "sin/cos approximation" addressed the Fstat part, while the recent "prefetching" affects the memory access of the Hough part.

Honestly I find it very hard to "tune" the credits of a workunit (i.e. predict the run-time for it) in a way that would do justice to everyone and every machine, especially if that should also serve as a reference for comparing speed of Apps.

For the latter I think it would be best to construct a "reference workunit" that covers the whole sky and thus averages out these variation, e.g. by taking a current workunit and picking some points from the skygrid file in the way described above. This would, however, mean that people interested in speed comparison would need to run it on their machine with every new App they get. Remember that you'd do this without getting BOINC credit for it, but just for comparison. How much (run-) time would you be willing to sacrifice for this?

BM