Nicholas Frechette's Blog2023-12-16T14:59:59+00:00http://nfrechette.github.ioNicholas Frechettezeno490@gmail.comChristmas came early: ACL 2.1 is out!2023-12-16T00:00:00+00:00http://nfrechette.github.io/2023/12/16/acl_v2.1.0<p>After over 30 months of work, the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> has finally reached <a href="https://github.com/nfrechette/acl/releases/tag/v2.1.0">v2.1</a> along with an updated <a href="https://github.com/nfrechette/acl-ue4-plugin/releases/tag/v2.1.0">v2.1 Unreal Engine plugin</a>.</p>
<p>Notable changes in this release include:</p>
<ul>
<li>Added support for <a href="/2022/01/23/anim_compression_bind_pose_stripping/">bind pose stripping</a> which reduces memory usage by <strong>3-5%</strong> on average</li>
<li>Added support for <a href="/2022/04/03/anim_compression_looping/">loop handling</a> through a looping policy</li>
<li>Added support for pre-processing</li>
<li>Added automatic compression level selection which reduces memory usage by <strong>1-2%</strong> on average</li>
<li>Optimized compression through <a href="/2023/02/26/dominance_based_error_estimation/">dominant shell computation</a> which reduces memory usage by up to <strong>10%</strong></li>
<li>Added support for <a href="/2022/06/05/anim_compression_rounding_time/">per sub-track rounding</a></li>
<li>Updated to <a href="https://github.com/nfrechette/rtm">Realtime Math</a> 2.2.0</li>
<li>Many other improvements!</li>
</ul>
<p>Overall, memory savings average between <strong>10-15%</strong> for this release compared to the prior version. Decompression performance remained largely the same if not slightly faster. Compression is slightly slower in part due to the default compression level being more aggressive thanks to many performance improvements that make it viable. Visual fidelity improved slightly as well.</p>
<p>This release took much longer than prior releases because a lot of time was spent on quality-of-life improvements. CI now runs regression tests and tests more platforms and toolchains. CI uses docker which makes finding and fixing regressions <em>much</em> easier. A <a href="https://github.com/nfrechette/acl-test-data">satellite project</a> was spun off to generate the regression test data. Together, these changes make it much faster for me to polish, test, and finalize a release.</p>
<p>This release saw many contributions from the community through GitHub issues, questions, code, and fixes. Special thanks to all <a href="https://github.com/nfrechette/acl#contributors-">ACL contributors</a>!</p>
<h1 id="who-uses-acl">Who uses ACL?</h1>
<p>The Unreal Engine plugin of ACL has proved to be quite popular over the years. The reverse engineering community over at <a href="https://www.gildor.org/smf/index.php/topic,8304.0.html">Gildor’s Forums</a> have a post that tracks who uses ACL in UE. At the time of writing, the list includes:</p>
<blockquote>
<p>Desktop games:
All latest Dontnod’s games: two last episodes of Life is Strange 2, Twin Mirror, Tell Me Why, most likely all their future releases as well.
Beyond a Steel Sky, Chivalry 2, Fortnite (current version), Kena: Bridge of Spirits, Remnant: From the Ashes (current version), Rogue Company (current version), Star Wars: Tales From The Galaxy’s Edge, Final Fantasy VII Remake (including Intergrade), The Ascent, The Dark Pictures Anthology (Man of Medan / Little Hope / House of Ashes / Devil in Me) (current version), Valorant, Evil Dead: The Game, The Quarry, The DioField Chronicle, Borderlands 3, Tiny Tina’s Wonderlands, Medieval Dynasty (current version), Divine Knockout, Lost Ark, SD Gundam Battle Alliance, Back 4 Blood, KartRider: Drift (current version), Dragon Quest X Offline, Gran Saga (current version), Gundam Evolution (current version), The Outlast Trials, Harvestella, Valkyrie Elysium, The Dark Pictures Anthology: The Devil In Me, The Callisto Protocol, Synced (current version), High On Life, PlayerUnknown’s Battlegrounds (current version), Deliver Us Mars, Sherlock Holmes The Awakened, Hogwarts Legacy, Wanted: Dead, Like a Dragon: Ishin, Atomic Heart, Crime Boss: Rockay City, Dead Island 2, Star Wars Jedi: Survivor, Redfall, Park Beyond, The Lord of the Rings: Gollum, Gangstar NY, Lies of P, Warhaven (current version), AEW: Fight Forever, Legend of the Condor, Crossfire: Sierra Squad, Mortal Kombat 1, My Hero Ultra Rumble, Battle Crush, Overkill’s The Walking Dead, Payday 3</p>
<p>Mobile games:
Apex Legends Mobile (current version), Dislyte, Marvel Future Revolution, Mir4, Ni no Kuni: Cross Worlds, PUBG Mobile (current version), Undecember, Blade & Soul Revolution (current version), Crystal of Atlan, Mortal Kombat: Onslaught (old versions), Farlight 84, ArcheAge War, Assassin’s Creed: Codename Jade, Undawn, Arena Breakout, High Energy Heroes, Dream Star</p>
<p>Console games:
No More Heroes 3</p>
</blockquote>
<p>Many others use ACL in their own internal game engines. If you aren’t using it yet, let me know if anything is missing!</p>
<h1 id="whats-next">What’s next</h1>
<p>I’ve already started fleshing out the task list for the next minor release <a href="https://github.com/nfrechette/acl/milestone/12">here</a>. This release will bring about more memory footprint improvements.</p>
<p>If you use ACL and would like to help prioritize the work I do, feel free to reach out and provide feedback or requests!</p>
<p>Now that ACL is the <a href="/2023/09/17/acl_in_ue/">default codec in Unreal Engine 5.3</a>, this reduces my maintenance burden considerably as Epic has taken over the plugin development. This will leave me to focus on improving the standalone ACL project and Realtime Math. Going forward, I will aim for smaller and more frequent releases.</p>
The Animation Compression Library in Unreal Engine 5.32023-09-17T00:00:00+00:00http://nfrechette.github.io/2023/09/17/acl_in_ue<p>The very first release of the <a href="https://github.com/nfrechette/acl-ue4-plugin">Animation Compression Library (ACL) Unreal Engine plugin</a> came out in July of 2018 and required custom engine changes. It was a bit rough around the edges. Still, in the releases that followed, despite the integration hurdles, it slowly grew in popularity in games large and small. Two years later, version 1.0 came out with official support of UE 4.25 through the <a href="https://www.unrealengine.com/marketplace/en-US/product/animation-compression-library?sessionInvalidated=true">Unreal Engine Marketplace</a>. Offering backwards compatibility with each new release and a solid alternative to the stock engine codecs, the plugin continued to grow in popularity. Today, many of the most demanding and popular console and mobile games that use Unreal already use ACL (Fortnite itself has been using it for many years). It is thus with great pleasure that I can announce that as of <a href="https://docs.unrealengine.com/5.3/en-US/unreal-engine-5.3-release-notes/">UE 5.3</a>, the ACL plugin comes out of the box with batteries included as the default animation compression codec!</p>
<p>This represents the culmination of many years of hard work and slow but steady progress. This new chapter is very exciting for a few reasons:</p>
<ul>
<li>Better quality, compression and decompression performance for all UE users out of the box.</li>
<li>A dramatically smaller memory footprint: ACL saves over 30% compared with the defaults codecs in UE 5.2 and more savings will come with the next ACL 2.1 release later this year (up to 46% smaller than UE 5.2).</li>
<li>Maintenance churn to keep up with each UE release was a significant burden. Epic will now take ownership of the plugin by continuing its development in their fork as part of the main Unreal Engine development branch.</li>
<li>With official UE distribution comes better documentation, localization, and support through the Unreal Developer Network (UDN). This means a reduced burden for me as I now can support the plugin as part of my day job at Epic as opposed to doing so on my personal time (burning that midnight oil).</li>
</ul>
<p>Best of all, by no longer having to maintain the plugin in my spare time, it will allow me to better focus on the <a href="https://github.com/nfrechette/acl">core of ACL</a> that benefits everyone. Many proprietary game engines leverage standalone ACL and in fact quite a few code contributions (and ideas) over the years came from them.</p>
<p>Unfortunately, it does mean that some tradeoffs had to be made:</p>
<ul>
<li>The GitHub version of the plugin will remain largely in read only mode, taking in only critical bug fixes. It will retain its current MIT License. Contributions should instead be directed to the Unreal Engine version of the plugin.</li>
<li>The Unreal Engine Marketplace version of the plugin will no longer be updated to keep up with each UE release (since it comes packaged with the engine already).</li>
<li>Any changes to the plugin will not be backwards compatible as they will be made in the Unreal Engine main development branch. To get the latest improvements, you’ll have to update your engine version (like any other built in feature).</li>
<li>Each plugin release will no longer publish statistics comparing ACL to the other UE codecs.</li>
<li>The ACL plugin within UE will take on the engine code license (it already had a dual license like all code plugins do on the Unreal Engine Marketplace).</li>
</ul>
<p>That being said, once ACL 2.1 comes out later this year, one last release of the plugin will be published on GitHub and the Unreal Engine Marketplace (for UE versions before 5.3). It brings many improvements to the memory footprint and visual fidelity. A future version of UE will include it (TBD).</p>
<p>To further clarify, the core ACL project will remain with me and continue to evolve on GitHub with the MIT License as it always has. I still have a long list of improvements and research topics to explore!</p>
Dominance based error estimation2023-02-26T00:00:00+00:00http://nfrechette.github.io/2023/02/26/dominance_based_error_estimation<p>As I’ve <a href="/2016/11/01/anim_compression_accuracy/">previously written about</a>, the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> measures the error introduced through compression by approximating the <a href="https://en.wikipedia.org/wiki/Skeletal_animation">skinning deformation</a> typically done with a graphical mesh. To do so, we define a rigid shell around each joint: a sphere. It is a crude approximation, but the method is otherwise very flexible, and any other shape could be used: a box, convex, etc. This rigid shell is then transformed using its joint transform at every point in time (provided as input through an animation clip). Using only individual joints, this process works in local space of the joint (local space) as transforms are relative to their parent joint in the skeleton hierarchy. When a joint chain that includes the root joint is used, the process works in local space of the object (or object space). While local space only accounts for a single transform (the joint’s), in object space we can account for any number of joints, it depends entirely on the topology of the joint hierarchy. As a result of these extra joints, measuring the error in object space is slower than in local space. Nevertheless, this method is very simple to implement and tune since the resulting error is a 3D displacement between the raw rigid shell and the lossy rigid shell undergoing compression.</p>
<p><a href="https://github.com/nfrechette/acl/blob/develop/docs/error_metrics.md">ACL leverages this technique</a> and uses it in two ways when optimizing the quantization bit rates (e.g finding how many bits to use for each joint transform).</p>
<p>First, for each joint, we proceed to find the lowest bit rate that meets our specified precision requirement (user provided, per joint). To do so, a pre-generated list of permutations is hardcoded of all possible bit rate combinations for a joint’s rotation, translation, and scale. That list is sorted by total transform size. We then simply iterate on that list in order, testing everything until we meet our precision target. We start with the most lossy bit rates and work our way up, adding more precision as we go along. This is an exhaustive search and to keep it fast, we do this processing in local space of each joint. Once this first pass is done, each joint has an approximate answer for its ideal bit rate. A joint may end up needing more precision to account for object space error (since error accumulates down the joint hierarchy), but it will never need less precision. This thus puts a lower bound on our compressed size.</p>
<p>Second, for each joint chain, we check the error in object space and try multiple permutations by slowly increasing the bit rate on various joints in the chain. This process is much slower and is controlled through the <code class="language-plaintext highlighter-rouge">compression level</code> setting which allows a user to tune how much time should be spent by ACL when attempting to optimize for size. Because the first pass only found an approximate answer in local space, this second pass is very important as it enforces our precision requirements in object space by accounting for the full joint hierarchy. Despite aggressive caching, unrolling, and many optimizations, this is by far the slowest part during compression.</p>
<p>This algorithm was implemented years ago, and it works great. But ever since I wrote it, I’ve been thinking of ways to improve it. After all, while it finds a local minimum, there is no guarantee that it is the true global minima which would yield the optimal compressed size. I’ve so far failed to come up with a way to find that optimal minima, but I’ve gotten one step closer!</p>
<h2 id="to-the-drawing-board">To the drawing board</h2>
<p>Why do we have two passes?</p>
<p>The second pass, in object space, is necessary because it is the one that accounts for the true error as it accumulates throughout the joint hierarchy. However, it is very hard to properly direct because the search space is huge. We need to try multiple bit rate permutations on multiple joint permutations within a chain. How do we pick which bit rate permutations to try? How do we pick which joint permutations to try? If we increase the bit rate towards the root of the hierarchy, we add precision that will improve every child that follows but there is no guarantee that even if we push it to retain full precision that we’ll be able to meet our precision requirements (some clips are exotic). This pass is thus trying to stumble to an acceptable solution and if it fails after spending a tunable amount of time, we give up and bump the bit rates of every joint in the chain as high as we need one by one, greedily. This is far from optimal…</p>
<p>To help it find a solution, it would be best if we could supply it with an initial guess that we hope is close to the ideal solution. The closer this initial guess is to the global minima, the higher our chances will be that we can find it in the second pass. A better guess thus directly leads to a lower memory footprint (the second pass can’t drift as far) and faster compression (the search space is vastly reduced). This is where the first pass comes in. Right off the bat, we know that bit rates that violate our precision requirements in local space will also violate them in object space since our parent joints can only add to our overall error (in practice, the error sometimes compensates but it only does so randomly and thus can’t be relied on). Since testing the error in local space is very fast, we can perform the exhaustive search mentioned above to trim the search space dramatically.</p>
<p>If only we could calculate the object space error using local space information… We can’t, but we can get close!</p>
<h2 id="long-range-attachments">Long-range attachments</h2>
<blockquote>
<p>Kim, Tae & Chentanez, Nuttapong & Müller, Matthias. (2012). Long Range Attachments - A Method to Simulate Inextensible Clothing in Computer Games. 305-310. 10.2312/SCA/SCA12/305-310.</p>
</blockquote>
<p><img src="/public/acl/long_range_attachments.jpg" alt="Long-range attachments" /></p>
<blockquote>
<p>Above: A static vertex in red with 3 attached dynamic vertices in black. A long-range constraint is added for each dynamic vertex, labeled <em>d1</em>, <em>d2</em>, and <em>d3</em>. <em>T0</em> shows the initial configuration and its evolution over time. When a vertex extends past the long-range constraint, a correction (in green) is applied. Vertices within the limits of the long-range constraints are unaffected.</p>
</blockquote>
<p>Back around 2014, I read the above paper on cloth simulation. It describes the concept of long-range attachments to accelerate and improve convergence of distance constraints (e.g. making sure the cloth doesn’t stretch). They start at a fixed point (where the cloth is rigidly attached to something) and they proceed to add additional clamping distance constraints (meaning the distance can be shorter than the desired distance, but no longer) between each simulated vertex on the cloth and the fixed point. These extra constraints help bring back the vertices when the constraints are violated under stress.</p>
<p>A few years later, it struck me: we can use the same general idea when estimating our error!</p>
<h2 id="dominance-based-error-estimation">Dominance based error estimation</h2>
<p>When we compress a joint transform, error is introduced. However, the error may or may not be visible to the naked eye. How important the error is depends on one thing: the distance of the rigid shell.</p>
<p><img src="/public/acl/translation_error_contribution.jpg" alt="Translation error contribution" /></p>
<p>For the translation and scale parts of each transform, the error is independent of the distance of the rigid shell. This means that a 1% error in those, yields a 1% error regardless of how far the rigid shell lives with respect to the joint.</p>
<p><img src="/public/acl/rotation_error_contribution.jpg" alt="Rotation error contribution" /></p>
<p>However, the story is different for a transform’s rotation: distance acts as a <a href="https://en.wikipedia.org/wiki/Lever">lever</a>. A 1% error on the rotation does not translate in the same way to the rigid shell. It depends on how close the rigid shell is to the joint. The closer it lives, the lower the error will be and the further it lives, the higher the error.</p>
<p>Using this knowledge, we can reframe our problem. When we measure the error in local space for a joint <code class="language-plaintext highlighter-rouge">X</code>, we wish to do so on the furthest rigid shell it can influence. That furthest rigid shell of joint <code class="language-plaintext highlighter-rouge">X</code> will be associated with a joint that lives in the same chain. It could be <code class="language-plaintext highlighter-rouge">X</code> itself or it could be some child joint <code class="language-plaintext highlighter-rouge">Y</code>. We’ll call this joint <code class="language-plaintext highlighter-rouge">Y</code> the <strong>dominant joint</strong> of <code class="language-plaintext highlighter-rouge">X</code>.</p>
<p>The <strong>dominant rigid shell</strong> of a dominant joint is formed through its associated virtual vertex, the <strong>dominant vertex</strong>.</p>
<p><img src="/public/acl/dominant_joint_step1.jpg" alt="Single joint chain" /></p>
<p>For leaf joints with no children, we have a single joint in our chain, and it is thus its own dominant joint.</p>
<p><img src="/public/acl/dominant_joint_step2.jpg" alt="Two joints chain" /></p>
<p>If we add a new joint, we look at it and its immediate children. Are the dominant joints of its children still dominant or is the new joint its own dominant joint? To find it, we look at the rigid shells formed by the dominant vertices of each joint and we pick the furthest.</p>
<p>After all, if we were to measure the error on any rigid shell enclosed within the dominant shell, the error would either be identical (if there is no error contribution from the rotation component) or lower. Distance is king and we wish to look for error as far as we can.</p>
<p><img src="/public/acl/dominant_joint_step3.jpg" alt="Three joints chain" /></p>
<p>As more joints are introduced, we iterate with the same logic at each joint. Iteration thus starts with leaf joints and proceeds towards the root. You can find the code for this <a href="https://github.com/nfrechette/acl/blob/develop/includes/acl/compression/impl/rigid_shell_utils.h">here</a>.</p>
<p>An important part of this process is the step wise nature. By evaluating joints one at a time, we find and update the dominant vertex distance by adding the current vertex distance (the distance between the vertex and its joint). This propagates the <a href="https://en.wikipedia.org/wiki/Geodesic"><em>geodesic distance</em></a>. This ensures that even if a joint chain folds on itself, the resulting dominant shell distance is only ever increasing as if every joint was in a straight line.</p>
<p>The last piece of the puzzle is the precision threshold when we measure the error. Because we use the dominant shell as a conservative estimate, we must account for the fact that intermediate joints in a chain will contain some error. Our precision threshold represents an upper bound on the error we’ll allow and a target that we’ll try to reach when optimizing the bit rates. For some bit rate, if the error is above the threshold, we’ll try a lower value, but if it is below the threshold, we’ll try a higher value since we have room to grow. As such, joints in a chain will add their precision threshold to their virtual vertex distance when computing the dominant shell distance except for the dominant joint. The dominant joint’s precision threshold will be used when measuring the error and accounted for there.</p>
<p>This process is done for every keyframe in a clip and the dominant rigid shell is retained for each joint. I also tried to do this processing for each segment ACL works with and there was no measurable impact.</p>
<p>Once computed, ACL then proceeds as before using this new dominant rigid shell distance and the dominant precision threshold. It is used in both passes. Even when measuring a joint in object space, we need to account for its children.</p>
<h2 id="but-wait-theres-more">But wait, there’s more!</h2>
<p>After implementing this, I realized I could use this trick to fix something else that had been bothering me: <a href="/2016/11/03/anim_compression_constant_tracks/">constant sub-track collapsing</a>. For each sub-track (rotation, translation, and scale), ACL attempts to determine if they are constant along the whole clip. If they are, we can store a single sample: the reference constant value (12 bytes). This considerably lowers the memory footprint. Furthermore, if this single sample is equal to the default value for that sub-track (either the identity or the bind pose value when <a href="/2022/01/23/anim_compression_bind_pose_stripping/">stripping the bind pose</a>), the constant sample can be removed as well and reconstructed during decompression. A default sub-track uses only 2 bits!</p>
<p>Until now, ACL used individual precision thresholds for rotation, translation, and scale sub-tracks. This has been a source of frustration for some time. While the error metric uses a single 3D displacement as its precision threshold with intuitive units (e.g. whatever the runtime uses to measure distance), the constant thresholds were this special edge case with non-intuitive units (scale doesn’t even have units). It also meant that ACL was not able to account for the error down the chain: it only looked in local space, not object space. It never occurred to me that I could have leveraged the regular error metric and computed the object space error until I implemented this optimization. Suddenly, it just made sense to use the same trick and skip the object space part altogether: we can use the dominant rigid shell.</p>
<p>This allowed me to remove the constant detection thresholds in favor of the single precision threshold used throughout the rest of the compression. A simpler, leaner API, with less room for user error while improving the resulting accuracy by properly accounting for every joint in a chain.</p>
<h2 id="show-me-the-money">Show me the money</h2>
<p>The results speak for themselves:</p>
<p><strong>Baseline:</strong></p>
<table>
<thead>
<tr>
<th>Data Set</th>
<th>Compressed Size</th>
<th>Compression Speed</th>
<th>Error 99th percentile</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>CMU</strong></td>
<td>72.14 MB</td>
<td>13055.47 KB/sec</td>
<td>0.0089 cm</td>
</tr>
<tr>
<td><strong>Paragon</strong></td>
<td>208.72 MB</td>
<td>10243.11 KB/sec</td>
<td>0.0098 cm</td>
</tr>
<tr>
<td><strong>Matinee Fight</strong></td>
<td>8.18 MB</td>
<td>16419.63 KB/sec</td>
<td>0.0201 cm</td>
</tr>
</tbody>
</table>
<p><strong>With dominant rigid shells:</strong></p>
<table>
<thead>
<tr>
<th>Data Set</th>
<th>Compressed Size</th>
<th>Compression Speed</th>
<th>Error 99th percentile</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>CMU</strong></td>
<td>65.83 MB (-8.7%)</td>
<td>34682.23 KB/sec (2.7x)</td>
<td>0.0088 cm</td>
</tr>
<tr>
<td><strong>Paragon</strong></td>
<td>184.41 MB (-11.6%)</td>
<td>20858.25 KB/sec (2.0x)</td>
<td>0.0088 cm</td>
</tr>
<tr>
<td><strong>Matinee Fight</strong></td>
<td>8.11 MB (-0.9%)</td>
<td>17097.23 KB/sec (1.0x)</td>
<td>0.0092 cm</td>
</tr>
</tbody>
</table>
<p>The memory footprint reduces by over <strong>10%</strong> in some data sets and compression is <strong>twice</strong> as fast (the median compression time per clip for Paragon is now 11 milliseconds)! Gains like this are <em>hard</em> to come by now that ACL has been so heavily optimized already; this is <em>really</em> significant. This optimization maintains the same great accuracy with no impact whatsoever to decompression performance.</p>
<p>The memory footprint reduces because our initial guess following the first pass is now much better: it accounts for the potential error of any children. This leaves less room for drifting off course in the second pass. And with our improved guess, the search space is considerably reduced leading to much faster convergence.</p>
<p>Now that this optimization has landed in ACL develop, I am getting close to finishing up the <a href="https://github.com/nfrechette/acl/milestone/11">upcoming ACL 2.1</a> release which will include it and much more. Stay tuned!</p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Manipulating the sampling time for fun and profit2022-06-05T00:00:00+00:00http://nfrechette.github.io/2022/06/05/anim_compression_rounding_time<p>The <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> works with uniformly sampled animation clips. For each animated data track, every sample falls at a predictable and regular rate (e.g. 30 samples/frames per second). This is done to keep the decompression code simple and fast: with samples being so regular, they can be packed efficiently, sorted by their timestamp.</p>
<p><img src="/public/uniform_sampling.jpg" alt="Uniform sampling" /></p>
<p>When we decompress, we simply find the surrounding samples to reconstruct the value we need.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">float</span> <span class="n">t</span> <span class="o">=</span> <span class="p">...</span> <span class="c1">// At what time in the clip we sample</span>
<span class="kt">float</span> <span class="n">duration</span> <span class="o">=</span> <span class="p">(</span><span class="n">num_samples</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">sample_rate</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">normalized_t</span> <span class="o">=</span> <span class="n">t</span> <span class="o">/</span> <span class="n">duration</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">sample_offset</span> <span class="o">=</span> <span class="n">normalized_t</span> <span class="o">*</span> <span class="p">(</span><span class="n">num_samples</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">first_sample</span> <span class="o">=</span> <span class="n">trunc</span><span class="p">(</span><span class="n">sample_offset</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">second_sample</span> <span class="o">=</span> <span class="n">min</span><span class="p">(</span><span class="n">first_sample</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="n">num_samples</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">interpolation_alpha</span> <span class="o">=</span> <span class="n">sample_offset</span> <span class="o">-</span> <span class="n">first_sample</span><span class="p">;</span>
<span class="n">interpolation_alpha</span> <span class="o">=</span> <span class="n">apply_rounding</span><span class="p">(</span><span class="n">interpolation_alpha</span><span class="p">,</span> <span class="n">rounding_mode</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">result</span> <span class="o">=</span> <span class="n">lerp</span><span class="p">(</span><span class="n">sample_values</span><span class="p">[</span><span class="n">first_sample</span><span class="p">],</span> <span class="n">sample_values</span><span class="p">[</span><span class="n">second_sample</span><span class="p">],</span> <span class="n">interpolation_alpha</span><span class="p">);</span>
</code></pre></div></div>
<p>This allows us to continously animate every value smoothly over time. However, that is not always necessary or desired. To support these use cases, ACL supports several <strong>sample time rounding modes</strong> in order to achieve various effects.</p>
<p>Rounding modes:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">none</code>: no rounding means that we linearly interpolate.</li>
<li><code class="language-plaintext highlighter-rouge">floor</code>: the first/earliest sample is returned with full weight (<code class="language-plaintext highlighter-rouge">interpolation alpha = 0.0</code>). This can be used to step animations forward. Visually, on a character animation, it would look like a stop motion performance.</li>
<li><code class="language-plaintext highlighter-rouge">ceil</code>: the second/latest sample is returned with full weight (<code class="language-plaintext highlighter-rouge">interpolation alpha = 1.0</code>). This might not have a valid use case but ACL supports it regardless for the sake of completeness. Unlike <code class="language-plaintext highlighter-rouge">floor</code> above, ceil returns values that are in the future which might not line up with other events that haven’t happened yet: particle effects, audio cues (e.g. footstep sounds), and other animations. <em>If you know of a valid use case for this, please reach out!</em></li>
<li><code class="language-plaintext highlighter-rouge">nearest</code>: the nearest sample is returned with full weight (<code class="language-plaintext highlighter-rouge">interpolation alpha = 0.0 or 1.0 using round to nearest</code>). ACL uses this internally when measuring the compression error. The error is measured at every keyframe and because of floating point rounding, we need to ensure that a whole sample is used. Otherwise, we might over/undershoot.</li>
<li><code class="language-plaintext highlighter-rouge">per_track</code>: each animated track can specify which rounding mode it wishes to use.</li>
</ul>
<h2 id="per-track-rounding">Per track rounding</h2>
<p><em>This is a new feature that has been requested by a few studios (included in the upcoming ACL 2.1 release).</em></p>
<p>When working with animated data, sometimes tracks within a clip are not homogenous and represent very different things. Synthetic joints can be introduced which do not represent something directly observable (e.g. camera, IK targets, pre-processed data). At times, these do not represent values that can be safely interpolated.</p>
<p>For example, imagine that we wish to animate a stop motion character along with a camera track in a single clip. We wish for the character joints to step forward in time and not interpolate (<code class="language-plaintext highlighter-rouge">floor</code> rounding) but the camera must move smoothly and interpolate (no rounding).</p>
<p>More commonly, this happens with scalar tracks. These are often used for animating blend shape (aka morph target) weights along with various other gameplay values. Gameplay tracks can mean anything and even though they are best stored as floating point values (to keep storage and their manipulation simple), they might not represent a smooth range (e.g. integral values, enums, etc).</p>
<p>When such mixing is used, each track can specify what rounding mode to use during decompression (ACL does not store this information in the compressed data).</p>
<p><em>Note: tracks that do not interpolate smoothly also do not blend smoothly (between multiple clips) and special care must be taken when doing so.</em></p>
<p>See <a href="https://github.com/nfrechette/acl/blob/develop/docs/handling_per_track_rounding.md">here</a> for details on how to use this new feature.</p>
<h2 id="performance-implications">Performance implications</h2>
<p>During decompression, ACL always interpolates between two samples even if the per track rounding feature is disabled during compilation. The most common rounding mode is <code class="language-plaintext highlighter-rouge">none</code> which means we interpolate. This allows us to keep the code simple. We simply snap the interpolation alpha to <code class="language-plaintext highlighter-rouge">0.0</code> or <code class="language-plaintext highlighter-rouge">1.0</code> when we <code class="language-plaintext highlighter-rouge">floor</code> and <code class="language-plaintext highlighter-rouge">ceil</code> respectively (or with <code class="language-plaintext highlighter-rouge">nearest</code>) and we make sure to use a <a href="https://fgiesen.wordpress.com/2012/08/15/linear-interpolation-past-present-and-future/">stable linear interpolation function</a>.</p>
<p>As a result, every rounding mode has the same performance with the exception of the <code class="language-plaintext highlighter-rouge">per track</code> one.</p>
<p>Per track rounding is disabled by default and the code is entirely stripped out because it isn’t commonly required. When enabled, it adds a small amount of overhead on the order of <strong>2-10%</strong> slower (AMD Zen2 with SSE2) regardless of which rounding mode individual tracks use.</p>
<p>During decompression, ACL unpacks 8 samples at a time and interpolates them in pairs. These are then cached in a structure on the execution stack. This is done ahead, before the interpolated samples are needed to be written out which means that during interpolation, it does not know which sample belongs to which track. That determination is only done when we read it from the cached structure. The reason for this will be the subject of a future blog post but it is done to speed up decompression by hiding memory latency.</p>
<p>As a result of this design decision, the only way to support per track rounding is to cache all possible rounding results. When we need to read an interpolated sample, we can simply index into the cache with the rounding mode chosen for that specific track. In practice, this is much cheaper than it sounds because even though it adds quite a few extra instructions to execute (SSE2 will be quite a bit worse than AVX here due to the lack of VEX prefix and blend instructions), they can do so independently of other surrounding instructions and in the shadow of expensive square root instructions for rotation sub-tracks (we need to reconstruct the quaternion <code class="language-plaintext highlighter-rouge">w</code> component and we need to normalize the resulting interpolated value). In practice, we don’t have to interpolate three times (one for each possible alpha value) because both <code class="language-plaintext highlighter-rouge">0.0</code> and <code class="language-plaintext highlighter-rouge">1.0</code> are trivial.</p>
<h2 id="caveats-when-key-frames-are-removed">Caveats when key frames are removed</h2>
<p>When the <a href="/2021/01/17/progressive_animation_streaming/">database feature</a> is used, things get a bit more complicated. Whole keyframes can be moved to the database and optionally stripped. This means that non-neighboring keyframes might interpolate together.</p>
<p>When all tracks interpolate with the same rounding mode, we can reconstruct the right interpolation alpha to use based on our closest keyframes present.</p>
<p>For example, let’s say that we have 3 keyframes: A, B, and C. Consider the case where we sample the time just after where B lies (at time <code class="language-plaintext highlighter-rouge">t</code>).</p>
<p><img src="/public/keyframe_reconstruction.jpg" alt="Keyframe reconstruction" /></p>
<p>If B has been moved to the database and isn’t present in memory during decompression, we have to reconstruct our value based on our closest remaining neighbors A and C (an inherently lossy process). When all tracks use the same rounding mode, this is clean and fast since only one interpolation alpha is needed for all tracks:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">none</code>: one alpha value is needed between A and C, past the 50% mark where B lies (see <code class="language-plaintext highlighter-rouge">x'</code> above)</li>
<li><code class="language-plaintext highlighter-rouge">floor</code>: one alpha value is needed at 50% between A and C to reconstruct B (see <code class="language-plaintext highlighter-rouge">B'</code> above)</li>
<li><code class="language-plaintext highlighter-rouge">ceil</code>: one alpha value is needed at 100% on C</li>
<li><code class="language-plaintext highlighter-rouge">nearest</code>: the same alpha value as <code class="language-plaintext highlighter-rouge">floor</code> and <code class="language-plaintext highlighter-rouge">ceil</code> is used</li>
</ul>
<p>However, consider what happens when individual tracks have their own rounding mode. We need three different interpolation alphas that are not trivial (<code class="language-plaintext highlighter-rouge">0.0</code> or <code class="language-plaintext highlighter-rouge">1.0</code>). Trivial alphas are very cheap since they fully match our known samples. To keep the code flexible and fast during decompression, we would have to fully interpolate three times:</p>
<ul>
<li>Once for <code class="language-plaintext highlighter-rouge">floor</code> since the interpolation alpha needed for it might not be a nice number like <code class="language-plaintext highlighter-rouge">0.5</code> if multiple consecutive keyframes have been removed.</li>
<li>Once for <code class="language-plaintext highlighter-rouge">ceil</code> for the same reason as <code class="language-plaintext highlighter-rouge">floor</code>.</li>
<li>Once for <code class="language-plaintext highlighter-rouge">none</code> for our actual interpolated value.</li>
</ul>
<p>As a result, the behavior will differ for the time being as ACL will return A with <code class="language-plaintext highlighter-rouge">floor</code> instead of the reconstructed B. This is unfortunate and I hope to address this in the next minor release (v2.2).</p>
<p><em>Note: This discrepancy will only happen when interpolating with missing keyframes. If the missing keyframes are streamed in, everything will work as expected as if no database was used.</em></p>
<p>In a future release, ACL will support <a href="https://github.com/nfrechette/acl/issues/392">splitting tracks into layers</a> to group them together. This will allow us to use different rounding modes for different layers. We will also be able to specify <a href="https://github.com/nfrechette/acl/issues/407">which tracks have values that can be interpolated or not</a>, allowing us to identify boundary keyframes that cannot be moved to the database.</p>
<p>I would have liked to properly handle this right off the bat however due to time constraints, I opted to defer this work until the next release. I’m hoping to finalize the release of <a href="https://github.com/nfrechette/acl/milestone/11">v2.1</a> this year. There is no ETA yet for v2.2 but it will focus on <a href="https://github.com/nfrechette/acl/milestone/12">streaming and other improvements</a>.</p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Compressing looping animation clips2022-04-03T00:00:00+00:00http://nfrechette.github.io/2022/04/03/anim_compression_looping<p>Animations that loop have a unique property in that the last character pose must exactly match the first, for playback to be seamless as they wrap around. As a result, this means that the repeating first/last pose is redundant. Looping animations are fairly common, and it thus makes sense to try and leverage this in order to reduce the memory footprint. However, there are important caveats with this, and special care must be taken.</p>
<p>The <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> now supports two looping modes: clamp and wrap.</p>
<p>To illustrate things, I will use a simple 1D repeating sequence but in practice, everything can be generalized in 3D (e.g. translations), 4D (e.g. rotation quaternions), and other dimensions.</p>
<h2 id="clamping-loops">Clamping loops</h2>
<p>Clamping requires the first and last samples to match exactly, and both will be retained in the final compressed data. This keeps things very simple.</p>
<p><img src="/public/acl/loop_clamp_mode.jpg" alt="Clamped loop" /></p>
<p>Calculating the clip duration can be achieved by taking the number of samples and subtracting one before multiplying by the sampling rate (e.g. 30 FPS). We subtract by one because we count how many intervals of time we have in between our samples.</p>
<p>If we wish to sample the clip at some time <code class="language-plaintext highlighter-rouge">t</code>, we normalize it by dividing by the clip duration to bring the resulting value between <code class="language-plaintext highlighter-rouge">[0.0, 1.0]</code> inclusive.</p>
<p>We can then multiply it with the last sample index to find which two values to use when interpolating:</p>
<ul>
<li>The first sample is the <code class="language-plaintext highlighter-rouge">floor</code> value (or the truncated value)</li>
<li>The second sample is the <code class="language-plaintext highlighter-rouge">ceil</code> value (or simply the first + 1) we clamp with the last sample index</li>
</ul>
<p>The interpolation alpha value between the two is the resulting fractional part.</p>
<p>Reconstructing our desired value can then be achieved by interpolating the two samples.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">float</span> <span class="n">duration</span> <span class="o">=</span> <span class="p">(</span><span class="n">num_samples</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">sample_rate</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">normalized_t</span> <span class="o">=</span> <span class="n">t</span> <span class="o">/</span> <span class="n">duration</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">sample_offset</span> <span class="o">=</span> <span class="n">normalized_t</span> <span class="o">*</span> <span class="p">(</span><span class="n">num_samples</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">first_sample</span> <span class="o">=</span> <span class="n">trunc</span><span class="p">(</span><span class="n">sample_offset</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">second_sample</span> <span class="o">=</span> <span class="n">min</span><span class="p">(</span><span class="n">first_sample</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="n">num_samples</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">interpolation_alpha</span> <span class="o">=</span> <span class="n">sample_offset</span> <span class="o">-</span> <span class="n">first_sample</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">result</span> <span class="o">=</span> <span class="n">lerp</span><span class="p">(</span><span class="n">sample_values</span><span class="p">[</span><span class="n">first_sample</span><span class="p">],</span> <span class="n">sample_values</span><span class="p">[</span><span class="n">second_sample</span><span class="p">],</span> <span class="n">interpolation_alpha</span><span class="p">);</span>
</code></pre></div></div>
<p>This works for any sequence with at least one sample, regardless of whether or not it loops. By design, we never interpolate between the first and last samples as playback loops around.</p>
<p>This is the behavior in ACL v2.0 and prior. No effort had been made to take advantage of looping animations.</p>
<h2 id="wrapping-loops">Wrapping loops</h2>
<p>Wrapping strips the last sample, and instead uses the first sample to interpolate with. As a result, unlike with the clamp mode, it will interpolate between the first and last samples as playback loops around.</p>
<p><img src="/public/acl/loop_wrap_mode.jpg" alt="Wrapped loop" /></p>
<p>This complicates things because it means that the duration of our clip is no longer a function of the number of samples actually present. One less sample is stored, but we have to remember to account for our repeating first sample.</p>
<p>Accounting for that fact, this gives us the following:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">float</span> <span class="n">duration</span> <span class="o">=</span> <span class="n">num_samples</span> <span class="o">*</span> <span class="n">sample_rate</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">normalized_t</span> <span class="o">=</span> <span class="n">t</span> <span class="o">/</span> <span class="n">duration</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">sample_offset</span> <span class="o">=</span> <span class="n">normalized_t</span> <span class="o">*</span> <span class="n">num_samples</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">first_sample</span> <span class="o">=</span> <span class="n">trunc</span><span class="p">(</span><span class="n">sample_offset</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">second_sample</span> <span class="o">=</span> <span class="n">first_sample</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">first_sample</span> <span class="o">==</span> <span class="n">num_samples</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Sampling the last sample with full weight, use the first sample</span>
<span class="n">first_sample</span> <span class="o">=</span> <span class="n">second_sample</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">sample_offset</span> <span class="o">=</span> <span class="mf">0.0</span><span class="n">f</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">else</span> <span class="nf">if</span> <span class="p">(</span><span class="n">second_sample</span> <span class="o">==</span> <span class="n">num_samples</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Interpolating with the last sample, use the first sample</span>
<span class="n">second_sample</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">float</span> <span class="n">interpolation_alpha</span> <span class="o">=</span> <span class="n">sample_offset</span> <span class="o">-</span> <span class="n">first_sample</span><span class="p">;</span>
<span class="kt">float</span> <span class="n">result</span> <span class="o">=</span> <span class="n">lerp</span><span class="p">(</span><span class="n">sample_values</span><span class="p">[</span><span class="n">first_sample</span><span class="p">],</span> <span class="n">sample_values</span><span class="p">[</span><span class="n">second_sample</span><span class="p">],</span> <span class="n">interpolation_alpha</span><span class="p">);</span>
</code></pre></div></div>
<p>By design, the above logic only works for clips which have had their last sample removed. As such, it is not suitable for non-looping clips.</p>
<h2 id="how-much-memory-does-it-save">How much memory does it save?</h2>
<p>I analyzed the animation clips from <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a> to see how much memory a single key frame (whole pose at a point in time) uses. The results were somewhat underwhelming to say the least.</p>
<table>
<thead>
<tr>
<th>Paragon</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>208.93 MB</td>
<td>208.71 MB</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0099 cm</td>
<td>0.0099 cm</td>
</tr>
</tbody>
</table>
<p>Roughly 200 KB were saved or about 0.1% even though 1454 clips (out of 6558) were identified as looping. Looking deeper into the data shows the following hard truth: <a href="/2020/08/09/animation_data_numbers/">animated key frame data</a> is pretty small to begin with.</p>
<table>
<thead>
<tr>
<th>Paragon</th>
<th>50th percentile</th>
<th>85th percentile</th>
<th>99th percentile</th>
</tr>
</thead>
<tbody>
<tr>
<td>Animated frame size</td>
<td>146.62 Bytes</td>
<td>328.88 Bytes</td>
<td>756.38 Bytes</td>
</tr>
</tbody>
</table>
<p>Half the clips have an animated key frame size of 146 bytes or smaller. Because joints are not all animated, they do not all benefit equally.</p>
<p>Here is the full distribution of how much memory was saved as a percentage of the original size:</p>
<p><img src="/public/acl/loop_optimization_paragon_distribution.png" alt="Distribution of percentage saved with loop optimization on Paragon animations" /></p>
<p>Interestingly, some clips end up larger when the last key frame is removed. How come? It all boils down to the fact that ACL tries to split the number of key frames evenly across all sub-segments it generates. <a href="/2016/11/10/anim_compression_uniform_segmenting/">Segments</a> are used to improve accuracy and lower the memory footprint but the current method is naive and will be improved on <a href="https://github.com/nfrechette/acl/issues/404">in the future</a>. As such, when a loop is detected, ACL can do one of two things: the last key frame can be removed <em>before</em> segments are created or <em>afterwards</em>. I opted to do so <em>before</em> to ensure proper balancing. As such, through the re-balancing that occurs, some clips can end up being larger. Either way, the small size of the animated key frame data puts a fairly low ceiling on how much this optimization can save.</p>
<p><em>Side node: thanks to this, I found a <a href="https://github.com/nfrechette/acl/issues/403">small bug</a> that impacts a very small subset of clips. Clips impacted by this ended up with segments containing more key frames than they otherwise would have resulting in a slightly larger memory footprint.</em></p>
<h2 id="caveats">Caveats</h2>
<p>Special care must be taken when removing the last sample, as joint tracks are not all equal. In a looping clip, the last sample will match the first for joints which have a direct impact on visual feedback to the user, but it might not be the case for joints which have an indirect impact, or those that have no impact. For example, while the joint for a character’s hand has direct visual feedback as we see it on screen, other joints like the root have only indirect feedback (through root motion), and any number of custom invisible joints might exist for things like inverse kinematics and other metadata required.</p>
<p>A looping clip with root motion might have a perfectly matching last sample for the visible portion of the character pose, but the root value might differ between the two. When this is the case, removing the last sample means truncating important data! For that reason, ACL only detects wrapping when <em>ALL</em> joints repeat perfectly between their first and last sample.</p>
<h2 id="why-is-root-motion-special">Why is root motion special?</h2>
<p>An animation clip contains a series of transforms for every joint at different points in time. To remove as much redundancy as possible, joints are stored relative to their parent (if they have one, local space). In order to display the visual mesh, each joint transform is combined with its parent, recursively, all the way to the root of the character (converting from local space to object space).</p>
<p>In an animation editor, a character will be animated relative to some origin point at <code class="language-plaintext highlighter-rouge">[0,0,0]</code> (object space). However, in practice, we need to be able to play that animation anywhere in the world (world space). As such, we need to transform joints from object space into world space by using the character’s transform.</p>
<p><img src="/public/acl/no_root_displacement.jpg" alt="Character motion without root joint" /></p>
<p>The pelvis joint is usually the highest most visible joint in the hierarchy with the legs and spine in turn parented to it, etc. When the character moves, every joint has direct visual feedback as their movement deforms the parts of the visual mesh attached to them (through the rigid skinning process).</p>
<p>Now what happens if we wish to make a walking animation where the character moves at some desired speed? If the pelvis joint is the highest most joint in the hierarchy, this motion will happen there, and it will drag every joint parented to it (because children transforms are stored relative to their parent to avoid having redundant motion in every joint).</p>
<p>This is problematic because now, the character motion is mixed in with what the pelvis does in this particular clip (e.g. swaying back and forth). It is no longer possible to extract cleanly that motion free from the noise of the animation. As the animation plays back in the world, things might be attached to the character (e.g. camera) that we wish to attach to the character himself, and not some particular joint. If only the joints move, this won’t work. As such, we often want to extract this character motion and apply it as a delta from the previous animation update. We would remove this motion from the joints, and instead transfer it onto the character.</p>
<p><img src="/public/acl/root_displacement.jpg" alt="Character motion with root joint" /></p>
<p>To simplify this process, character motion is generally stored in a root joint that sits highest in the joint hierarchy. This joint does not directly move anything, and we instead extract the character motion from it and apply it onto the character.</p>
<p>As a result, character motion is stored as an absolute displacement value relative to that <code class="language-plaintext highlighter-rouge">[0,0,0]</code> origin mentioned earlier. In a 1-second-long animation where the character moves 1 meter, on the first sample the root value will be zero meters and on the last sample, the value is 1 meter. Values in between can vary depending on how the character velocity changes. As such, even if the character pose is identical between the first and last samples, their root joints differ and <em>CANNOT</em> be interpolated between.</p>
<p>When we sample the animation, we can find out our character motion delta as follow: <code class="language-plaintext highlighter-rouge">current_root_position - previous_root_position</code>. This gives us how much the character moved since the last update. When the animation loops, we must break this calculation down into two parts:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">last_root_position - previous_root_position</code> to get the trailing end of the motion</li>
<li><code class="language-plaintext highlighter-rouge">current_root_position - first_root_position</code> to get the rest of the motion (note that <code class="language-plaintext highlighter-rouge">first_root_position</code> is usually zero and can be omitted)</li>
</ul>
<p>There is also one more special case to consider: if the animation loops more than once since the last update. This can happen for any number of reasons: playback rate changes, the character was off screen for a while and is now visible and updating again, etc. When this happens, we must extract the total displacement of the animation and multiply it by how many times we looped through. The total displacement value lives in our last sample and can simply be read from there: <code class="language-plaintext highlighter-rouge">last_root_position * num_loops</code>.</p>
<p>If, for whatever reason, we forcibly remove the last sample, we will no longer have information on how fast the root was moving as it wrapped around. This velocity information is permanently lost and cannot be reconstructed.</p>
<p>It is generally desirable to keep the number of samples for every joints the same for compression purposes within an animation clip. If we don’t, we would need to either store how many samples we have per joints (and there can be many) or whether the last sample has been removed or not. This added overhead and complexity is ideally one we’d like to do without.</p>
<p>It is thus tempting to remove the last sample and estimate the missing value somehow. While this might work with few to no visible artifacts, it can have subtle implications when other things that should be moving at the same overall rate are visible on screen. For example, two characters might be running side by side at the same velocity, but with different animations. Just because the overall velocity is the same (e.g. both move at 1 meter per second), it does not mean that the velocity on the last sample matches. One animation might start fast and slow down before looping, while the other does the opposite. If this is the case, both characters will end up slowly drifting from each other even though they should not.</p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Bind pose stripping2022-01-23T00:00:00+00:00http://nfrechette.github.io/2022/01/23/anim_compression_bind_pose_stripping<p>A little over three years ago, <a href="/2018/05/08/anim_compression_additive_bind/">I wrote about storing animation clips relative to the bind pose</a> in order to improve the compression ratio. In a gist, this stores each clip as an additive onto the base <a href="/2016/10/26/anim_compression_terminology/">bind pose</a>.</p>
<p>This does two things for us:</p>
<ul>
<li>It reduces the range of each sub-track (rotation, translation, scale) which allows more accuracy to be retained (e.g. instead of having the pelvis bone animating at 60cm above the root on the ground, it will animate around 0cm relative to the bind position 60cm above said ground).</li>
<li>It increases the likelihood that sub-tracks will become equal to their identity value as very often joints are not animated and will be equal to their bind pose value. <a href="/2016/11/03/anim_compression_constant_tracks/">This allows us to remove their values entirely</a> from the compressed byte stream since we only need a bit set to reconstruct them: whether the value is equal to the identity or not.</li>
</ul>
<p>It is very common for sub-tracks to not be animated and to retain the bind pose value, especially for translations. For example, upper body animations might have all of the lower body be identical to the bind pose. Facial animations might have the rest of the body equal to it. For that reason, at the time, I reported memory savings of up to <strong>8%</strong> with that method.</p>
<p>The main drawback of the technique is that in order to reconstruct the clip, we have to apply the output pose onto the bind pose much like we would with an additive clip. This means performing an expensive transform multiplication. With a QVV (quat-vec3-vec3) format, this means performing three quaternion multiplications which isn’t cheap.</p>
<p>However, there is an alternate way to leverage the bind pose to achieve similar memory savings without the expensive overhead: stripping it entirely.</p>
<h2 id="how-it-works">How it works</h2>
<p>The <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> now allows you to specify per joint what its default value should be. Default values are not stored within the compressed clip and instead rely on a simple bit set.</p>
<p>If a sub-track is not animated, it has a constant value across all its samples. If that constant value is equal to the default value specified, then ACL simply strips it. Later, during decompression, if a sub-track has a default value, we simply write it into the output pose.</p>
<p>ACL is very flexible here and it allows you to specify either a constant value for every default sub-track (e.g. the identity) or you can use a unique value per sub-track (e.g. the bind pose). This way, the full output pose can be safely reconstructed. As a bonus, it also allows default sub-tracks to be skipped entirely. This is very handy when you pre-fill the output pose buffer with the bind pose before decompressing a clip. This can be achieved efficiently with memcpy (or similar) and during decompression default sub-tracks will be skipped, leaving the value untouched in the output pose.</p>
<p>By removing the bind pose, we achieve a very similar result as we would storing the clip as an additive on top of it. We can leverage the fact that many sub-tracks are not animated and are equal to their bind pose value. However, we do not reduce the range of motion.</p>
<p>Crucially, reconstructing the original pose is now much cheaper as it does not involve any expensive arithmetic and the bind pose will often be warm in the CPU cache as multiple clips will use it and it might be used for other things as part of the animation update/evaluation.</p>
<p><em>Side note: Reducing the range of motion can be partially achieved for translation and scale by simply removing the bind pose value with component wise subtraction. This allows us to reconstruct the original value by adding the bind pose values which is very cheap.</em></p>
<h2 id="results">Results</h2>
<p>Now that ACL supports this, I measured bind pose stripping against two data sets:</p>
<ul>
<li><a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">Carnegie-Mellon University’s motion capture database</a> which contains 2534 clips.</li>
<li><a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a> which contains 6558 clips.</li>
</ul>
<p>I measured the final compressed size before and after as well as the 99th percentile error (99% of joint samples have an error below this value):</p>
<table>
<thead>
<tr>
<th>CMU</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>75.55 MB</td>
<td>75.50 MB</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0088 cm</td>
<td>0.0088 cm</td>
</tr>
</tbody>
</table>
<p>For CMU, the small gain is expected due to the nature of motion capture data and bind pose stripping performs about as well as storing clips relative to it. Motion capture is often noisy and sub-tracks are unlikely to be constant, let alone equal to the bind pose.</p>
<table>
<thead>
<tr>
<th>Paragon</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>224.30 MB</td>
<td>220.85 MB</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0095 cm</td>
<td>0.0090 cm</td>
</tr>
</tbody>
</table>
<p>However, for Paragon, the results are <strong>1.54%</strong> smaller. It turns out that quite a few clips are adversely impacted by bind pose stripping. They can end up with a size far higher which I found surprising.</p>
<p><img src="/public/acl/bind_pose_stripping_paragon_distribution.png" alt="Paragon size delta distribution" /></p>
<p>In the above image, I plotted the size delta as a percentage. Positive values denote a reduction in size.</p>
<p>As we can see, for the vast majority of clips, we observe a reduction in size: 5697 (87% of) clips ended up smaller. Over 1889 (29% of) clips saw a reduction of 10% or more. The median represents 4.8% in savings. Not bad!</p>
<p>Sadly, 59 (1% of) clips saw an increase in size of 10% or more with the largest increase at 67.7%.</p>
<p>Looking at some of the clips that perform terribly helped shed some light as to what is going on. Clips that have long bone chains of default sub-track values equal to their bind pose can end up with very high error. To compensate ACL retains more bits to preserve quality. This is caused by two things.</p>
<p>First, we use an error threshold to detect when a sub-track is equal to its default value or not. Even though the error threshold is very conservative, a very small amount of error is introduced and it can compound in long bone chains.</p>
<p>Second, when we measure the compression error, we do so by using the original raw clip. Because the bind pose values we strip rely on the error threshold mentioned above, the optimization algorithm can end up trying to reach a pose that is can never reach. For example, if a joint is stripped by being equal to the bind pose and doing so introduces an error of 1cm on some distant child, even if every joint in between remains with full raw precision values, the error will remain.</p>
<p><em>Side note, constant sub-tracks use the same error threshold to detect if they are animated or not which can lead to the same issues happening.</em></p>
<p>This is not easily remedied without some form of error compensation which ACL does not currently support. However, I’m hoping to integrate a partial solution in the coming months. Stay tuned!</p>
<p><em>Side note, we could pre-process the raw data to ensure that constant and default sub-tracks are clean with every sample perfectly repeating. This would ensure that the error metric does not need to compensate. However, ACL does not own the raw data and as such it cannot do such transformations safely. A future release might expose such functions to clean up the raw data prior to compression.</em></p>
<h2 id="conclusion">Conclusion</h2>
<p>For the time being I recommend that if you use this new feature, you should also try not stripping the bind pose and pick the best of the two results (for most clips, ACL compresses very fast). The develop branch of the <a href="https://github.com/nfrechette/acl-ue4-plugin">Unreal Engine 4 ACL plugin</a> now supports this experimental feature and testing both codec variations can easily be achieved there (and in parallel too).</p>
<p>Anedoctally, a few people have reached out to me about leveraging this feature and they reported memory savings in the <strong>3-5%</strong> range. YMMV.</p>
<p>While the memory savings of this technique aren’t as impressive as storing clips as additives of the bind pose, their dramatically lower decompression cost makes it a very attractive optimization.</p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
ACL 2.0 is hot off the press2021-05-04T00:00:00+00:00http://nfrechette.github.io/2021/05/04/acl_v2.0.0<p>After 18 months of work, the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> has finally reached <a href="https://github.com/nfrechette/acl/releases/tag/v2.0.0">v2.0</a> along with an updated <a href="https://github.com/nfrechette/acl-ue4-plugin/releases/tag/v2.0.1">v2.0 Unreal Engine 4 plugin</a>.</p>
<p>Notable changes in this release include:</p>
<ul>
<li>Unified and cleaned up APIs</li>
<li>Cleaned up naming convention to match C++ stdlib, boost</li>
<li><a href="/2021/01/17/progressive_animation_streaming/">Introduced streaming database support</a></li>
<li>Decompression profiling now uses <a href="https://github.com/google/benchmark">Google Benchmark</a></li>
<li>Decompression has been heavily optimized</li>
<li>Compression has been heavily optimized</li>
<li>First release to support backwards compatibility going forward</li>
<li>Migrated all math to <a href="https://github.com/nfrechette/rtm">Realtime Math</a></li>
<li>Clips now support 4 billion samples/tracks</li>
<li><a href="https://github.com/nfrechette/acl-js">WebAssembly support added through emscripten</a></li>
<li>Many other improvements</li>
</ul>
<p>Overall, this release is cleaner, leaner, and much faster.</p>
<p><a href="https://github.com/nfrechette/acl-ue4-plugin/blob/release/2.0/Docs/decompression_performance.md">Decompression is now <strong>1.4x</strong> (PC) to <strong>1.8x</strong> (Mobile) faster than ACL 1.3</a> which is no small feat! I’ll be writing a blog post in the next few months with the juicy details. To make this possible, the memory footprint may increase slightly (mostly header related changes, a few bytes here and there, and alignment padding) but many datasets showed no increase at all. Quality remains unchanged.</p>
<h1 id="whats-next">What’s next</h1>
<p>I’ve already started fleshing out the task list for the next minor release <a href="https://github.com/nfrechette/acl/milestone/11">here</a>. This release will bring about more memory footprint improvements.</p>
<p>If you use ACL and would like to help prioritize the work I do, feel free to reach out and provide feedback or requests!</p>
<p>ACL 2.0 turned out to be a massive undertaking and it took countless hours, late nights, and weekends to make it happen (over 1700 commits since the project began 4 years ago). As such, I’ll pause development for a month or two (or three) while I focus on writing a few blog posts I’ve been meaning to get to and take a much needed break. However, I’ll continue to make patch releases during this time if anything important pops up.</p>
<p>Special thanks to <a href="https://kr.ncsoft.com">NCSOFT</a> for sponsoring many of the improvements that came with this major release!</p>
Controlling animation quality through streaming2021-01-17T00:00:00+00:00http://nfrechette.github.io/2021/01/17/progressive_animation_streaming<p>For a few years now, I’ve had ideas on how to leverage streaming to further improve compression with the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a>. Thanks to the support of <a href="https://kr.ncsoft.com">NCSOFT</a>, I’ve been able to try them out and integrate progressive quality streaming for the upcoming 2.0 release next month.</p>
<p>Progressive quality streaming is perfectly suited for modern games on high end consoles all the way down to mobile devices and web browsers. Its unique approach will empower animators to better control animation quality and when to pay the price for it.</p>
<h2 id="space-is-precious">Space is precious</h2>
<p>For many mobile and web games out there, the size of animation data remains an ever present issue. All of this data needs to be downloaded on disk and later loaded into memory. This takes time and resources that devices do not always have in large supply. Moreover, on a small screen, the animation quality doesn’t matter as much as compression artifacts are often much less visible than on a large 4K monitor. Although it might seem like an ancient problem older consoles and early mobile phones had to worry about, many modern games still contend with it today.</p>
<p>A popular technique to deal with this is to use <a href="/2016/11/17/anim_compression_sub_sampling/">sub-sampling</a>: take the key frames of an input animation clip and re-sample it with fewer key frames (e.g. going from 30 FPS to 24 FPS). Unreal Engine 4 implements a special case of this called <a href="https://docs.unrealengine.com/en-US/API/Runtime/Engine/Animation/UAnimSequence/index.html">frame stripping</a>: every other key frame is removed.</p>
<p>By their nature, these techniques are indiscriminate and destructive: the data is permanently removed and cannot be recovered. Furthermore, if a specific key frame is of particular importance (e.g. highest point of a jump animation), it could end up being removed leading to undesirable artifacts. Despite these drawbacks, they remain very popular.</p>
<p>In practice, some data within each animation cannot be removed (metadata, etc) and as I have documented in a <a href="/2020/08/09/animation_data_numbers/">previous blog post</a>, the animated portion of the data isn’t always dominant. For that reason, frame stripping often yields a memory reduction around <strong>30-40%</strong> despite the fact that we remove every other key frame.</p>
<h2 id="bandwidth-is-limited">Bandwidth is limited</h2>
<p>Animation data also competes with every other asset for space and bandwidth. Even with modern SSDs, loading and copying hundreds of megabytes of data still takes time. Modern games now have hundreds of megabytes of high quality textures to load as well as dense geometry meshes. These often have solutions to alleviate their cost both at runtime and at load time in the form of Levels of Details (e.g. mip maps). However, animation data does not have an equivalent to deal with this problem because animations are closer in spirit to audio assets: most of them are either very short (a 2 second long jump and its 200 millisecond sound effect) or very long (a cinematic and its background music).</p>
<p>Most assets that leverage streaming end up doing so in one of two ways: on demand (e.g. texture/mip map streaming) or ahead of time (e.g. video/audio streaming). Sadly, neither solution is popular with animation data.</p>
<p>When a level starts, it generally has to wait for all animation data to be loaded. Stalling and waiting for IO to complete during gameplay would be unacceptable and similarly playing a generic <a href="https://en.wikipedia.org/wiki/T-pose">T-stance</a> would quickly become an obvious eye sore. By their nature, gameplay animations can start and end at any moment based on player input or world events. Equally worse, gameplay animations are often very short and we wouldn’t have enough time to stream them in part (or whole) before the information is needed.</p>
<p>On the other hand, long cinematic sequences that contain lots of data that play linearly and predictably can benefit from streaming. This is often straightforward to implement as the whole animation clip can be split into smaller segments each compressed and loaded independently. In practice, cinematics are often loaded on demand through higher level management constructs and as such progressive streaming is not very common (UE4 does support it but to my knowledge it is not currently documented).</p>
<h2 id="the-gist">The gist</h2>
<p>Here are the constraints we work with and our wish list:</p>
<ul>
<li>Animations can play at any time and must retain some quality for their full duration</li>
<li>Not all key frames are equally important, some must always be present while others can be discarded or loaded later</li>
<li>Most animations are short</li>
<li>Large contiguous reads from disk are better than many small random reads</li>
<li>Decompression must remain as fast as possible</li>
</ul>
<h2 id="enter-progressive-quality-streaming">Enter progressive quality streaming</h2>
<p>Because most animations are short, it makes sense to attempt to load their data in bulk. As such, ACL now supports aggregating multiple animation clips into a single database. Important key frames and other metadata required to decompress will remain in the animation clip and optionally at runtime, the database can be provided to feed it the remaining data.</p>
<p>This leads us to the next question: how do we partition our data? How do we determine what remains in the animation clip and what can be streamed later? Crucially, how do we make sure that decompression remains fast now that our data can live in multiple locations?</p>
<p>To solve this second part of the problem, during compression ACL will tag each whole key frame with how much error removing it contributes. We use this information to construct a variant of <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> where only whole key frames are removed. However, in our case, they will simply be moved to the database instead of being lost forever. This allows us to quickly find the data we need when we sample our animation clip with a single search for which key frames we need. This helps keep the cost constant regardless of how many key frames or joints an animation clip has. By further limiting the number of key frames in a segment to a maximum of 32, finding the data we need boils down to a few bit scanning operations efficiently implemented in hardware.</p>
<p>The algorithm is straightforward. We first assume that every key frame is retained. We’ll pick the first that is movable (e.g. not first/last in a segment) and measure the resulting error when we remove it (both on itself and on its neighbors that might have already been removed). Any missing key frames are reconstructed using linear interpolation from their neighbors. To measure the error, we use the same <a href="/2016/11/01/anim_compression_accuracy/">error metric</a> used to optimize the variable bit rates. We record how much error is contributed and we add back the key frame. We’ll iterate over every key frame doing so. Once we have the contributing error of every key frame, we’ll pick the lowest error and remove that key frame permanently. We’ll then repeat the whole process with the remaining key frames. Each step will remove the key frame that contributes the least error until they are all removed. This yields us a sorted list of how important they are. While not perfect and exhaustive, this gives us a pretty good approximation. This error contribution is then stored as extra metadata within the animation clip. This metadata is only required to build the database and it is stripped when we do so.</p>
<p><img src="/public/error_contribution_step1.jpg" alt="Calculating the error contribution step 1" /></p>
<p><img src="/public/error_contribution_step2.jpg" alt="Calculating the error contribution step 2" /></p>
<p><img src="/public/database_frame_importance.jpg" alt="Final importance tier" /></p>
<p>Now that we know which key frames are important and which aren’t, we’ll iterate over every animation clip and move the least important key frames out into the database first. Our most important key frames will remain within each animation clip to be able to retain some quality if we need to play back either with no database or with partial database data. How much data each tier will contain is user controlled and optionally the lowest tier can be stripped.</p>
<p><img src="/public/database_settings.jpg" alt="Database settings" /></p>
<p>We consider three quality tiers for now:</p>
<ul>
<li>High importance key frames will remain in the animation clips as they are not optional</li>
<li>Medium and low importance key frames are moved to the database each in its separate tier which can be streamed independently</li>
</ul>
<p>Since we know every clip that is part of the database, we can find the globally optimal distribution. As such, if we wish to move the least important 50% percent, we will remove as much data as frame stripping but now the operation is far less destructive. Some clips will contribute more key frames than others to reach the same memory footprint. This is frame stripping on steroids!</p>
<p><img src="/public/importance_tier_vs_fidelity.jpg" alt="Importance tiers VS visual fidelity" /></p>
<p>This partitioning allows us to represent three visual fidelity levels:</p>
<ul>
<li>Highest visual fidelity requires all three importance quality tiers to be loaded</li>
<li>Medium visual fidelity requires only the high and medium importance tiers to be loaded</li>
<li>Lowest visual fidelity requires only the high importance tier to be loaded (the clip itself)</li>
</ul>
<p><img src="/public/streaming_blueprint_editing.jpg" alt="Controlling visual fidelity with UE4 blueprints" /></p>
<p>Under the hood, ACL allows you to stream both tiers independently and in any order you wish. For simplicity, the UE4 plugin exposes the desired visual fidelity level and the streaming request size granularity while abstracting what needs to stream in or out. This allows the game to allocate memory on demand when data is streamed in while also allowing the game to unload tiers that are no longer needed. In a pinch, the entire database can be unloaded or destroyed and animations can continue to play at the lowest visual fidelity level.</p>
<h2 id="unprecedented-control">Unprecedented control</h2>
<p>What this means is that you can now group animations into as many databases as makes sense for your game. Some animations always need the highest fidelity and shouldn’t belong to any database (e.g. main character locomotion) while general gameplay animations and exotic animations (e.g. emotes) can be split into separate databases for ultimate control. You can now decide at a high level how much data to stream later and when to stream it. Crucially, this means that you can decide ahead of time if a quality tier isn’t required and strip it entirely from disk or you can make that decision at runtime based on the device the game runs on.</p>
<p>You can make a single package for your mobile game that can run with reduced quality on lower end devices while retaining full quality for higher end ones.</p>
<p>Your multiplayer game can stream in the hundreds of emotes by grouping them by popularity lazily in the background.</p>
<iframe src="https://giphy.com/embed/QIiqoufLNmWo8" width="480" height="360" frameborder="0" class="giphy-embed" allowfullscreen=""></iframe>
<p><a href="https://giphy.com/gifs/QIiqoufLNmWo8">via GIPHY</a></p>
<h2 id="room-to-grow">Room to grow</h2>
<p>Because the feature is new, I had to make a few executive decisions to keep the scope down while leaving room for future changes. Here are a few points that could be improved on over time.</p>
<p>I had to settle on three quality tiers both for simplicity and performance. Making the number arbitrary complicates authoring while at the same time might degrade decompression performance (which now only adds 100-150 instructions and 2 cache misses compared to the normal path without a database lookup). That being said, if a good case can be made, that number could be increased to five without too much work or overhead.</p>
<p>Evaluating how much error each key frame contributes works fine but it ends up treating every joint equally. In practice, some joints contribute more while others contribute less. Facial and finger joints are often far less important. Joints that move fast are also less important as any error will be less visible to the naked eye (see <a href="http://www.jp.square-enix.com/tech/publications.html">Velocity-based compression of 3D rotation, translation, and scale animations</a> by David Goodhue about using velocity to compress animations). Instead of selecting the contributing error solely based on one axis (which key frame), we could split into a second axis: how important joints are. This would allow us to retain more quality for a given quality tier while reaching the same memory footprint. The downside though is that this will increase slightly the decompression cost as we’ll now need to search for four key frames to interpolate (we’ll need two for high and two for low importance joints). Adding more partitioning axes increases that cost linearly.</p>
<p>Any key frame can currently be moved to the database with two exceptions: each internal segment retains its first and last key frame. If the settings are aggressive enough, everything else can be moved out into the database. In practice, this is achieved by simply pinning their contributing error to infinity which prevents them from being moved. This same trick could be used to prevent specific key frames from being moved if an animator wished to author that information and feed it to ACL.</p>
<p>In order to avoid small reads from disk, data is split into chunks of 1 MB. At runtime, we specify how many chunks to stream at a time. This means that each chunk contains multiple animation clips. No metadata is currently kept for this mapping and as a result it is not currently possible to stream in specific animation clips as it would somewhat defeat the bulk streaming nature of the work. Should this be needed, we can introduce metadata to stream in individual chunks but I hope that it won’t be necessary. In practice, you can split your animations into as many databases as you need: one per character, per character type, per gameplay mode, etc. Managing streaming in bulk ensures a more optimal usage of resources while at the same time lowering the complexity of what to stream and when.</p>
<h2 id="coming-soon">Coming soon</h2>
<p>All of this work has now landed into the main development branches for ACL and its <a href="https://github.com/nfrechette/acl-ue4-plugin">UE4 plugin</a>. You can try it out now if you wish but if not, the next major release scheduled for late February 2021 will include this and many more improvements. Stay tuned!</p>
<p><a href="/2016/10/21/anim_compression_toc/">Animation Compression Table of Contents</a></p>
Windows Hyper-V woes2020-12-27T00:00:00+00:00http://nfrechette.github.io/2020/12/27/windows_hyper_v_woes<p>This weekend, I noticed that <a href="https://www.amd.com/en/technologies/ryzen-master">AMD Ryzen Master</a> (a great tool to control your CPU when profiling code performance) and <a href="https://www.virtualbox.org/">VirtualBox</a> both stopped working for me under Windows 10. I wasn’t too sure how it happened and I spent a good deal of time chasing what went wrong so I am documenting my process and result here in hope that it might help others as well.</p>
<h2 id="why-arent-they-working">Why aren’t they working?</h2>
<p>Over the last few years, Windows has been roling out a hypervisor of its own: <a href="https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/about/">Hyper-V</a>. This allows you to run virtual machines (without VMWare, VirtualBox, or some other similar program). It also allows isolation of various <a href="https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-requirements">security components</a>, further improving kernel safety. When the hypervisor is used, Windows runs inside it, much like a guest OS runs inside a host OS in a virtual machine setup. This is the root cause of my woes.</p>
<p>AMD Ryzen Master <a href="https://community.amd.com/t5/drivers-software/amd-ryzen-master-1-3-0-618-vbs-error/td-p/179477">complains</a> (see also <a href="https://github.com/microsoft/WSL/issues/4771">this</a>) that <a href="https://docs.microsoft.com/en-us/windows-hardware/design/device-experiences/oem-vbs">Virtualization Based Security (VPS)</a> is enabled. From my understanding, what this means is that running this program within a virtual machine isn’t supported or recommended. <a href="https://github.com/milesburchell/RyzenMasterVBSFix">Some hacks</a> are floating around to patch the executable to allow the check to be skipped and it does appear to work (although I haven’t tested it myself). Attempting to disable VPS lead me to pain and misery and in the end nothing worked.</p>
<p>In turn, VirtualBox attempts to run a virtual machine inside the hypervisor virtual machine. This is problematic because to keep performance fast under virtualization, the CPU exposes hardware a feature to accelerate virtual memory translation and now that feature is used by Windows and it cannot be shared and used by VirtualBox. While the latest version will still run your guest OS (Ubuntu in my case), it will be terribly slow. So slow as to be unusable. Older VBox versions might simply refuse to start the guest OS. <a href="https://forums.virtualbox.org/viewtopic.php?f=6&t=90853">This thread</a> is filled with sorrow.</p>
<p>I spent hours pouring over forum threads trying to piece together the puzzle.</p>
<h2 id="how-did-it-suddenly-get-turned-on">How did it suddenly get turned on?</h2>
<p>I was confused at first, because I hadn’t changed anything related to Hyper-V or anything of the sort. How then, did it suddenly start being used? As it turns out, I installed <a href="https://www.docker.com/">Docker</a> which is built on top of this.</p>
<h2 id="fixing-this-once-and-for-all">Fixing this once and for all</h2>
<p>Finding a solution wasn’t easy. Microsoft documents the many security features that virtualization provides and since I use my desktop for work as well, I wanted to make sure that turning it off was safe for me. Many forum posts offer various suggestions on what to do from modifying the registry, to uninstalling various things.</p>
<p>The end goal for me was to be able to stop using the virtualization as it cannot coexist with Ryzen Master and VirtualBox. To do so, you must uninstall every piece of software that requires it. <a href="https://forums.virtualbox.org/viewtopic.php?f=1&t=62339#p452019">This VirtualBox forum post</a> lists known components that use it:</p>
<ul>
<li>Application Guard</li>
<li>Credential Guard</li>
<li>Device Guard</li>
<li>
<any> * Guard
</any>
</li>
<li>Containers</li>
<li>Hyper-V</li>
<li>Virtual Machine Platform</li>
<li>Windows Hypervisor Platform</li>
<li>Windows Sandbox</li>
<li>Windows Server Containers</li>
<li>Windows Subsystem for Linux 2 (WSL2) (WSL1 does not enable Hyper-V)</li>
</ul>
<p>To remove them, press the Windows Key + X -> App and Features -> Programs and Features -> Turn Windows features on or off.</p>
<p>You’ll of course need to uninstall Docker and other similar software.</p>
<p>To figure out if you are using Device Guard and Credential Guard, Microsoft provides <a href="https://www.microsoft.com/en-us/download/details.aspx?id=53337">this tool</a>. Run it from a PowerShell instance with administrative privileges. Make sure the execution environment is unrestricted as well (as per the readme). And run it with the <code class="language-plaintext highlighter-rouge">-Ready</code> switch to see whether or not they are used. Those features might be required by your system administrator if you work in an office or perhaps required by your VPN to connect. In my case, the features weren’t used and as such it was safe for me to remove the virtualization.</p>
<p>Once everything that might require virtualization has been removed, it is time to turn it off. Whether Windows boots inside the virtualized environment or not is determined by the boot loader. <a href="https://stackoverflow.com/questions/30496116/how-to-disable-hyper-v-in-command-line">See this thread for examples</a>. You can create new bootloader entries if you like but I opted to simply turn it off by executing with administrative privileges in a command prompt: <code class="language-plaintext highlighter-rouge">bcdedit /set hypervisorlaunchtype off</code>. If you need to turn it back on, you can execute this: <code class="language-plaintext highlighter-rouge">bcdedit /set hypervisorlaunchtype auto</code>.</p>
<p>Simply reboot and you should be good to go!</p>
Animation clip metadata packing2020-08-11T00:00:00+00:00http://nfrechette.github.io/2020/08/11/clip_metadata_packing<p>In my <a href="/2020/08/09/animation_data_numbers/">previous blog post</a>, I analyzed over 15000 animations. This offered a number of key insights about what animation data looks like for compression purposes. If you haven’t read it yet, I suggest you do so first as it covers the terminology and basics for this post. I’ll also be using the same datasets as described in the previous post.</p>
<p>As mentioned, most clips are fairly short and the metadata we retain ends up having a disproportionate impact on the memory footprint. This is because long and short clips have the same amount of metadata everything else being equal.</p>
<h2 id="packing-range-values">Packing range values</h2>
<p>Each animated track has its samples normalized within the full range of motion for each clip. This ends up being stored as a minimum value and a range extent. Both are three full precision 32 bit floats. Reconstructing our original sample is done like this:</p>
<p><code class="language-plaintext highlighter-rouge">sample = (normalized sample * range extent) + range minimum</code></p>
<p>This is quick and efficient.</p>
<p>Originally, I decided to retain full precision here out of convenience and expediency during the original implementation. But for a while now, I’ve been wondering if perhaps we can quantize the range values as well on fewer bits. In particular, each sample is also normalized a second time within the range of motion of the segment it lies in and those per segment range values are quantized to 8 bits per component. This works great as range values do not need to be all that accurate. In fact, over two years ago I tried to quantize the segment range values on 16 bits instead to see if things would improve (accuracy or memory footprint) and to my surprise, the result was about the same. The larger metadata footprint did allow higher accuracy and fewer bits retained per animated sample but over a large dataset, the two canceled out.</p>
<p>In order to quantize our range values, we must first extract the total range: every sample from every track. This creates a sort of Axis Aligned Bounding Box for rotation, translation, and scale. Ideally we want to treat those separately since by their very nature, their accepted range of values can differ by quite a bit. For translation and scale, things are a bit complicated as some tracks require full precision and the range can be very dynamic from track to track. In order to test out this optimization idea, I opted to try with rotations first. Rotations are much easier to handle since quaternions have their components already normalized within <code class="language-plaintext highlighter-rouge">[-1.0, 1.0]</code>. I went ahead and quantized each component to 8 bits with padding to maintain the alignment. Instead of storing 6x float32 (24 bytes), we are storing 8x uint8 (8 bytes). This represents a 3x reduction in size.</p>
<p>Here are the results:</p>
<table>
<thead>
<tr>
<th>CMU</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>70.61 MB</td>
<td>70.08 MB</td>
</tr>
<tr>
<td>Compressed size 50th percentile</td>
<td>15.35 KB</td>
<td>15.20 KB</td>
</tr>
<tr>
<td>Compression ratio</td>
<td>20.24 : 1</td>
<td>20.40 : 1</td>
</tr>
<tr>
<td>Max error</td>
<td>0.0725 cm</td>
<td>0.0741 cm</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0089 cm</td>
<td>0.0089 cm</td>
</tr>
<tr>
<td>Error threshold percentile rank (0.01 cm)</td>
<td>99.86 %</td>
<td>99.86 %</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Paragon</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>208.04 MB</td>
<td>205.92 MB</td>
</tr>
<tr>
<td>Compressed size 50th percentile</td>
<td>15.83 KB</td>
<td>15.12 KB</td>
</tr>
<tr>
<td>Compression ratio</td>
<td>20.55 : 1</td>
<td>20.77 : 1</td>
</tr>
<tr>
<td>Max error</td>
<td>2.8824 cm</td>
<td>3.5543 cm</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0099 cm</td>
<td>0.0111 cm</td>
</tr>
<tr>
<td>Error threshold percentile rank (0.01 cm)</td>
<td>99.04 %</td>
<td>98.89 %</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Fortnite</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>482.65 MB</td>
<td>500.95 MB</td>
</tr>
<tr>
<td>Compressed size 50th percentile</td>
<td>9.69 KB</td>
<td>9.65 KB</td>
</tr>
<tr>
<td>Compression ratio</td>
<td>36.72 : 1</td>
<td>35.38 : 1</td>
</tr>
<tr>
<td>Max error</td>
<td>69.375 cm</td>
<td>69.375 cm</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0316 cm</td>
<td>0.0319 cm</td>
</tr>
<tr>
<td>Error threshold percentile rank (0.01 cm)</td>
<td>97.69 %</td>
<td>97.62 %</td>
</tr>
</tbody>
</table>
<p>At first glance, it appears a small net win with CMU and Paragon but then everything goes downhill with Fortnite. Even though all three datasets see a win in the compressed size for 50% of their clips, the end result is a significant net loss for Fortnite. The accuracy is otherwise slightly lower. As I’ve <a href="/2017/09/10/acl_v0.4.0/">mentioned before</a>, the max error although interesting can be very misleading.</p>
<p>It is clear that for some clips this is a win, but not always nor overall. Due to the added complexity and the small gain for CMU and Paragon, I’ve opted not to enable this optimization nor push further at this time. It requires more nuance to get it right but it is regardless worth revisiting at some point in the future. In particular, I want to wait until I have rewritten how constant tracks are identified. Nearly every animation compression implementation out there that detects constant tracks (ACL included) does so by using a local space error threshold. This means that it ignores the object space error that it contributes to. In turn, this sometimes causes undesirable artifacts in very exotic cases where a track needs to be animated below the threshold where it is detected to be constant. I plan to handle this more holistically by integrating it as part of the global optimization algorithm: a track will be constant for the clip only if it contributes an acceptable error in object space.</p>
<h2 id="packing-constant-samples">Packing constant samples</h2>
<p>Building on the previous range packing work, we can also use the same trick to quantize our constant track samples. Here however, 8 bits is too little so I quantized the constant rotation components to 16 bits. Instead of storing 3x float32 (12 bytes) for each constant rotation sample, we’ll be storing 4x uint16 (8 bytes): a 1.33x reduction in size.</p>
<p><em>Before results contain packed range values as described above.</em></p>
<table>
<thead>
<tr>
<th>CMU</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>70.08 MB</td>
<td>72.54 MB</td>
</tr>
<tr>
<td>Compressed size 50th percentile</td>
<td>15.20 KB</td>
<td>15.72 KB</td>
</tr>
<tr>
<td>Compression ratio</td>
<td>20.40 : 1</td>
<td>19.70 : 1</td>
</tr>
<tr>
<td>Max error</td>
<td>0.0741 cm</td>
<td>0.0734 cm</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0089 cm</td>
<td>0.0097 cm</td>
</tr>
<tr>
<td>Error threshold percentile rank (0.01 cm)</td>
<td>99.86 %</td>
<td>99.38 %</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Paragon</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>205.92 MB</td>
<td>213.17 MB</td>
</tr>
<tr>
<td>Compressed size 50th percentile</td>
<td>15.12 KB</td>
<td>15.43 KB</td>
</tr>
<tr>
<td>Compression ratio</td>
<td>20.77 : 1</td>
<td>20.06 : 1</td>
</tr>
<tr>
<td>Max error</td>
<td>3.5543 cm</td>
<td>5.8224 cm</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0111 cm</td>
<td>0.0344 cm</td>
</tr>
<tr>
<td>Error threshold percentile rank (0.01 cm)</td>
<td>98.89 %</td>
<td>96.84 %</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Fortnite</th>
<th>Before</th>
<th>After</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compressed size</td>
<td>500.95 MB</td>
<td>663.01 MB</td>
</tr>
<tr>
<td>Compressed size 50th percentile</td>
<td>9.65 KB</td>
<td>9.83 KB</td>
</tr>
<tr>
<td>Compression ratio</td>
<td>35.38 : 1</td>
<td>26.73 : 1</td>
</tr>
<tr>
<td>Max error</td>
<td>69.375 cm</td>
<td>5537580.0 cm</td>
</tr>
<tr>
<td>Track error 99th percentile</td>
<td>0.0319 cm</td>
<td>0.9272 cm</td>
</tr>
<tr>
<td>Error threshold percentile rank (0.01 cm)</td>
<td>97.62 %</td>
<td>88.53 %</td>
</tr>
</tbody>
</table>
<p>Even though our clip metadata size does reduce considerably, overall it yields a significant net loss. The reduced accuracy forces animated samples to retain more bits. It seems that lossless compression techniques might work better here although it would still be quite hard since each constant sample is disjoint: there is little redundancy to take advantage of.</p>
<p>With constant rotation tracks, the quaternion W component is dropped just like for animated samples. I also tried to retain the W component with full precision along with the other three. The idea being that if reducing accuracy increases the footprint, would increasing the accuracy reduce it? Sadly, it doesn’t. The memory footprint ended up being marginally higher. It seems like the sweet spot is to drop one of the quaternion components.</p>
<h2 id="is-there-any-hope-left">Is there any hope left?</h2>
<p>Although both of these optimizations turned out to be failures, I thought it best to document them here anyway. With each idea I try, whether it pans out or not I learn more about the domain and I grow wiser.</p>
<p>There still remains opportunities to optimize the clip metadata but they require a bit more engineering to test out. For one, many animation clips will have constant tracks in common. For example, if the same character is animated in many different ways over several animations, each of them might find that many sub-tracks are not animated. In particular, translation is rarely animated but very often constant as it often holds the bind pose. To better optimize these, animation clips must be compressed as a whole in a database of sorts. It gives us the opportunity to identity redundancies across many clips.</p>
<p>In a few weeks I’ll begin implementing animation streaming and to do so I’ll need to create such a database. This will open the door to these kind of optimizations. Stay tuned!</p>
<p><a href="/2016/10/21/anim_compression_toc/">Animation Compression Table of Contents</a></p>
Animation data in numbers2020-08-09T00:00:00+00:00http://nfrechette.github.io/2020/08/09/animation_data_numbers<p>For a while now, I’ve been meaning to take some time to instrument the <a href="https://github.com/nfrechette/acl">Animation Compression Library (aka ACL)</a> and look more in depth at the animations I have. Over the years I’ve had a lot of ideas on what could be improved and good data is critical in deciding where my time is best spent optimizing. I’m not sure if this will be of interest to many people out there but since I went ahead and did all that work, I might as well publish it!</p>
<h2 id="the-basics">The basics</h2>
<p>An animation clip is made up of a number of tracks each containing an equal number of transform or scalar samples (we use uniform sampling). Transform tracks are further broken down into sub-tracks for rotation, translation, and scale. Each sub-track is in one of three states:</p>
<ul>
<li>Animated: every sample is retained and quantized</li>
<li>Constant: a single sample is repeating and retained</li>
<li>Default: a single repeating sample equal to the sub-track identity, no sample retained</li>
</ul>
<p>Each sub-track type has its own identity. Rotations have the quaternion identity, translations have <code class="language-plaintext highlighter-rouge">0.0, 0.0, 0.0</code>, and scale tracks have <code class="language-plaintext highlighter-rouge">0.0, 0.0, 0.0</code> or <code class="language-plaintext highlighter-rouge">1.0, 1.0, 1.0</code> depending on whether the clip is additive or not and its additive type.</p>
<p>Being able to collapse constant and default tracks is <a href="/2016/11/03/anim_compression_constant_tracks/">an important optimization</a>. They are fairly common and allow us to considerably trim down on the number of samples we need to compress.</p>
<p>Sub-tracks that are animated end up having their <a href="/2016/11/09/anim_compression_range_reduction/">samples normalized</a> within the full range of values within the clip. This range information is later used to reconstruct our original sample.</p>
<p><em>Note that constant sample values and range values are currently stored with full float32 precision.</em></p>
<h2 id="the-compression-format">The compression format</h2>
<p>When an animation is compressed with ACL, it is first broken down into <a href="/2016/11/10/anim_compression_uniform_segmenting/">segments</a> of approximately 16 samples per track. As such, we end up with data that is needed regardless where we sample our clip and data that is needed only when we need a particular segment.</p>
<p>The memory layout roughly breaks down like this:</p>
<ul>
<li>Per clip metadata
<ul>
<li>Headers</li>
<li>Offsets</li>
<li>Track sub-type states</li>
<li>Per clip constant samples</li>
<li>Per clip animated sample ranges</li>
</ul>
</li>
<li>Our segments</li>
<li>Optional metadata</li>
</ul>
<p>Each segment ends up containing the following:</p>
<ul>
<li>Per segment metadata
<ul>
<li>Number of bits used per sub-track</li>
<li>Sub-track range information (we also normalize our samples per segment)</li>
</ul>
</li>
<li>The animated samples packed on a variable number of bits</li>
</ul>
<p>In order to figure out what to try and optimize next, I wanted to see where the memory goes within the above categories.</p>
<h2 id="the-datasets">The datasets</h2>
<p>I use three datasets for regression testing and for research and development:</p>
<ul>
<li><a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">Carnegie-Mellon University motion capture database</a></li>
<li><a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a></li>
<li><a href="/2019/01/25/compressing_fortnite_animations/">Fortnite</a></li>
</ul>
<p>We’ll focus more on Paragon and Fortnite since they are more representative and substantial but CMU is included regardless.</p>
<p><em>Special thanks to Epic for letting me use their animations for research purposes!</em></p>
<p>When calculating the raw size of an animation clip, I assume that each track is animated and that nothing is stripped. As such, the raw size can be calculated easily: <code class="language-plaintext highlighter-rouge">raw size = num bones * num samples * 10 * sizeof(float)</code>. Each sample is made up of 10 floats: 4x for the rotation, 3x for the translation, and 3x for the scale.</p>
<p>Here is how they look at a glance:</p>
<table>
<thead>
<tr>
<th> </th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Number of clips</td>
<td>2534</td>
<td>6558</td>
<td>8310</td>
</tr>
<tr>
<td>Raw size</td>
<td>1429.38 MB</td>
<td>4276.11 MB</td>
<td>17724.75 MB</td>
</tr>
<tr>
<td>Compressed size</td>
<td>71.00 MB</td>
<td>208.39 MB</td>
<td>483.54 MB</td>
</tr>
<tr>
<td>Compression ratio</td>
<td>20.13 : 1</td>
<td>20.52 : 1</td>
<td>36.66 : 1</td>
</tr>
<tr>
<td>Number of tracks</td>
<td>111496</td>
<td>816700</td>
<td>1559545</td>
</tr>
<tr>
<td>Number of sub-tracks</td>
<td>334488</td>
<td>2450100</td>
<td>4678635</td>
</tr>
</tbody>
</table>
<h3 id="sub-track-breakdown">Sub-track breakdown</h3>
<table>
<thead>
<tr>
<th> </th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Number of default sub-tracks</td>
<td>34.09%</td>
<td>37.26%</td>
<td>42.01%</td>
</tr>
<tr>
<td>Number of constant sub-tracks</td>
<td>52.29%</td>
<td>43.95%</td>
<td>50.33%</td>
</tr>
<tr>
<td>Number of animated sub-tracks</td>
<td>13.62%</td>
<td>18.78%</td>
<td>7.66%</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>CMU</th>
<th>Default</th>
<th>Constant</th>
<th>Animated</th>
</tr>
</thead>
<tbody>
<tr>
<td>Number of rotation sub-tracks</td>
<td>0.00%</td>
<td>64.41%</td>
<td>38.59%</td>
</tr>
<tr>
<td>Number of translation sub-tracks</td>
<td>2.27%</td>
<td>95.46%</td>
<td>2.27%</td>
</tr>
<tr>
<td>Number of scale sub-tracks</td>
<td>100.00%</td>
<td>0.00%</td>
<td>0.00%</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Paragon</th>
<th>Default</th>
<th>Constant</th>
<th>Animated</th>
</tr>
</thead>
<tbody>
<tr>
<td>Number of rotation sub-tracks</td>
<td>10.85%</td>
<td>47.69%</td>
<td>41.45%</td>
</tr>
<tr>
<td>Number of translation sub-tracks</td>
<td>3.32%</td>
<td>82.98%</td>
<td>13.70%</td>
</tr>
<tr>
<td>Number of scale sub-tracks</td>
<td>97.62%</td>
<td>1.19%</td>
<td>1.19%</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Fortnite</th>
<th>Default</th>
<th>Constant</th>
<th>Animated</th>
</tr>
</thead>
<tbody>
<tr>
<td>Number of rotation sub-tracks</td>
<td>21.15%</td>
<td>62.84%</td>
<td>16.01%</td>
</tr>
<tr>
<td>Number of translation sub-tracks</td>
<td>7.64%</td>
<td>86.78%</td>
<td>5.58%</td>
</tr>
<tr>
<td>Number of scale sub-tracks</td>
<td>97.23%</td>
<td>1.38%</td>
<td>1.39%</td>
</tr>
</tbody>
</table>
<p>Overall, across all three data sets, about half the tracks are constant. Translation tracks tend to be constant much more often. Most tracks aren’t animated but rotation tracks tend to be animated the most. Rotation tracks are 3x more likely to be animated than translation tracks and scale tracks are very rarely animated (~1.3% of the time). As such, segment data mostly contains animated rotation data.</p>
<h3 id="segment-breakdown">Segment breakdown</h3>
<table>
<thead>
<tr>
<th>Number of segments</th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total</td>
<td>51909</td>
<td>49213</td>
<td>121175</td>
</tr>
<tr>
<td>50th percentile</td>
<td>10</td>
<td>3</td>
<td>2</td>
</tr>
<tr>
<td>85th percentile</td>
<td>42</td>
<td>9</td>
<td>18</td>
</tr>
<tr>
<td>99th percentile</td>
<td>116</td>
<td>117</td>
<td>187</td>
</tr>
</tbody>
</table>
<p>Half the clips are very short and only contain 2 or 3 segments for Paragon and Fortnite. Those clips are likely to be 50 frames or less, or about 1-2 seconds at 30 FPS. For Paragon, 85% of the clips have 9 segments or less and in Fortnite we have 18 segments or less.</p>
<h3 id="compressed-size-breakdown">Compressed size breakdown</h3>
<table>
<thead>
<tr>
<th>Compressed size</th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total</td>
<td>71.00 MB</td>
<td>208.39 MB</td>
<td>483.54 MB</td>
</tr>
<tr>
<td>50th percentile</td>
<td>15.42 KB</td>
<td>15.85 KB</td>
<td>9.71 KB</td>
</tr>
<tr>
<td>85th percentile</td>
<td>56.36 KB</td>
<td>48.18 KB</td>
<td>72.22 KB</td>
</tr>
<tr>
<td>99th percentile</td>
<td>153.68 KB</td>
<td>354.54 KB</td>
<td>592.13 KB</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Clip metadata size</th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total size</td>
<td>4.22 MB (5.94%)</td>
<td>24.38 MB (11.70%)</td>
<td>38.32 MB (7.92%)</td>
</tr>
<tr>
<td>50th percentile</td>
<td>9.73%</td>
<td>22.03%</td>
<td>46.06%</td>
</tr>
<tr>
<td>85th percentile</td>
<td>18.68%</td>
<td>46.06%</td>
<td>97.43%</td>
</tr>
<tr>
<td>99th percentile</td>
<td>37.38%</td>
<td>98.48%</td>
<td>98.64%</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Segment metadata size</th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total size</td>
<td>6.23 MB (8.78%)</td>
<td>22.61 MB (10.85%)</td>
<td>54.21 MB (11.21%)</td>
</tr>
<tr>
<td>50th percentile</td>
<td>8.07%</td>
<td>6.88%</td>
<td>0.75%</td>
</tr>
<tr>
<td>85th percentile</td>
<td>9.28%</td>
<td>11.37%</td>
<td>10.95%</td>
</tr>
<tr>
<td>99th percentile</td>
<td>10.59%</td>
<td>21.00%</td>
<td>26.21%</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Segment animated data size</th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total size</td>
<td>60.44 MB (85.13%)</td>
<td>160.98 MB (77.25%)</td>
<td>390.15 MB (80.69%)</td>
</tr>
<tr>
<td>50th percentile</td>
<td>81.92%</td>
<td>70.55%</td>
<td>48.22%</td>
</tr>
<tr>
<td>85th percentile</td>
<td>87.08%</td>
<td>79.62%</td>
<td>81.65%</td>
</tr>
<tr>
<td>99th percentile</td>
<td>88.55%</td>
<td>87.93%</td>
<td>89.25%</td>
</tr>
</tbody>
</table>
<p>From this data, we can conclude that our efforts might be best spent optimizing the clip metadata where the constant track data and the animated range data will contribute much more relative to the overall footprint. Short clips have less animated data but just as much metadata as longer clips with an equal number of bones. Even though overall the clip metadata doesn’t contribute much, for the vast majority of clips it does contribute a significant amount (for half the Fortnite clips, clip metadata represented 46.06% or more of the total clip size).</p>
<p>The numbers are somewhat skewed by the fact that a few clips are <em>very</em> long. Their animated footprint ends up dominating the overall numbers hence why breaking things down by percentile is insightful here.</p>
<p>The clip metadata isn’t as optimized and it contains more low hanging fruits to attack. I’ve spent most of my time focusing on the segment animated data but as a result, pushing things further on that front is much harder and requires more work and a higher risk that what I try won’t pan out.</p>
<h3 id="animated-data-breakdown">Animated data breakdown</h3>
<table>
<thead>
<tr>
<th> </th>
<th>CMU</th>
<th>Paragon</th>
<th>Fortnite</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total compressed size</td>
<td>71.00 MB</td>
<td>208.39 MB</td>
<td>483.54 MB</td>
</tr>
<tr>
<td>Animated data size</td>
<td>60.44 MB (85.13%)</td>
<td>160.98 MB (77.25%)</td>
<td>390.15 MB (80.69%)</td>
</tr>
<tr>
<td>70% of animated data removed</td>
<td>28.69 MB (2.47x smaller)</td>
<td>95.70 MB (2.18x smaller)</td>
<td>210.44 MB (2.30x smaller)</td>
</tr>
<tr>
<td>50% of animated data removed</td>
<td>40.78 MB (1.74x smaller)</td>
<td>127.90 MB (1.63x smaller)</td>
<td>288.47 MB (1.68x smaller)</td>
</tr>
<tr>
<td>25% of animated data removed</td>
<td>55.89 MB (1.27x smaller)</td>
<td>168.15 MB (1.24x smaller)</td>
<td>386.00 MB (1.25x smaller)</td>
</tr>
</tbody>
</table>
<p>A common optimization is to strip a number of frames from the animation data (aka sub-sampling or frame stripping). This is very destructive but can yield good memory savings. Since we know how much animated data we have and its relative footprint, we can compute ballpark numbers for how much smaller removing 70%, 50%, or 25% of our animated data might be. The numbers above represent the total compressed size after stripping and the reduction factor.</p>
<p>In my next post, I’ll explore the results of <a href="/2020/08/11/clip_metadata_packing/">quantizing the constant sample values and the clip range values</a>, stay tuned!</p>
<p><a href="/2016/10/21/anim_compression_toc/">Animation Compression Table of Contents</a></p>
Zero overhead backward compatibility2020-07-18T00:00:00+00:00http://nfrechette.github.io/2020/07/18/zero_overhead_backward_compat<p>The <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> finally <a href="https://github.com/nfrechette/acl/pull/307">supports backward compatibility</a> (going forward once 2.0 comes out). I’m really happy with how it turned out so I thought I would write a bit about how the ACL decompression is designed.</p>
<h2 id="the-api">The API</h2>
<p>At runtime, decompressing animation data could not be easier:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">acl</span><span class="o">::</span><span class="n">decompression_context</span><span class="o"><</span><span class="n">custom_decompression_settings</span><span class="o">></span> <span class="n">context</span><span class="p">;</span>
<span class="n">context</span><span class="p">.</span><span class="n">initialize</span><span class="p">(</span><span class="n">compressed_data</span><span class="p">);</span>
<span class="c1">// Seek 1.0 second into our compressed animation</span>
<span class="c1">// and don't use rounding to interpolate</span>
<span class="n">context</span><span class="p">.</span><span class="n">seek</span><span class="p">(</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="n">acl</span><span class="o">::</span><span class="n">sample_rounding_policy</span><span class="o">::</span><span class="n">none</span><span class="p">);</span>
<span class="n">custom_writer</span> <span class="nf">writer</span><span class="p">(</span><span class="n">output_data</span><span class="p">);</span>
<span class="n">context</span><span class="p">.</span><span class="n">decompress_tracks</span><span class="p">(</span><span class="n">writer</span><span class="p">);</span>
</code></pre></div></div>
<p>A small context object is created and bound to our compressed data. Its construction is cheap enough that it can be done on the stack on demand. It can then subsequently be used (and re-used) to seek and decompress.</p>
<p>A key design goal is to have as little overhead as possible: pay only for what you use. This is achieved through templating in two ways:</p>
<ul>
<li>The <code class="language-plaintext highlighter-rouge">custom_decompression_settings</code> argument on the context object controls what features are enabled or disabled.</li>
<li>The <code class="language-plaintext highlighter-rouge">custom_writer</code> wraps whatever container you might be using in your game engine to represent the animation data. This is to make sure no extra copying is required.</li>
</ul>
<p>The decompression settings are where the magic happens.</p>
<h2 id="compile-time-user-control">Compile time user control</h2>
<p>There are many game engines out there each handling animation in their own specific way. In order to be able to integrate as seamlessly as possible, ACL exposes a small struct that can be overriden to control its behavior. By leveraging <code class="language-plaintext highlighter-rouge">constexpr</code> and templating, features that aren’t used or needed can be removed entirely at compile time to ensure zero cost at runtime.</p>
<p><a href="https://github.com/nfrechette/acl/blob/8f1849adbf52c6f4af7aa3ba88e59c93f57404b6/includes/acl/decompression/decompress.h#L64">Here is how it looks</a>:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="nc">decompression_settings</span>
<span class="p">{</span>
<span class="c1">// Whether or not to clamp the sample time when `seek(..)`</span>
<span class="c1">// is called. Defaults to true.</span>
<span class="k">static</span> <span class="k">constexpr</span> <span class="kt">bool</span> <span class="n">clamp_sample_time</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="nb">true</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// Whether or not the specified track type is supported.</span>
<span class="c1">// Defaults to true.</span>
<span class="c1">// If a track type is statically known not to be supported,</span>
<span class="c1">// the compiler can strip the associated code.</span>
<span class="k">static</span> <span class="k">constexpr</span> <span class="kt">bool</span> <span class="n">is_track_type_supported</span><span class="p">(</span><span class="n">track_type8</span> <span class="cm">/*type*/</span><span class="p">)</span>
<span class="p">{</span> <span class="k">return</span> <span class="nb">true</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// Other stuff ...</span>
<span class="c1">// Which version we should optimize for.</span>
<span class="c1">// If 'any' is specified, the decompression context will</span>
<span class="c1">// support every single version with full backwards</span>
<span class="c1">// compatibility.</span>
<span class="c1">// Using a specific version allows the compiler to</span>
<span class="c1">// statically strip code for all other versions.</span>
<span class="c1">// This allows the creation of context objects specialized</span>
<span class="c1">// for specific versions which yields optimal performance.</span>
<span class="k">static</span> <span class="k">constexpr</span> <span class="n">compressed_tracks_version16</span> <span class="n">version_supported</span><span class="p">()</span>
<span class="p">{</span> <span class="k">return</span> <span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">any</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// Whether the specified rotation/translation/scale format</span>
<span class="c1">// are supported or not.</span>
<span class="c1">// Use this to strip code related to formats you do not need.</span>
<span class="k">static</span> <span class="k">constexpr</span> <span class="kt">bool</span> <span class="n">is_rotation_format_supported</span><span class="p">(</span><span class="n">rotation_format8</span> <span class="cm">/*format*/</span><span class="p">)</span>
<span class="p">{</span> <span class="k">return</span> <span class="nb">true</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// Other stuff ...</span>
<span class="c1">// Whether rotations should be normalized before being</span>
<span class="c1">// output or not. Some animation runtimes will normalize</span>
<span class="c1">// in a separate step and do not need the explicit</span>
<span class="c1">// normalization.</span>
<span class="c1">// Enabled by default for safety.</span>
<span class="k">static</span> <span class="k">constexpr</span> <span class="kt">bool</span> <span class="n">normalize_rotations</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="nb">true</span><span class="p">;</span> <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Extending this is simple and clean:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="nc">default_transform_decompression_settings</span> <span class="o">:</span> <span class="k">public</span> <span class="n">decompression_settings</span>
<span class="p">{</span>
<span class="c1">// Only support transform tracks</span>
<span class="k">static</span> <span class="k">constexpr</span> <span class="kt">bool</span> <span class="n">is_track_type_supported</span><span class="p">(</span><span class="n">track_type8</span> <span class="n">type</span><span class="p">)</span>
<span class="p">{</span> <span class="k">return</span> <span class="n">type</span> <span class="o">==</span> <span class="n">track_type8</span><span class="o">::</span><span class="n">qvvf</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// By default, we only support the variable bit rates as</span>
<span class="c1">// they are generally optimal</span>
<span class="k">static</span> <span class="k">constexpr</span> <span class="kt">bool</span> <span class="n">is_rotation_format_supported</span><span class="p">(</span><span class="n">rotation_format8</span> <span class="n">format</span><span class="p">)</span>
<span class="p">{</span> <span class="k">return</span> <span class="n">format</span> <span class="o">==</span> <span class="n">rotation_format8</span><span class="o">::</span><span class="n">quatf_drop_w_variable</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// Other stuff ...</span>
<span class="p">};</span>
</code></pre></div></div>
<p>A new struct is created to inherit from the desired decompression settings and specific functions are defined to hide the base implementations, thus replacing them.</p>
<p>By templating the <code class="language-plaintext highlighter-rouge">decompression_context</code> with the settings structure, it can be used to determine everything needed at compile time:</p>
<ul>
<li>Which memory representation is needed depending on whether we are decompressing scalar or transform tracks.</li>
<li>Which algorithm version to support and optimize for.</li>
<li>Which features to strip when they aren’t needed.</li>
</ul>
<p>This is much nicer than the common C way to use <code class="language-plaintext highlighter-rouge">#define</code> macros. By using a template argument, multiple setting objects can easily be created (with type safety) and used within the same application or file.</p>
<h2 id="backward-compatibility">Backward compatibility</h2>
<p>By using the <code class="language-plaintext highlighter-rouge">decompression_settings</code>, we can specify which version we optimize for. If no specific version is provided (the default behavior), we will branch and handle all supported versions. However, if a specific version is provided, we can strip the code for all other versions removing any runtime overhead. This is clean and simple thanks to <a href="https://github.com/nfrechette/acl/blob/develop/includes/acl/decompression/impl/decompression_version_selector.h">templates</a>.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">template</span><span class="o"><</span><span class="n">compressed_tracks_version16</span> <span class="n">version</span><span class="p">></span>
<span class="k">struct</span> <span class="nc">decompression_version_selector</span> <span class="p">{};</span>
<span class="c1">// Specialize for ACL 2.0's format</span>
<span class="k">template</span><span class="o"><</span><span class="p">></span> <span class="k">struct</span>
<span class="nc">decompression_version_selector</span><span class="o"><</span><span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">v02_00_00</span><span class="o">></span>
<span class="p">{</span>
<span class="k">static</span> <span class="kt">bool</span> <span class="n">is_version_supported</span><span class="p">(</span><span class="n">compressed_tracks_version16</span> <span class="n">version</span><span class="p">)</span>
<span class="p">{</span> <span class="k">return</span> <span class="n">version</span> <span class="o">==</span> <span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">v02_00_00</span><span class="p">;</span> <span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">class</span> <span class="nc">decompression_settings_type</span><span class="p">,</span> <span class="k">class</span> <span class="nc">context_type</span><span class="p">></span>
<span class="n">ACL_FORCE_INLINE</span> <span class="k">static</span> <span class="kt">bool</span> <span class="n">initialize</span><span class="p">(</span><span class="n">context_type</span><span class="o">&</span> <span class="n">context</span><span class="p">,</span> <span class="k">const</span> <span class="n">compressed_tracks</span><span class="o">&</span> <span class="n">tracks</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">acl_impl</span><span class="o">::</span><span class="n">initialize_v0</span><span class="o"><</span><span class="n">decompression_settings_type</span><span class="o">></span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">tracks</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// Other stuff ...</span>
<span class="p">};</span>
<span class="c1">// Specialize to support all versions</span>
<span class="k">template</span><span class="o"><</span><span class="p">></span> <span class="k">struct</span>
<span class="nc">decompression_version_selector</span><span class="o"><</span><span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">any</span><span class="o">></span>
<span class="p">{</span>
<span class="k">static</span> <span class="kt">bool</span> <span class="n">is_version_supported</span><span class="p">(</span><span class="n">compressed_tracks_version16</span> <span class="n">version</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">version</span> <span class="o">>=</span> <span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">first</span> <span class="o">&&</span> <span class="n">version</span> <span class="o"><=</span> <span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">latest</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">class</span> <span class="nc">decompression_settings_type</span><span class="p">,</span> <span class="k">class</span> <span class="nc">context_type</span><span class="p">></span>
<span class="k">static</span> <span class="kt">bool</span> <span class="n">initialize</span><span class="p">(</span><span class="n">context_type</span><span class="o">&</span> <span class="n">context</span><span class="p">,</span> <span class="k">const</span> <span class="n">compressed_tracks</span><span class="o">&</span> <span class="n">tracks</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// TODO: Note that the `any` decompression can be optimized further to avoid a complex switch on every call.</span>
<span class="k">const</span> <span class="n">compressed_tracks_version16</span> <span class="n">version</span> <span class="o">=</span> <span class="n">tracks</span><span class="p">.</span><span class="n">get_version</span><span class="p">();</span>
<span class="k">switch</span> <span class="p">(</span><span class="n">version</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">case</span> <span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">v02_00_00</span><span class="p">:</span>
<span class="k">return</span> <span class="n">decompression_version_selector</span><span class="o"><</span><span class="n">compressed_tracks_version16</span><span class="o">::</span><span class="n">v02_00_00</span><span class="o">>::</span><span class="n">initialize</span><span class="o"><</span><span class="n">decompression_settings_type</span><span class="o">></span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">tracks</span><span class="p">);</span>
<span class="nl">default:</span>
<span class="n">ACL_ASSERT</span><span class="p">(</span><span class="nb">false</span><span class="p">,</span> <span class="s">"Unsupported version"</span><span class="p">);</span>
<span class="k">return</span> <span class="nb">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// Other stuff ...</span>
<span class="p">};</span>
</code></pre></div></div>
<p>This is ideal for many game engines. For example <em>Unreal Engine 4</em> always compresses locally and caches the result in its <em>Derived Data Cache</em>. This means that the compressed format is always the latest one used by the plugin. As such, UE4 only needs to support a single version and it can do so without any overhead.</p>
<p>Other game engines might choose to support the latest two versions, emiting a warning to recompress old animations while still being able to support them with very little overhead: a single branch to pick which context to use.</p>
<p>More general applications might opt to support every version (e.g. a glTF viewer).</p>
<p><em>Note that backward compatibility will only be supported for official releases as the develop branch is constantly subject to change.</em></p>
<h2 id="conclusion">Conclusion</h2>
<p>This C++ customization pattern is clean and simple to use and it allows a compact API with a rich feature set. It was present in a slightly different form ever since ACL 0.1 and the more I use it, the more I love it.</p>
<p>In fact, in my opinion the Animation Compression Library and <a href="https://github.com/nfrechette/rtm">Realtime Math</a> contain some of the best code (quality wise) that I’ve ever written in my career. Free from time or budget constraints, I can carefully craft each facet to the best of my ability.</p>
<p>ACL 2.0 continues to progress nicely. It is still missing a few features but it is already an amazing step up from 1.3.</p>
Realtime Math 2.0 is out, cleaner, and faster!2020-06-28T00:00:00+00:00http://nfrechette.github.io/2020/06/28/rtm_v2.0.0<p>A lot of work went into this latest release and here is the gist of what changed:</p>
<ul>
<li>Added support for GCC10, clang8, clang9, clang10, VS 2019 clang, and emscripten</li>
<li>Added a lot of matrix math</li>
<li>Added trigonometric functions (scalar and vector)</li>
<li>Angle types have been removed</li>
<li>Lots of optimizations and improvements</li>
<li>Tons of cleanup to ensure a consistent API</li>
</ul>
<p>It should now contain everything needed by most realtime applications. The one critical feature that is missing at the moment, is a proper browsable documentation. While every function is currently documented, the lack of a web browsable documentation makes using it a bit more difficult than I’d like. Hopefully I can remedy this in the coming months.</p>
<h2 id="migrating-from-1x">Migrating from 1.x</h2>
<p>Most of the APIs haven’t changed materially and simply recompiling should work depending on what you use. Where compilation fails, a few minor fixes might be required. There are two main reasons why this release is a major one:</p>
<ul>
<li>The <code class="language-plaintext highlighter-rouge">anglef</code> and <code class="language-plaintext highlighter-rouge">angled</code> types have been removed</li>
<li>Extensive usage of return type overloading</li>
</ul>
<p>The angle types have been removed because I could not manage to come up with a clean API for angular constants that would work well without introducing a LOT more complexity while remaining optimal for code generation. Angular constants (and constants in general) are used with all sorts of code. In particular, SIMD code (such as SSE2 or NEON) often ends up needing to use them and I wanted to be able to efficiently do so. As such they are now simple typedefs for floating point types and can easily be used with ordinary scalar or SIMD code. The <a href="https://github.com/nfrechette/rtm/blob/develop/includes/rtm/constants.h">pattern used for constants</a> is inspired from Boost.</p>
<p>I had originally introduced them in hope of providing added type safety but the constants weren’t really usable in RTM 1.1. For now, it is easier to document that all angles are represented as radians. The typedef remains to clarify the API.</p>
<h3 id="return-type-overloading">Return type overloading</h3>
<p>C++ doesn’t really have return type overloading but it can be faked. It looks like this in action:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">vector4f</span> <span class="n">vec1</span> <span class="o">=</span> <span class="n">vector_set</span><span class="p">(</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">2.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">3.0</span><span class="n">f</span><span class="p">);</span>
<span class="n">vector4f</span> <span class="n">vec2</span> <span class="o">=</span> <span class="n">vector_set</span><span class="p">(</span><span class="mf">5.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">6.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">7.0</span><span class="n">f</span><span class="p">);</span>
<span class="n">scalarf</span> <span class="n">dot_sse2</span> <span class="o">=</span> <span class="n">vector_dot3</span><span class="p">(</span><span class="n">vec1</span><span class="p">,</span> <span class="n">vec2</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">dot_x87</span> <span class="o">=</span> <span class="n">vector_dot3</span><span class="p">(</span><span class="n">vec1</span><span class="p">,</span> <span class="n">vec2</span><span class="p">);</span>
<span class="n">vector4f</span> <span class="n">dot_broadcast</span> <span class="o">=</span> <span class="n">vector_dot3</span><span class="p">(</span><span class="n">vec1</span><span class="p">,</span> <span class="n">vec2</span><span class="p">);</span>
</code></pre></div></div>
<p>Usage is very clean and the compiler can figure out what to do fairly easily in most cases. The implementation behind the scene is a bit complicated but it is worth it for the flexibility it provides:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// A few things omited for brevity</span>
<span class="k">struct</span> <span class="nc">dot3_helper</span>
<span class="p">{</span>
<span class="kr">inline</span> <span class="k">operator</span> <span class="kt">float</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">do_the_math_here1</span><span class="p">();</span>
<span class="p">}</span>
<span class="kr">inline</span> <span class="k">operator</span> <span class="n">scalarf</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">do_the_math_here2</span><span class="p">();</span>
<span class="p">}</span>
<span class="kr">inline</span> <span class="k">operator</span> <span class="n">vector4f</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">do_the_math_here3</span><span class="p">();</span>
<span class="p">}</span>
<span class="n">vector4f</span> <span class="n">left_hand_side</span><span class="p">;</span>
<span class="n">vector4f</span> <span class="n">right_hand_side</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">constexpr</span> <span class="n">dot3_helper</span> <span class="nf">vector_dot3</span><span class="p">(</span><span class="n">vector4f</span> <span class="n">left_hand_side</span><span class="p">,</span> <span class="n">vector4f</span> <span class="n">right_hand_side</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">dot3_helper</span><span class="p">{</span> <span class="n">left_hand_side</span><span class="p">,</span> <span class="n">right_hand_side</span> <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<p>One motivating reason for this pattern is that very often we perform some operation and return a scalar value. Depending on the architecture, it might be optimal to return it as a SIMD register type instead of a regular <code class="language-plaintext highlighter-rouge">float</code> as those do not always mix well. ARM NEON doesn’t suffer from this issue and for that platform, <code class="language-plaintext highlighter-rouge">scalarf</code> is a typedef for <code class="language-plaintext highlighter-rouge">float</code>. But for x86 with SSE2 and for some PowerPC processors, this distinction is very important in order to achieve optimal performance. It doesn’t stop there though, even when floating point arithmetic uses the same registers as SIMD arithmetic (such as x64 with SSE2), there is sometimes a benefit to having a different type in order to improve code generation. VS2019 still struggles today to avoid extra shuffles when ordinary scalar and SIMD code are mixed. The type distinction allows for improved performance.</p>
<p>This pattern was present from day one inside RTM but it wasn’t as widely used. Usage of <code class="language-plaintext highlighter-rouge">scalarf</code> wasn’t as widespread. The latest release pushes its usage much further and as such a lot of code was modified to support both return types. This can sometime lead to ambiguous function calls (and those will need fixing in user code) but it is fairly rare in practice. It forces the programmer to be explicit about what types are used which is in line with RTM’s philosophy.</p>
<h2 id="quaternion-math-improvements">Quaternion math improvements</h2>
<p>The <a href="https://github.com/nfrechette/acl">Animation Compression Library (ACL)</a> heavily relies on quaternions and as such I spend a good deal of time trying to optimize them. This release introduces the important <code class="language-plaintext highlighter-rouge">quat_slerp</code> function as well as many optimizations for ARM processors.</p>
<h3 id="arm-neon-performance-can-be-surprising">ARM NEON performance can be surprising</h3>
<p>RTM supports both ARMv7 and ARM64 and very often what is optimal for one isn’t optimal for the other. Worse, different devices disagree about what code is optimal, sometimes by quite a bit.</p>
<p>I spent a good deal of time trying to optimize two functions: quaternion multiplication and rotating a 3D vector with a quaternion. Rotating a 3D vector uses two quaternion multiplications.</p>
<p>For quaternion multiplication, I tried a few variations:</p>
<ul>
<li><a href="https://github.com/nfrechette/rtm/blob/bc5e3fa1b00fd1b99c62120281c8b5505237a907/tools/bench/sources/bench_quat_mul.cpp#L185">Swizzling + floating point multiplication to flip the right signs</a></li>
<li><a href="https://github.com/nfrechette/rtm/blob/bc5e3fa1b00fd1b99c62120281c8b5505237a907/tools/bench/sources/bench_quat_mul.cpp#L221">Swizzling + XOR to flip the right signs (what SSE2 uses)</a></li>
<li><a href="https://github.com/nfrechette/rtm/blob/bc5e3fa1b00fd1b99c62120281c8b5505237a907/tools/bench/sources/bench_quat_mul.cpp#L253">Swizzling + floating point negation</a></li>
<li><a href="https://github.com/nfrechette/rtm/blob/bc5e3fa1b00fd1b99c62120281c8b5505237a907/tools/bench/sources/bench_quat_mul.cpp#L31">A scalar implementation with no SIMD</a></li>
</ul>
<p>The first two implementations are inspired from the classic SSE2 implementation. This is the same code used by <em>DirectX Math</em> on SSE2 and ARM as well.</p>
<p>The third implementation is a bit more clever. Instead of using constants that must be loaded and applied in order to align our signs to leverage fused-multiply-add, we use the floating point negation instruction. This is done once and mixed in with the various swizzle instructions that NEON supports. This ends up being extremely compact and uses only 12 instructions with ARM64!</p>
<p>I measured extensively using micro benchmarks (with <em>Google Benchmark</em>) as well as within ACL. <a href="https://github.com/nfrechette/rtm/issues/61">The results</a> turned out to be quite interesting.</p>
<p>On a Pixel 3 android phone, with ARMv7 the scalar version was fastest. It beat the multiplication variant by 1.05x and the negation variant by 1.10x. However, with ARM64, the negation variant was best. It beat the multiplication variant by 1.05x and the scalar variant by 1.16x.</p>
<p>On a Samsung S8 android phone, the results were similar: scalar wins with ARMv7 and negation wins with ARM64 (both by a significant margin again).</p>
<p>On an iPad Pro with ARM64 the results agreed again with the negation variant being fastest.</p>
<p>I hadn’t seen that particular variant used anywhere else so I was quite pleased to see it perform so well with ARM64. In light of these results, RTM now uses the scalar version with ARMv7 and the negation version with ARM64.</p>
<p>Since rotating a 3D vector with a quaternion is two quaternion multiplications back-to-back, I set out to use the same tricks as above with one addition.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">vector4f</span> <span class="nf">quat_mul_vector3</span><span class="p">(</span><span class="n">vector4f</span> <span class="n">vector</span><span class="p">,</span> <span class="n">quatf</span> <span class="n">rotation</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">quatf</span> <span class="n">vector_quat</span> <span class="o">=</span> <span class="n">quat_set_w</span><span class="p">(</span><span class="n">vector_to_quat</span><span class="p">(</span><span class="n">vector</span><span class="p">),</span> <span class="mf">0.0</span><span class="n">f</span><span class="p">);</span>
<span class="n">quatf</span> <span class="n">inv_rotation</span> <span class="o">=</span> <span class="n">quat_conjugate</span><span class="p">(</span><span class="n">rotation</span><span class="p">);</span>
<span class="k">return</span> <span class="n">quat_to_vector</span><span class="p">(</span><span class="n">quat_mul</span><span class="p">(</span><span class="n">quat_mul</span><span class="p">(</span><span class="n">inv_rotation</span><span class="p">,</span> <span class="n">vector_quat</span><span class="p">),</span> <span class="n">rotation</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We first extend our <code class="language-plaintext highlighter-rouge">vector3</code> into a proper <code class="language-plaintext highlighter-rouge">vector4</code> by padding it with <code class="language-plaintext highlighter-rouge">0.0</code>. Using this information, we can strip out a few operations from the first quaternion multiplication.</p>
<p>Again, I tested all four variants and surprisingly, the scalar variant won out every time both with ARMv7 and ARM64 on both android devices. The iPad saw the negation variant as fastest. Code generation was identical yet it seems that the iPad CPU has very different performance characteristics. As a compromise, the scalar variant is used with all ARM flavors. It isn’t optimal on the iPad but it remains much better than the reference implementation.</p>
<p>I suspect that the scalar implementation performs better because more operations are independent. Despite having way more instructions, there must be fewer stalls and this leads to an overall win. It is possible that this information can be better leveraged to further improve things but that is a problem for another day.</p>
<h2 id="compiler-bugs-bonanza">Compiler bugs bonanza</h2>
<p>Realtime Math appears to put considerable stress on compilers and often ends up breaking them. In the first 10 years of my career, I found maybe 2-3 C++ compiler bugs. Here are just some of the bugs I remember from the past year:</p>
<ul>
<li><a href="https://github.com/nfrechette/rtm/issues/37">clang7 and clang8 hang, not fixed</a></li>
<li><a href="https://github.com/nfrechette/rtm/issues/35">VS2019 code generation bug in scalar_sqrt_reciprocal, fixed</a></li>
<li><a href="https://github.com/nfrechette/rtm/issues/34">VS2019 code generation bug in mask_set, fixed</a></li>
<li><a href="https://travis-ci.org/github/nfrechette/rtm/jobs/695261257">clang5 crashes, not fixed</a></li>
<li><a href="https://github.com/nfrechette/rtm/commit/b0ae7fbb20a1bbd38a36b73a86a4c68c7efaa94d">GCC doesn’t generate the right assembly with x86 and SSE2, not fixed</a></li>
<li><a href="https://developercommunity.visualstudio.com/content/problem/940424/sub-optimal-xmm-register-allocation-with-msvc.html">VS2019 sometimes spills registers on the stack when it isn’t necessary, not fixed</a></li>
<li><a href="https://developercommunity.visualstudio.com/content/problem/1055298/vs-2019-sometimes-incorrectly-reads-return-value-f.html">VS2019 sometimes incorrectly reads the return value of a function that uses __vectorcall, not fixed</a></li>
<li><a href="https://developercommunity.visualstudio.com/content/problem/1076955/vs2019-vs2017-vs2015-crash-when-mm-set-epi64x-is-u.html">VS2015, 2017, and 2019 crash when _mm_set_epi64x is used in debug with x86, not fixed</a></li>
</ul>
<p>And those are just the ones I remember from the top of my head. I also found one or two with ACL not in the above list. Some of those will never get fixed because the compiler versions are too old but thankfully the Microsoft Visual Studio team has been very quick to address some of the above issues.</p>
<iframe src="https://giphy.com/embed/oSGgdCmx0igJa" width="480" height="360" frameborder="0" class="giphy-embed" allowfullscreen=""></iframe>
Keep an eye out for buffer security checks2020-06-26T00:00:00+00:00http://nfrechette.github.io/2020/06/26/buffer_security_checks<p>By default, when compiling C++, Visual Studio enables the <code class="language-plaintext highlighter-rouge">/GS</code> flag for <a href="https://docs.microsoft.com/en-us/cpp/build/reference/gs-buffer-security-check?view=vs-2019">buffer security checks</a>.</p>
<p>In functions that the compiler deems vulnerable to the stack getting overwritten, it adds buffer security checks. To detect if the stack has been tampered with during execution of the function, it first writes a sentinel value past the end of the reserved space the function needs. This sentinel value is random per process to avoid an attacker guessing its value. Just before the function exits, it calls a small function that validates the sentinel value: <code class="language-plaintext highlighter-rouge">__security_check_cookie()</code>.</p>
<p>The rules on what can trigger this are as follow (from the MSDN documentation):</p>
<ul>
<li>The function contains an array on the stack that is larger than 4 bytes, has more than two elements, and has an element type that is not a pointer type.</li>
<li>The function contains a data structure on the stack whose size is more than 8 bytes and contains no pointers.</li>
<li>The function contains a buffer allocated by using the <code class="language-plaintext highlighter-rouge">_alloca</code> function.</li>
<li>The function contains a data structure that contains another which triggers one of the above checks.</li>
</ul>
<p>This is all fine and well and <strong>you should never disable it program/file wide</strong>. But you should keep an eye out. In very performance critical code, this small overhead can have a sizable impact. I’ve observed this over the years a few times but it now popped up somewhere I didn’t expect it: my math library <a href="https://github.com/nfrechette/rtm">Realtime Math (RTM)</a>.</p>
<h2 id="non-zero-cost-abstractions">Non-zero cost abstractions</h2>
<p>The SSE2 headers define a few types. Of interest to us today is <code class="language-plaintext highlighter-rouge">__m128</code> but others suffer from the same issue as well (including wider types such as <code class="language-plaintext highlighter-rouge">__m256</code>). Those define a register wide value suitable for SIMD intrinsics: <code class="language-plaintext highlighter-rouge">__m128</code> contains four 32 bit floats. As such, it takes up 16 bytes.</p>
<p>Because it is considered a native type by the compiler, it does not trigger buffer security checks to be inserted despite being larger than 8 bytes without containing pointers.</p>
<p>However, the same is not true if you wrap it in a struct or class.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="nc">scalarf</span>
<span class="p">{</span>
<span class="n">__m128</span> <span class="n">value</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div></div>
<p>The above struct might trigger security checks to be inserted: it is a struct larger than 8 bytes that does not contain pointers.</p>
<p>Similarly, many other common math types suffer from this:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="nc">matrix3x3f</span>
<span class="p">{</span>
<span class="n">__m128</span> <span class="n">x_axis</span><span class="p">;</span>
<span class="n">__m128</span> <span class="n">y_axis</span><span class="p">;</span>
<span class="n">__m128</span> <span class="n">z_axis</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Many years ago, in a discussion about an unrelated compiler bug, someone at Microsoft mentioned to me that it is best to typedef SIMD types than it is to wrap them in a concrete type; it should lead to better code generation. They didn’t offer any insights as to why that might be (and I didn’t ask) and honestly I had never noticed any difference until security buffer checks came into play, <em>last Friday</em>. Their math library <em>DirectX Math</em> uses a typedef for its vector type and so does RTM everywhere it can. But sometimes it can’t be avoided.</p>
<p>RTM also extensively uses a helper struct pattern to help keep the code clean and flexible. Some code such as a vector dot product returns a scalar value. But on some architectures, it isn’t desirable to treat it as a <code class="language-plaintext highlighter-rouge">float</code> for performance reasons (PowerPC, x86, etc). For example, with x86 float arithmetic does not use SSE2 unless you explicitly use intrinsics for it: by default it uses the x87 floating point stack (with MSVC at least). If this value is later needed as part of more SSE2 vector code (such as vector normalization), the value will be calculated from two SSE2 vectors, be stored on the x87 float stack, only to be passed back to SSE2. To avoid this roundtrip when using SSE2 with x86, RTM exposes the <code class="language-plaintext highlighter-rouge">scalarf</code> type. However, sometimes you really need the value as a float. The usage dictates what you need. To support both variants with as little syntactic overhead as possible, RTM leverages return type overloading (it’s not really a thing in C++ but it can be faked with implicit coercion). It makes the following possible:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">vector4f</span> <span class="n">vec1</span> <span class="o">=</span> <span class="n">vector_set</span><span class="p">(</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">2.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">3.0</span><span class="n">f</span><span class="p">);</span>
<span class="n">vector4f</span> <span class="n">vec2</span> <span class="o">=</span> <span class="n">vector_set</span><span class="p">(</span><span class="mf">5.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">6.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">7.0</span><span class="n">f</span><span class="p">);</span>
<span class="n">scalarf</span> <span class="n">dot_sse2</span> <span class="o">=</span> <span class="n">vector_dot3</span><span class="p">(</span><span class="n">vec1</span><span class="p">,</span> <span class="n">vec2</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">dot_x87</span> <span class="o">=</span> <span class="n">vector_dot3</span><span class="p">(</span><span class="n">vec1</span><span class="p">,</span> <span class="n">vec2</span><span class="p">);</span>
<span class="n">vector4f</span> <span class="n">dot_broadcast</span> <span class="o">=</span> <span class="n">vector_dot3</span><span class="p">(</span><span class="n">vec1</span><span class="p">,</span> <span class="n">vec2</span><span class="p">);</span>
</code></pre></div></div>
<p>This is very clean to use and the compiler can figure out what code to call easily. But it is ugly to implement; a small price to pay for readability.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// A few things omited for brevity</span>
<span class="k">struct</span> <span class="nc">dot3_helper</span>
<span class="p">{</span>
<span class="kr">inline</span> <span class="k">operator</span> <span class="kt">float</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">do_the_math_here1</span><span class="p">();</span>
<span class="p">}</span>
<span class="kr">inline</span> <span class="k">operator</span> <span class="n">scalarf</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">do_the_math_here2</span><span class="p">();</span>
<span class="p">}</span>
<span class="kr">inline</span> <span class="k">operator</span> <span class="n">vector4f</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">do_the_math_here3</span><span class="p">();</span>
<span class="p">}</span>
<span class="n">vector4f</span> <span class="n">left_hand_side</span><span class="p">;</span>
<span class="n">vector4f</span> <span class="n">right_hand_side</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">constexpr</span> <span class="n">dot3_helper</span> <span class="nf">vector_dot3</span><span class="p">(</span><span class="n">vector4f</span> <span class="n">left_hand_side</span><span class="p">,</span> <span class="n">vector4f</span> <span class="n">right_hand_side</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">dot3_helper</span><span class="p">{</span> <span class="n">left_hand_side</span><span class="p">,</span> <span class="n">right_hand_side</span> <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<blockquote>
<p>Side note, on ARM <code class="language-plaintext highlighter-rouge">scalarf</code> is a typedef for <code class="language-plaintext highlighter-rouge">float</code> in order to achieve optimal performance.</p>
</blockquote>
<p>There is a lot of boilerplate but the code is very simple and either <em>constexpr</em> or marked <em>inline</em>. We create a small struct and return it, and at the call site the compiler invokes the right implicit coercion operator. It works just fine and the compiler optimizes everything away to yield the same lean assembly you would expect. We leverage inlining to create what should be a zero cost abstraction. Except, in rare cases (at least with Visual Studio as recent as 2019), inlining fails and everything goes wrong.</p>
<blockquote>
<p>Side note, the above is one example why <code class="language-plaintext highlighter-rouge">scalarf</code> cannot be a typedef because we need it distinct from <code class="language-plaintext highlighter-rouge">vector4f</code> both of which are represented in SSE2 as a <code class="language-plaintext highlighter-rouge">__m128</code>. To avoid this issue, <code class="language-plaintext highlighter-rouge">vector4f</code> is a typedef while <code class="language-plaintext highlighter-rouge">scalarf</code> is a wrapping struct.</p>
</blockquote>
<p>Many math libraries out there wrap SIMD types in a proper type and use similar patterns. And while generally most math functions are small and get inlined fine, it isn’t always the case. In particular, these security buffer checks can harm the ability of the compiler to inline while at the same time degrading performance of perfectly safe code.</p>
<h2 id="8-instructions-is-too-much">8 instructions is too much</h2>
<p>All of this worked well until I noticed, out of the blue, a performance regression in the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> when I updated to the latest RTM version. This was very strange as it should have only contained performance optimizations.</p>
<p><a href="https://github.com/nfrechette/acl/blob/1cd782149e59c4d682791f4cc23b104a51627c16/includes/acl/compression/skeleton_error_metric.h#L342">Here</a> is where the code generation changed:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// A few things omited for brevity</span>
<span class="kr">__declspec</span><span class="p">(</span><span class="n">safebuffers</span><span class="p">)</span> <span class="n">rtm</span><span class="o">::</span><span class="n">scalarf</span> <span class="nf">calculate_error_no_scale</span><span class="p">(</span><span class="k">const</span> <span class="n">calculate_error_args</span><span class="o">&</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvvf</span><span class="o">&</span> <span class="n">raw_transform_</span> <span class="o">=</span> <span class="o">*</span><span class="k">static_cast</span><span class="o"><</span><span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvvf</span><span class="o">*></span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">transform0</span><span class="p">);</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvvf</span><span class="o">&</span> <span class="n">lossy_transform_</span> <span class="o">=</span> <span class="o">*</span><span class="k">static_cast</span><span class="o"><</span><span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvvf</span><span class="o">*></span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">transform1</span><span class="p">);</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector4f</span> <span class="n">vtx0</span> <span class="o">=</span> <span class="n">args</span><span class="p">.</span><span class="n">shell_point_x</span><span class="p">;</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector4f</span> <span class="n">vtx1</span> <span class="o">=</span> <span class="n">args</span><span class="p">.</span><span class="n">shell_point_y</span><span class="p">;</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector4f</span> <span class="n">raw_vtx0</span> <span class="o">=</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvv_mul_point3_no_scale</span><span class="p">(</span><span class="n">vtx0</span><span class="p">,</span> <span class="n">raw_transform_</span><span class="p">);</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector4f</span> <span class="n">raw_vtx1</span> <span class="o">=</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvv_mul_point3_no_scale</span><span class="p">(</span><span class="n">vtx1</span><span class="p">,</span> <span class="n">raw_transform_</span><span class="p">);</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector4f</span> <span class="n">lossy_vtx0</span> <span class="o">=</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvv_mul_point3_no_scale</span><span class="p">(</span><span class="n">vtx0</span><span class="p">,</span> <span class="n">lossy_transform_</span><span class="p">);</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector4f</span> <span class="n">lossy_vtx1</span> <span class="o">=</span> <span class="n">rtm</span><span class="o">::</span><span class="n">qvv_mul_point3_no_scale</span><span class="p">(</span><span class="n">vtx1</span><span class="p">,</span> <span class="n">lossy_transform_</span><span class="p">);</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">scalarf</span> <span class="n">vtx0_error</span> <span class="o">=</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector_distance3</span><span class="p">(</span><span class="n">raw_vtx0</span><span class="p">,</span> <span class="n">lossy_vtx0</span><span class="p">);</span>
<span class="k">const</span> <span class="n">rtm</span><span class="o">::</span><span class="n">scalarf</span> <span class="n">vtx1_error</span> <span class="o">=</span> <span class="n">rtm</span><span class="o">::</span><span class="n">vector_distance3</span><span class="p">(</span><span class="n">raw_vtx1</span><span class="p">,</span> <span class="n">lossy_vtx1</span><span class="p">);</span>
<span class="k">return</span> <span class="n">rtm</span><span class="o">::</span><span class="n">scalar_max</span><span class="p">(</span><span class="n">vtx0_error</span><span class="p">,</span> <span class="n">vtx1_error</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<blockquote>
<p>Side note, as part of a prior effort to optimize that performance critical function, I had already disabled buffer security checks which are unnecessary here.</p>
</blockquote>
<p>The above code is fairly simple. We take two 3D vertices in local space and transform them to world space. We do this for our source data (which is raw) and for our compressed data (which is lossy). We calculate the 3D distance between the raw and lossy vertices and it yields our compression error. We take the maximum value as our final result.</p>
<p><code class="language-plaintext highlighter-rouge">qvv_mul_point3_no_scale</code> is <a href="https://github.com/nfrechette/rtm/blob/bc5e3fa1b00fd1b99c62120281c8b5505237a907/includes/rtm/qvvf.h#L117">fairly heavy</a> instruction wise and it doesn’t get fully inlined. Some of it does but the <code class="language-plaintext highlighter-rouge">quat_mul_vector3</code> it contains does not inline.</p>
<p>By the time the compiler gets to the <code class="language-plaintext highlighter-rouge">vector_distance3</code> calls, the compiler struggles. Both VS2017 and VS2019 fail to inline 8 instructions (the <code class="language-plaintext highlighter-rouge">vector_length3</code> <a href="https://github.com/nfrechette/rtm/blob/bc5e3fa1b00fd1b99c62120281c8b5505237a907/includes/rtm/vector4f.h#L1368">it contains</a>).</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// const rtm::scalarf vtx0_error = rtm::vector_distance3(raw_vtx0, lossy_vtx0);</span>
<span class="n">vsubps</span> <span class="n">xmm1</span><span class="p">,</span><span class="n">xmm0</span><span class="p">,</span><span class="n">xmm6</span>
<span class="n">vmovups</span> <span class="n">xmmword</span> <span class="n">ptr</span> <span class="p">[</span><span class="n">rsp</span><span class="o">+</span><span class="mx">20h</span><span class="p">],</span><span class="n">xmm1</span>
<span class="n">lea</span> <span class="n">rcx</span><span class="p">,[</span><span class="n">rsp</span><span class="o">+</span><span class="mx">20h</span><span class="p">]</span>
<span class="n">vzeroupper</span>
<span class="n">call</span> <span class="n">rtm</span><span class="o">::</span><span class="n">rtm_impl</span><span class="o">::</span><span class="n">vector4f_vector_length3</span><span class="o">::</span><span class="k">operator</span> <span class="n">rtm</span><span class="o">::</span><span class="n">scalarf</span>
</code></pre></div></div>
<blockquote>
<p>Side note, when AVX is enabled, Visual Studio often ends up attempting to use wider registers when they aren’t needed, causing the addition of <code class="language-plaintext highlighter-rouge">vzeroupper</code> and other artifacts that can degrade performance.</p>
</blockquote>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// inline operator scalarf() const</span>
<span class="c1">// {</span>
<span class="n">sub</span> <span class="n">rsp</span><span class="p">,</span><span class="mx">18h</span>
<span class="n">mov</span> <span class="n">rax</span><span class="p">,</span><span class="n">qword</span> <span class="n">ptr</span> <span class="p">[</span><span class="n">__security_cookie</span><span class="p">]</span>
<span class="n">xor</span> <span class="n">rax</span><span class="p">,</span><span class="n">rsp</span>
<span class="n">mov</span> <span class="n">qword</span> <span class="n">ptr</span> <span class="p">[</span><span class="n">rsp</span><span class="p">],</span><span class="n">rax</span>
<span class="c1">// const scalarf len_sq = vector_length_squared3(input);</span>
<span class="n">vmovups</span> <span class="n">xmm0</span><span class="p">,</span><span class="n">xmmword</span> <span class="n">ptr</span> <span class="p">[</span><span class="n">rcx</span><span class="p">]</span>
<span class="n">vmulps</span> <span class="n">xmm2</span><span class="p">,</span><span class="n">xmm0</span><span class="p">,</span><span class="n">xmm0</span>
<span class="n">vshufps</span> <span class="n">xmm1</span><span class="p">,</span><span class="n">xmm2</span><span class="p">,</span><span class="n">xmm2</span><span class="p">,</span><span class="mi">1</span>
<span class="n">vaddss</span> <span class="n">xmm0</span><span class="p">,</span><span class="n">xmm2</span><span class="p">,</span><span class="n">xmm1</span>
<span class="n">vshufps</span> <span class="n">xmm2</span><span class="p">,</span><span class="n">xmm2</span><span class="p">,</span><span class="n">xmm2</span><span class="p">,</span><span class="mi">2</span>
<span class="n">vaddss</span> <span class="n">xmm0</span><span class="p">,</span><span class="n">xmm0</span><span class="p">,</span><span class="n">xmm2</span>
<span class="c1">// return scalar_sqrt(len_sq);</span>
<span class="n">vsqrtss</span> <span class="n">xmm0</span><span class="p">,</span><span class="n">xmm0</span><span class="p">,</span><span class="n">xmm0</span>
<span class="c1">// }</span>
<span class="n">mov</span> <span class="n">rcx</span><span class="p">,</span><span class="n">qword</span> <span class="n">ptr</span> <span class="p">[</span><span class="n">rsp</span><span class="p">]</span>
<span class="n">xor</span> <span class="n">rcx</span><span class="p">,</span><span class="n">rsp</span>
<span class="n">call</span> <span class="n">__security_check_cookie</span>
<span class="n">add</span> <span class="n">rsp</span><span class="p">,</span><span class="mx">18h</span>
<span class="n">ret</span>
</code></pre></div></div>
<p>And that is where everything goes wrong. Those 8 instructions calculate the 3D dot product and the square root required to get the length of the vector between our two points. They balloon up to 16 instructions because of the security check overhead. Behind the scenes, VS doesn’t fail to inline 8 instructions, it fails to inline something larger than we see when everything goes right.</p>
<p>This was quite surprising to me, because until now I had never seen this behavior kick in this way. Indeed, try as I might, I could not reproduce this behavior in a small playground (yet).</p>
<p>In light of this, and because the math code within Realtime Math is very simple, I have decided to annotate every function to explicitly disable buffer security checks. Not every function requires this but it is easier to be consistent for maintenance. This restores the optimal code generation and <code class="language-plaintext highlighter-rouge">vector_distance3</code> finally inlines properly.</p>
<p>I am now wondering if I should explicitly force inline short functions…</p>
ACL right in your browser2020-05-28T00:00:00+00:00http://nfrechette.github.io/2020/05/28/acl_js_v0.1.0<p>This week, the first beta release of the <a href="https://github.com/nfrechette/acl-js/releases/tag/v0.1.0">Animation Compression Library JavaScript</a> module has been released. You can find it on NPM <a href="https://www.npmjs.com/package/@nfrechette/acl">here</a> and you can try it live in your browser right <a href="https://nfrechette.github.io/acl_viewer">here</a> by drag and dropping glTF files.</p>
<p><em>The library should be usable but keep in mind until ACL reaches version 2.0 with backwards compatibility you might have to recompress when you upgrade to future versions.</em></p>
<p>The module uses the powerful <a href="https://emscripten.org">emscripten</a> compiler toolchain to take C++ and compile it into WebAssembly. That means that as part of this effort, <a href="https://github.com/nfrechette/acl">ACL</a> and <a href="https://github.com/nfrechette/rtm">Realtime Math</a> have been upgraded to support emscripten as well. Note that WASM SIMD isn’t supported yet (contributions welcome).</p>
<p>Both compression and decompression are supported although if you enable dead code stripping in your JavaScript bundler, you should be able to only pay for what you use.</p>
<p><em>Special thanks to <a href="https://zeux.io">Arseny Kapoulkine</a> and his excellent <a href="https://github.com/zeux/meshoptimizer">meshoptimizer</a> library. Using his blog posts and his code as a guide, I was able to get up and running fairly quickly.</em></p>
<h2 id="next-steps">Next steps</h2>
<p>Progress continues towards ACL 2.0 but I will take a few weeks to finish up RTM 2.0 first. It is now used extensively by the main ACL development branch. A few new features will be introduced, some cleanup remains, and I want to double check some optimizations before releasing it.</p>
Morph target animation compression2020-05-04T00:00:00+00:00http://nfrechette.github.io/2020/05/04/morph_target_compresion<p>The <a href="https://github.com/nfrechette/acl/releases/tag/v1.3.0">Animation Compression Library v1.3</a> introduced support for compressing animated floating point tracks but it wasn’t until now that I managed to get around to trying it inside the <a href="https://github.com/nfrechette/acl-ue4-plugin">Unreal Engine 4 Plugin</a>. Scalar tracks are used in various ways from animating the <a href="https://en.wikipedia.org/wiki/Field_of_view_in_video_games">Field Of View</a> to animating <a href="https://en.wikipedia.org/wiki/Inverse_kinematics">Inverse Kinematic</a> blend targets. While they have many practical uses in modern video games, they are most commonly used to animate <a href="https://en.wikipedia.org/wiki/Morph_target_animation">morph targets</a> (aka blend shapes) in facial and cloth animations.</p>
<p><em>TL;DR: By using the morph target deformation information, we can better control how much precision each blend weight needs while yielding a lower memory footprint.</em></p>
<h2 id="how-do-morph-targets-work">How do morph targets work?</h2>
<p>Morph targets are essentially a set of meshes that get blended together per vertex. It is a way to achieve vertex based animation. Let’s use a simple triangle as an example:</p>
<p><img src="/public/morph_target_concept.jpg" alt="Morph Targets Explained" /></p>
<p>In blue we have our reference mesh and in green our morph target. To animate our vertices, we apply a scaled displacement to every vertex. This scale factor is called the blend weight. Typically it lies between <strong>0.0</strong> (our reference mesh) and <strong>1.0</strong> (our target mesh). Values in between end up being a linear combination of both:</p>
<p><code class="language-plaintext highlighter-rouge">final vertex = reference vertex + (target vertex - reference vertex) * blend weight</code></p>
<p>Each morph target is thus controlled by a single blend weight and each vertex can end up having a contribution from multiple targets. With our blend weights being independent, we can compress them as a set of floating point curves. This is typically done by specifying a desired level of precision to retain on each curve. However, in practice this can be hard to control: blend weights have no units.</p>
<p>At the time of writing:</p>
<ul>
<li>Unreal Engine 4 leaves its curves in a compact raw form (a spline) by default but they can be optionally compressed with various codecs.</li>
<li>Unity, Lumberyard, and CRYENGINE do not document how they are stored and they do not appear to expose a way to further compress them (as far as I know).</li>
</ul>
<p>All of this animated data does add up and it can benefit from being compressed. In order to achieve this, the UE4 ACL plugin uses an intuitive technique to control the resulting quality.</p>
<h2 id="compressing-blend-weights">Compressing blend weights</h2>
<p>Our animated blend weights ultimately yield a mesh deformation. Can we use that information to our advantage? Let’s take a closer look at the math behind morph targets. Here is how each vertex is transformed:</p>
<p><code class="language-plaintext highlighter-rouge">final vertex = reference vertex + (target vertex - reference vertex) * blend weight</code></p>
<p>As we saw earlier, <code class="language-plaintext highlighter-rouge">(target vertex - reference vertex)</code> is the displacement delta to apply to achieve our deformation. Let’s simplify our equation a bit:</p>
<p><code class="language-plaintext highlighter-rouge">final vertex = reference vertex + vertex delta * blend weight</code></p>
<p>Let’s plug in some displacement numbers with some units and see what happens.</p>
<table>
<thead>
<tr>
<th>Reference Position</th>
<th>Target Position</th>
<th>Delta</th>
</tr>
</thead>
<tbody>
<tr>
<td>5 cm</td>
<td>5 cm</td>
<td>0 cm</td>
</tr>
<tr>
<td>5 cm</td>
<td>5.1 cm</td>
<td>0.1 cm</td>
</tr>
<tr>
<td>5 cm</td>
<td>50 cm</td>
<td>45 cm</td>
</tr>
</tbody>
</table>
<p>Let us assume that at a particular moment in time, our raw blend weight is <strong>0.2</strong>. Due to compression, that value will change by the introduction of a small amount of imprecision. Let’s see what happens if our lossy blend weight is <strong>0.22</strong> instead.</p>
<table>
<thead>
<tr>
<th>Delta</th>
<th>Delta * Raw Blend Weight</th>
<th>Delta * Lossy Blend Weight</th>
<th>Error</th>
</tr>
</thead>
<tbody>
<tr>
<td>0 cm</td>
<td>0 cm</td>
<td>0 cm</td>
<td>0 cm</td>
</tr>
<tr>
<td>0.1 cm</td>
<td>0.02 cm</td>
<td>0.022 cm</td>
<td>0.002 cm</td>
</tr>
<tr>
<td>45 cm</td>
<td>9 cm</td>
<td>9.9 cm</td>
<td>0.9 cm</td>
</tr>
</tbody>
</table>
<p>From this, we can observe some important facts:</p>
<ul>
<li>When our vertex doesn’t move and has no displacement delta, the error introduced doesn’t matter.</li>
<li>The error introduced is linearly proportional to the displacement delta: when the error is fixed, a small delta will have a small amount of error while a large delta will have a large amount of error.</li>
</ul>
<p>This means that it is not suitable to set a single precision value (in blend weight space) for all our blend weights. Some morph targets require more precision than others and ideally we would like to take advantage of that fact.</p>
<p>Specifying the amount of precision that a blend weight should have isn’t easy if we don’t know how much deformation it ultimately drives. A better way to specify how much precision we need is by specifying it in meaningful units. If we assume that we want our vertices to have a precision of <strong>0.01 cm</strong> instead, how much precision do their blend weights need?</p>
<p><code class="language-plaintext highlighter-rouge">blend weight precision = vertex precision / vertex displacement delta</code></p>
<p><em>Notice how the <code class="language-plaintext highlighter-rouge">vertex precision</code> and <code class="language-plaintext highlighter-rouge">vertex displacement delta</code> units cancel out.</em></p>
<table>
<thead>
<tr>
<th>Delta</th>
<th>Blend Weight Precision</th>
</tr>
</thead>
<tbody>
<tr>
<td>0 cm</td>
<td>Division by zero!</td>
</tr>
<tr>
<td>0.1 cm</td>
<td>0.1</td>
</tr>
<tr>
<td>45 cm</td>
<td>0.00022</td>
</tr>
</tbody>
</table>
<p>When a vertex doesn’t move, it doesn’t matter what precision our blend weight retains since the final vertex position will be identical regardless. However, smaller displacements require less precision while larger displacements require more precision retained.</p>
<p>This is what the ACL plugin does: when a <em>Skeletal Mesh</em> is provided to the compression codec, ACL will use a vertex displacement precision value instead of a generic scalar track precision value. This is intuitive to tune for an artist because the units now have meaning: we can easily find out how much <strong>0.01 cm</strong> represents within the context of our 3D model. Underneath the hood, this translates into unique precision requirements tailored to each morph target blend weight curve. This allows ACL to attain an even lower memory footprint while providing a strict guarantee on the visual fidelity of the animated deformations.</p>
<p><img src="/public/ue4_curve_compression_codec.jpg" alt="Curve Compression Example" /></p>
<h2 id="a-boy-and-his-kite">A Boy and His Kite</h2>
<p>In order to test this out, I used the <em>GDC 2015</em> demo from <em>Epic</em>: <a href="https://www.youtube.com/watch?v=JNgsbNvkNjE"><em>A Boy and His Kite</em></a>. The demo is available for free on the <em>Unreal Marketplace</em> under the <em>Learning</em> section.</p>
<p>It shows a boy running through a landscape along with his kite. He contains <strong>811</strong> animated curves within each of the <strong>31</strong> animation sequences that comprise the full cinematic. <strong>692</strong> of those curves end up driving morph targets for his facial and clothing deformations. Some of the shots have the camera very close to his face and as such retaining as much quality as possible is critical.</p>
<p>I decided to compare <strong>5</strong> codecs:</p>
<ul>
<li>Compressed Rich Curves with an error threshold of <strong>0.0</strong> (default within UE 4.25)</li>
<li>Compressed Rich Curves with an error threshold of <strong>0.001</strong></li>
<li>Uniform Sampling</li>
<li>ACL with a generic precision of <strong>0.001</strong></li>
<li>ACL with a generic precision of <strong>0.001</strong> and a morph target deformation precision of <strong>0.01 cm</strong></li>
</ul>
<p>The <em>Compressed Rich Curves</em> and <em>Uniform Sampling</em> codecs are built into UE4 and provide a trade-off between memory and speed. The rich curves will tend to have a lower memory footprint but evaluating them at runtime is slower than when uniform sampling is used.</p>
<p>ACL uses uniform sampling internally for very fast evaluation but it is also much more aggressive with its compression.</p>
<table>
<thead>
<tr>
<th> </th>
<th>Compressed Size</th>
<th>Compression Rate</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed Rich Curves 0.0</strong></td>
<td>3620.66 KB</td>
<td>1.0x (baseline)</td>
</tr>
<tr>
<td><strong>Compressed Rich Curves 0.001</strong></td>
<td>1458.77 KB</td>
<td>2.5x smaller</td>
</tr>
<tr>
<td><strong>Uniform Sampling</strong></td>
<td>2052.82 KB</td>
<td>1.8x smaller</td>
</tr>
<tr>
<td><strong>ACL 0.001</strong></td>
<td>540.24 KB</td>
<td>6.7x smaller</td>
</tr>
<tr>
<td><strong>ACL with morph 0.01 cm</strong></td>
<td>381.10 KB</td>
<td><strong>9.5x smaller</strong></td>
</tr>
</tbody>
</table>
<p>I manually inspected the visual fidelity of each codec and everything looked flawless. By leveraging our knowledge of the individual morph target deformations, ACL manages to reduce the memory footprint by an extra <strong>159.14 KB (29%)</strong>.</p>
<p>The new <a href="https://github.com/nfrechette/acl-ue4-plugin/issues/38">curve compression</a> will be available shortly in the ACL plugin v0.6 (suitable for UE 4.24) as well as v1.0 (suitable for UE 4.25 and later). The ACL plugin v1.0 isn’t out yet but it will come to the <em>Unreal Marketplace</em> as soon as UE 4.25 is released.</p>
ACL 1.3 is out: smaller, better, faster than ever2019-11-18T00:00:00+00:00http://nfrechette.github.io/2019/11/18/acl_v1.3.0<p>After 7 months of work, the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> has finally reached <a href="https://github.com/nfrechette/acl/releases/tag/v1.3.0">v1.3</a> along with an updated <a href="https://github.com/nfrechette/acl-ue4-plugin/releases/tag/v0.5.0">v0.5 Unreal Engine 4 plugin</a>. Notable changes in this release include:</p>
<ul>
<li>Added support for VS2019, GCC 9, clang7, and Xcode 11</li>
<li>Optimized compression and decompression significantly</li>
<li>Added support for multiple root bones</li>
<li>Added support for scalar track compression</li>
</ul>
<p>Compared to <strong>UE 4.23.1</strong>, the ACL plugin compresses up to <strong>2.9x smaller</strong>, is up to <strong>4.7x more accurate</strong>, up to <strong>52.9x faster to compress</strong>, and up to <strong>6.8x faster to decompress</strong> (results may vary depending on the platform and data).</p>
<p>This latest release is a bit more accurate than the previous one and it also reduces the memory footprint by about 4%. Numbers vary a bit but decompression is roughly 1.5x faster on every platform.</p>
<h2 id="realtime-compression">Realtime compression</h2>
<p>The compression speed improvements are massive. Compared to the previous release, it is <strong>over 2.6x faster</strong>!</p>
<table>
<thead>
<tr>
<th> </th>
<th>ACL v1.3</th>
<th>ACL v1.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">CMU</a></td>
<td>2m 22.3s (10285.52 KB/sec)</td>
<td>6m 9.71s (3958.99 KB/sec)</td>
</tr>
<tr>
<td><a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a></td>
<td>10m 23.05s (7027.87 KB/sec)</td>
<td>28m 56.48s (2521.62 KB/sec)</td>
</tr>
<tr>
<td><a href="https://github.com/nfrechette/acl/blob/develop/docs/fight_scene_performance.md">Matinee fight scene</a></td>
<td>4.89s (13074.59 KB/sec)</td>
<td>20.27s (3150.43 KB/sec)</td>
</tr>
</tbody>
</table>
<p>It is so fast that when I tried it in Unreal Engine 4 and the popup dialog showed instantaneously after compression, I thought something was wrong. I got curious and decided to take a look at how fast it is for individual clips. The results were quite telling. In the Carnegie-Mellon University motion capture database, 50% of the clips compress in less than 29ms, 85% take less than 114ms, and 99% compress in 313ms or less. In Paragon, 50% of the clips compress in less than 31ms, 85% take less than 128ms, and 99% compress in 1.461 seconds or less. Half of ordinary animations compress fast enough to do so in realtime!</p>
<p>I had originally planned more improvements to the compression speed as part of this release but ultimately opted to stop there for now. I tried to switch from <em>Arrays of Structures</em> to <em>Structures of Arrays</em> and while it was faster, the added complexity was not worth the very minimal gain. There remains lots of room for improvement though and the next release should be even faster.</p>
<h1 id="whats-next">What’s next</h1>
<p>The next release will be a major one: <a href="https://github.com/nfrechette/acl/milestone/7">v2.0</a> is scheduled around <strong>Summer 2020</strong>. A number of significant changes and additions are planned.</p>
<p>While <a href="https://github.com/nfrechette/rtm">Realtime Math</a> was integrated this release for the new scalar track compression API, the next release will see it replace <em>all</em> of the math done in ACL. Preliminary results show that it will speed up compression by about 10%. This will reduce significantly the maintenance burden and speed up CI builds. This will constitute a fairly minor API break. RTM is already included within every release of ACL (through a sub-module). Integrations will simply need to add the required include path as well as change the few places that interface with the library.</p>
<p>The error metric functions will also change a bit to allow further optimization opportunities. This will constitute a minor API break as well <em>if</em> integrations have implemented their own error metric (which is unlikely).</p>
<p>So far, ACL is what might be considered a <em>runtime</em> compression format. It was designed to change with every release and as such requires recompression whenever a new version comes out. This isn’t a very big burden as more often than not, game engines already recompress animations on demand, often storing raw animations in source control. Starting with the next release, backwards compatibility will be introduced. In order to do this, the format will be standardized, modernized, and documented. The decompression code path will remain optimal through templating. This step is necessary to allow ACL to be suitable for long term storage. As part of this effort, a <a href="https://github.com/nfrechette/acl-gltf">glTF extention</a> will be created as well as tools to pack and unpack glTF files.</p>
<p>Last but not least, a new <a href="https://github.com/nfrechette/acl-js">javascript/web assembly</a> module will be created in order to support ACL on the modern web. Through the glTF extension and a ThreeJS integration, it will become a first class citizen in your browser.</p>
<p>ACL is already in use on millions of consoles and mobile devices today and this next major release will make its adoption easier than ever.</p>
<p>If you use ACL and would like to help prioritize the work I do, feel free to reach out and provide feedback or requests!</p>
<p>Thanks to <a href="https://github.com/sponsors">GitHub Sponsors</a>, you can <a href="https://github.com/sponsors/nfrechette">sponsor me</a>! All funds donated will go towards purchasing new devices to optimize for as well as other related costs (like coffee). The best way to ensure that ACL continues to move forward is to sponsor me for specific feature work, custom integrations, through GitHub, or some other arrangement.</p>
Realtime Math: faster than ever2019-11-14T00:00:00+00:00http://nfrechette.github.io/2019/11/14/realtime_math_11<p>Today <a href="https://github.com/nfrechette/rtm">Realtime Math</a> has finally reached <a href="https://github.com/nfrechette/rtm/releases/tag/v1.1.0">v1.1.0</a>! This release brings a lot of good things:</p>
<ul>
<li>Added support for Windows ARM64</li>
<li>Added support for VS2019, GCC9, clang7, and Xcode 11</li>
<li>Added support for Intel FMA and ARM64 FMA</li>
<li>Many optimizations, minor fixes, and cleanup</li>
</ul>
<p>I spent a great deal of time optimizing the quaternion arithmetic for NEON and SSE and as a result, many functions are now among the fastest out there. In order to make sure not to introduce regressions, the <a href="https://github.com/google/benchmark">Google Benchmark</a> library has been integrated and allows me to quickly whip up tests to try various ideas and variants.</p>
<p>RTM will be used in the upcoming <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> release for the new scalar track compression API. The subsequent ACL release will remove all of its internal math code and switch everything to RTM. Preliminary tests show that it speeds up its compression by about 10%.</p>
<h2 id="is-intel-fma-worth-it">Is Intel FMA worth it?</h2>
<p>As part of this release, <a href="https://github.com/nfrechette/rtm/issues/21">support for AVX2 and FMA was added</a>. Seeing how libraries like <a href="https://github.com/Microsoft/DirectXMath">DirectX Math</a> already use FMA, I expected it to give a measurable performance win. However, on my Haswell MacBook Pro and my Ryzen 2950X desktop, it turned out to be significantly slower. As a result of these findings, I opted to not use FMA (although the relevant defines are present and handled). If you are aware of a CPU where FMA is faster, please reach out!</p>
<p>It is also worth noting that typically when <em>Fast Math</em> type compiler optimizations are enabled, FMA instructions are often automatically generated when it detects a pattern where they can be used. RTM explicitly disables this behavior with Visual Studio (by forcing <em>Precise Math</em> with a pragma) and as such even when it is enabled, the functions retain the SSE/AVX instructions that showed the best performance.</p>
<h2 id="arm-neon-performance-notes">ARM NEON performance notes</h2>
<p>RTM, DirectX Math, and many other libraries make extensive use NEON SIMD intrinsics. However, while measuring various implementation variants for quaternion multiplication I noticed that using simple scalar math is considerably faster on both ARMv7 and ARM64 on my Pixel 3 phone and my iPad. Going forward I will make sure to always measure the scalar code path as a baseline.</p>
<h2 id="compiler-bugs">Compiler bugs</h2>
<p>Surprisingly, RTM triggered 3 compiler code generation bugs this year. <a href="https://github.com/nfrechette/rtm/issues/34">#34</a> and <a href="https://github.com/nfrechette/rtm/issues/35">#35</a> in VS2019 as well as <a href="https://github.com/nfrechette/rtm/issues/37">#37</a> in clang7. As soon as those are fixed and validated by continuous integration, I will release a new version (minor or patch).</p>
<h2 id="whats-next">What’s next</h2>
<p>The development of RTM is largely driven by my work with ACL. If you’d like to see specific things in the next release, feel free to reach out or to create GitHub issues and I will prioritize them accordingly. As always, contributions welcome!</p>
<p>Thanks to <a href="https://github.com/sponsors">GitHub Sponsors</a>, you can <a href="https://github.com/sponsors/nfrechette">sponsor me</a>! All funds donated will go towards purchasing new devices to optimize for as well as other related costs (like coffee).</p>
Faster floating point arithmetic with Exclusive OR2019-10-22T00:00:00+00:00http://nfrechette.github.io/2019/10/22/float_xor_optimization<p>Today it’s time to talk about <a href="/2019/05/08/sign_flip_optimization/">another floating point arithmetic trick</a> that sometimes can come in very handy with SSE2. This trick isn’t novel, and I don’t often get to use it but a few days ago inspiration struck me late at night in the middle of a long 3 hour drive. The results inspired this post.</p>
<p>I’ll cover three functions that use it for <a href="https://en.wikipedia.org/wiki/Quaternion">quaternion</a> arithmetic which have already been merged into the <a href="https://github.com/nfrechette/rtm">Realtime Math (RTM)</a> library as well as the <a href="https://github.com/nfrechette/acl">Animation Compression Library (ACL)</a>. ACL uses quaternions heavily and I’m always looking for ways to make them faster.</p>
<p><em>TL;DR: With SSE2, XOR (and other logical operators) can be leveraged to speed up common floating point operations.</em></p>
<h1 id="xor-the-lesser-known-logical-operator">XOR: the lesser known logical operator</h1>
<p>Most programmers are familiar with <code class="language-plaintext highlighter-rouge">AND</code>, <code class="language-plaintext highlighter-rouge">OR</code>, and <code class="language-plaintext highlighter-rouge">NOT</code> logical operators. They form the bread and butter of everyday programming. And while we all learn about their cousin <code class="language-plaintext highlighter-rouge">XOR</code>, it doesn’t come in handy anywhere near as often. Here is a quick recap of what it does.</p>
<table>
<thead>
<tr>
<th>A</th>
<th>B</th>
<th>A XOR B</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>0</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
<tr>
<td>1</td>
<td>1</td>
<td>0</td>
</tr>
</tbody>
</table>
<p>We can infer a few interesting properties from it:</p>
<ul>
<li>Any input that we apply XOR to with zero yields the input</li>
<li>XOR can be used to flip a bit from <code class="language-plaintext highlighter-rouge">true</code> to <code class="language-plaintext highlighter-rouge">false</code> or vice versa by XOR-ing it with one</li>
<li>Using XOR when both inputs are identical yields zero (every zero bit remains zero and every one bit flips to zero)</li>
</ul>
<p>Exclusive OR comes in handy mostly with bit twiddling hacks when squeezing every cycle counts. While it is commonly used with integer inputs, it can also be used with floating point values!</p>
<h1 id="xor-with-floats">XOR with floats</h1>
<p>SSE2 contains support for XOR (and other logical operators) on both integral (<code class="language-plaintext highlighter-rouge">_mm_xor_si128</code>) and floating point values (<code class="language-plaintext highlighter-rouge">_mm_xor_ps</code>). Usually when you transition from the integral domain to the floating point domain of operations on a register (or vice versa), the CPU will incur a 1 cycle penalty. By implementing a different instruction for both domains, this hiccup can be avoided. Logical operations can often execute on more than one execution port (even on older hardware) which can enable multiple instructions to dispatch in the same cycle.</p>
<p>The question then becomes, when does it make sense to use them?</p>
<h1 id="quaternion-conjugate">Quaternion conjugate</h1>
<p>For a quaternion <code class="language-plaintext highlighter-rouge">A</code> where <code class="language-plaintext highlighter-rouge">A = [x, y, z] | w</code> with real (<code class="language-plaintext highlighter-rouge">[x, y, z]</code>) and imaginary (<code class="language-plaintext highlighter-rouge">w</code>) parts, its <a href="https://en.wikipedia.org/wiki/Quaternion#Conjugation,_the_norm,_and_reciprocal">conjugate</a> can be expressed as follow: <code class="language-plaintext highlighter-rouge">conjugate(A) = [-x, -y, -z] | w</code>. The conjugate simply flips the sign of each component of the real part.</p>
<p>The most common way to achieve this is by multiplying a constant just like <a href="https://github.com/microsoft/DirectXMath/blob/939c1a86b28f0d10858601b80faa7845070687fb/Inc/DirectXMathMisc.inl#L251">DirectX Math</a> and Unreal Engine 4 do.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">quatf</span> <span class="nf">quat_conjugate</span><span class="p">(</span><span class="n">quatf</span> <span class="n">input</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">constexpr</span> <span class="n">__m128</span> <span class="n">signs</span> <span class="o">=</span> <span class="p">{</span> <span class="o">-</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">1.0</span><span class="n">f</span> <span class="p">};</span>
<span class="k">return</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">input</span><span class="p">,</span> <span class="n">signs</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This yields a single instruction (the constant will be loaded from memory as part of the multiply instruction) but we can do better. Flipping the sign bit can also be achieved by XOR-ing our input with the sign bit. To avoid flipping the sign of the <code class="language-plaintext highlighter-rouge">w</code> component, we can simply XOR it with zero which will leave the original value unchanged.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">quatf</span> <span class="nf">quat_conjugate</span><span class="p">(</span><span class="n">quatf</span> <span class="n">input</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">constexpr</span> <span class="n">__m128</span> <span class="n">signs</span> <span class="o">=</span> <span class="p">{</span> <span class="o">-</span><span class="mf">0.0</span><span class="n">f</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.0</span><span class="n">f</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">0.0</span><span class="n">f</span> <span class="p">};</span>
<span class="k">return</span> <span class="n">_mm_xor_ps</span><span class="p">(</span><span class="n">input</span><span class="p">,</span> <span class="n">signs</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Again, this yields a single instruction but this time it is much faster. See full results <a href="https://github.com/nfrechette/rtm/issues/6">here</a> but on my MacBook Pro it is <strong>33.5% faster</strong>!</p>
<h1 id="quaternion-interpolation">Quaternion interpolation</h1>
<p>Linear interpolation for scalars and vectors is simple: <code class="language-plaintext highlighter-rouge">result = ((end - start) * alpha) + start</code>.</p>
<p>While this can be used for quaternions as well, it breaks down if both quaternions are not on the <a href="https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation#The_hypersphere_of_rotations">same side of the hypersphere</a>. Both quaternions <code class="language-plaintext highlighter-rouge">A</code> and <code class="language-plaintext highlighter-rouge">-A</code> represent the same 3D rotation but lie on opposite ends of the 4D hypersphere represented by unit quaternions. In order to properly handle this case, we first need to calculate the dot product of both inputs being interpolated and depending its sign, we must flip one of the inputs so that it can lie on the same side of the hypersphere.</p>
<p>In code it looks like this:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">quatf</span> <span class="nf">quat_lerp</span><span class="p">(</span><span class="n">quatf</span> <span class="n">start</span><span class="p">,</span> <span class="n">quatf</span> <span class="n">end</span><span class="p">,</span> <span class="kt">float</span> <span class="n">alpha</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// To ensure we take the shortest path, we apply a bias if the dot product is negative</span>
<span class="kt">float</span> <span class="n">dot</span> <span class="o">=</span> <span class="n">vector_dot</span><span class="p">(</span><span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">bias</span> <span class="o">=</span> <span class="n">dot</span> <span class="o">>=</span> <span class="mf">0.0</span><span class="n">f</span> <span class="o">?</span> <span class="mf">1.0</span><span class="n">f</span> <span class="o">:</span> <span class="o">-</span><span class="mf">1.0</span><span class="n">f</span><span class="p">;</span>
<span class="n">vector4f</span> <span class="n">rotation</span> <span class="o">=</span> <span class="n">vector_neg_mul_sub</span><span class="p">(</span><span class="n">vector_neg_mul_sub</span><span class="p">(</span><span class="n">end</span><span class="p">,</span> <span class="n">bias</span><span class="p">,</span> <span class="n">start</span><span class="p">),</span> <span class="n">alpha</span><span class="p">,</span> <span class="n">start</span><span class="p">);</span>
<span class="k">return</span> <span class="n">quat_normalize</span><span class="p">(</span><span class="n">rotation</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p><em>The double <code class="language-plaintext highlighter-rouge">vector_neg_mul_sub</code> trick was explained in a <a href="/2019/05/08/sign_flip_optimization/">previous</a> blog post.</em></p>
<p>As mentioned, we take the sign of the dot product to calculate a bias and simply multiply it with the <code class="language-plaintext highlighter-rouge">end</code> input. This can be achieved with a compare instruction between the dot product and zero to generate a mask and using that mask to select between the positive and negative value of <code class="language-plaintext highlighter-rouge">end</code>. This is entirely branchless and boils down to a few instructions: 1x compare, 1x subtract (to generate <code class="language-plaintext highlighter-rouge">-end</code>), 1x blend (with AVX to select the bias), and 1x multiplication (to apply the bias). If AVX isn’t present, the selection is done with 3x logical operation instructions instead. This is what Unreal Engine 4 and many others do.</p>
<p>Here again, we can do better with logical operators.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">quatf</span> <span class="nf">quat_lerp</span><span class="p">(</span><span class="n">quatf</span> <span class="n">start</span><span class="p">,</span> <span class="n">quatf</span> <span class="n">end</span><span class="p">,</span> <span class="kt">float</span> <span class="n">alpha</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">__m128</span> <span class="n">dot</span> <span class="o">=</span> <span class="n">vector_dot</span><span class="p">(</span><span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">);</span>
<span class="c1">// Calculate the bias, if the dot product is positive or zero, there is no bias</span>
<span class="c1">// but if it is negative, we want to flip the 'end' rotation XYZW components</span>
<span class="n">__m128</span> <span class="n">bias</span> <span class="o">=</span> <span class="n">_mm_and_ps</span><span class="p">(</span><span class="n">dot</span><span class="p">,</span> <span class="n">_mm_set_ps1</span><span class="p">(</span><span class="o">-</span><span class="mf">0.0</span><span class="n">f</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">rotation</span> <span class="o">=</span> <span class="n">_mm_add_ps</span><span class="p">(</span><span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">_mm_sub_ps</span><span class="p">(</span><span class="n">_mm_xor_ps</span><span class="p">(</span><span class="n">end</span><span class="p">,</span> <span class="n">bias</span><span class="p">),</span> <span class="n">start</span><span class="p">),</span> <span class="n">_mm_set_ps1</span><span class="p">(</span><span class="n">alpha</span><span class="p">)),</span> <span class="n">start</span><span class="p">);</span>
<span class="k">return</span> <span class="n">quat_normalize</span><span class="p">(</span><span class="n">rotation</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>What we really want to achieve is to do nothing (use <code class="language-plaintext highlighter-rouge">end</code> as-is) if the sign of the bias is zero and to flip the sign of <code class="language-plaintext highlighter-rouge">end</code> if the bias is negative. This is a perfect fit for XOR! All we need to do is XOR <code class="language-plaintext highlighter-rouge">end</code> with the sign bit of the bias. We can easily extract it with a logical AND instruction and a mask of the sign bit. This boils down to just two instructions: 1x logical AND and 1x logical XOR. We managed to remove expensive floating point operations while simultaneously using fewer and cheaper instructions.</p>
<p><em>Measuring is left as an exercise for the reader.</em></p>
<h1 id="quaternion-multiplication">Quaternion multiplication</h1>
<p>While the previous two use cases have been in RTM for some time now, this one is brand new and is what crossed my mind the other night: quaternion multiplication can use the same trick!</p>
<p>Multiplying two quaternions with scalar arithmetic is done like this:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">quatf</span> <span class="nf">quat_mul</span><span class="p">(</span><span class="n">quatf</span> <span class="n">lhs</span><span class="p">,</span> <span class="n">quatf</span> <span class="n">rhs</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">float</span> <span class="n">x</span> <span class="o">=</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">w</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">x</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">y</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">z</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">z</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">y</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">y</span> <span class="o">=</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">w</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">y</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">x</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">z</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">y</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">z</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">x</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">z</span> <span class="o">=</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">w</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">z</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">x</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">y</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">y</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">z</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">w</span><span class="p">);</span>
<span class="kt">float</span> <span class="n">w</span> <span class="o">=</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">w</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">w</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">x</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">x</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">y</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">y</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">z</span> <span class="o">*</span> <span class="n">lhs</span><span class="p">.</span><span class="n">z</span><span class="p">);</span>
<span class="k">return</span> <span class="n">quat_set</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">z</span><span class="p">,</span> <span class="n">w</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Floating point multiplications and additions being expensive, we can reduce their number by converting this to SSE2 and shuffling our inputs to line everything up. A few shuffles can line the values up for our multiplications but it is clear that in order to use addition (or subtraction), we have to flip the signs of a few components. Again, <a href="https://github.com/microsoft/DirectXMath/blob/939c1a86b28f0d10858601b80faa7845070687fb/Inc/DirectXMathMisc.inl#L143">DirectX Math</a> does just this (and so does Unreal Engine 4).</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">quatf</span> <span class="nf">quat_mul</span><span class="p">(</span><span class="n">quatf</span> <span class="n">lhs</span><span class="p">,</span> <span class="n">quatf</span> <span class="n">rhs</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">constexpr</span> <span class="n">__m128</span> <span class="n">control_wzyx</span> <span class="o">=</span> <span class="p">{</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">,</span><span class="o">-</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">,</span><span class="o">-</span><span class="mf">1.0</span><span class="n">f</span> <span class="p">};</span>
<span class="k">constexpr</span> <span class="n">__m128</span> <span class="n">control_zwxy</span> <span class="o">=</span> <span class="p">{</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">,</span><span class="o">-</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span><span class="o">-</span><span class="mf">1.0</span><span class="n">f</span> <span class="p">};</span>
<span class="k">constexpr</span> <span class="n">__m128</span> <span class="n">control_yxwz</span> <span class="o">=</span> <span class="p">{</span> <span class="o">-</span><span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">,</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">,</span><span class="o">-</span><span class="mf">1.0</span><span class="n">f</span> <span class="p">};</span>
<span class="n">__m128</span> <span class="n">r_xxxx</span> <span class="o">=</span> <span class="n">_mm_shuffle_ps</span><span class="p">(</span><span class="n">rhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">,</span> <span class="n">_MM_SHUFFLE</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">r_yyyy</span> <span class="o">=</span> <span class="n">_mm_shuffle_ps</span><span class="p">(</span><span class="n">rhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">,</span> <span class="n">_MM_SHUFFLE</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">r_zzzz</span> <span class="o">=</span> <span class="n">_mm_shuffle_ps</span><span class="p">(</span><span class="n">rhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">,</span> <span class="n">_MM_SHUFFLE</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">r_wwww</span> <span class="o">=</span> <span class="n">_mm_shuffle_ps</span><span class="p">(</span><span class="n">rhs</span><span class="p">,</span> <span class="n">rhs</span><span class="p">,</span> <span class="n">_MM_SHUFFLE</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">lxrw_lyrw_lzrw_lwrw</span> <span class="o">=</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">r_wwww</span><span class="p">,</span> <span class="n">lhs</span><span class="p">);</span>
<span class="n">__m128</span> <span class="n">l_wzyx</span> <span class="o">=</span> <span class="n">_mm_shuffle_ps</span><span class="p">(</span><span class="n">lhs</span><span class="p">,</span> <span class="n">lhs</span><span class="p">,</span><span class="n">_MM_SHUFFLE</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">lwrx_lzrx_lyrx_lxrx</span> <span class="o">=</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">r_xxxx</span><span class="p">,</span> <span class="n">l_wzyx</span><span class="p">);</span>
<span class="n">__m128</span> <span class="n">l_zwxy</span> <span class="o">=</span> <span class="n">_mm_shuffle_ps</span><span class="p">(</span><span class="n">l_wzyx</span><span class="p">,</span> <span class="n">l_wzyx</span><span class="p">,</span><span class="n">_MM_SHUFFLE</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">lwrx_nlzrx_lyrx_nlxrx</span> <span class="o">=</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">lwrx_lzrx_lyrx_lxrx</span><span class="p">,</span> <span class="n">control_wzyx</span><span class="p">);</span> <span class="c1">// flip!</span>
<span class="n">__m128</span> <span class="n">lzry_lwry_lxry_lyry</span> <span class="o">=</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">r_yyyy</span><span class="p">,</span> <span class="n">l_zwxy</span><span class="p">);</span>
<span class="n">__m128</span> <span class="n">l_yxwz</span> <span class="o">=</span> <span class="n">_mm_shuffle_ps</span><span class="p">(</span><span class="n">l_zwxy</span><span class="p">,</span> <span class="n">l_zwxy</span><span class="p">,</span><span class="n">_MM_SHUFFLE</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">));</span>
<span class="n">__m128</span> <span class="n">lzry_lwry_nlxry_nlyry</span> <span class="o">=</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">lzry_lwry_lxry_lyry</span><span class="p">,</span> <span class="n">control_zwxy</span><span class="p">);</span> <span class="c1">// flip!</span>
<span class="n">__m128</span> <span class="n">lyrz_lxrz_lwrz_lzrz</span> <span class="o">=</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">r_zzzz</span><span class="p">,</span> <span class="n">l_yxwz</span><span class="p">);</span>
<span class="n">__m128</span> <span class="n">result0</span> <span class="o">=</span> <span class="n">_mm_add_ps</span><span class="p">(</span><span class="n">lxrw_lyrw_lzrw_lwrw</span><span class="p">,</span> <span class="n">lwrx_nlzrx_lyrx_nlxrx</span><span class="p">);</span>
<span class="n">__m128</span> <span class="n">nlyrz_lxrz_lwrz_wlzrz</span> <span class="o">=</span> <span class="n">_mm_mul_ps</span><span class="p">(</span><span class="n">lyrz_lxrz_lwrz_lzrz</span><span class="p">,</span> <span class="n">control_yxwz</span><span class="p">);</span> <span class="c1">// flip!</span>
<span class="n">__m128</span> <span class="n">result1</span> <span class="o">=</span> <span class="n">_mm_add_ps</span><span class="p">(</span><span class="n">lzry_lwry_nlxry_nlyry</span><span class="p">,</span> <span class="n">nlyrz_lxrz_lwrz_wlzrz</span><span class="p">);</span>
<span class="k">return</span> <span class="n">_mm_add_ps</span><span class="p">(</span><span class="n">result0</span><span class="p">,</span> <span class="n">result1</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The code this time is a bit harder to read but here is the gist:</p>
<ul>
<li>We need 7x shuffles to line everything up</li>
<li>With everything lined up, we need 4x multiplications and 3x additions</li>
<li>3x multiplications are also required to flip our signs ahead of each addition (which conveniently can also be done with fused-multiply-add)</li>
</ul>
<p>I’ll omit the code for brevity but by using <code class="language-plaintext highlighter-rouge">-0.0f</code> and <code class="language-plaintext highlighter-rouge">0.0f</code> as our control values to flip the sign bits with XOR instead, quaternion multiplication becomes much <a href="https://github.com/nfrechette/rtm/issues/29">faster</a>. On my MacBook Pro it is <strong>14%</strong> faster while on my Ryzen 2950X it is <strong>10%</strong> faster! I also measured with ACL to see what the speed up would be in a real world use case: compressing lots of animations. With the data sets I measure with, this new quaternion multiplication accelerates the compression by up to <strong>1.3%</strong>.</p>
<p><em>Most x64 CPUs in use today (including those in the PlayStation 4 and Xbox One) do not yet support fused-multiply-add and when I add support for it in RTM, I will measure again.</em></p>
<h1 id="is-it-safe">Is it safe?</h1>
<p>In all three of these examples, the results are binary exact and identical to their reference implementations. Flipping the sign bit on normal floating point values (and infinities) with XOR yields a binary exact result. If the input is <code class="language-plaintext highlighter-rouge">NaN</code>, XOR will not yield the same output but it will yield a <code class="language-plaintext highlighter-rouge">NaN</code> with the sign bit flipped which is entirely valid and consistent (the sign bit is typically left unused on <code class="language-plaintext highlighter-rouge">NaN</code> values).</p>
<p>I also measured this trick with NEON on ARMv7 and ARM64 but sadly it is slower on those platforms (for now). It appears that there is indeed a penalty there for switching between the two domains and perhaps in time it will go away or perhaps something else is slowing things down.</p>
<p><em>ARM64 already uses fused-multiply-add where possible.</em></p>
<h1 id="progress-update">Progress update</h1>
<p>It has been almost a year since the <a href="https://nfrechette.github.io/2019/01/19/introducing_realtime_math/">first release</a> of RTM. The next release should happen very soon now, just ahead of the next ACL release which will introduce it as a dependency for its <a href="https://github.com/nfrechette/acl/issues/71">scalar track compression</a> API. I am on track to finish both releases before the end of the year.</p>
<p>Thanks to the new <a href="https://github.com/sponsors">GitHub Sponsors</a> program, you can now <a href="https://github.com/sponsors/nfrechette">sponsor me</a>! All funds donated will go towards purchasing new devices to optimize for as well as other related costs (like coffee).</p>
Pitfalls of linear sample reduction: Part 42019-07-31T00:00:00+00:00http://nfrechette.github.io/2019/07/31/pitfalls_linear_reduction_part4<p><strong>A quick recap:</strong> animation clips are formed from a set of time series called tracks. Tracks have a fixed number of samples per second and each track has the same length. The <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> retains every sample while <em>Unreal Engine</em> uses the popular method of <a href="/2016/12/07/anim_compression_key_reduction/">removing samples that can be linearly interpolated</a> from their neighbors.</p>
<p>The <a href="/2019/07/23/pitfalls_linear_reduction_part1/">first post</a> showed how removing samples negates some of the benefits that come from <a href="/2016/11/10/anim_compression_uniform_segmenting/">segmenting</a>, rendering the technique a lot less effective.</p>
<p>The <a href="/2019/07/25/pitfalls_linear_reduction_part2/">second post</a> explored how sorting (or not) the retained samples impacts the decompression performance.</p>
<p>The <a href="/2019/07/29/pitfalls_linear_reduction_part3/">third post</a> took a deep dive into the memory overhead of the three techniques we have been discussing so far:</p>
<ul>
<li>Retaining every sample with ACL</li>
<li>Sorted and unsorted linear sample reduction</li>
</ul>
<p>This fourth and final post in the series shows exactly how many samples are removed in practice.</p>
<h3 id="how-often-are-samples-removed">How often are samples removed?</h3>
<p>In order to answer this question, I instrumented the <a href="https://github.com/nfrechette/acl-ue4-plugin">ACL UE4 plugin</a> to extract how many samples per pose, per clip, and per track were dropped. I then ran this over the <a href="http://mocap.cs.cmu.edu/"><em>Carnegie-Mellon University</em> motion capture database</a> as well as <em>Paragon</em> and <em>Fortnite</em>. I did this while keeping every sample with full precision (quantizing the samples can only make things worse) with and without <a href="/2016/12/22/anim_compression_error_compensation/">error compensation</a> (retargeting). The idea behind retargeting is to compensate the error by altering the samples slightly as we optimize a bone chain. While it will obviously be slower to compress than not using it, it should in theory reduce the overall error and possibly allow us to remove more samples as a result.</p>
<p><em>Note that constant and default tracks are ignored when calculating the number of samples dropped.</em></p>
<table>
<thead>
<tr>
<th>Carnegie-Mellon University</th>
<th>With retargeting</th>
<th>Without retargeting</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total size</td>
<td>204.40 MB</td>
<td>204.40 MB</td>
</tr>
<tr>
<td>Compression speed</td>
<td>6487.83 KB/sec</td>
<td>9756.59 KB/sec</td>
</tr>
<tr>
<td>Max error</td>
<td>0.1416 cm</td>
<td>0.0739 cm</td>
</tr>
<tr>
<td>Median dropped per clip</td>
<td>0.13 %</td>
<td>0.13 %</td>
</tr>
<tr>
<td>Median dropped per pose</td>
<td>0.00 %</td>
<td>0.00 %</td>
</tr>
<tr>
<td>Median dropped per track</td>
<td>0.00 %</td>
<td>0.00 %</td>
</tr>
</tbody>
</table>
<p>As expected, the CMU motion capture database performs very poorly with sample reduction. By its very nature, motion capture data can be quite noisy as it comes from cameras or sensors.</p>
<table>
<thead>
<tr>
<th>Paragon</th>
<th>With retargeting</th>
<th>Without retargeting</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total size</td>
<td>575.55 MB</td>
<td>572.60 MB</td>
</tr>
<tr>
<td>Compression speed</td>
<td>2125.21 KB/sec</td>
<td>3319.38 KB/sec</td>
</tr>
<tr>
<td>Max error</td>
<td>80.0623 cm</td>
<td>14.1421 cm</td>
</tr>
<tr>
<td>Median dropped per clip</td>
<td>12.37 %</td>
<td>12.70 %</td>
</tr>
<tr>
<td>Median dropped per pose</td>
<td>13.04 %</td>
<td>13.41 %</td>
</tr>
<tr>
<td>Median dropped per track</td>
<td>2.13 %</td>
<td>2.25 %</td>
</tr>
</tbody>
</table>
<p>Now the data is more interesting. Retargeting continues to be slower to compress but surprisingly, it fails to reduce the memory footprint as well as the number of samples dropped. It even fails to improve the compression accuracy.</p>
<table>
<thead>
<tr>
<th>Fortnite</th>
<th>With retargeting</th>
<th>Without retargeting</th>
</tr>
</thead>
<tbody>
<tr>
<td>Total size</td>
<td>1231.97 MB</td>
<td>1169.01 MB</td>
</tr>
<tr>
<td>Compression speed</td>
<td>1273.36 KB/sec</td>
<td>2010.08 KB/sec</td>
</tr>
<tr>
<td>Max error</td>
<td>283897.4062 cm</td>
<td>172080.7500 cm</td>
</tr>
<tr>
<td>Median dropped per clip</td>
<td>7.11 %</td>
<td>7.72 %</td>
</tr>
<tr>
<td>Median dropped per pose</td>
<td>10.81 %</td>
<td>11.76 %</td>
</tr>
<tr>
<td>Median dropped per track</td>
<td>15.37 %</td>
<td>16.13 %</td>
</tr>
</tbody>
</table>
<p>The retargeting trend continues with <em>Fortnite</em>. One possible explanation for these disappointing results is that error compensation within UE4 does not measure the error in the same way that the engine does after compression is done: it does not take into account <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/error_measurements.md">virtual vertices or leaf bones</a>. This discrepancy leads to the optimizing algorithm thinking the error is lower than it really is.</p>
<p>This is all well and good but how does the full distribution look? <em>Retargeting will be omitted since it doesn’t appear to contribute much.</em></p>
<p><img src="/public/samples_dropped_distribution.png" alt="Samples dropped distribution" /></p>
<p><em>Note that Paragon and Fortnite have ~460 clips (7%) and ~2700 clips (32%) respectively with one or two samples and thus no reduction can happen in those clips.</em></p>
<p>The graph is quite telling: more often than not we fail to drop enough samples to match ACL with segmenting. Very few clips end up dropping over 35% of their samples: none do in CMU, 18% do in <em>Paragon</em>, and 23% in <em>Fortnite</em>.</p>
<p>This is despite using very optimistic memory overhead estimates. In practice, the overhead is almost always higher than what we used in our calculations and removing samples might negatively impact our quantization bit rates, further increasing the overall memory footprint.</p>
<p><em>Note that curve fitting might allow us to remove more samples but it would slow down compression and decompression.</em></p>
<h2 id="removing-samples-just-isnt-worth-it">Removing samples just isn’t worth it</h2>
<p>I have been implementing animation compression algorithms in one form or another for many years now and I have grown to believe that removing samples just isn’t worth it: retaining every sample is the overall best strategy.</p>
<p>Games often play animations erratically or in unpredictable manner. Some randomly seek while others play forward and backward. Various factors control when and where an animation starts playing and when it stops. Clips are often sampled at various sample rates that differ from their runtime playback rates. The ideal default strategy must handle all of these cases equally well. The last thing animators want to do is mess around with compression parameters of individual clips to avoid an algorithm butchering their work.</p>
<p>When samples are removed, sorting what is retained and using a persistent context is a challenging idea in large scale games. Even if decompression has the potential to be the fastest under specific conditions, in practice the gains might not materialize. Regardless, whether the retained samples are sorted or not, metadata must be added to compensate and it eats away at the memory gains achieved. While the <em>Unreal Engine</em> codecs (which use unsorted sample reduction) could be optimized, the amount of cache misses cannot be significantly reduced and ultimately proves to be the bottleneck.</p>
<p>Furthermore, as ACL continues to show, removing samples is not necessary in order to achieve a low and competitive memory footprint. By virtue of having its data simply laid out in memory, very little metadata overhead is required and performance remains consistent and lightning fast. This also dramatically simplifies and streamlines compression as we do not need to consider which samples to retain while attempting to find the optimal bit rate for every track.</p>
<p>It is also worth noting that while we assumed that it is possible to achieve the same bit rates as ACL while removing samples, it might not be the case as the resulting error will combine in subtle ways. Despite being very conservative in our estimates with the sample reduction variants, ACL emerges a clear winner.</p>
<p>That being said, I do have some ideas of my own on how to tackle the problem of efficient sample reduction and maybe someday I will get the chance to try them even if only for the sake of research.</p>
<h2 id="a-small-note-about-curves">A small note about curves</h2>
<p>It is worth nothing that by uniformly sampling an input curve, some amount of precision loss can happen. If the apex of a curve happens between two samples, it will be smoothed out and lost.</p>
<p>The most common curve formats (cubic) generally require 4 values in order to interpolate. This means that the context footprint also increases by a factor of two. In theory, a curve might need fewer samples to represent the same time series but that is not always the case. Animations that come from motion capture or offline simulations such as cloth or hair will often have very noisy data and will not be well approximated by a curve. Such animations might see the number of samples removed drop below 10% as can be seen with the CMU motion capture database.</p>
<p>Curves might also need arbitrary time values that do not fall on uniformly distributed values. When this is the case, the time cannot be quantized too much as it will lower the resulting accuracy, further complicating things and increasing the memory footprint. If the data is non-uniform, a context object is required in order to keep decompression fast and everything I mentioned earlier applies. This is also true of techniques that store their data relative to previous samples (e.g. a delta or velocity change).</p>
<h2 id="special-thanks">Special thanks</h2>
<p>I spend a great deal of time implementing ACL and writing about the nuggets of knowledge I find along the way. All of this is made possible, in part, thanks to <em>Epic</em> which is generously allowing me to use the <em>Paragon</em> and <em>Fortnite</em> animations for research purposes. <a href="https://github.com/CodyDWJones">Cody Jones</a>, <a href="https://github.com/tirpidz">Martin Turcotte</a>, and <a href="https://keybase.io/visualphoenix">Raymond Barbiero</a> continue to contribute code, ideas, and proofread my work and their help is greatly appreciated. Many others have contributed to ACL and its UE4 plugin as well. Thank you for keeping me motivated and your ongoing support!</p>
Pitfalls of linear sample reduction: Part 32019-07-29T00:00:00+00:00http://nfrechette.github.io/2019/07/29/pitfalls_linear_reduction_part3<p><strong>A quick recap:</strong> animation clips are formed from a set of time series called tracks. Tracks have a fixed number of samples per second and each track has the same length. The <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> retains every sample while <em>Unreal Engine</em> uses the popular method of <a href="/2016/12/07/anim_compression_key_reduction/">removing samples that can be linearly interpolated</a> from their neighbors.</p>
<p>The <a href="/2019/07/23/pitfalls_linear_reduction_part1/">first post</a> showed how removing samples negates some of the benefits that come from <a href="/2016/11/10/anim_compression_uniform_segmenting/">segmenting</a>, rendering the technique a lot less effective.</p>
<p>The <a href="/2019/07/25/pitfalls_linear_reduction_part2/">second post</a> explored how sorting (or not) the retained samples impacts the decompression performance.</p>
<p>This third post will take a deep dive into the memory overhead of the three techniques we have been discussing so far:</p>
<ul>
<li>Retaining every sample with ACL</li>
<li>Sorted and unsorted linear sample reduction</li>
</ul>
<h2 id="should-we-remove-samples-or-retain-them">Should we remove samples or retain them?</h2>
<p>ACL does not yet implement a sample reduction algorithm while UE4 is missing a number of features that ACL provides. As such, in order to keep things as fair as possible, some assumptions will be made for the sample reduction variants and we will thus extrapolate some results using ACL as a baseline.</p>
<p>In order to find out if it is worth it to remove samples and how to best go about storing the remaining samples, we will use the <em>Main Trooper</em> from the <a href="2017-10-05-acl_in_ue4">Matinee fight scene</a>. It has 541 bones with no animated 3D scale (1082 tracks in total) and the sequence has 1991 frames (~66 seconds long) per track. A total of 71 tracks are constant, 1 is default, and 1010 are animated. Where segmenting is concerned, I use the arbitrarily chosen segment #13 as a baseline. ACL splits this clip in 124 segments of about 16 frames each. This clip has a <em>LOT</em> of data and a high bone count which should highlight how well these algorithms scale.</p>
<p>We will track three things we care about:</p>
<ul>
<li>The number of bytes touched during decompression for a full pose</li>
<li>The number of cache lines touched during decompression for a full pose</li>
<li>The total compressed size</li>
</ul>
<p>Due to the simple nature of animation decompression, the number of cache lines touched is a good indicator of overall performance as it often can be memory bound.</p>
<p>All numbers will be rounded up to the nearest byte and cache line.</p>
<p>All three algorithms use linear interpolation and as such require two poses to interpolate our final result.</p>
<p><em>See the annex at the end of the post for how the math breaks down</em></p>
<h3 id="the-shared-base">The shared base</h3>
<p>Some features are always a win and will be assumed present in our three algorithms:</p>
<ul>
<li>If a track has a single repeating sample (within a threshold), it will be deemed a <a href="/2016/11/03/anim_compression_constant_tracks/">constant track</a>. Constant tracks are collapsed into a single full resolution sample with 3x floats (even for rotations) with a single bit per track to tell them apart.</li>
<li>If a constant track is identical to the identity value for that track type (e.g. quaternion identity), it will be deemed a default track. Default tracks have no sample stored, just a single bit per track to tell them apart.</li>
<li>All animated tracks (not constant or default) will be normalized within their min/max values by performing <a href="/2016/11/09/anim_compression_range_reduction/">range reduction</a> over the whole clip. This increases the accuracy which leads to a lower memory footprint for the majority of clips. To do so, we store our range minimum and extent values as 3x floats each (even for rotations).</li>
</ul>
<p>The current UE4 codecs do not have special treatment for constant and default tracks but they do support range reduction. There is room for improvement here but that is what ACL uses right now and it will be good enough for our calculations.</p>
<table>
<thead>
<tr>
<th> </th>
<th>Bytes touched</th>
<th>Cache lines touched</th>
<th>Compressed size</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Default bit set</strong></td>
<td>136</td>
<td>3</td>
<td>136 bytes</td>
</tr>
<tr>
<td><strong>Constant bit set</strong></td>
<td>136</td>
<td>3</td>
<td>136 bytes</td>
</tr>
<tr>
<td><strong>Constant values</strong></td>
<td>852</td>
<td>14</td>
<td>852 bytes</td>
</tr>
<tr>
<td><strong>Range values</strong></td>
<td>24240</td>
<td>379</td>
<td>24240 bytes</td>
</tr>
<tr>
<td><strong>Total</strong></td>
<td>25364</td>
<td>399</td>
<td>25 KB</td>
</tr>
</tbody>
</table>
<p>In order to support variable bit rates where each animated track can have its own bit rate, ACL stores 1 byte per animated track (and per segment). This is overkill as only 19 bit rates are currently supported but it keeps things simple and in the future the extra bits will be used for other things. When segmenting is enabled with ACL, range reduction is also performed per segment and adds 6 bytes of overhead per animated track for the quantized range values.</p>
<table>
<thead>
<tr>
<th> </th>
<th>Bytes touched</th>
<th>Cache lines touched</th>
<th>Compressed size</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Bit rates</strong></td>
<td>1010</td>
<td>16</td>
<td>123 KB</td>
</tr>
<tr>
<td><strong>Segment range values</strong></td>
<td>6060</td>
<td>95</td>
<td>734 KB</td>
</tr>
</tbody>
</table>
<h3 id="the-acl-results">The ACL results</h3>
<p>We will consider two cases for ACL. The default compression settings have segmenting enabled which is great for the memory footprint and compression speed but due to the added memory overhead and range reduction, decompression is a bit slower. As such, we will also consider the case where we disable segmenting in order to bias for faster decompression.</p>
<p>With segmenting, the animated pose size (just the samples, without the bit rate and segment range values) for the segment #13 is 3777 bytes (60 cache lines). This represents about 3.74 bytes per sample or about 30 <strong>b</strong>its <strong>p</strong>er <strong>s</strong>ample (<strong>bps</strong>).</p>
<p>Without segmenting, the animated pose size is 5903 bytes (93 cache lines). This represents about 5.84 bytes per sample or about 46 <strong>bps</strong>.</p>
<p>Although the UE4 codecs do not support variable bit rates the way ACL does, we will assume that we use the same algorithm and as such these numbers of 30 and 46 bits per samples will be used in our projections. Because 30 bps is only attainable with segmenting enabled, we will also assume it is enabled for the sample reduction algorithms when using that bit rate.</p>
<table>
<thead>
<tr>
<th> </th>
<th>Bytes touched</th>
<th>Cache lines touched</th>
<th>Compressed size</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>With segmenting</strong></td>
<td>39990</td>
<td>625</td>
<td>7695 KB</td>
</tr>
<tr>
<td><strong>Without segmenting</strong></td>
<td>38172</td>
<td>597</td>
<td>11503 KB</td>
</tr>
</tbody>
</table>
<p>As we can see, while segmenting reduces considerably the overall memory footprint (by 33%), it does contribute to quite a few extra cache lines being touched (4.7% more) during decompression despite the animated pose being 36% smaller. This highlights how normalizing the samples within the range of each segment increases their overall accuracy and reduces the number of bits required to maintain it.</p>
<h3 id="unsorted-sample-reduction">Unsorted sample reduction</h3>
<p>In order to decompress when samples are missing, we need to store the sample time (or index). For every track, we will search for the two samples that bound the current time we are interpolating at and reconstruct the correct interpolation alpha from them. To keep things simple, we will store this time value on 1 byte per sample retained (for a maximum of 256 samples per clip or segment) along with the total number of samples retained per track on 1 byte. Supporting arbitrary track decompression efficiently also requires storing an offset map where each track begins. For simplicity’s sake, we will omit this overhead but UE4 uses 4 bytes per track. When decompressing, we will assume that we immediately find the two samples we need within a single cache line and that both samples are within another cache line (2 cache misses per track).</p>
<p>These estimates are very conservative. In practice, the offsets are required to support <em>Levels of Detail</em> as well as efficient single bone decompression and more than 1 byte is often required for sample indices and their count in larger clips. In the wild, the memory footprint is surely going to be larger than these projections will show.</p>
<p><em>Values below assume every sample is retained, for now.</em></p>
<table>
<thead>
<tr>
<th> </th>
<th>Bytes touched</th>
<th>Cache lines touched</th>
<th>Compressed size</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Sample times</strong></td>
<td>2020</td>
<td>1010</td>
<td>1964 KB</td>
</tr>
<tr>
<td><strong>Sample count with segmenting</strong></td>
<td>1010</td>
<td>16</td>
<td>123 KB</td>
</tr>
<tr>
<td><strong>Sample values with segmenting</strong></td>
<td>7575</td>
<td>1010</td>
<td>7346 KB</td>
</tr>
<tr>
<td><strong>Total with segmenting</strong></td>
<td>43039</td>
<td>2546</td>
<td>10315 KB</td>
</tr>
<tr>
<td><strong>Sample count without segmenting</strong></td>
<td>1010</td>
<td>16</td>
<td>1 KB</td>
</tr>
<tr>
<td><strong>Sample values without segmenting</strong></td>
<td>11615</td>
<td>1010</td>
<td>11470 KB</td>
</tr>
<tr>
<td><strong>Total without segmenting</strong></td>
<td>40009</td>
<td>2435</td>
<td>13583 KB</td>
</tr>
</tbody>
</table>
<p>When our samples are unsorted, it becomes obvious why decompression is quite slow. The number of cache lines touched is staggering: 2435 cache lines which represents 153 KB! This scales linearly with the number of animated tracks. We can also see that despite the added overhead of segmenting, the overall memory footprint is lower (by 23%) but not by as much as ACL.</p>
<p><em>Despite my claims from the first post, segmenting appears attractive here. This is a direct result of our estimates being conservative and UE4 not supporting the aggressive per track quantization that ACL provides.</em></p>
<h3 id="sorted-sample-reduction">Sorted sample reduction</h3>
<p>With our samples sorted, we will add 16 bits of metadata per sample to store the sample time, the track index, and track type. This is optimistic. In reality, some samples would likely require more than that.</p>
<p>Our context will only store animated tracks to keep it as small as possible. We will consider two scenarios: when our samples are stored with full precision (96 bits per sample) and when they are packed with the same format as the compressed byte stream (in which case we simply copy the values into our context when seeking, no unpacking occurs). As previously mentioned, linear interpolation requires us to store two samples per animated track.</p>
<table>
<thead>
<tr>
<th> </th>
<th>Bytes touched</th>
<th>Cache lines touched</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Context values @ 96 bps</strong></td>
<td>24240</td>
<td>379</td>
</tr>
<tr>
<td><strong>Context values @ 46 bps</strong></td>
<td>12808</td>
<td>198</td>
</tr>
<tr>
<td><strong>Context values @ 30 bps</strong></td>
<td>14626</td>
<td>230</td>
</tr>
</tbody>
</table>
<p>Right off the bat, it is clear that if we want interpolation to be as fast as possible (with no unpacking), our context is quite large and requires evicting quite a bit of CPU cache. To keep the context footprint as low as possible, going forward we will assume that we store the values packed inside it. Storing packed samples into our context comes with challenges. Each segment will have a different pose size and as such we either need to resize the context or allocate it with the largest pose size. When packed in the compressed byte stream, each sample is often bit aligned and copying into another bit aligned buffer is more expensive than a regular <code class="language-plaintext highlighter-rouge">memcpy</code> operation. Keeping the context size low requires some work.</p>
<p><em>Note that the above numbers also include the bytes touched for the bit rates and segment range values (where relevant) because they are needed for interpolation when samples are packed.</em></p>
<table>
<thead>
<tr>
<th> </th>
<th>Bytes touched</th>
<th>Cache lines touched</th>
<th>Compressed size</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed values with segmenting</strong></td>
<td>5798</td>
<td>91</td>
<td>11274 KB</td>
</tr>
<tr>
<td><strong>Compressed values without segmenting</strong></td>
<td>7919</td>
<td>123</td>
<td>15398 KB</td>
</tr>
<tr>
<td><strong>Total with segmenting</strong></td>
<td>45788</td>
<td>720</td>
<td>12156 KB</td>
</tr>
<tr>
<td><strong>Total without segmenting</strong></td>
<td>46091</td>
<td>720</td>
<td>15424 KB</td>
</tr>
</tbody>
</table>
<p><em>Note that the compressed size above does not consider the footprint of the context required at runtime to decompress but the bytes and cache lines touched do.</em></p>
<p>Compared to the unsorted algorithm, the memory overhead goes up quite a bit: the constant struggle between size and speed.</p>
<p>The number of cache lines touched during decompression is quite a bit higher (15%) than ACL. In order to match ACL, 100% of the samples must be dropped which makes sense considering that we use the same bit rate as ACL to estimate and our context stores two poses which is also what ACL touches. Reading the compressed pose is added on top of this. As such, if every sample is dropped within a pose and already present in our context the decompression cost will be at best identical to ACL since the two will perform about the same amount of work. Reading compressed samples and copying them into our context will take some time and lead to a net win for ACL.</p>
<p>If instead we keep the context at full precision and unpack samples once into it, the picture becomes a bit more complicated. If no unpacking occurs and we simply interpolate from the context, we will be touching more cache lines overall but everything will be very hardware friendly. Decompression is likely to beat ACL but the evicted CPU cache might slow down the caller slightly. Whether this yields a net win is uncertain. Any amount of unpacking that might be required will slow things down further as well. Ultimately, even if enough samples are dropped and it is faster, it will come with a noticeable increase in runtime memory footprint and the complexity to manage a persistent context.</p>
<h3 id="three-challengers-enter-only-one-emerges-victorious">Three challengers enter, only one emerges victorious</h3>
<table>
<thead>
<tr>
<th> </th>
<th>Bytes touched</th>
<th>Cache lines touched</th>
<th>Compressed size</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Uniform with segmenting</strong></td>
<td>39990</td>
<td>625</td>
<td>7695 KB</td>
</tr>
<tr>
<td><strong>Unsorted with segmenting</strong></td>
<td>43039</td>
<td>2546</td>
<td>10315 KB</td>
</tr>
<tr>
<td><strong>Sorted with segmenting</strong></td>
<td>45788</td>
<td>720</td>
<td>12156 KB</td>
</tr>
<tr>
<td><strong>Uniform without segmenting</strong></td>
<td>38172</td>
<td>597</td>
<td>11503 KB</td>
</tr>
<tr>
<td><strong>Unsorted without segmenting</strong></td>
<td>40009</td>
<td>2435</td>
<td>13583 KB</td>
</tr>
<tr>
<td><strong>Sorted without segmenting</strong></td>
<td>46091</td>
<td>720</td>
<td>15424 KB</td>
</tr>
</tbody>
</table>
<p>ACL retains every sample and touches the least amount of memory and it applies the lower CPU cache pressure. However, so far our estimates assumed that all samples were retained and as such, we cannot make a determination as to whether or not it also wins on the overall memory footprint. What we can do however, is determine how many samples we need to drop in order to match it.</p>
<p>With unsorted samples, we have to drop roughly 30% of our samples in order to match the compressed memory footprint of ACL with segmenting and 15% without. However, regardless of how many samples we remove, the decompression performance will never come close to the other two techniques due to the extremely high number of cache misses.</p>
<p>With sorted samples, we have to drop roughly 40% of our samples in order to match the compressed memory footprint of ACL with segmenting and 25% without. The number of cache lines touched during decompression is now quite a bit closer to ACL compared to the unsorted algorithm. It may or may not end up being faster to decompress depending on how many samples are removed, the context format used, and how many samples need unpacking but it will always evict more of the CPU cache.</p>
<p>It is worth noting that most of these numbers remain true if cubic interpolation is used instead. While the context object will double in size and thus require more cache lines to be touched during decompression, the total compressed size will remain the same if the same number of samples are retained.</p>
<p><a href="/2019/07/31/pitfalls_linear_reduction_part4/">The fourth</a> and last blog post in the series will look at how many samples are actually removed in <em>Paragon</em> and <em>Fortnite</em>. This will complete the puzzle and paint a clear picture of the strengths and weaknesses of linear sample reduction techniques.</p>
<h3 id="annex">Annex</h3>
<p>Inputs:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">num_segments = 124</code></li>
<li><code class="language-plaintext highlighter-rouge">num_samples_per_track = 1991</code></li>
<li><code class="language-plaintext highlighter-rouge">num_tracks = 1082</code></li>
<li><code class="language-plaintext highlighter-rouge">num_animated_tracks = 1010</code></li>
<li><code class="language-plaintext highlighter-rouge">num_constant_tracks = 71</code></li>
<li><code class="language-plaintext highlighter-rouge">bytes_per_sample_with_segmenting = 3.74</code></li>
<li><code class="language-plaintext highlighter-rouge">bytes_per_sample_without_segmenting = 5.84</code></li>
<li><code class="language-plaintext highlighter-rouge">num_animated_samples = num_samples_per_track * num_animated_tracks = 2010910</code></li>
<li><code class="language-plaintext highlighter-rouge">num_pose_to_interpolate = 2</code></li>
</ul>
<p>Shared math:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">bitset_size = num_tracks / 8 = 136 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">constant_values_size = num_constant_tracks * sizeof(float) * 3 = 852 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">range_values_size = num_animated_tracks * sizeof(float) * 6 = 24240 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">clip_shared_size = bitset_size * 2 + constant_values_size + range_values_size = 25364 bytes = 25 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">bit_rates_size = num_animated_tracks * 1 = 1010 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">bit_rates_size_total_with_segmenting = bit_rates_size * num_segments = 123 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">segment_range_values_size = num_animated_tracks * 6 = 6060 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">segment_range_values_size_total = segment_range_values_size * num_segments = 734 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">animated_pose_size_with_segmenting = bytes_per_sample_with_segmenting * num_animated_tracks = 3778 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">animated_pose_size_without_segmenting = bytes_per_sample_without_segmenting * num_animated_tracks = 5899 bytes</code></li>
</ul>
<p>ACL math:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">acl_animated_size_with_segmenting = animated_pose_size_with_segmenting * num_samples_per_track = 7346 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">acl_animated_size_without_segmenting = animated_pose_size_without_segmenting * num_samples_per_track = 11470 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">acl_decomp_bytes_touched_with_segmenting = clip_shared_size + bit_rates_size + segment_range_values_size + acl_animated_pose_size_with_segmenting * num_pose_to_interpolate = 39990 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">acl_decomp_bytes_touched_without_segmenting = clip_shared_size + bit_rates_size + acl_animated_pose_size_without_segmenting * num_pose_to_interpolate = 38172 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">acl_size_with_segmenting = clip_shared_size + bit_rates_size_total_with_segmenting + segment_range_values_size_total + acl_animated_size_with_segmenting = 8106 KB</code> (actual size is lower due to the bytes per sample changing from segment to segment)</li>
<li><code class="language-plaintext highlighter-rouge">acl_size_without_segmenting = clip_shared_size + bit_rates_size + acl_animated_size_without_segmenting = 11496 KB</code> (actual size is higher by a few bytes due to misc. clip overhead)</li>
</ul>
<p>Unsorted linear sample reduction math:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">unsorted_decomp_bytes_touched_sample_times = num_animated_tracks * 1 * num_pose_to_interpolate = 1010 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_sample_times_size_total = num_animated_samples * 1 = 1964 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_sample_counts_size = num_animated_tracks * 1 = 1010 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_sample_counts_size_total = unsorted_sample_counts_size * num_segments = 123 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_animated_size_with_segmenting = acl_animated_size_with_segmenting = 7346 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_animated_size_without_segmenting = acl_animated_size_without_segmenting = 11470 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_total_size_with_segmenting = clip_shared_size + bit_rates_size_total_with_segmenting + segment_range_values_size_total + unsorted_sample_times_size_total + unsorted_sample_counts_size_total + unsorted_animated_size_with_segmenting = 10315 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_total_size_without_segmenting = clip_shared_size + bit_rates_size_total + unsorted_sample_times_size_total + unsorted_sample_counts_size + unsorted_animated_size_without_segmenting = 13583 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_total_sample_size_with_segmenting = unsorted_sample_times_size_total + unsorted_animated_size_with_segmenting = 9310 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_total_sample_size_without_segmenting = unsorted_sample_times_size_total + unsorted_animated_size_without_segmenting = 13434 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_drop_rate_with_segmenting = (unsorted_total_size_with_segmenting - acl_size_with_segmenting) / unsorted_total_sample_size_with_segmenting = 28 %</code></li>
<li><code class="language-plaintext highlighter-rouge">unsorted_drop_rate_without_segmenting = (unsorted_total_size_without_segmenting - acl_size_without_segmenting) / unsorted_total_sample_size_without_segmenting = 15.5 %</code></li>
</ul>
<p>Sorted linear sample reduction math:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">full_resolution_context_size = num_animated_tracks * num_pose_to_interpolate * sizeof(float) * 3 = 24240 bytes = 24 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">with_segmenting_context_size = num_pose_to_interpolate * animated_pose_size_with_segmenting = 7556 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">without_segmenting_context_size = num_pose_to_interpolate * animated_pose_size_without_segmenting = 11798 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">with_segmenting_context_decomp_bytes_touched = with_segmenting_context_size + bit_rates_size + segment_range_values_size = 14626 bytes = 15 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">without_segmenting_context_decomp_bytes_touched = without_segmenting_context_size + bit_rates_size = 12808 bytes = 13 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_decomp_compressed_bytes_touched_with_segmenting = num_animated_tracks * sizeof(uint16) + animated_pose_size_with_segmenting = 5798 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_decomp_compressed_bytes_touched_without_segmenting = num_animated_tracks * sizeof(uint16) + animated_pose_size_without_segmenting = 7919 bytes</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_animated_size_with_segmenting = num_animated_samples * sizeof(uint16) + acl_animated_size_with_segmenting = 11274 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_animated_size_without_segmenting = num_animated_samples * sizeof(uint16) + acl_animated_size_without_segmenting = 15398 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_decomp_bytes_touched_with_segmenting = with_segmenting_context_decomp_bytes_touched + sorted_decomp_compressed_bytes_touched_with_segmenting + clip_shared_size = 45788 bytes = 45 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_decomp_bytes_touched_without_segmenting = without_segmenting_context_decomp_bytes_touched + sorted_decomp_compressed_bytes_touched_without_segmenting + clip_shared_size = 46091 bytes = 46 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_total_size_with_segmenting = clip_shared_size + bit_rates_size_total_with_segmenting + segment_range_values_size_total + sorted_animated_size_with_segmenting = 12156 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_total_size_without_segmenting = clip_shared_size + bit_rates_size + sorted_animated_size_without_segmenting = 15424 KB</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_drop_rate_with_segmenting = (sorted_total_size_with_segmenting - acl_size_with_segmenting) / sorted_animated_size_with_segmenting = 40 %</code></li>
<li><code class="language-plaintext highlighter-rouge">sorted_drop_rate_without_segmenting = (sorted_total_size_without_segmenting - acl_size_without_segmenting) / sorted_animated_size_without_segmenting = 25.5 %</code></li>
</ul>
Pitfalls of linear sample reduction: Part 22019-07-25T00:00:00+00:00http://nfrechette.github.io/2019/07/25/pitfalls_linear_reduction_part2<script async="" src="//s.imgur.com/min/embed.js" charset="utf-8"></script>
<p><strong>A quick recap:</strong> animation clips are formed from a set of time series called tracks. Tracks have a fixed number of samples per second and each track has the same length. The <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> retains every sample while the most commonly used <em>Unreal Engine</em> codecs use the popular method of <a href="/2016/12/07/anim_compression_key_reduction/">removing samples that can be linearly interpolated</a> from their neighbors.</p>
<p>The <a href="/2019/07/23/pitfalls_linear_reduction_part1/">first post</a> showed how removing samples negates some of the benefits that come from segmenting, rendering the technique a lot less effective.</p>
<p>Another area where it struggles is decompression performance. When we want to sample a clip at a particular point in time, we have to search and find the closest samples in order to interpolate them. <em>Unreal Engine</em> lays out the data per track: the indices for all the retained samples are followed by their sample values.</p>
<p><img src="/public/offset_map.jpg" alt="Offset map" /></p>
<p>This comes with a major performance drawback: each track will incur a cache miss for the sample indices in order to find the neighbors and another cache miss to read the two samples we need to interpolate. This is <em>very</em> slow. Each memory access will be random, preventing the hardware prefetcher from hiding the memory access latency. Even if we manage to prefetch it by hand, we still touch a very large number of cache lines. Equally worse, each cache line is only used partially as they also contain data we will not need. In the end, a significant portion of our CPU cache will be evicted with data that will only be read once.</p>
<p><img src="/public/sorted_uniform_samples.jpg" alt="Sorted uniform samples" /></p>
<p>In contrast, ACL retains every sample and sorts them by time (sample index). This ensures that all the samples we need at a particular point in time are contiguous in memory. Sampling our clip becomes very fast:</p>
<ul>
<li>We don’t need to search for our neighbors, just where the first sample lives</li>
<li>We don’t need to read indices, offsets, or the number of samples retained</li>
<li>Each cache line is fully used</li>
<li>The hardware prefetcher will detect our predictable access pattern and work properly</li>
</ul>
<p>Sorting is clearly the key to fast decompression.</p>
<h2 id="sorting-retained-samples">Sorting retained samples</h2>
<p>Back in 2017, if you searched for ‘‘<em>animation compression</em>’’, the most popular blog posts were <a href="http://bitsquid.blogspot.com/2011/10/low-level-animation-part-2.html">one by Bitsquid</a> which advocates using curve fitting with sorted samples for fast and cache friendly decompression and <a href="https://technology.riotgames.com/news/compressing-skeletal-animation-data">a post by Riot Games</a> about trying the same technique with some success.</p>
<p><img src="/public/sorted_samples.jpg" alt="Sorted samples" /></p>
<p>Without getting into the details too much (the two posts above explain it quite well), you sort the samples by the time you need them <em>at</em> (<strong>NOT</strong> by their sample time) and you keep track from frame to frame where and what you last decompressed from the compressed byte stream. Once decompressed, samples are stored (often raw, unpacked) in a persistent context object that is reused from frame to frame. This allows you to touch the least possible amount of <em>compressed</em> contiguous data every frame by unpacking only the new samples that you need to interpolate at the current desired time. Once all your samples are unpacked inside the context, interpolation is <em>very</em> fast. You can use tons of tricks like <em>Structure of Arrays</em>, wider SIMD registers with AVX, and you can easily interpolate two or three samples at a time in order to use all available registers and minimize pipeline stalls. This requires keeping a context object around all the time but it is by far the fastest way to interpolate a compressed animation because you can avoid unpacking samples that have already been cached.</p>
<p>Sorting our samples keeps things contiguous and CPU cache friendly and as such it stood to reason that it was a good idea worth trying. Some <em>Unreal Engine</em> codecs already support linear sample reduction and as such the remaining samples were simply sorted in the same manner.</p>
<p>With this new technique, decompression was up to <strong>4x</strong> faster on PC. It looked <strong>phenomenal</strong> in a <em>gym</em>.</p>
<iframe src="https://giphy.com/embed/qTK56RyKJIZ9K" width="480" height="400" frameborder="0" class="giphy-embed" allowfullscreen=""></iframe>
<p>Unfortunately, this technique has a number of drawbacks that are either skimmed briefly, downplayed, or not mentioned at all in the two blog posts advocating it. Those proved too significant to ignore in <em>Fortnite</em>. Sometimes being the fastest isn’t the best idea.</p>
<h3 id="no-u-turn-allowed">No U-turn allowed</h3>
<p>By sorting the samples, the playback direction must now match the sort direction. If you attempt to play a sequence backwards that has its samples sorted forward, you must seek and read everything until you find the ones you need. You can sort the samples backwards to fix this but forward playback will now have the same issue. There is no optimal sort order for both playback directions. Similarly, random seeks in a sequence have equally abysmal performance.</p>
<p>As <em>Bitsquid</em> mentions, this can be mitigated by introducing a full frame of data at specific intervals to avoid fully reading everything or segments can be used for the same purpose. This comes at the cost of a larger memory footprint and it does not offset entirely the extra work done when seeking. One would think that most clips play forward in time and it isn’t that big a deal. Sure some clips play backward or randomly but those aren’t that common, right?</p>
<iframe src="https://giphy.com/embed/kZGxyEX1m9t7y" width="480" height="469" frameborder="0" class="giphy-embed" allowfullscreen=""></iframe>
<p>In practice, things are not always this way. Many prop animations will play forward <em>and</em> backward at runtime. For example, a chest opening and closing might have a single animation played in both directions. The same idea can be used with all sorts of other objects like doors, windows, etc. A more subtle problem are clips that play forward in time but that do not start playing at the beginning. With motion matching, often when you transition from one clip to another you will not start playing the new clip at frame 0. When playback starts, you will have to read and skip samples, creating a performance spike. This can also happen with clips that advance in time but do not decompress due to <em>Level of Detail (LOD)</em> constraints. As soon as those characters start animating, performance spikes. It is also worth noting that even if you start playing at the first frame, you need to unpack everything needed to prime the context which creates a performance spike regardless.</p>
<h3 id="30-fps-is-more-cinematic">30 FPS is ‘more cinematic’</h3>
<p>It is not unusual for clips to be exported with varying sample rates such as 30, 60, or 120 FPS (and everything in between). Equally common are animations that play at various rates. However, unlike other techniques, these properties combine and can become problematic. If we play an animation faster than its sample rate (e.g. a 60 FPS game with 30 FPS animations) we will have frames where no data needs to be unpacked from the compressed stream and we can interpolate entirely from the context. This is very fast but it does mean that our decompression performance is inconsistent as some frames will need to unpack samples while others will not. This typically isn’t that big a deal but things get much worse if we play back slower than the sample rate (e.g. a 30 FPS game with 60 FPS animations). Our clip contains many more samples that we will not need to interpolate and because they are sorted, we will have to read and skip them in order to reach the ones we need. When samples are removed and sorted, decompression performance becomes a function of the playback speed and the animation sample rate. Such a problematic scenario can arise if an animation (such as a cinematic) requires a very high sample rate to maintain its quality (perhaps due to cloth and hair simulations).</p>
<h3 id="just-one-please">Just one please</h3>
<p>Although not as common, single bone decompression performance is pretty bad. Because all the data is mixed together, decompressing a specific bone requires decompressing (or at least skipping) all the other data. This is fine if you rarely do it or if it’s done at the same time as sampling the full pose while sharing the context between calls but this is not always possible. In some cases you have to sample individual bones at runtime for various gameplay or AI purposes (e.g. to predict where a bone will land in the future). This same property of the data means that lowering the LOD does not speed up seeking nor does it reduce the context memory footprint as everything needs to be unpacked just in case the LOD changes suddenly (although you can get away with interpolating only what you need).</p>
<h3 id="one-more-byte">One more byte</h3>
<blockquote class="imgur-embed-pub" lang="en" data-id="zussD" data-context="false"><a href="//imgur.com/zussD">Just one more bite</a></blockquote>
<p>Sorting the samples means that there is no pattern to them and metadata per sample needs to be introduced. You need to be able to tell what type a sample is (rotation, translation, 3D scale, scalar), at what time the sample appears, and which bone it belongs to. This overhead being per sample adds up quickly. You can use all sorts of clever tricks to use fewer bits if the bone index and sample time index are small compared to the previous one but ultimately it is quite a bit larger than alternative techniques and it cannot be hidden or removed entirely.</p>
<p>With all of this data, the context object becomes quite large to the point where its memory footprint cannot be ignored in a game with many animations playing at the same time. In order to interpolate, you need at least two full poses with linear interpolation (four if you use cubic curves) stored inside along with other metadata to keep track of things. For example, if a bone transform needs a <em>quaternion</em> (rotation) and a <em>vector3</em> (translation) for a total of 28 bytes, 100 bones will require 2.7 KB for a single pose and 5.5 KB for two (and often 3D scale is needed, adding even more data). With curves, those balloon to 11 KB and by touching them you evict over 30% of your CPU L1 cache (most CPUs have 32 KB of L1) for data that will not be needed again until the next frame. This is not cheap.</p>
<p>It is clear that while we touch less compressed memory and avoid the price of unpacking it, we end up accessing quite a bit of uncompressed memory, evicting precious CPU cache lines in the process. Typically, once decompression ends, the resulting pose will be blended with another intermediate pose later to be blended with another, and another. All of these intermediate poses benefit from remaining in the cache because they are needed over and over often reused by new decompression and pose blending calls. As such, the true cost of decompression cannot be measured easily: the cache impact can slow down the calling code as well. While sorting is definitely more cache friendly than not doing so when samples are removed, whether this is more so than retaining every sample is not as obvious.</p>
<p>You can keep the poses packed in some way within the context, either with the same format as the compressed stream or packed in a friendlier format at the cost of interpolation performance. Regardless, the overhead adds up and in a game like <em>Fortnite</em> where you can have <strong>50 vs 50</strong> players fighting with props, pets, and other things animating all at the same time, the overall memory footprint ended up too large to be acceptable on mobile devices. We attempted to not retain a context object per animation that was playing back, sharing them across characters and threads but this added a lot of complexity and we still had performance spikes from the higher amount of seeking. You can have a moderate amount of expensive seeks or a lower runtime memory footprint but not both.</p>
<p><em>This last point ended up being an issue even without any sorting (just with segmenting). Even though the memory overhead was not as significant, it still proved to be above what we would have liked, the complexity too high, and the decompression performance too unpredictable.</em></p>
<h3 id="a-lot-of-complexity-for-not-much">A lot of complexity for not much</h3>
<p>Overall, sorting the samples retained increased slightly the compressed size, it increased the runtime memory footprint, decompression performance became erratic, while peak decompression speed increased. Overcoming these issues would have required a lot more time and effort. With the complexity already being very high, I was not confident I could beat the unsorted codecs consistently. We ultimately decided not to use this technique. The runtime memory footprint increased beyond what we considered acceptable for <em>Fortnite</em> and the decompression performance too erratic.</p>
<p>Although the technique appeared very attractive as presented by <em>Bitsquid</em>, it ended up being quite underwhelming. This was made all the more apparent by my parallel efforts with ACL that retained every sample yet achieved remarkable results with little to no complexity. ACL has consistent and fast decompression performance regardless of the playback rate, playback direction, or sample rate and it does this without the need for a persistent context.</p>
<p>When linear sample reduction is used, both sorted and unsorted algorithms have significant drawbacks when it comes to decompression performance and memory usage. While both techniques require extra metadata that increases their memory footprint, if enough samples are removed, the overhead can be offset to yield a net win. The <a href="/2019/07/29/pitfalls_linear_reduction_part3/">next post</a> will look into how many samples need to be removed in order to beat ACL which retains all of them.</p>
Pitfalls of linear sample reduction: Part 12019-07-23T00:00:00+00:00http://nfrechette.github.io/2019/07/23/pitfalls_linear_reduction_part1<p>Two years ago I worked with <em>Epic</em> to try to improve the <em>Unreal Engine</em> animation compression codecs. <a href="/2019/06/14/acl_plugin_v0.4.0/">While I managed to significantly improve the compression performance</a>, despite my best intentions, a lot of research, and hard work, some of the ideas I tried failed to improve the decompression speed and memory footprint. In the end their numbers just didn’t add up when working at the scale <em>Fortnite</em> operates at.</p>
<p>I am now refactoring Unreal’s codec API to natively support animation compression plugins. As part of that effort, I am removing the experimental ideas I had added and I thought they deserved their own blog posts. There are many academic papers and posts about things that worked but very few about those that didn’t.</p>
<p>The UE4 codecs rely heavily on <a href="/2016/12/07/anim_compression_key_reduction/">linear sample reduction</a>: samples in a time series that can be linearly interpolated from their neighbors are removed to achieve compression. This technique is very common but it introduces a number of nuances that are often overlooked. We will look at some of these in this multi-part series:</p>
<ul>
<li>Splitting animation sequences into independent segments doesn’t work too well</li>
<li><a href="/2019/07/25/pitfalls_linear_reduction_part2/">Sorting the samples in a CPU cache friendly memory layout has far reaching implications</a></li>
<li><a href="/2019/07/29/pitfalls_linear_reduction_part3/">The added memory overhead when samples are removed is fairly large</a></li>
<li><a href="/2019/07/31/pitfalls_linear_reduction_part4/">Empirical data shows that in practice not many samples are removed</a></li>
</ul>
<p>TL;DR: As codecs grow in complexity, they can sometimes have unintended side-effects and ultimately be outperformed by simpler codecs.</p>
<h2 id="what-we-are-working-with">What we are working with?</h2>
<p><a href="/2016/10/27/anim_compression_data/">Animation data</a> consists of a series of values at various points in time (a time series). These drive the rotation, translation, and scale of various bones in a character’s skeleton or in some object (prop). Both <em>Unreal Engine</em> and the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> work with a fixed number of samples (frames) per second. Storing animation curves where the samples can be placed at arbitrary points in time isn’t as common in video games. The series of rotation, translation, and scale are called tracks and all have the same length. Together they combine to form an animation clip that describes how multiple bone transforms evolve over time.</p>
<p>Compression is commonly achieved by:</p>
<ul>
<li>Removing samples that can be reconstructed by interpolating between the remaining neighbors</li>
<li>Storing each sample using a reduced number of bits</li>
</ul>
<p><em>Unreal Engine</em> uses linear interpolation to reconstruct the samples it removes and it stores each track using 32, 48, or 96 bits per sample.</p>
<p>ACL retains every sample and stores each track using 9, 12, 16, … 96 bits per sample (19 possible bit rates).</p>
<h2 id="slice-it-up">Slice it up!</h2>
<p><img src="/public/segmenting_explained.jpg" alt="Segmenting Explained" /></p>
<p><a href="/2016/11/10/anim_compression_uniform_segmenting/">Segmenting a clip</a> is the act of splitting it into a number of independent and contiguous segments. For example, we can split an animation sequence that has tracks with 31 samples each (1 second at 30 samples per second) into two segments with 16 and 15 samples per track.</p>
<p>Segmenting has a number of advantages:</p>
<ul>
<li>Values within a segment have a <a href="/2016/11/09/anim_compression_range_reduction/">reduced range of motion</a> which allows fewer bits to be used to represent them, achieving compression.</li>
<li>When samples are removed we have to search for their neighbors to reconstruct their value. Segmenting allows us to narrow down the search faster, reducing the cost of seeking.</li>
<li>Because all the data needed to sample a time <code class="language-plaintext highlighter-rouge">T</code> is within a contiguous segment, we can easily stream it from a slower medium or prefetch it.</li>
<li>If segments are small enough, they fit entirely within the processor L1 or L2 cache which leads to faster compression.</li>
<li>Independent segments can trivially be compressed in parallel.</li>
</ul>
<p>Around that time in the summer of 2017, I introduced segmenting into ACL and <a href="/2017/09/10/acl_v0.4.0/">saw massive gains</a>: the memory footprint reduced by roughly <strong>36%</strong>.</p>
<p>ACL uses 16 samples per segment on average and having access to the animations from <em>Paragon</em> I looked at how many segments it had: <strong>6558</strong> clips turned into <strong>49214</strong> segments.</p>
<p>What a <em>great</em> idea to try with the UE4 codecs as well!</p>
<h2 id="segmenting-in-unreal">Segmenting in Unreal</h2>
<p>Unfortunately, the <em>Unreal Engine</em> codecs were not designed with this concept in mind.</p>
<p><img src="/public/offset_map.jpg" alt="Offset map" /></p>
<p>In order to keep decompression fast, each track stores an offset into the compressed byte stream where its data starts as well as the number of samples retained. This allows great performance if all you need is a single bone or when bones are decompressed in an order different from the one they were compressed in (this is quite common in UE4 for various reasons).</p>
<p>Unfortunately, this overhead must be repeated in every segment. To avoid the compressed clip’s size increasing too much, I settled on a segment size of 64 samples. But these larger segments came with some drawbacks:</p>
<ul>
<li>They are less likely to fit in the CPU L1 cache when compressing</li>
<li>There are fewer opportunities for parallelism when compressing</li>
<li>They don’t narrow the sample search as much when decompressing</li>
</ul>
<p>Most of the <em>Paragon</em> clips are short. Roughly <strong>60%</strong> of them need only a single segment. Only <strong>19%</strong> had more than two segments. This meant most clips consumed the same amount of memory while being slightly slower to decompress. Only long cinematics like the <a href="/2017/10/05/acl_in_ue4/">Matinee fight scene</a> showed significant gains on their memory footprint, compression performance, and decompression performance. In my experience working on multiple games, short clips are by far the most common and <em>Paragon</em> isn’t an outlier.</p>
<p><strong>Overall, segmenting worked but it was very underwhelming within the UE4 codecs. It did not deliver what I had hoped it would and what I had seen with ACL.</strong></p>
<p>In an effort to fix the decompression performance regression, a context object was introduced, adding even more complexity. A context object persists from frame to frame for each clip being played back. It allows data to be reused from a previous decompression call to speed up the next call. It is also necessary in order to support sorting the samples which I tried next and will be covered in my <a href="/2019/07/25/pitfalls_linear_reduction_part2/">next post</a>.</p>
Comparing past and present Unreal Engine 4 animation compression2019-06-14T00:00:00+00:00http://nfrechette.github.io/2019/06/14/acl_plugin_v0.4.0<p>Ever since the <a href="https://github.com/nfrechette/acl-ue4-plugin">Animation Compression Library UE4 Plugin</a> was released in <em>July 2018</em>, I have been comparing and tracking its progress against <em>Unreal Engine 4.19.2</em>. For the sake of consistency, even when newer UE versions came out, I did not integrate ACL and measure from scratch. This was convenient and practical since animation compression doesn’t change very often in UE and the numbers remain fairly close over time. However, in <em>UE 4.21</em> significant changes were made and comparing against an earlier release was no longer a fair and accurate comparison.</p>
<p>As such, I have updated my baseline to <em>UE 4.22.2</em> and am releasing a new version of the plugin to go along with it: <a href="https://github.com/nfrechette/acl-ue4-plugin/releases/tag/v0.4.0">v0.4</a>. This new plugin release does not bring significant improvements on the previous one (they both still use <a href="https://github.com/nfrechette/acl/releases/tag/v1.2.0">ACL v1.2</a>) but it does bring the necessary changes to integrate cleanly with the newer UE API. One of the most notable changes in this release is the introduction of <a href="https://github.com/nfrechette/acl-ue4-plugin/tree/develop/Docs#engine-integration">git patches</a> for the custom engine changes required to support the ACL plugin. Note that earlier engine versions are not officially supported although if there is interest, feel free to reach out to me.</p>
<p>One benefit of the thorough measurements that I regularly perform is that not only can I track ACL’s progress over time but I can also do the same with Unreal Engine. Today we’ll talk about <em>Unreal</em> a bit more.</p>
<h2 id="what-changed-in-ue-421">What changed in UE 4.21</h2>
<p>Two years ago <em>Epic</em> asked me to improve their own animation compression. The primary focus was on improving the compression speed of their automatic compression codec while maintaining the existing memory footprint and decompression speed. Improvements to the other metrics was considered a stretch goal.</p>
<p>The automatic compression codec in <em>UE 4.20</em> and earlier tried <strong>35+</strong> codec permutations and picked the best overall. Understandably, this could be quite slow in many cases.</p>
<p>To speed it up, two important changes were made:</p>
<ul>
<li>The codecs tried were distilled into a white list of the 11 most commonly used codecs</li>
<li>The codecs are now evaluated in parallel</li>
</ul>
<p>This brought a significant win. As mentioned at the <a href="https://youtu.be/tWVZ6KO4lRs?t=1256">Games Developer Conference 2019</a>, the introduction of the white list brought the time to compress <em>Fortnite</em> (while cooking for the <em>Xbox One</em>) from <strong>6h 25mins</strong> down to <strong>1h 50mins</strong>, a speed up of <strong>3.5x</strong>. Executing them in parallel lowered this even more, down to <strong>40mins</strong> or about <strong>2.75x</strong> faster. Overall, it ended up <strong>9.6x</strong> faster.</p>
<p>While none of this impacted the codecs or the API in any way, <em>UE 4.21</em> also saw significant changes to the codec API. The reason for this will be the subject of my <a href="/2019/07/23/pitfalls_linear_reduction_part1/">next blog post</a>: failed optimization attempts and the lessons learned from them. In particular, I will show why removing samples that can be reconstructed through interpolation has significant drawbacks. Sometimes despite your best intentions and research, things just don’t pan out.</p>
<h2 id="the-juicy-numbers">The juicy numbers</h2>
<p>For the <em>GDC</em> presentation we only measured a few metrics and only while cooking <em>Fortnite</em>. However, every release I use a much more extensive set of animations to track progress and many more metrics. Here are all the numbers.</p>
<p><em>Note: The ACL plugin v0.3 numbers are from its integration in UE 4.19.2 while v0.4 is with UE 4.22.2. Because the Unreal codecs didn’t change, the decompression performance remained roughly the same. The ACL plugin accuracy numbers are slightly different due to fixes in the commandlet used to extract them. The compression speedup for the ACL plugin largely comes from the switch to Visual Studio 2017 that came with UE 4.20.</em></p>
<h3 id="carnegie-mellon-university-database-performance">Carnegie-Mellon University database performance</h3>
<p>For details about this data set and the metrics used, see <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/cmu_performance.md">here</a>.</p>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.4.0</th>
<th>ACL Plugin v0.3.0</th>
<th>UE v4.22.2</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>74.42 MB</td>
<td>74.42 MB</td>
<td>100.15 MB</td>
<td>99.94 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>19.21 : 1</td>
<td>19.21 : 1</td>
<td>14.27 : 1</td>
<td>14.30 : 1</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>5m 10s</td>
<td>6m 24s</td>
<td>11m 11s</td>
<td>1h 27m 40s</td>
</tr>
<tr>
<td><strong>Compression speed</strong></td>
<td>4712 KB/sec</td>
<td>3805 KB/sec</td>
<td>2180 KB/sec</td>
<td>278 KB/sec</td>
</tr>
<tr>
<td><strong>Max ACL error</strong></td>
<td>0.0968 cm</td>
<td>0.0702 cm</td>
<td>0.1675 cm</td>
<td>0.1520 cm</td>
</tr>
<tr>
<td><strong>Max UE4 error</strong></td>
<td>0.0816 cm</td>
<td>0.0816 cm</td>
<td>0.0995 cm</td>
<td>0.0996 cm</td>
</tr>
<tr>
<td><strong>ACL Error 99<sup>th</sup> percentile</strong></td>
<td>0.0089 cm</td>
<td>0.0088 cm</td>
<td>0.0304 cm</td>
<td>0.0271 cm</td>
</tr>
<tr>
<td><strong>Samples below ACL error threshold</strong></td>
<td>99.90 %</td>
<td>99.93 %</td>
<td>47.81 %</td>
<td>49.34 %</td>
</tr>
</tbody>
</table>
<h3 id="paragon-database-performance">Paragon database performance</h3>
<p>For details about this data set and the metrics used, see <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/paragon_performance.md">here</a>.</p>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.4.0</th>
<th>ACL Plugin v0.3.0</th>
<th>UE v4.22.2</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>234.76 MB</td>
<td>234.76 MB</td>
<td>380.37 MB</td>
<td>392.97 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>18.22 : 1</td>
<td>18.22 : 1</td>
<td>11.24 : 1</td>
<td>10.88 : 1</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>23m 58s</td>
<td>30m 14s</td>
<td>2h 5m 11s</td>
<td>15h 10m 23s</td>
</tr>
<tr>
<td><strong>Compression speed</strong></td>
<td>3043 KB/sec</td>
<td>2412 KB/sec</td>
<td>582 KB/sec</td>
<td>80 KB/sec</td>
</tr>
<tr>
<td><strong>Max ACL error</strong></td>
<td>0.8623 cm</td>
<td>0.8623 cm</td>
<td>0.8619 cm</td>
<td>0.8619 cm</td>
</tr>
<tr>
<td><strong>Max UE4 error</strong></td>
<td>0.8601 cm</td>
<td>0.8601 cm</td>
<td>0.6424 cm</td>
<td>0.6424 cm</td>
</tr>
<tr>
<td><strong>ACL Error 99<sup>th</sup> percentile</strong></td>
<td>0.0100 cm</td>
<td>0.0094 cm</td>
<td>0.0438 cm</td>
<td>0.0328 cm</td>
</tr>
<tr>
<td><strong>Samples below ACL error threshold</strong></td>
<td>99.00 %</td>
<td>99.19 %</td>
<td>81.75 %</td>
<td>84.88 %</td>
</tr>
</tbody>
</table>
<h3 id="matinee-fight-scene-performance">Matinee fight scene performance</h3>
<p>For details about this data set and the metrics used, see <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/fight_scene_performance.md">here</a>.</p>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.4.0</th>
<th>ACL Plugin v0.3.0</th>
<th>UE v4.22.2</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>8.77 MB</td>
<td>8.77 MB</td>
<td>23.67 MB</td>
<td>23.67 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>7.11 : 1</td>
<td>7.11 : 1</td>
<td>2.63 : 1</td>
<td>2.63 : 1</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>16s</td>
<td>20s</td>
<td>9m 36s</td>
<td>54m 3s</td>
</tr>
<tr>
<td><strong>Compression speed</strong></td>
<td>3790 KB/sec</td>
<td>3110 KB/sec</td>
<td>110 KB/sec</td>
<td>19 KB/sec</td>
</tr>
<tr>
<td><strong>Max ACL error</strong></td>
<td>0.0588 cm</td>
<td>0.0641 cm</td>
<td>0.0426 cm</td>
<td>0.0671 cm</td>
</tr>
<tr>
<td><strong>Max UE4 error</strong></td>
<td>0.0617 cm</td>
<td>0.0617 cm</td>
<td>0.0672 cm</td>
<td>0.0672 cm</td>
</tr>
<tr>
<td><strong>ACL Error 99<sup>th</sup> percentile</strong></td>
<td>0.0382 cm</td>
<td>0.0382 cm</td>
<td>0.0161 cm</td>
<td>0.0161 cm</td>
</tr>
<tr>
<td><strong>Samples below ACL error threshold</strong></td>
<td>94.61 %</td>
<td>94.52 %</td>
<td>94.23 %</td>
<td>94.22 %</td>
</tr>
</tbody>
</table>
<h2 id="conclusion-and-what-comes-next">Conclusion and what comes next</h2>
<p>Overall, the new parallel white list is a clear winner. It is dramatically faster to compress and none of the other metrics measurably suffered. However, despite this massive improvement ACL remains <em>much</em> faster.</p>
<p>For the past few months I have been working with Epic to refactor the engine side of things to ensure that animation compression plugins are natively supported by the engine. This effort is ongoing and will hopefully land soon in an Unreal Engine near you.</p>
Faster arithmetic by flipping signs2019-05-08T00:00:00+00:00http://nfrechette.github.io/2019/05/08/sign_flip_optimization<p>Over the years, I picked up a number of tricks to optimize code. Today I’ll talk about one of them.</p>
<p>I first picked it up a few years ago when I was tasked with optimizing the cloth simulation code in <a href="https://en.wikipedia.org/wiki/Shadow_of_the_Tomb_Raider">Shadow of the Tomb Raider</a>. It had been fine tuned extensively with <em>PowerPC</em> intrinsics for the <em>Xbox 360</em> but its performance was lacking on <em>XboxOne</em> (<em>x64</em>). While I hoped to talk about it <a href="http://nfrechette.github.io/2017/04/11/modern_simd_hardware/">2 years ago</a>, I ended up side tracked with the <a href="https://github.com/nfrechette/acl">Animation Compression Library (ACL)</a>. However, last week while introducing <em>ARM64</em> fused muptiply-add support into ACL and the <a href="https://github.com/nfrechette/rtm">Realtime Math (RTM)</a> library, I noticed an opportunity to use this trick when linearly interpolating <a href="https://en.wikipedia.org/wiki/Quaternion">quaternions</a> and it all came back to me.</p>
<p>TL;DR: Flipping the sign of some arithmetic operations can lead to shorter and faster assembly.</p>
<h1 id="flipping-for-fun-and-profit-on-x64">Flipping for fun and profit on x64</h1>
<p>To understand how this works, we need to give a bit of context first.</p>
<p>On <em>PowerPC</em>, <em>ARM</em>, and many other platforms, before an instruction can use a value it must first be loaded from memory into a register explicitly with a load type instruction. This is not always the case with <em>x86</em> and <em>x64</em> where many instructions can take either a register or a memory address as input. In the later case, the load still happens behind the scenes and while it isn’t really much faster by itself it does have a few benefits.</p>
<p>Not having an explicit load instruction means that we do not use one of the precious named registers. While under the hood the CPU has tons of registers (e.g modern processors have <strong>50+ XMM registers</strong>), the instructions can only reference a few of them: only <strong>16</strong> named XMM registers can be referenced. This can be very important if a piece of code places considerable pressure on the amount of registers it needs. Fewer named registers used means a reduced likelihood that registers have to spill on the stack and spilling introduces quite a few instructions as well. Altogether, removing that single load instruction can considerably improve the chances of a function getting inlined.</p>
<p>Fewer instructions also means a lower code cache footprint and better overall performance although in practice, I have found this impact very hard to measure.</p>
<p>Two things are important to keep in mind:</p>
<ul>
<li>An instruction that operates directly from memory will be slower than the same instruction working from a register: it has to perform the load operation and even if the value resides in the CPU cache, it still takes time.</li>
<li>Most arithmetic instructions that take two inputs only support having one of them come from memory: addition, subtraction, multiplication, etc. Typically, only the second input (on the right) can come from memory.</li>
</ul>
<p>To illustrate how flipping the sign of arithmetic operations can lead to improved code generation, we will use the following example:</p>
<script src="https://gist.github.com/nfrechette/0654f9ca2d722524844dd1b99f9b5e85.js"></script>
<p>Both <em>MSVC 19.20</em> and <em>GCC 9</em> generate the above assembly. The <code class="language-plaintext highlighter-rouge">1.0f</code> constant must be loaded from memory because <code class="language-plaintext highlighter-rouge">subss</code> only supports the second argument coming from memory. Interestingly, <em>Clang 8</em> figures out that it can use the sign flip trick all on its own:</p>
<script src="https://gist.github.com/nfrechette/b36d26bebfecf67b2bacace7382ad07d.js"></script>
<p>Because we multiply our input value by a constant, we are in control of its sign and we can leverage that fact to change our <code class="language-plaintext highlighter-rouge">subss</code> instruction into an <code class="language-plaintext highlighter-rouge">addss</code> instruction that can work with another constant from memory directly. Both functions are mathematically equivalent and their results are identical down to every last bit.</p>
<p>Short and sweet!</p>
<p>Not every mainstream compiler today is smart enough to do this on its own especially in more complex cases where other instructions will end up in between the two sign flips. Doing it by hand ensures that they have all the tools to do it right. Should the compiler think that performance will be better by loading the constant into a register anyway, it will also be able to do so (for example if the function is inlined in a loop).</p>
<h1 id="but-wait-theres-more">But wait, there’s more!</h1>
<p>While that trick is very handy on <em>x64</em>, it can also be used in a different context and I found a good example on <em>ARM64</em>: when linearly interpolating quaternions.</p>
<p>Typically linear interpolation isn’t recommended with quaternions as it is not guaranteed to give very accurate results if the two quaternions are far apart. However, in the context of animation compression, successive quaternions are often very close to one another and linear interpolation works just fine. Here is what the function looked like before I used the trick:</p>
<script src="https://gist.github.com/nfrechette/83f989059160c2befdcc67d3e2496db1.js"></script>
<p>The code is fairly simple:</p>
<ul>
<li>We calculate a bias and if both quaternions are on opposite sides of the <a href="https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation#The_hypersphere_of_rotations">hypersphere</a> (negative dot product), we apply a bias to flip one of them to make sure that they lie on the same side. This guarantees that the path taken when interpolating will be the shortest.</li>
<li>We linearly interpolate: <code class="language-plaintext highlighter-rouge">(end - start) * alpha + start</code></li>
<li>Last but not least, we normalize our quaternion to make sure that it represents a valid 3D rotation.</li>
</ul>
<p>When I introduced the fused multiply-add support to <em>ARM64</em>, I looked at the above code and its generated assembly and noticed that we had a multiplication instruction followed by a subtraction instruction before our final fused multiply-add instruction. Can we do better?</p>
<p>While <a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/#techs=FMA"><em>FMA3</em></a> has a myriad of instructions for all sorts of fused multiply-add variations, <a href="https://developer.arm.com/architectures/instruction-sets/simd-isas/neon/intrinsics"><em>ARM64</em> does not</a>: it only has 2 such instructions: <code class="language-plaintext highlighter-rouge">fmla</code> ((a * b) + c) and <code class="language-plaintext highlighter-rouge">fmls</code> (-(a * b) + c).</p>
<p>Here is the interpolation broken down a bit more clearly:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">x = (end * bias) - start</code> (<code class="language-plaintext highlighter-rouge">fmul</code>, <code class="language-plaintext highlighter-rouge">fsub</code>)</li>
<li><code class="language-plaintext highlighter-rouge">result = (x * alpha) + start</code> (<code class="language-plaintext highlighter-rouge">fmla</code>)</li>
</ul>
<p>After a bit of thinking, it becomes obvious what the solution is:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">-x = -(end * bias) + start</code> (<code class="language-plaintext highlighter-rouge">fmls</code>)</li>
<li><code class="language-plaintext highlighter-rouge">result = -(-x * alpha) + start</code> (<code class="language-plaintext highlighter-rouge">fmls</code>)</li>
</ul>
<p>By flipping the sign of our intermediate value with the <code class="language-plaintext highlighter-rouge">fmls</code> instruction, we can flip it again and cancel it out by using it once more while removing an instruction in the process. This simple change resulted in a <strong>2.5 %</strong> speedup during animation decompression (which also does a bunch of other stuff) on my iPad.</p>
<p><em>Note: because <code class="language-plaintext highlighter-rouge">fmls</code> reuses one of the input registers for its output, a new <code class="language-plaintext highlighter-rouge">mov</code> instruction was added in the final inlined decompression code but it executes much faster than a floating point instruction.</em></p>
<p>You can find the change in RTM <a href="https://github.com/nfrechette/rtm/pull/17">here</a>.</p>
The Animation Compression Library just got even faster2019-04-15T00:00:00+00:00http://nfrechette.github.io/2019/04/15/acl_v1.2.0<p>Slowly but surely, the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> has now reached <a href="https://github.com/nfrechette/acl/releases/tag/v1.2.0">v1.2</a> along with an updated <a href="https://github.com/nfrechette/acl-ue4-plugin/releases/tag/v0.3.0">v0.3 Unreal Engine 4 plugin</a>. The most notable changes in this release are as follow:</p>
<ul>
<li>More compilers and architectures added to continuous integration</li>
<li>Accuracy bug fixes</li>
<li>Floating point sample rate support</li>
<li>Dramatically faster compression through the introduction of a compression level setting</li>
</ul>
<p>TL;DR: Compared to <strong>UE 4.19.2</strong>, the ACL plugin compresses up to <strong>1.7x smaller</strong>, is up to <strong>3x more accurate</strong>, up to <strong>158x faster to compress</strong>, and up to <strong>7.5x faster to decompress</strong> (results may vary depending on the platform and data).</p>
<p><em>Note that UE 4.21 introduced changes that significantly sped up the compression with its Automatic Compression codec but I haven’t had the time to setup a new branch with it to measure.</em></p>
<h2 id="ue-4-plugin-support-and-progress">UE 4 plugin support and progress</h2>
<p>Now that ACL properly supports a floating point sample rate, the UE4 plugin has reached feature parity with the stock codecs.</p>
<p>As announced at the <em>GDC 2019</em>, work is ongoing to refactor the <em>Unreal Engine</em> to natively support animation compression plugins and is currently on track to land with <strong>UE 4.23</strong>. Once it does, the plugin will be updated once more, finally reaching <strong>v1.0</strong> on the <em>Unreal marketplace</em> for <strong>free</strong>.</p>
<h2 id="lighting-fast-compression">Lighting fast compression</h2>
<p>One of the most common feedback I received from those that use ACL in the wild (both within UE4 and outside) was the desire for faster compression. The optimization algorithm is very aggressive and despite its impressive performance overall (as highlighted in prior releases), some clips with deep bone hierarchies could take a very long time to compress, prohibitively so.</p>
<p>In order to address this, a new compression level was introduced in the compression settings to better control how much time should be spent attempting to find an optimal bit rate. Higher levels take more time but yield a lower memory footprint. A total of five levels were introduced but the lowest three currently behave the same for now: <em>Lowest, Low, Medium, High, Highest</em>. The <em>Highest</em> level corresponds to what prior releases did by default. After carefully reviewing the impact of each level, a decision was made to make the default level be <em>Medium</em> instead. This translates in dramatically faster compression, identical accuracy, with a very small and acceptable increase in memory footprint. This should provide for a much better experience for animators during production. Once the game is ready to be released, the animations can easily and safely be recompressed with the <em>Highest</em> setting in order to squeeze out every byte.</p>
<p>In order to extract the following results, I compressed the <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">Carnegie-Mellon University motion capture database</a>, <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a>, and Fortnite in parallel with <strong>4</strong> threads using ACL standalone. Numbers in parenthesis represent the delta again <em>Highest</em>.</p>
<table>
<thead>
<tr>
<th>Compressed Size</th>
<th>Highest</th>
<th>High</th>
<th>Medium</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>CMU</strong></td>
<td>67.05 MB</td>
<td>68.85 MB (+2.7%)</td>
<td>71.01 MB (+5.9%)</td>
</tr>
<tr>
<td><strong>Paragon</strong></td>
<td>206.87 MB</td>
<td>211.81 MB (+2.4%)</td>
<td>218.58 MB (+5.7%)</td>
</tr>
<tr>
<td><strong>Fortnite</strong></td>
<td>491.79 MB</td>
<td>497.60 MB (+1.2%)</td>
<td>507.11 MB (+3.1%)</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Compression Time</th>
<th>Highest</th>
<th>High</th>
<th>Medium</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>CMU</strong></td>
<td>24m 57.59s</td>
<td>11m 51.48s</td>
<td>6m 20.89s</td>
</tr>
<tr>
<td><strong>Paragon</strong></td>
<td>4h 55m 42.57s</td>
<td>1h 19m 36.01s</td>
<td>29m 21.65s</td>
</tr>
<tr>
<td><strong>Fortnite</strong></td>
<td>8h 13m 1.66s</td>
<td>2h 29m 59.37s</td>
<td>1h 3m 18.17s</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Compression Speed</th>
<th>Highest</th>
<th>High</th>
<th>Medium</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>CMU</strong></td>
<td>977.36 KB/sec</td>
<td>2057.24 KB/sec (+2.1x)</td>
<td>3842.79 KB/sec (+3.9x)</td>
</tr>
<tr>
<td><strong>Paragon</strong></td>
<td>246.79 KB/sec</td>
<td>916.82 KB/sec (+3.7x)</td>
<td>2485.58 KB/sec (+10.1x)</td>
</tr>
<tr>
<td><strong>Fortnite</strong></td>
<td>613.56 KB/sec</td>
<td>2016.82 KB/sec (+3.3x)</td>
<td>4778.65 KB/sec (+7.8x)</td>
</tr>
</tbody>
</table>
<p>And here are the default settings in action on the <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/paragon_performance.md">animations from Paragon</a> with the ACL plugin inside UE4:</p>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.3.0</th>
<th>ACL Plugin v0.2.0</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>234.76 MB</td>
<td>226.09 MB</td>
<td>392.97 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>18.22 : 1</td>
<td>18.91 : 1</td>
<td>10.88 : 1</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>30m 14.69s</td>
<td>6h 4m 18.21s</td>
<td>15h 10m 23.56s</td>
</tr>
<tr>
<td><strong>Compression speed</strong></td>
<td>2412.94 KB/sec</td>
<td>200.32 KB/sec</td>
<td>80.16 KB/sec</td>
</tr>
<tr>
<td><strong>Max ACL error</strong></td>
<td>0.8623 cm</td>
<td>0.8590 cm</td>
<td>0.8619 cm</td>
</tr>
<tr>
<td><strong>Max UE4 error</strong></td>
<td>0.8601 cm</td>
<td>0.8566 cm</td>
<td>0.6424 cm</td>
</tr>
<tr>
<td><strong>ACL Error 99<sup>th</sup> percentile</strong></td>
<td>0.0094 cm</td>
<td>0.0116 cm</td>
<td>0.0328 cm</td>
</tr>
<tr>
<td><strong>Samples below ACL error threshold</strong></td>
<td>99.19 %</td>
<td>98.85 %</td>
<td>84.88 %</td>
</tr>
</tbody>
</table>
<p>The <strong>99th</strong> percentile and the number of samples below the <strong>0.01 cm</strong> error threshold are calculated by measuring the error of every bone at every sample in each of the <strong>6558</strong> animation clips. More details on how the error is measured can be found <a href="https://github.com/nfrechette/acl/blob/develop/docs/error_metrics.md">here</a>.</p>
<p>In this new release, the decompression performance remains largely unchanged. It is worth noting that a month ago my <em>Google Nexus 5X</em> died abruptly and as such performance numbers will no longer be tracked on it. Instead, my new <em>Google Pixel 3</em> will be used from here on out.</p>
<h1 id="whats-next">What’s next</h1>
<p>The <a href="https://github.com/nfrechette/acl/milestone/9">next release v1.3</a> currently scheduled for the <strong>Fall 2019</strong> will aim to tackle commonly requested features:</p>
<ul>
<li>Faster decompression in long clips by optimizing seeking</li>
<li>Multiple root transform support (e.g. rigid body simulation compression)</li>
<li>Scalar track support (e.g. float curves for blend shapes)</li>
<li>Faster compression in part by using multiple threads to compress a single clip (which will help the UE4 plugin a lot)</li>
</ul>
<p>If you use ACL and would like to help prioritize the work I do, feel free to reach out and provide feedback or requests!</p>
Compressing Fortnite Animations2019-01-25T00:00:00+00:00http://nfrechette.github.io/2019/01/25/compressing_fortnite_animations<p>New year, new stats! A few months ago, <em>Epic</em> agreed to let me use their <em>Fortnite</em> animations for my open source research with the <a href="https://github.com/nfrechette/acl">Animation Compression Library (ACL)</a>. Following months of work to refactor <em>Unreal Engine 4</em> in order to natively support animation compression plugins, it has finally entered the review stage on <em>Epic’s</em> end. While I had hoped the changes could make it in time for <em>Unreal Engine 4.22</em>, due to unforeseen delays, <em>4.23</em> seems a much more likely candidate.</p>
<p>Even though the code isn’t public yet, the new updated ACL plugin kicks ass and <em>Fortnite</em> is a great title to showcase it with. The real game uses the classic UE4 codecs but I recompressed everything with the latest and greatest. After spending several hundred hours compressing the animations, fixing bugs, and iterating I can finally present the results.</p>
<p><strong>TL;DR:</strong> Inside <em>Fortnite</em>, ACL shines bright with <strong>2x</strong> faster compression, <strong>2x</strong> smaller memory footprint, and higher accuracy. Decompression is <strong>1.6x</strong> faster on desktop, <strong>2.3x</strong> faster on a <em>Samsung S8</em>, and <strong>1.2x</strong> faster on the <em>Xbox One</em>.</p>
<h1 id="methodology">Methodology</h1>
<p>For the UE4 measurements I used a modified <em>UE 4.21</em> with its default <strong>Automatic Compression</strong>. It tries a list of codecs in parallel and selects the optimal result by considering both size and accuracy.</p>
<p>ACL uses a modified version of the open source <a href="https://github.com/nfrechette/acl-ue4-plugin">ACL Plugin v0.2.2</a>. It uses its own default compression settings and in the rare event where the error is above <strong>1cm</strong>, it falls back automatically to safer settings.</p>
<p>Although the UE4 refactor doesn’t change the legacy codecs, it does speed up their decompression a bit compared to previous UE4 releases. That is one of many benefits everyone will get to enjoy as a result of my refactor work regardless of which codec is used.</p>
<h2 id="error-measurements">Error measurements</h2>
<p>While the UE4 and ACL error measurements never exactly matched, they historically have been very close for every single clip I had tried, until <em>Fortnite</em>. As it turns out, some exotic animations brought to light the fact that some subtle differences in how they both measure the error can lead to some large perceived discrepancies. This has now been documented in the plugin <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/error_measurements.md">here</a>.</p>
<p><strong>Three</strong> differences stand out: how the error is measured, where the error is measured in space, and where the error is measured in time. You can follow the link above for the gory details but the jist is that ACL is more conservative and more accurate in how it measures the error and it should always be trusted over what UE4 reports in case of doubt or disagreement.</p>
<p>It is worth noting that because ACL does not currently support a floating point sample rate (e.g 28.3 FPS), those clips (and there are many) have a higher reported error with UE4 because by rounding, we are effectively time stretching those clips a tiny bit. They still look just as good though. This will be fixed in the <a href="https://github.com/nfrechette/acl-ue4-plugin/issues/7">next version</a>.</p>
<h1 id="the-animations">The animations</h1>
<p>I extracted all the non-additive animations regardless of whether they were used by the game or not: a grand total of <strong>8304</strong> clips! A total raw size of <strong>17 GB</strong> and roughly <strong>17.5 hours</strong> worth of playback.</p>
<p><em>Fortnite</em> has a surprising number of exotic clips. Some take <strong>hours</strong> to compress with UE4 and others have a range of motion as wide as the <em>distance between the earth and the moon</em>! These allowed me to identify a number of very subtle bugs in ACL and to fix them.</p>
<h1 id="compression-stats">Compression stats</h1>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin</th>
<th>UE4</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>498.21 MB</td>
<td>1011.84 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>35.55 : 1</td>
<td>17.50 : 1</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>12h 38m 04.99s</td>
<td>23h 8m 58.94s</td>
</tr>
<tr>
<td><strong>Compression speed</strong></td>
<td>398.72 KB/sec</td>
<td>217.62 KB/sec</td>
</tr>
<tr>
<td><strong>Max ACL error</strong></td>
<td>0.9565 cm</td>
<td>8392339 cm</td>
</tr>
<tr>
<td><strong>Max UE4 error</strong></td>
<td>108904.6797 cm</td>
<td>8397727 cm</td>
</tr>
<tr>
<td><strong>ACL Error 99<sup>th</sup> percentile</strong></td>
<td>0.0309 cm</td>
<td>2.1856 cm</td>
</tr>
<tr>
<td><strong>Samples below ACL error threshold</strong></td>
<td>97.71 %</td>
<td>77.37 %</td>
</tr>
</tbody>
</table>
<p>Once again, ACL performs admirably: the compression speed is twice as fast (<strong>1.83x</strong>), the memory footprint reduces in half (<strong>2.03x</strong> smaller), and the accuracy is right where we want it. This is also in line with the previous results from <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/paragon_performance.md">Paragon</a>.</p>
<p><img src="/public/acl/fortnite_max_error_distribution.png" alt="Fortnite Max Error Distribution" /></p>
<p>UE4’s accuracy struggles a bit with a few clips but in practice the error might not be visible as the overwhelming majority of samples are very accurate. This is consistent as well with <a href="http://nfrechette.github.io/2017/12/05/acl_paragon/">previous results</a>.</p>
<p><img src="/public/acl/ue4_importing.png" alt="UE4 Import Comic" /></p>
<p>A handful of clips contribute to a large portion of the UE4 compression time and its high error. One clip in particular stands out: it has <strong>1167 bones</strong>, <strong>8371 samples</strong> at <strong>120 FPS</strong>, and a total raw size of <strong>372.66 MB</strong>. Its range of motion peaks at <strong>477000 kilometers</strong> away from the origin! It truly pushes the codecs to their absolute limits.</p>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin</th>
<th>UE4</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>71.53 MB</td>
<td>220.87 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>5.21 : 1</td>
<td>1.69 : 1</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>1m 38.07s</td>
<td>4h 51m 59.13s</td>
</tr>
<tr>
<td><strong>Compression speed</strong></td>
<td>3891.19 KB/sec</td>
<td>21.78 KB/sec</td>
</tr>
<tr>
<td><strong>Max ACL error</strong></td>
<td>0.0625 cm</td>
<td>8392339 cm</td>
</tr>
<tr>
<td><strong>Max UE4 error</strong></td>
<td>108904.6797 cm</td>
<td>8397727 cm</td>
</tr>
</tbody>
</table>
<p>It takes almost <strong>5 hours</strong> to compress with UE4! In comparison, ACL zips through in well under <strong>2 minutes</strong>. While it tries its best with the default settings it ultimately ends up using the safety fallback and thus compresses <em>twice</em> in that amount of time.</p>
<p>Overall, if you added the ACL codec to the <em>Automatic Compression</em> list, here is how it would perform:</p>
<ul>
<li>ACL is smaller for <strong>7711</strong> clips (<strong>92.86 %</strong>)</li>
<li>ACL is more accurate for <strong>7576</strong> clips (<strong>91.23 %</strong>)</li>
<li>ACL has faster compression for <strong>5704</strong> clips (<strong>68.69 %</strong>)</li>
<li>ACL is smaller, better, and faster for <strong>5017</strong> clips (<strong>60.42 %</strong>)</li>
<li>ACL wins Automatic Compression for <strong>7863</strong> clips (<strong>94.69 %</strong>)</li>
</ul>
<h1 id="decompression-stats">Decompression stats</h1>
<p><em>Fortnite</em> has the handy ability to create replays. These make gathering deterministic profiling numbers a breeze. The numbers that follow are from a <em>50 vs 50</em> replay. On each platform, a section of the replay with some high intensity action was profiled.</p>
<h2 id="desktop">Desktop</h2>
<p><img src="/public/acl/fortnite_pc_decompression_time.png" alt="Fortnite Desktop Decompression Time" /></p>
<p>The performance on desktop looks pretty good. ACL is consistently faster, about <strong>38%</strong> on average. It also appears a bit less noisy, a direct benefit of the improved cache friendliness of its algorithm.</p>
<h2 id="samsung-s8">Samsung S8</h2>
<p><img src="/public/acl/fortnite_s8_decompression_time_both.png" alt="Fortnite Samsung S8 Decompression Time" /></p>
<p><img src="/public/acl/fortnite_s8_decompression_time_acl.png" alt="Fortnite Samsung S8 ACL Decompression Time" /></p>
<p>ACL really shines on mobile. On average it is <strong>56%</strong> faster but that is only part of the story. On the S8 it appears that the core is hyperthreaded and another thread does heavy work and applies cache pressure. This causes all sorts of spikes with UE4 but in comparison, the cache aware ACL allows it to maintain a clean and consistent performance.</p>
<p>Hyperthreading on the CPU (and the GPU) works, roughly speaking, by the processor switching to another thread already executing when it notices that the current thread is stalled waiting on a slow operation, typically when memory needs to be pulled into the cache. Both threads are executing in the sense that they have data being held in registers but only one of them advances at a time on that core. When one stalls, the other executes.</p>
<p>When you have a piece of code that triggers a lot of cache misses, such as some of the legacy UE4 codecs, the processor will be more likely to switch to the other hyperthread. When this happens, execution is suspended and it will only resume once the other thread stalls or the time slice expires. This could be a long time especially if the other hyperthread is executing cache friendly code and doesn’t otherwise stall often.</p>
<p>This translates into the type of graph above where there is heavy fluctuation as the execution time varies widely from the noise of the neighbor hyperthread.</p>
<p>On the other hand, when the code is cache friendly, it doesn’t give the chance to the other thread to run. This gives a nice and smooth graph for that current thread as the risk of long interruptions is reduced. When the code is that optimized, hyperthreading typically doesn’t help speed things up much as both threads compete for the same time slice with few opportunities to hide stalling latency. This is also what I observed when <a href="http://nfrechette.github.io/2018/01/10/acl_v0.6.0/">measuring the compression performance</a>. In theory due to the higher cache pressure, performance could even degrade with hyperthreading but in practice I haven’t observed it, not with ACL at least.</p>
<h2 id="xbox-one">Xbox One</h2>
<p><img src="/public/acl/fortnite_xb1_decompression_time.png" alt="Fortnite Xbox One Decompression Time" /></p>
<p>On the <em>Xbox One</em> ACL is about <strong>13%</strong> faster on average. Both lines seem to have very similar shapes unlike the previous two platforms due in large part to the absence of hyperthreading. There are a few possibilities as to why the gain isn’t as significant on this platform:</p>
<ul>
<li>The MSVC compiler does not generate assembly that is as clean as it generates on PC, it’s certainly sub-optimal on a few points. It fails to inline many trivial functions and it leaves around unnecessary shuffle instructions.</li>
<li>Perhaps the other threads that share the L2 thrash the hardware prefetcher, preventing it from kicking in. ACL benefits heavily from hardware prefetching.</li>
<li>The CPU is quite slow compared to the speed of its memory. This reduces the benefit of cache locality as it keeps L2 cache misses fairly cheap in comparison.</li>
</ul>
<p>The last two points seems the most likely culprits. ACL does a bit more work per sample decompressed than UE4 but everything is cache friendly. This gives it a massive edge when memory is slow compared to the CPU clock as is the case on my desktop, the <em>Samsung S8</em>, and lots of other platforms.</p>
<h1 id="conclusion">Conclusion</h1>
<p>With faster compression, faster decompression on every platform, a smaller memory footprint, and higher accuracy, ACL continues to shine in UE4 and it won’t be long now before everyone can find it on the marketplace for free.</p>
<p>In the meantime, in the next few months I will perform another release of ACL and its plugin with all the latest fixes made possible with <em>Fortnite’s</em> data.</p>
<h1 id="status-update">Status update</h1>
<p>To my knowledge, the first game released with ACL came out in <em>November 2018</em> with the public UE4 plugin: <a href="https://store.steampowered.com/app/717690/OVERKILLs_The_Walking_Dead/">OVERKILL’s The Walking Dead</a>. I was told it reduced their animation memory footprint by over <strong>50%</strong> helping them fit within their console budgets.</p>
<p>A number of people have also integrated it into their own custom game engines and although I have no idea if they are using it or not, <a href="https://github.com/Remedy-Entertainment/acl">Remedy Entertainment</a> has forked ACL!</p>
<p>Last but not least, I’d like to extend a special shout-out to <em>Epic</em> for allowing me to do this and to the ACL contributors!</p>
Introducing Realtime Math v1.02019-01-19T00:00:00+00:00http://nfrechette.github.io/2019/01/19/introducing_realtime_math<p>Almost two years ago now, I began writing the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a>. I set out to build it to be production quality which meant I needed a whole lot of optimized math. At the time, I took a look at the landscape of math libraries and I opted to roll out my own. It has served me well, propelling ACL to success with some of the <a href="http://nfrechette.github.io/2018/09/08/acl_v1.1.0/">fastest compression and decompression</a> performance in the industry. I am now proud to announce that the code has been refactored out into its own open source library: <a href="https://github.com/nfrechette/rtm">Realtime Math v1.0 (RTM)</a> (MIT license).</p>
<p>There were a few reasons that motivated the choice to move the code out on its own:</p>
<ul>
<li>A significant amount of the ACL Continuous Integration build time is compiling and running the math unit tests which slows things down a bit more than I’d like</li>
<li>It decouples code that will benefit from being on its own</li>
<li>I believe it has its place in the landscape of math libraries out there</li>
</ul>
<p>In order to support that last point, I reviewed <strong>9</strong> other popular and usable math libraries for realtime applications. I looked at these with the lenses of my own needs and experience, your mileage may vary.</p>
<p><em>Disclaimer: the list of reviewed libraries is in no way exhaustive but I believe it is representative. Note that Unreal Engine 4 is included for informational purposes as it isn’t really usable on its own. Libraries are listed in no particular order and I tried to be as objective as possible. If you spot any inaccuracies, don’t hesitate to reach out.</em></p>
<p>The list: <em>Realtime Math, MathFu, vectorial, VectorialPlusPlus, C OpenGL Graphics Math (CGLM), OpenGL Graphics Math (GLM), Industrial Light & Magic Base (ILMBase), DirectX Math, and Unreal Engine 4</em>.</p>
<h2 id="tldr-how-realtime-math-stands-out">TL;DR: How Realtime Math stands out</h2>
<p>I believe <em>Realtime Math</em> stands out for a few reasons.</p>
<p>It is geared for high performance, deeply hot code. Most functions will end up inlined but the price to pay is an API that is a bit more verbose as a result of being C-style. When the need arises to use intrinsics, it gets out of the way and lets you do your thing. Only two libraries had what I would call optimal inlinability: <em>Realtime Math</em> and <em>DirectX Math</em>. Only those two libraries properly support the <em>__vectorcall</em> calling convention explicitly and only RTM handles GCC and Clang argument passing explicitly.</p>
<p>While it still needs a bit of love, quaternions are a first class citizen and it is the only standalone open source library I could find that supports QVV transforms (a rotation quaternion, a 3d scale vector, and a translation vector).</p>
<p><em>Realtime Math</em> uses a coding style similar to the C++ standard library and feels clean and natural to read and write.</p>
<p>It consists entirely of C++11 headers, it runs almost everywhere, it supports 64 bit floating point arithmetic, and it sports a very permissive MIT license.</p>
<h2 id="license">License</h2>
<p>ACL is open source and uses the MIT license. I am never keen on adding dependencies and if I really have to, I want a permissive license free of constraints.</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>License</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>MIT</td>
</tr>
<tr>
<td>MathFu</td>
<td>Apache 2.0</td>
</tr>
<tr>
<td>vectorial</td>
<td>BSD 2-clause</td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>BSD 2-clause</td>
</tr>
<tr>
<td>CGLM</td>
<td>MIT</td>
</tr>
<tr>
<td>GLM</td>
<td>Modified MIT</td>
</tr>
<tr>
<td>ILMBase</td>
<td>Custom but permissive</td>
</tr>
<tr>
<td>DirectX Math</td>
<td>MIT</td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>UE4 EULA</td>
</tr>
</tbody>
</table>
<h2 id="header-only">Header only</h2>
<p>For simplicity and ease of integration, I want ACL to be entirely made of C++11 headers. This also constrains any dependencies to the same requirement.</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Header Only</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>Yes</td>
</tr>
<tr>
<td>MathFu</td>
<td>Yes</td>
</tr>
<tr>
<td>vectorial</td>
<td>Yes</td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>Yes</td>
</tr>
<tr>
<td>CGLM</td>
<td>Yes (optional lib)</td>
</tr>
<tr>
<td>GLM</td>
<td>Yes</td>
</tr>
<tr>
<td>ILMBase</td>
<td>No</td>
</tr>
<tr>
<td>DirectX Math</td>
<td>Yes</td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>No</td>
</tr>
</tbody>
</table>
<h2 id="verbosity-readability-and-power">Verbosity, readability, and power</h2>
<p>An important requirement for a math library is to be reasonably concise with average code without getting in the way if the need arises to dive right into raw intrinsics. In my experience, general math type abstractions take you very far but in order to squeeze out every cycle it is sometimes necessary to write custom per platform code. When this is required, it is important for the library to not hide its internals and leave the door open.</p>
<p>I am personally more a fan of C-style interfaces for a math library for various reasons: I can infer very well what happens under the hood (I have seen many libraries make <em>fancy</em> use of some operators that leave many newcomers to wonder what they do) and they are optimal for performance as we will discuss later. The downside of course is that they tend to be a bit more verbose. However, this largely boils down to a matter of personal taste.</p>
<p><em>vectorial</em> is one of the few libraries that offers both a C-style interface and C++ wrappers and at the other end of the spectrum <em>DirectX Math</em> has both a namespace and a prefix for every type, constant and function.</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Verbosity</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>Medium (C-style)</td>
</tr>
<tr>
<td>MathFu</td>
<td>Light (C++ wrappers)</td>
</tr>
<tr>
<td>vectorial</td>
<td>Light (C++ wrappers) and Medium (C-style)</td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>Light (C++ wrappers)</td>
</tr>
<tr>
<td>CGLM</td>
<td>Medium (C-style)</td>
</tr>
<tr>
<td>GLM</td>
<td>Light (C++ wrappers)</td>
</tr>
<tr>
<td>ILMBase</td>
<td>Light (C++ wrappers)</td>
</tr>
<tr>
<td>DirectX Math</td>
<td>Medium++ (C-style with prefix and namespace)</td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>Light (C++ wrappers)</td>
</tr>
</tbody>
</table>
<p>It is very common for C-style math APIs to <em>typedef</em> their types to the underlying SIMD type. <em>Realtime Math</em>, <em>DirectX Math</em>, and many others do this. While this is great for performance, it does raise one problem: type safety is reduced. While usually those interfaces will opt to not expose proper vector2 or vector3 types and instead rely on functions that simply ignore the extra components, it doesn’t work so well when vector4 and quaternions are mixed. Only <em>Realtime Math</em>, <em>DirectX Math</em> and <em>CGLM</em> have quaternions with C-style interfaces but only the first two have a distinct type for quaternions when SIMD intrinsics are disabled. This somewhat mitigates the issue because with both <em>Realtime Math</em> and <em>DirectX Math</em> you can compile without intrinsics and still have type safety validated there. Although at the end of the day, all three have functions with distinct prefixes for vector and quaternion math and as such type safety is unlikely to be an issue.</p>
<h2 id="type-and-feature-support">Type and feature support</h2>
<p>By virtue or being an animation compression library, ACL’s needs are a bit different from a traditional realtime application. This dictated the need I had for specific types and features. I had no need for general 3x3 or 4x4 matrices as well as 2D vectors which are more commonly used in gameplay and rendering. However, 3x4 affine matrices, 3D and 4D vectors, quaternions, and QVV transforms (a quaternion, a vector3 translation, and a vector3 scale) are of critical importance. Those types are front and center in an animation runtime and I needed them to be fully featured and fast. Most of the libraries under review had way more features than I cared for (mostly for rendering) but generally missed proper or any support for quaternions and QVV transforms.</p>
<p><em>MathFu appears to have a <a href="https://github.com/google/mathfu/issues/33">bug</a> where the Matrix 4x4 SIMD template specialization isn’t included by default and its <a href="https://github.com/google/mathfu/issues/34">quaternions are 32 bytes</a> instead of the ideal 16 due to alignment constraints.</em></p>
<p><em>VectorialPlusPlus quaternions also take 32 bytes instead of 16 due to alignment constraints and most of their quaternion code appears to be scalar.</em></p>
<p><em>UE 4 is notable for being the only other library to support QVV and it does offer a VectorRegister type to support SIMD for Vector2/3/4 although most of the code written in the engine uses the scalar version.</em></p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Vector2</th>
<th>Vector3</th>
<th>Vector4</th>
<th>Quaternion</th>
<th>Matrix 3x3</th>
<th>Matrix 4x4</th>
<th>Matrix 3x4</th>
<th>QVV</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td> </td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
</tr>
<tr>
<td>MathFu</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>Partial SIMD</td>
<td>Scalar</td>
<td>SIMD</td>
<td>Scalar</td>
<td> </td>
</tr>
<tr>
<td>vectorial</td>
<td> </td>
<td>SIMD</td>
<td>SIMD</td>
<td> </td>
<td> </td>
<td>SIMD</td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>Scalar</td>
<td>SIMD</td>
<td>SIMD</td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>CGLM</td>
<td> </td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td> </td>
</tr>
<tr>
<td>GLM</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>Partial SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td> </td>
</tr>
<tr>
<td>ILMBase</td>
<td>Scalar</td>
<td>Scalar</td>
<td>Scalar</td>
<td>Scalar</td>
<td>Scalar</td>
<td>Scalar</td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>DirectX Math</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td>SIMD</td>
<td> </td>
<td>SIMD</td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>Scalar</td>
<td>Scalar</td>
<td>Scalar</td>
<td>SIMD</td>
<td> </td>
<td>SIMD</td>
<td> </td>
<td>SIMD</td>
</tr>
</tbody>
</table>
<h2 id="simd-architecture-support">SIMD architecture support</h2>
<p>Equally important was the SIMD architecture support. I want to run ACL everywhere with the best performance possible, especially on mobile. SSE, AVX, and NEON are all equally important to me.</p>
<p><em>Worth noting that 2 years ago DirectX NEON support appeared almost exclusively to be for Windows ARM NEON and I have no idea if it runs on iOS or Android even today.</em></p>
<table>
<thead>
<tr>
<th>Library</th>
<th>SSE</th>
<th>AVX</th>
<th>NEON</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td>MathFu</td>
<td>Yes</td>
<td> </td>
<td>Yes</td>
</tr>
<tr>
<td>vectorial</td>
<td>Yes</td>
<td> </td>
<td>Yes</td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>Yes</td>
<td> </td>
<td>Partial</td>
</tr>
<tr>
<td>CGLM</td>
<td>Yes</td>
<td>Yes</td>
<td>Partial</td>
</tr>
<tr>
<td>GLM</td>
<td>Yes</td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>ILMBase</td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td>DirectX Math</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
</tbody>
</table>
<h2 id="platform-and-compiler-support">Platform and compiler support</h2>
<p>Here things are a bit more complicated as libraries will list platforms but not compilers or compilers but not platforms. I need ACL to run everywhere and this means limiting myself to C++11 features.</p>
<ul>
<li>Realtime Math: Windows (VS2015, VS2017) x86 and x64, Linux (gcc5, gcc6, gcc7, gcc8, clang4, clang5, clang6) x86 and x64, OS X (Xcode 8.3, Xcode 9.4, Xcode 10.1) x86 and x64, Android clang ARMv7-A and ARM64, iOS (Xcode 8.3, Xcode 9.4, Xcode 10.1) ARM64</li>
<li>MathFu: Windows, Linux, OS X, Android</li>
<li>vectorial: Unlisted but probably Windows, Linux, OS X, Android, and iOS</li>
<li>VectorialPlusPlus: Unlisted but probably Windows</li>
<li>CGLM: Windows, Unix, and probably everywhere</li>
<li>GLM: VS2013+, Apple Clang 6, GCC 4.7+, ICC XE 2013+, LLVM 3.4+, CUDA 7+</li>
<li>ILMBase: Unlisted but probably Windows, Linux, OS X</li>
<li>DirectX Math: VS2015 and VS2017, possibly elsewhere</li>
<li>Unreal Engine 4: Windows (VS2015, VS2017) x64, Linux x64, OS X x64, Android ARMv7-A (no NEON) and ARM64, iOS ARM64</li>
</ul>
<h2 id="continuous-integration-support">Continuous integration support</h2>
<p>Continuous integration is a critical part of modern software development especially with C++ when multiple platforms are supported and maintained.</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Continuous Integration</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>Yes</td>
</tr>
<tr>
<td>MathFu</td>
<td>No</td>
</tr>
<tr>
<td>vectorial</td>
<td>No</td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>No</td>
</tr>
<tr>
<td>CGLM</td>
<td>Yes</td>
</tr>
<tr>
<td>GLM</td>
<td>Yes</td>
</tr>
<tr>
<td>ILMBase</td>
<td>No</td>
</tr>
<tr>
<td>DirectX Math</td>
<td>No</td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>Not public</td>
</tr>
</tbody>
</table>
<h2 id="dependencies">Dependencies</h2>
<p>I’m not personally a big fan of pulling in tons of dependencies, especially for a math library. As mentioned earlier, the Unreal Engine 4 math library isn’t really usable on its own because of this but is included regardless.</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Dependencies</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td> </td>
</tr>
<tr>
<td>MathFu</td>
<td>vectorial (BSD 2-clause)</td>
</tr>
<tr>
<td>vectorial</td>
<td> </td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>HandyCPP (custom license)</td>
</tr>
<tr>
<td>CGLM</td>
<td> </td>
</tr>
<tr>
<td>GLM</td>
<td> </td>
</tr>
<tr>
<td>ILMBase</td>
<td> </td>
</tr>
<tr>
<td>DirectX Math</td>
<td> </td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>Unreal Engine 4</td>
</tr>
</tbody>
</table>
<h2 id="floating-point-support">Floating point support</h2>
<p>When I got started with ACL, I wasn’t sure at the time if 64 bit floating point arithmetic might offer superior accuracy or not and if it would be worth using. As a result, I needed the math code to support both float32 and float64 types for everything with a seamless API between the two for quick testing. It later <a href="http://nfrechette.github.io/2017/12/29/acl_research_arithmetic/">turned out</a> that the extra floating point precision isn’t helping enough to be worth using.</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Float 32 Support</th>
<th>Float 64 Support</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>Yes</td>
<td>Yes (partial SIMD)</td>
</tr>
<tr>
<td>MathFu</td>
<td>Yes</td>
<td>Yes (no SIMD)</td>
</tr>
<tr>
<td>vectorial</td>
<td>Yes</td>
<td> </td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>Yes</td>
<td>Yes (partial SIMD)</td>
</tr>
<tr>
<td>CGLM</td>
<td>Yes</td>
<td> </td>
</tr>
<tr>
<td>GLM</td>
<td>Yes</td>
<td> </td>
</tr>
<tr>
<td>ILMBase</td>
<td>Yes (no SIMD)</td>
<td>Yes (no SIMD)</td>
</tr>
<tr>
<td>DirectX Math</td>
<td>Yes</td>
<td> </td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>Yes</td>
<td> </td>
</tr>
</tbody>
</table>
<h2 id="inlinability">Inlinability</h2>
<p>Due to the critical need for ACL to be as fast as possible on every platform, having the bulk of the math operations be inline is very important. Many things impact whether a function is inlined by the compiler but two stand out:</p>
<ul>
<li>Simple and short functions inline better</li>
<li>Passing arguments by register needs fewer instructions which inlines better</li>
</ul>
<p>Thankfully, most math function are fairly simple and short: add, mul, div, etc. C-style functions will generally have a slight advantage over C++ wrappers mainly because they also must track the implicit <em>this</em> pointer being passed around even if ultimately it is optimized out inside the caller. When the compiler needs to determine if it can inline a function, it uses a heuristic and the size of the intermediate assembly/IR/AST most likely plays a role. Generally speaking, C++ wrapper functions that are short will inline just fine but some operations have a harder time due to their size: matrix 4x4 multiplication, quaternion multiplication, and quaternion interpolation. For this reason, I personally favor a C-style API for this sort of code.</p>
<p>The second point is not to be underestimated. Most of the libraries in the list either take the arguments by value or by <em>const</em> reference. While passing SIMD types by value does the right thing on ARM and passes them by register (up to 4), it does not work for aggregate types like matrices and it does not work with the default <em>x64</em> calling convention with MSVC. In order to be able to pass SIMD types by register with MSVC, you must use its <em>__vectorcall</em> <a href="https://msdn.microsoft.com/en-us/library/dn375768.aspx">calling convention</a>. It also works for aggregate and wrapper types. Up to <strong>6</strong> registers can be used for this. On desktop and Xbox One, using <em>__vectorcall</em> is critical for high performance code and sadly, most libraries do not support it explicitly (and not all support it implicitly if the whole compilation unit is forced to use that calling convention). With <em>Visual Studio 2015</em>, <em>__vectorcall</em> is the difference between having quaternion interpolation getting inlined or not. When I added support for it in ACL, I measured a roughly <strong>5%</strong> speedup during the decompression.</p>
<p>Note that once a function is inlined, whether the arguments are passed by register or not typically does not impact the generated assembly although it sometimes does (at least with MSVC especially when AVX is enabled).</p>
<p><em>Some libraries which use a generic vector template class with specializations for SIMD (like MathFu) sometime end up passing *float32</em> arguments by const-reference instead of by value which is often suboptimal when not inlined.*</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Inlinability</th>
<th>Register Passing</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>Optimal (C-style + by register)</td>
<td>Explicit (everywhere)</td>
</tr>
<tr>
<td>MathFu</td>
<td>Decent (C++ wrappers)</td>
<td>None</td>
</tr>
<tr>
<td>vectorial</td>
<td>Good (C-style), Decent (C++ wrappers)</td>
<td>Implicit (C-style and ARM only)</td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>Decent (C++ wrappers)</td>
<td>None</td>
</tr>
<tr>
<td>CGLM</td>
<td>Good (C-style)</td>
<td>None</td>
</tr>
<tr>
<td>GLM</td>
<td>Decent (C++ wrappers)</td>
<td>None</td>
</tr>
<tr>
<td>ILMBase</td>
<td>Decent (C++ wrappers)</td>
<td>None</td>
</tr>
<tr>
<td>DirectX Math</td>
<td>Optimal (C-style + by register)</td>
<td>Explicit (vectorcall and ARM only)</td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>Decent (C++ wrappers)</td>
<td>None</td>
</tr>
</tbody>
</table>
<h2 id="multiplication-order">Multiplication order</h2>
<p>An important point of contention is how things are multiplied. As the list below shows, the OpenGL way is by far the most popular for open source math libraries.</p>
<p>It all boils down to whether vectors are represented as a <em>row</em> or as a <em>column</em>. In the former case, multiplication with a matrix takes the form <code class="language-plaintext highlighter-rouge">v' = vM</code> while in the later case we have <code class="language-plaintext highlighter-rouge">v' = Mv</code>. Linear algebra typically treats vectors as <em>columns</em> and OpenGL opted to use that convention for <a href="http://steve.hollasch.net/cgindex/math/matrix/column-vec.html">that reason</a>. If you think of matrices as functions that modify an input and return an output it ends up reading like this: <code class="language-plaintext highlighter-rouge">result = object_to_world(local_to_object(input))</code>. This reads right-to-left as is common with nested function evaluation. In my opinion, this is quite awkward to work with as most modern programming languages (and western languages) read left-to-right. Most linear algebra formulas use abstract letters and names for things which somewhat hides this nuance but when I write code, I try to keep my matrix names as clear as possible: what space are the input and output in. While you could technically reverse the naming <code class="language-plaintext highlighter-rouge">result = world_from_object * object_from_local * input</code> so it at least reads decently right-to-left, it’s still harder to reason with because just about everything we work with in the world goes from somewhere to somewhere else and not the other way around: trains, buses, planes, Monday to Friday, 5@7, etc.</p>
<p>On the other hand, DirectX uses <em>row</em> vectors and ends up with the much more natural: <code class="language-plaintext highlighter-rouge">result = input * local_to_object * object_to_world</code>. Your input is in local space, it gets transformed into object space before finally ending up in world space. Clean, clear, and readable. If you instead multiply the two matrices together on their own, you get the clear <code class="language-plaintext highlighter-rouge">local_to_world = local_to_object * object_to_world</code> instead of the awkward <code class="language-plaintext highlighter-rouge">local_to_world = object_to_world * local_to_object</code> you would get with OpenGL and <em>column</em> vectors.</p>
<p>At the end of the day, which way you choose largely boils down to a personal choice (or whatever library you use for rendering) as I don’t think there’s a big performance difference between the two on modern hardware. For ACL, all its output data is in local space and although we evaluate the error in world space internally, this is entirely transparent to the client application and it is free to use either convention.</p>
<table>
<thead>
<tr>
<th>Library</th>
<th>Multiplication Style</th>
</tr>
</thead>
<tbody>
<tr>
<td>Realtime Math</td>
<td>DirectX</td>
</tr>
<tr>
<td>MathFu</td>
<td>OpenGL</td>
</tr>
<tr>
<td>vectorial</td>
<td>OpenGL</td>
</tr>
<tr>
<td>VectorialPlusPlus</td>
<td>OpenGL</td>
</tr>
<tr>
<td>CGLM</td>
<td>OpenGL</td>
</tr>
<tr>
<td>GLM</td>
<td>OpenGL</td>
</tr>
<tr>
<td>ILMBase</td>
<td>OpenGL</td>
</tr>
<tr>
<td>DirectX Math</td>
<td>DirectX</td>
</tr>
<tr>
<td>Unreal Engine 4</td>
<td>DirectX</td>
</tr>
</tbody>
</table>
<h2 id="conclusion">Conclusion</h2>
<p>Ultimately, which math library you choose for a particular project boils down to a matter of personal preference to a large extent. For the vast majority of the code you’ll write, the performance and code generation is likely to be very close if not identical. Two years ago, I knew regardless of which option I picked I would have to do a lot of work to add what was missing. This greatly motivated me to just start from scratch as many middleware do and I do not regret the experience or results.</p>
<p>My top two favorite libraries are <em>Realtime Math</em> and <em>DirectX Math</em>. Both are quite similar today although <em>DirectX Math</em> wasn’t quite as attractive when I started.</p>
<h2 id="next-steps">Next steps</h2>
<p>Over the next few days I will populate various issues on GitHub to document things that are missing or that could benefit from some love.</p>
<p>A core part that is partially missing at the moment is the quantization and packing logic that ACL already contains. I have not migrated that code yet in large part because I am not sure how to best expose it in a clean and consistent API. I do believe it belongs in RTM where everyone can benefit from it.</p>
<p>ACL does not yet use RTM but that migration is planned for ACL <a href="https://github.com/nfrechette/acl/issues/170">v2.0</a>.</p>
Smaller, faster: ACL lets you cut your animation costs in half2018-09-08T00:00:00+00:00http://nfrechette.github.io/2018/09/08/acl_v1.1.0<p>I am excited to announce the open source <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> has reached <a href="https://github.com/nfrechette/acl/releases/tag/v1.1.0">v1.1</a> along with an updated <a href="https://github.com/nfrechette/acl-ue4-plugin/releases/tag/v0.2.0">Unreal Engine 4 plugin v0.2</a>.</p>
<p>ACL now beats <em>Unreal Engine 4</em> on all the important metrics. The plugin is <strong>1.7x</strong> smaller, <strong>3x</strong> more accurate, <strong>2.5x</strong> faster to compress, and up to <strong>4.8x</strong> faster to decompress!</p>
<h1 id="whats-new">What’s new</h1>
<p>The latest release focused on decompression performance. <em>ACL v1.1</em> is about <strong>30%</strong> faster than the previous version on every platform when decompressing a whole pose. Decompressing a single bone, a common operation in Unreal, is now about <strong>3-4x</strong> faster. Also, <em>ARM-based</em> products will now use <em>NEON SIMD</em> acceleration when available.</p>
<p>The UE4 plugin was reworked to be more tightly integrated and is about <strong>twice as fast</strong> compared to the previous version.</p>
<p>ACL has now reached a point where I can confidently say that it is the best overall animation compression algorithm in the video games industry. While other techniques might beat ACL on some of these metrics, beating it simultaneously on <em>speed, size, and accuracy</em> will prove to be very challenging. In particular, unlike other algorithms that offer very fast decompression, ACL has <em>no extra runtime memory cost</em> beyond the compressed clip data.</p>
<h1 id="new-data">New data!</h1>
<p>One year ago, <em>Epic</em> generously agreed to let me use the <a href="https://www.unrealengine.com/en-US/paragon">Paragon</a> animations <a href="http://nfrechette.github.io/2017/12/05/acl_paragon/">for research purposes</a>. This helped me find and fix bugs in <em>Unreal Engine</em> and ACL, and see how well both animation compression approaches perform in a real game. <em>Paragon</em> also allows each release to be rigorously tested against a large, relevant, and varied data set.</p>
<p>I am excited to announce that <em>Epic</em> is allowing me to use <a href="https://www.epicgames.com/fortnite/en-US/home">Fortnite</a> to further my research as well! While <em>Paragon</em> will continue to play its role in tracking compression performance and regression testing, <em>Fortnite</em> will allow me to measure decompression performance in real world scenarios much more easily. Testing with <em>Fortnite</em> should highlight new ways ACL can be improved further.</p>
<h1 id="whats-next">What’s next</h1>
<p>I am shifting my focus to add animation compression plugin support to UE4 during the next few months. If everything goes well, when <strong>UE 4.22</strong> is released next year, I will be able to add the ACL plugin to the <em>Unreal Engine Marketplace</em> for everyone to use, for <strong>free</strong>.</p>
<p>Proper plugin support will remove overhead and help make ACL’s in-game decompression faster still.</p>
<p>Due to the rigorous testing and extensive statistics extraction every release now requires, I expect the release cycle to slow down. I will aim to perform non-bug fix releases about twice a year.</p>
<h1 id="compression-performance-overview">Compression performance overview</h1>
<p>Here is a quick glance of how well it performs on the <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/paragon_performance.md">animations from Paragon</a>:</p>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.2.0</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>226.09 MB</td>
<td>392.97 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>18.91 : 1</td>
<td>10.88 : 1</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>6h 4m 18.21s</td>
<td>15h 10m 23.56s</td>
</tr>
<tr>
<td><strong>Bone Error 99<sup>th</sup> percentile</strong></td>
<td>0.0116 cm</td>
<td>0.0328 cm</td>
</tr>
<tr>
<td><strong>Samples below 0.01 cm error threshold</strong></td>
<td>98.85 %</td>
<td>84.88 %</td>
</tr>
</tbody>
</table>
<p>The <strong>99<sup>th</sup></strong> percentile and the number of samples below the <strong>0.01 cm</strong> error threshold are calculated by measuring the world-space error of every bone at every sample in each of the <strong>6558</strong> animation clips. To put this into perspective, over <strong>99 %</strong> of the compressed data has an error lower than the width of a human hair. More details on how the error is measured can be found <a href="https://github.com/nfrechette/acl/blob/develop/docs/error_metrics.md">here</a>.</p>
<h1 id="decompression-performance-overview">Decompression performance overview</h1>
<p><a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/decompression_performance.md">Decompression performance</a> is currently tracked with the <a href="http://nfrechette.github.io/2017/10/05/acl_in_ue4/">Matinee fight scene</a>. The troopers have around 70 bones each while the main trooper has 541.</p>
<p><img src="/public/acl/acl_plugin_v020_decomp_s8_matinee.png" alt="Matinee S8 Median Performance" /></p>
<p>Much care was taken to ensure that ACL has consistent decompression performance. The following two images show the time taken to decompress a pose at every point of the Matinee fight scene which highlights how regular ACL is.</p>
<p><img src="/public/acl/acl_plugin_v020_decomp_x8_matinee_ue4_variance.png" alt="Matinee UE4 S8 Performance Variance" />
<img src="/public/acl/acl_plugin_v020_decomp_x8_matinee_acl_variance.png" alt="Matinee ACL S8 Performance Variance" /></p>
<p>It also has consistent decompression performance <a href="https://github.com/nfrechette/acl/blob/develop/docs/decompression_performance.md#uniformly-sampled-algorithm">regardless of the playback direction</a> and it works on every modern platform making it a safe choice when using it as the default algorithm in your games.</p>
<p>Overall, ACL is ideal for games with large amounts of animations playing concurrently such as those with large crowds, MMOs, and e-sports as well as those that run on mobile or slower platforms.</p>
Animation Compression Library: Release 1.0.02018-07-21T00:00:00+00:00http://nfrechette.github.io/2018/07/21/acl_v1.0.0<p>The long awaited <strong>ACL v1.0</strong> release is finally <a href="https://github.com/nfrechette/acl/releases/tag/v1.0.0">here</a>! And it comes with the brand new <a href="https://github.com/nfrechette/acl-ue4-plugin/releases/tag/v0.1.0">Unreal Engine 4 plugin v0.1</a>! It took over <strong>15</strong> months of late nights, days off, and weekends to reach this point and I couldn’t be more pleased with the results.</p>
<h1 id="recap">Recap</h1>
<p>The core idea behind ACL was to explore a different way to perform animation compression, one that departed from classic methods. Unlike the vast majority of algorithms in the wild, it uses bit aligned values as opposed to naturally aligned integers. This is slower to unpack but I hoped to compensate by not performing any sort of key reduction. By retaining every sample, the data is uniform in memory and offsets are trivially calculated, keeping things fast, the memory touched contiguous, and the hardware happy. While the technique itself isn’t novel and is often used with compression algorithms in other fields, to my knowledge it had never been tried to the extent ACL pushes it with animation compression, at least not publicly.</p>
<p>Very early, the technique proved competitive and over time it emerged as a superior alternative over traditional techniques involving key reduction. I then spent about 8 months writing the necessary infrastructure to make ACL not only production ready but production quality: unit tests were written, extensive regression tests were introduced, documentation was added as well as comments, scripts to replicate the results, cross platform support (ACL now runs on <strong><em>every</em></strong> platform!), etc. All that good stuff that one would expect from a professional product.</p>
<p>But don’t take my word for it! Check out the <strong>100% C++</strong> code (<em>MIT license</em>), the statistics below, and take the plugin out for a spin!</p>
<h1 id="performance">Performance</h1>
<p>While ACL provides various synthetic test hardnesses to benchmark and extract statistics, nothing beats running it within a real game engine. This is where the <em>UE4</em> plugin comes in and really shines. Just as with ACL, three data sets are measured: CMU, Paragon, and the Matinee fight scene.</p>
<p><em>Note that there are small differences between measuring with the UE4 plugin and with the ACL test harnesses due to implementation choices in the plugin.</em></p>
<h2 id="carnegie-mellon-university-cmu">Carnegie-Mellon University (CMU)</h2>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.1.0</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>70.60 MB</td>
<td>99.94 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>20.25 : 1</td>
<td>14.30 : 1</td>
</tr>
<tr>
<td><strong>Max error</strong></td>
<td>0.0722 cm</td>
<td>0.0996 cm</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>34m 30.51s</td>
<td>1h 27m 40.15s</td>
</tr>
</tbody>
</table>
<p>ACL was smaller for <strong>2532</strong> clips (<strong>99.92 %</strong>)<br />
ACL was more accurate for <strong>2486</strong> clips (<strong>98.11 %</strong>)<br />
ACL has faster compression for <strong>2534</strong> clips (<strong>100.00 %</strong>)<br />
ACL was smaller, better, and faster for <strong>2484</strong> clips (<strong>98.03 %</strong>)</p>
<p>Would the <em>ACL Plugin</em> have been included in the <em>Automatic Compression</em> permutations tried, it would have won for <strong>2534</strong> clips (<strong>100.00 %</strong>)</p>
<p>Data tracked <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/cmu_performance.md">here</a> by the plugin, and <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">here</a> by ACL.</p>
<h2 id="paragon">Paragon</h2>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.1.0</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>226.02 MB</td>
<td>392.97 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>18.92 : 1</td>
<td>10.88 : 1</td>
</tr>
<tr>
<td><strong>Max error</strong></td>
<td>0.8566 cm</td>
<td>0.6424 cm</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>6h 35m 03.24s</td>
<td>15h 10m 23.56s</td>
</tr>
</tbody>
</table>
<p>ACL was smaller for <strong>6413</strong> clips (<strong>97.79 %</strong>)<br />
ACL was more accurate for <strong>4972</strong> clips (<strong>75.82 %</strong>)<br />
ACL has faster compression for <strong>5948</strong> clips (<strong>90.70 %</strong>)<br />
ACL was smaller, better, and faster for <strong>4499</strong> clips (<strong>68.60 %</strong>)</p>
<p>Would the <em>ACL Plugin</em> have been included in the <em>Automatic Compression</em> permutations tried, it would have won for <strong>6098</strong> clips (<strong>92.99 %</strong>)</p>
<p>Data tracked <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/paragon_performance.md">here</a> by the plugin, and <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">here</a> by ACL.</p>
<h2 id="matinee-fight-scene">Matinee fight scene</h2>
<table>
<thead>
<tr>
<th> </th>
<th>ACL Plugin v0.1.0</th>
<th>UE v4.19.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Compressed size</strong></td>
<td>8.67 MB</td>
<td>23.67 MB</td>
</tr>
<tr>
<td><strong>Compression ratio</strong></td>
<td>7.20 : 1</td>
<td>2.63 : 1</td>
</tr>
<tr>
<td><strong>Max error</strong></td>
<td>0.0674 cm</td>
<td>0.0672 cm</td>
</tr>
<tr>
<td><strong>Compression time</strong></td>
<td>52.44s</td>
<td>54m 03.18s</td>
</tr>
</tbody>
</table>
<p>ACL was smaller for <strong>1</strong> clip (<strong>20 %</strong>)<br />
ACL was more accurate for <strong>4</strong> clips (<strong>80 %</strong>)<br />
ACL has faster compression for <strong>5</strong> clips (<strong>100 %</strong>)<br />
ACL was smaller, better, and faster for <strong>0</strong> clip (<strong>0 %</strong>)</p>
<p>Would the <em>ACL Plugin</em> have been included in the <em>Automatic Compression</em> permutations tried, it would have won for <strong>3</strong> clips (<strong>60 %</strong>)</p>
<p>Data tracked <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/fight_scene_performance.md">here</a> by the plugin, and <a href="https://github.com/nfrechette/acl/blob/develop/docs/fight_scene_performance.md">here</a> by ACL.</p>
<h2 id="decompression-performance">Decompression performance</h2>
<p><img src="/public/acl/acl_plugin_v010_decomp_s8_matinee.png" alt="Matinee S8 Median Performance" /></p>
<p><img src="/public/acl/acl_plugin_v010_decomp_s8_playground.png" alt="Playground S8 Median Performance" /></p>
<p>Data tracked <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/fight_scene_performance.md">here</a> by the plugin, and <a href="https://github.com/nfrechette/acl/blob/develop/docs/fight_scene_performance.md">here</a> by ACL (they also include other platforms and more data).</p>
<h2 id="performance-summary">Performance summary</h2>
<p>As the numbers clearly show, ACL beats <em>UE4</em> across every compression metric, sometimes by a significant margin: it is <em>MUCH</em> faster to compress, the quality is just as good, and the memory footprint is significantly reduced. ACL achieves all of this with default settings that animators rarely if <em>ever</em> need to tweak. What’s not to love?</p>
<p>However, the ACL decompression performance is sometimes ahead, sometimes behind, or the same. There are a few reasons for this, most of which I am hoping to fix in the next version to take the lead: <em>NEON</em> (<em>SIMD</em>) is not yet used on <em>ARM</em>, the ACL plugin needlessly performs <em>MUCH</em> more work than <em>UE4</em> when decompressing, and many low hanging fruits were left to be fixed post-1.0 release.</p>
<p><strong><em>ACL is just getting started!</em></strong></p>
<h1 id="how-to-use-the-acl-plugin">How to use the ACL Plugin</h1>
<p>As the documentation states <a href="https://github.com/nfrechette/acl-ue4-plugin/blob/develop/Docs/README.md">here</a>, a few minor engine changes are required in order to support the ACL plugin. These changes mostly consist of bug fixes and changes to expose the necessary hooks to plugins.</p>
<p>For the time being, the plugin is not yet on the marketplace as it is not fully plug-and-play. However, this summer I am working with <em>Epic</em> to introduce the necessary changes in order to publish the ACL plugin on the marketplace. <strong>Stay tuned!</strong></p>
<p><em>Note that the ACL Plugin will reach <strong>v1.0</strong> once it can be published on the marketplace but it is production ready regardless.</em></p>
<h1 id="whats-new-in-acl-v10">What’s new in ACL v1.0</h1>
<p>Few things actually changed in between <em>v0.8</em> and <em>v1.0</em>. Most of the changes revolved around minor additions, documentation updates, etc. There are two notable changes:</p>
<ul>
<li>The first is visible in the <a href="https://github.com/nfrechette/acl/blob/develop/docs/decompression_performance.md">decompression graphs</a>: we now yield the thread before measuring every sample. This helps ensure more stable results by reducing the likelihood that the kernel will swap out the thread and interrupt it while executing the decompression code.</li>
<li>The second is visible in the <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">compression stats</a> for Paragon: a bug was causing the visible error to sometimes be partially hidden when 3D scale is present. While the new version is not less accurate than the previous, the measured error can be higher in very rare cases (only <strong>1</strong> clip is higher).</li>
</ul>
<p>Regardless, the measuring should now be much more stable.</p>
<h1 id="whats-next">What’s next</h1>
<p>The next release of ACL will focus on improving the <a href="https://github.com/nfrechette/acl/milestone/6">compression and decompression performance</a>. While ACL was built from the ground up to be fast to decompress; so far the focus has been on making sure things function properly and safely to establish a solid baseline to work with. Now that this work is done, the fun part can begin: making it the best it can be! I have many improvements planned and while some of them will make it in <strong>v1.1</strong>, others will have to wait for future versions.</p>
<p>Special care will be taken to make sure ACL performs at its best in <em>UE4</em> but there is no reason why it couldn’t be used in your own favorite game engine or animation middleware. Developing with <em>UE4</em> is easier for me in large part because of my past experience with it, my relationship with <em>Epic</em>, and the fact that it is open source. Other game engines like Unity explicitly forbid their use for benchmarking purposes in their <em>EULA</em> which prevents me from publishing any results without prior written agreement form their legal departement. Furthermore, without access to the source code, creating a plugin for it requires a lot more work. In due time, I hope to support Unity, Godot, and anyone else willing to try it out.</p>
Animation Compression Library: Release 0.8.02018-05-12T00:00:00+00:00http://nfrechette.github.io/2018/05/12/acl_v0.8.0<p>Today marks the <strong>v0.8</strong> <a href="https://github.com/nfrechette/acl/releases/tag/v0.8.0">release</a> of the <em>Animation Compression Library</em>. It contains lots of goodies but by far the most significant point is the fact that it has now reached feature parity with <em>Unreal 4</em>. For the first time <em>Unreal 4</em> games should be able to run exclusively with <em>ACL</em>. The focus for the next 2 months will be to validate this with my custom <em>UE 4.15</em> integration and implement whatever might be missing as well as to create a proper and free plugin to bring <em>ACL</em> to the marketplace.</p>
<p>While I have already published some <a href="/2018/05/11/acl_decompression_baseline/">decompression performance numbers</a> earlier this week, once a proper integration has been made, new numbers will be published to showcase how <em>ACL</em> performs against <em>Unreal 4</em> within the game engine itself. The existing numbers for the <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">Carnegie-Mellon University</a> database, <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a>, and the <a href="https://github.com/nfrechette/acl/blob/develop/docs/fight_scene_performance.md">Matinee fight scene</a> already clearly show <em>ACL</em> to be ahead in terms of compression time, compression ratio, and accuracy. However, while it remains to be seen if it will also be ahead with its decompression performance, I fully expect that it will.</p>
ACL decompression performance2018-05-11T00:00:00+00:00http://nfrechette.github.io/2018/05/11/acl_decompression_baseline<p>At long last I finally got around to measuring the decompression performance of ACL. This blog post will detail the baseline performance from which we will measure future progress. As I have <a href="http://nfrechette.github.io/2018/04/02/acl_v0.7.0/">previously mentioned</a>, no effort has been made so far to optimize the decompression and I hope to remedy that following the v1.0 release scheduled around June 2018.</p>
<p>In order to establish a reliable data set to measure against, I use the same <strong>42</strong> clips used for <a href="https://github.com/nfrechette/acl/blob/develop/test_data/README.md">regression testing</a> plus <strong>5</strong> more from the <a href="https://github.com/nfrechette/acl/blob/develop/docs/fight_scene_performance.md">Matinee fight scene</a>. To keep things interesting, I measure performance on everything I have on hand:</p>
<ul>
<li>Desktop: <a href="https://ark.intel.com/products/94188/Intel-Core-i7-6850K-Processor-15M-Cache-up-to-3_80-GHz">Intel i7-6850K @ 3.8 GHz</a></li>
<li>Laptop: <a href="https://support.apple.com/kb/sp703?locale=en_US">MacBook Pro mid 2014 @ 2.6 GHz</a></li>
<li>Phone: <a href="https://www.androidcentral.com/nexus-5x-specs">Android Nexus 5X @ 1.8 GHz</a></li>
<li>Tablet: <a href="https://en.wikipedia.org/wiki/IPad_Pro">iPad Pro 10.5 inch @ 2.39 GHz</a></li>
</ul>
<p>The first two use both <em>x86</em> and <em>x64</em> while the later two use <em>armv7-a</em> and <em>arm64</em> respectively. Furthermore, on the desktop I also compare <em>VS 2015</em>, <em>VS 2017</em>, <em>GCC 7</em>, and <em>Clang 5</em>. The more data, the merrier!</p>
<p>Decompression is measured both with a warm CPU cache to remove the memory fetches as much as possible from the equation as well as with a cold CPU cache to simulate a more realistic game engine playback scenario.</p>
<p>Three forms of playback are measured: forward, backward, and random.</p>
<p>Each clip is sampled <strong>3</strong> times at every key frame based on the clip sample rate and the smallest value is retained for that key.</p>
<p>Finally, two ways to decompress are profiled: decompressing a whole pose in one go (<code class="language-plaintext highlighter-rouge">decompress_pose</code>), and decompressing a whole pose bone by bone (<code class="language-plaintext highlighter-rouge">decompress_bone</code>).</p>
<p>The profiling harness is not perfect but I hope the extensive data pulled from it will be sufficient for our purposes.</p>
<h2 id="playback-direction">Playback direction</h2>
<p>In a real game, the overwhelming majority of clips play forward in time. Some clips play backwards (e.g. opening and closing a chest might use the same animation played in reverse) and a few others play randomly (e.g. driven by a thumb stick).</p>
<p>Not all algorithms will exhibit the same performance regardless of playback direction. In particular, forms of delta encoding as well as any caching of the last played position will severely degrade when the playback isn’t the one optimized for (as is often the case with <a href="http://nfrechette.github.io/2016/12/10/anim_compression_curve_fitting/">key reduction</a> techniques due to the data being sorted by time).</p>
<p>ACL currently uses the <a href="https://github.com/nfrechette/acl/blob/develop/docs/algorithm_uniformly_sampled.md">uniformly sampled</a> algorithm which offers consistent performance regardless of the playback direction. To validate this claim, I hand picked <strong>3</strong> clips that are fairly long: <em>104_30</em> (<strong>44</strong> bones, <strong>11</strong> seconds) from CMU, and <em>Trooper_1</em> (<strong>71</strong> bones, <strong>66</strong> seconds) and <em>Trooper_Main</em> (<strong>541</strong> bones, <strong>66</strong> seconds) from the Matinee fight scene. To visualize the performance, I used a <a href="http://www.statisticshowto.com/probability-and-statistics/descriptive-statistics/box-plot/">box and whiskers chart</a> which shows concisely the min/max as well as the quartiles. Forward playback is shown in <strong>Red</strong>, backward in <strong>Green</strong>, and random in <strong>Blue</strong>.</p>
<p><img src="/public/acl/acl_decomp_v080_vs2015_x64_playback.png" alt="VS 2015 x64 Playback Performance" /></p>
<p>As we can see, the performance is identical for all intents and purposes regardless of the playback direction on my desktop with <em>VS 2015 x64</em>. Let’s see if this claim holds true on my <em>iPad</em> as well.</p>
<p><img src="/public/acl/acl_decomp_v080_ios_arm64_playback.png" alt="iOS arm64 Playback Performance" /></p>
<p>Here again we see that the performance is consistent. One thing that shows up on this chart is that, surprisingly, the <em>iPad</em> performance is <em>often</em> better than my <em>desktop</em>! That is <strong>INSANE</strong> and I nearly fell off my chair when I first saw this. Not only is the CPU clocked at a lower frequency but the desktop code makes use of <em>SSE</em> and <em>AVX</em> where it can for all basic vector arithmetic while there is currently no corresponding <em>NEON</em> SIMD support. I double and triple checked the numbers and the code. Suspecting that the compiler might be playing a big part in this, I undertook to dump all the compiler stats on desktop; something I did not originally intend to do. Read on!</p>
<h2 id="the-cpu-cache">The CPU cache</h2>
<p>Because animation clips are typically sampled once per rendered image, the CPU cache will generally always be cold during decompression. Fortunately for us, modern CPUs offer hardware prefetching which greatly helps when reads are linear. The uniformly sampled algorithm ACL uses is uniquely optimized for this with <strong>ALL</strong> reads being linear and split into <strong>4</strong> streams: constant track values, clip range data, segment range data, and the animated segment data.</p>
<p><em>Notes: ACL does not currently have any software prefetching and the constant track and clip range data will <a href="https://github.com/nfrechette/acl/issues/72">later</a> be merged into a single stream since a track is one of three types: default (in which case there is neither constant nor range data), constant with no range data, or animated with range data and thus not constant.</em></p>
<p>For this reason, a cold cache is what will most interest us. That being said, I also measured with a warm CPU cache. This will allow us to see how much time is spent waiting on memory versus executing instructions. It will also allow us to compare the various platforms in terms of CPU and memory speed.</p>
<p>In the following graphs, the <em>x86</em> performance was omitted because for every compiler it is slower than <em>x64</em> (ranging from 25% slower up to 200%) except for my <em>OS X</em> laptop where the performance was nearly identical. I also omitted the <em>VS 2017</em> performance because it was identical to <em>VS 2015</em>. Forward playback is used along with <code class="language-plaintext highlighter-rouge">decompress_pose</code>. The median decompression time is shown.</p>
<p>Two new clips were added to the graphs to get a better picture.</p>
<p><img src="/public/acl/acl_decomp_v080_cold_cache1.png" alt="Cold CPU Cache Performance" />
<img src="/public/acl/acl_decomp_v080_cold_cache2.png" alt="Cold CPU Cache Performance cont." /></p>
<p>Again, we can see that the <em>iPad</em> outperforms almost everything with a cold cache except on the desktop with <em>GCC 7</em> and <em>Clang 5</em>. It is clear that <em>Clang</em> does an outstanding job and plays an integral part in the surprising <em>iPad</em> performance. Another point worth noting is that its memory is faster than what I have in my desktop. My <em>iPad</em> has memory clocked at <strong>1600 MHz (25 GB/s)</strong> while my desktop has its memory clocked at <strong>1067 MHz (16.6 GB/s)</strong>.</p>
<p>And now with a warm cache:</p>
<p><img src="/public/acl/acl_decomp_v080_warm_cache1.png" alt="Warm CPU Cache Performance" />
<img src="/public/acl/acl_decomp_v080_warm_cache2.png" alt="Warm CPU Cache Performance cont." /></p>
<p>We can see that the <em>iPad</em> now loses out to <em>VS 2015</em> with one exception: <em>Trooper_Main</em>. Why is that? That particular clip should easily fit within the CPU cache: only about <strong>40KB</strong> is touched when sampling (or about <strong>650</strong> cache lines). Further research led to another interesting fact: the <em>iPad A10X</em> processor has a <strong>64KB L1</strong> data cache per core (and <strong>8 MB L2</strong> shared) while my <em>i7-6850K</em> has a <strong>32KB L1</strong> data cache and a <strong>256KB L2</strong> (with <strong>15MB L3</strong> shared). The clip thus fits entirely within the L1 on the <em>iPad</em> but needs to be fetched from the L2 on desktop.</p>
<p>Another takeaway from these graphs is that <em>GCC 7</em> beats <em>VS 2015</em> and <em>Clang 5</em> beats both hands down on my desktop.</p>
<p>Finally, my <em>Nexus 5X</em> is <strong>really</strong> slow. On all the graphs, it exceeded any reasonable scale and I had to truncate it. I included it for the sake of completeness and to get a sense of how much slower it was.</p>
<h2 id="decompression-method">Decompression method</h2>
<p>ACL currently offers two ways to decompress: <code class="language-plaintext highlighter-rouge">decompress_pose</code> and <code class="language-plaintext highlighter-rouge">decompress_bone</code>. The former is more efficient if the whole pose is required but in practice it is very common to decompress specific bones individually or to decompress a pose bone by bone.</p>
<p>The following charts use the median decompression time with a cold CPU cache and forward playback.</p>
<p><img src="/public/acl/acl_decomp_v080_function_performance_cmu.png" alt="Function Performance on CMU" />
<img src="/public/acl/acl_decomp_v080_function_performance_matinee.png" alt="Function Performance on the Matinee Fight" /></p>
<p>Once more, we see very clearly how outstanding and consistent the <em>iPad</em> performance is. The numbers for the <em>Nexus 5X</em> are very noisy in comparison in large part because of the slower memory and larger footprint of some clips (<code class="language-plaintext highlighter-rouge">decompress_bone</code> is not shown for <em>Android</em> because it was far too slow and prevented a clean view of everything else).</p>
<p>We can clearly see that decompressing each bone separately is much slower and this is entirely because at the time of writing, each bone not required needs to be skipped over instead of using a direct look up with an offset. This will be optimized soon and the performance should end up much closer.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Despite having no external reference frame to compare them against, I could confirm and validate my hunches as well as observe a few interesting things:</p>
<ul>
<li>My <em>Nexus 5X</em> is really slow …</li>
<li>Both <em>GCC 7</em> and <em>Clang 5</em> generate much better code than <em>VS 2017</em></li>
<li><code class="language-plaintext highlighter-rouge">decompress_bone</code> is much slower than it needs to be</li>
<li>The playback direction has no impact on performance</li>
</ul>
<p>By far the most surprising thing to me was the <em>iPad</em> performance. Even though what I measure is not representative of ordinary application code, the numbers clearly demonstrate that the single core decompression performance matches that of a modern desktop. It might even exceed the single core performance of an <em>Xbox One</em> or <em>PlayStation 4</em>! <strong>Wow!!</strong></p>
<p>I do have some baseline Unreal 4 numbers on hand but this blog post is already getting long and the next ACL version aims to be integrated into a native Unreal 4 plugin which will allow for a superior comparison to be made. However, they do show that ACL will be very close and will likely <strong>exceed</strong> the UE 4.15 decompression performance; stay tuned!</p>
How much does additive bind pose help?2018-05-08T00:00:00+00:00http://nfrechette.github.io/2018/05/08/anim_compression_additive_bind<p>A common trick when compressing an animation clip is to store it relative to the bind pose. The conventional wisdom is that this allows to <a href="/2016/11/09/anim_compression_range_reduction/">reduce the range of motion</a> of many bones, increasing the accuracy and the likelihood that constant bones will turn into the identity, and thus allowing a lower memory footprint as a result. I have implemented this specific feature many times in the past and the results were consistent: a memory reduction of <strong>3-5%</strong> was generally observed.</p>
<p>Now that the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> supports additive animation clips, I thought it would be a good idea to test this claim once more.</p>
<h2 id="how-it-works">How it works</h2>
<p>The concept is very simple to implement:</p>
<ul>
<li>Before compression happens, the bind pose is removed from the clip by subtracting it from every key.</li>
<li>Then, the clip is compressed as usual.</li>
<li>Finally, after we decompress a pose, we simply add back the bind pose.</li>
</ul>
<p>The transformation is lossless aside from whatever loss happens as a result of floating point precision. It has two primary side effects.</p>
<p>The first is that bone translations end up having a much shorter range of motion. For example, a walking character might have the pelvic bone about <strong>60cm</strong> up from the ground (and root bone). The range of motion will thus circle around this large value for the whole track. Removing the bind pose brings the track closer to zero since the bind pose value of that bone is likely very near <strong>60cm</strong>. Smaller floating point values generally retain higher accuracy. The principle is identical to normalizing a track within its range.</p>
<p>The second impacts constant tracks. If the pelvic bone is not animated in a clip, it will retain some constant value. This value is often the bind pose itself. When this happens, removing the bind pose yields the identity rotation and translation. Since these values are trivial to reconstruct at runtime, instead of having to store the constant floating point values, we can store a simple bit set.</p>
<p>As a result, hand animated clips with the bind pose removed often find themselves with a lower memory footprint following compression.</p>
<p>Mathematically speaking, how the bind pose is added or removed can be done in a number of ways, much like additive animation clips. While additive animation clips heavily depend on the animation runtime, ACL now supports <a href="https://github.com/nfrechette/acl/blob/develop/docs/additive_clips.md">three variants</a>:</p>
<ul>
<li>Relative space</li>
<li>Additive space 0</li>
<li>Additive space 1</li>
</ul>
<p><em>The last two names are not very creative or descriptive… Suggestions welcome!</em></p>
<h2 id="relative-space">Relative space</h2>
<p>In this space, the clip is reconstructed by multiplying the bind pose with a normal <code class="language-plaintext highlighter-rouge">transform_mul</code> operation. For example, this is the same operation used to convert from local space to object space. Performance wise, this is the slowest: to reconstruct our value we end up having to perform <strong>3</strong> quaternion multiplications and if negative scale is present in the clip, it is even slower (extra code not shown below, see <a href="https://github.com/nfrechette/acl/blob/develop/includes/acl/math/transform_32.h">here</a>).</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Transform</span> <span class="nf">transform_mul</span><span class="p">(</span><span class="k">const</span> <span class="n">Transform</span><span class="o">&</span> <span class="n">lhs</span><span class="p">,</span> <span class="k">const</span> <span class="n">Transform</span><span class="o">&</span> <span class="n">rhs</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Quat</span> <span class="n">rotation</span> <span class="o">=</span> <span class="n">quat_mul</span><span class="p">(</span><span class="n">lhs</span><span class="p">.</span><span class="n">rotation</span><span class="p">,</span> <span class="n">rhs</span><span class="p">.</span><span class="n">rotation</span><span class="p">);</span>
<span class="n">Vector4</span> <span class="n">translation</span> <span class="o">=</span> <span class="n">vector_add</span><span class="p">(</span><span class="n">quat_rotate</span><span class="p">(</span><span class="n">rhs</span><span class="p">.</span><span class="n">rotation</span><span class="p">,</span> <span class="n">vector_mul</span><span class="p">(</span><span class="n">lhs</span><span class="p">.</span><span class="n">translation</span><span class="p">,</span> <span class="n">rhs</span><span class="p">.</span><span class="n">scale</span><span class="p">)),</span> <span class="n">rhs</span><span class="p">.</span><span class="n">translation</span><span class="p">);</span>
<span class="k">return</span> <span class="n">transform_set</span><span class="p">(</span><span class="n">rotation</span><span class="p">,</span> <span class="n">translation</span><span class="p">,</span> <span class="n">scale</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="additive-space-0">Additive space 0</h2>
<p>This is the first of the two classic additive spaces. It simply multiplies the rotations, it adds the translations, and multiplies the scales. The animation runtime <a href="http://guillaumeblanc.github.io/ozz-animation/">ozz-animation</a> uses this format. Performance wise, this is the fastest implementation.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Transform</span> <span class="nf">transform_add0</span><span class="p">(</span><span class="k">const</span> <span class="n">Transform</span><span class="o">&</span> <span class="n">base</span><span class="p">,</span> <span class="k">const</span> <span class="n">Transform</span><span class="o">&</span> <span class="n">additive</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Quat</span> <span class="n">rotation</span> <span class="o">=</span> <span class="n">quat_mul</span><span class="p">(</span><span class="n">additive</span><span class="p">.</span><span class="n">rotation</span><span class="p">,</span> <span class="n">base</span><span class="p">.</span><span class="n">rotation</span><span class="p">);</span>
<span class="n">Vector4</span> <span class="n">translation</span> <span class="o">=</span> <span class="n">vector_add</span><span class="p">(</span><span class="n">additive</span><span class="p">.</span><span class="n">translation</span><span class="p">,</span> <span class="n">base</span><span class="p">.</span><span class="n">translation</span><span class="p">);</span>
<span class="n">Vector4</span> <span class="n">scale</span> <span class="o">=</span> <span class="n">vector_mul</span><span class="p">(</span><span class="n">additive</span><span class="p">.</span><span class="n">scale</span><span class="p">,</span> <span class="n">base</span><span class="p">.</span><span class="n">scale</span><span class="p">);</span>
<span class="k">return</span> <span class="n">transform_set</span><span class="p">(</span><span class="n">rotation</span><span class="p">,</span> <span class="n">translation</span><span class="p">,</span> <span class="n">scale</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="additive-space-1">Additive space 1</h2>
<p>This last additive space combines the base pose in the same way as the previous except for the scale component. This is the format used by Unreal 4. Performance wise, it is very close to the previous space but requires an extra instruction or two.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Transform</span> <span class="nf">transform_add1</span><span class="p">(</span><span class="k">const</span> <span class="n">Transform</span><span class="o">&</span> <span class="n">base</span><span class="p">,</span> <span class="k">const</span> <span class="n">Transform</span><span class="o">&</span> <span class="n">additive</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Quat</span> <span class="n">rotation</span> <span class="o">=</span> <span class="n">quat_mul</span><span class="p">(</span><span class="n">additive</span><span class="p">.</span><span class="n">rotation</span><span class="p">,</span> <span class="n">base</span><span class="p">.</span><span class="n">rotation</span><span class="p">);</span>
<span class="n">Vector4</span> <span class="n">translation</span> <span class="o">=</span> <span class="n">vector_add</span><span class="p">(</span><span class="n">additive</span><span class="p">.</span><span class="n">translation</span><span class="p">,</span> <span class="n">base</span><span class="p">.</span><span class="n">translation</span><span class="p">);</span>
<span class="n">Vector4</span> <span class="n">scale</span> <span class="o">=</span> <span class="n">vector_mul</span><span class="p">(</span><span class="n">vector_add</span><span class="p">(</span><span class="n">vector_set</span><span class="p">(</span><span class="mf">1.0</span><span class="n">f</span><span class="p">),</span> <span class="n">additive</span><span class="p">.</span><span class="n">scale</span><span class="p">),</span> <span class="n">base</span><span class="p">.</span><span class="n">scale</span><span class="p">);</span>
<span class="k">return</span> <span class="n">transform_set</span><span class="p">(</span><span class="n">rotation</span><span class="p">,</span> <span class="n">translation</span><span class="p">,</span> <span class="n">scale</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It is worth noting that because these two additive spaces differ only by how they handle scale, if the animation clip has none, both methods will yield identical results.</p>
<h2 id="results">Results</h2>
<p>Measuring the impact is simple: I simply enabled all three modes one by one and compressed all of the <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">Carnegie-Mellon University</a> motion capture database as well as all of the <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a> data set. Decompression performance was not measured on its own but the compression time will serve as a hint as to how it would perform.</p>
<p>Everything has been measured with my desktop using <em>Visual Studio 2015</em> with <em>AVX</em> support enabled with up to <strong>4</strong> clips being compressed in parallel. All measurements were performed with the upcoming <em>ACL v0.8</em> release.</p>
<p><img src="/public/acl/acl_cmu_bind_additive_results.png" alt="CMU Results" /></p>
<p>CMU has no scale and it is thus no surprise that the two additive formats perform the same. The memory footprint and the max error remain overall largely identical but as expected the compression time degrades. No gain is observed from this technique which further highlights how this data set differs from hand authored animations.</p>
<p><img src="/public/acl/acl_paragon_bind_additive_results.png" alt="Paragon Results" /></p>
<p>Paragon shows the results I was expecting. The memory footprint reduces by about <strong>7.9%</strong> which is quite significant and the max error improves as well. Again, we can see both additive methods performing equally well. The relative space clearly loses out here and fails to show significant gains to compensate for the dramatically worse compression performance.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Overall it seems clear that any potential gains from this technique are heavily data dependent. A nearly <strong>8%</strong> smaller memory footprint is nothing to spit at but in the grand scheme of things, it might no longer be worth it in 2018 when decompression performance is likely much more important, especially on mobile devices. It is not immediately clear to me if the reduction in memory footprint could save enough to translate into fewer cache lines being fetched but even so it seems unlikely that it would offset the extra cost of the math involved.</p>
<p>See also the related <a href="/2022/01/23/anim_compression_bind_pose_stripping/">bind pose stripping</a> optimization.</p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression Library: Release 0.7.02018-04-02T00:00:00+00:00http://nfrechette.github.io/2018/04/02/acl_v0.7.0<p>Almost a year ago, I began working on <em>ACL</em> and it is now one step closer to being production ready with the new <strong>v0.7</strong> <a href="https://github.com/nfrechette/acl/releases/tag/v0.7.0">release</a>!</p>
<p>This new release is significant for several reasons but two stand out above all others:</p>
<ul>
<li>Exhaustive automated testing</li>
<li>Full multi-platform support</li>
</ul>
<p>Unlike previous releases, the performance remained unchanged since <strong>v0.6</strong> but I went ahead and updated the <a href="https://github.com/nfrechette/acl/tree/develop/docs#performance-metrics">stats and graphs</a> regardless.</p>
<h1 id="testing-testing-one-two">Testing, Testing, One, Two</h1>
<p>This new release introduces extensive unit testing for all the <em>core</em> and <em>math</em> functions on top of which <em>ACL</em> is built. There is still lots of room for improvement here, contributions welcome! Continuous integration also executes the unit tests for every platform except <em>iOS</em> and <em>Android</em> where they must be executed manually for now.</p>
<p>More significant is the addition of exhaustive regression testing. A total of <strong>42</strong> clips from the <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">Carnegie-Mellon University motion capture database</a> are each compressed under a mix of <strong>7</strong> configurations. At the moment, these must be run manually for every platform but scripts are present to automate the whole process. The primary reason why it remains manual is that the data is too large for <em>GitHub</em> and I do not have a webserver to host it. The instructions can be found <a href="https://github.com/nfrechette/acl/blob/develop/test_data/README.md">here</a>.</p>
<h1 id="it-runs-everywhere">It runs everywhere</h1>
<p>ACL now officially supports <strong>12</strong> different compiler toolchains and all the <a href="https://github.com/nfrechette/acl#supported-platforms">major platforms</a>: <em>Windows</em>, <em>Linux</em>, <em>OS X</em>, <em>iOS</em>, and <em>Android</em>. Both compression and decompression are supported and can easily be tested with the provided unit and regression tests.</p>
<p>But this is only the list of platforms I can reliably and easily test. In practice, since all the code is now pure <strong>C++11</strong>, if it compiles it should run just fine as-is. Although I cannot test them yet, I fully expect all major consoles to work out of the box: <em>Nintendo Switch (ARM)</em>, <em>PlayStation 4 (x64)</em>, and <em>Xbox One (x64)</em>.</p>
<h1 id="paragon-data">Paragon data</h1>
<p>The data I obtained from Paragon under NDA last year may or may not differ from what has now been <a href="https://www.unrealengine.com/en-US/paragon">publicly released</a> by Epic. As soon as I get the chance, I will update the published stats with the new public data. This also means that I will be able to include Paragon clips into the regression tests as well to increase our coverage.</p>
<h1 id="next-steps">Next steps</h1>
<p>The next <strong>v0.8</strong> release aims to achieve three goals (<a href="https://github.com/nfrechette/acl/milestone/5">roadmap</a>):</p>
<ul>
<li>Document as much as possible</li>
<li>Add the remaining features to support real games</li>
<li>Add the necessary decompression profiling infrastructure</li>
</ul>
<p>Decompression performance is one of the most important metric for modern games on both mobile and consoles. Measuring it accurately and reliably in an environment that is as close to a real game as possible is challenging which is why it was left last. However, ACL was built from the ground up in order to decompress as fast as possible: all memory reads are contiguous and linear and writes can be too depending on the host game engine integration. I am quite confident it will end up competitive with the state of the art codecs within UE4 and there are many opportunities left to optimize that I have delayed until I can measure their individual impact properly.</p>
<p>This upcoming release is likely to be the last before the first production release which aims to be a drop-in replacement within UE4. If everything goes according to plan and no delays surface, at the current pace, I should be able to reach this milestone around June 2018.</p>
Animation Compression Library: Release 0.6.02018-01-10T00:00:00+00:00http://nfrechette.github.io/2018/01/10/acl_v0.6.0<p>Hot off the press, ACL <strong>v0.6</strong> has just been <a href="https://github.com/nfrechette/acl/releases/tag/v0.6.0">released</a> and contains <a href="https://github.com/nfrechette/acl/blob/develop/CHANGELOG.md">lots of great things</a>!</p>
<p>This release focused primarily on extending the platform support as well as improving the accuracy. Proper <strong>Linux</strong> and <strong>OS X</strong> support was added as well as the <strong>x86</strong> architecture. As always, the list of supported platforms is in the <a href="https://github.com/nfrechette/acl">readme</a>. This was made possible thanks to continuous build integration which has been added and contributed in part by <a href="https://github.com/janisozaur">Michał Janiszewski</a>!</p>
<p>Another notable mention is that the backlog and roadmap have been migrated to <a href="https://github.com/nfrechette/acl/issues">GitHub Issues</a>. This ensures complete transparency with where the project is going.</p>
<h1 id="compiler-battle-royal">Compiler battle royal</h1>
<p>Now that we have all of these compilers and platforms supported, I thought it would make sense to measure everything at least once on the full data set from <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">Carnegie-Mellon University</a>.</p>
<p>Another thing I wanted to measure is how much do we gain from hyper-threading and last but not least, I thought it would be interesting to include <em>x86</em> as well as <em>x64</em>.</p>
<p>Here is my setup to measure:</p>
<ul>
<li><em>Windows 10</em> running on an <em>Intel i7-6850K</em> with <strong>6</strong> physical cores and <strong>12</strong> logical cores</li>
<li><em>Ubuntu 16.04</em> running in <em>VirtualBox</em> with <strong>6</strong> cores assigned</li>
<li><em>OS X</em> running on an <em>Intel i5-4288U</em> with <strong>2</strong> physical cores and <strong>4</strong> logical cores</li>
</ul>
<p>The <a href="https://github.com/nfrechette/acl/tree/develop/tools/acl_compressor">acl_compressor.py</a> script is used to compress multiple clips in parallel in independent processes. Each clip runs in its own process.</p>
<p>Every platform used a <em>Release</em> build with <em>AVX</em> enabled. The <em>wall clock time</em> is the cummulative time it took to run everything: compression, decompression to measure accuracy, reading the clip, writing the stats, etc. On the other hand, the <em>total thread time</em> measures the total sum of time the threads each spent on compression.</p>
<p><img src="/public/acl/acl_cmu_v060_compiler_performance.png" alt="Compiler performance" /></p>
<p>A number of things stand out:</p>
<ul>
<li><em>x86</em> is slower for <strong>VS 2015</strong> (<strong>66.5%</strong> slower) , <strong>VS 2017</strong> (<strong>64.0%</strong> slower with <em>11</em> cores, <strong>108.8%</strong> slower with <em>3</em> cores), and <strong>Clang 5</strong> (<strong>36.0%</strong> slower) but it seems to be faster for <strong>GCC 5</strong> (<strong>10.9%</strong> faster)</li>
<li>Hyper-threading barely helps at all: going from <em>6</em> cores to <em>11</em> with <strong>VS 2017</strong> was only <strong>7.8%</strong> faster but the total thread time increases by 69.9%</li>
<li><strong>Clang 5</strong> with <em>x64</em> wins hands down, it is <strong>25.2%</strong> faster than <strong>VS 2017</strong> and <strong>220.8%</strong> faster than <strong>GCC 5</strong></li>
</ul>
<p><strong>GCC 5</strong> performs so bad here that I am wondering if the default <em>CMake</em> compiler flags for <em>Release</em> builds are sane or if I made a mistake somewhere. <strong>Clang 5</strong> really blew me away: despite running in a VM it significantly outperforms all the other compilers with both <em>x86</em> and <em>x64</em>.</p>
<p>As expected, hyper-threading does not help all that much. When clips are compressed, most of the data manipulated can fit within the L2 or L3 caches. With so little IO made, animation compression is primarily CPU bound. Overall this leaves very little opportunity for a neighbor thread to execute since they hardly ever stall on expensive operations.</p>
<h1 id="accuracy-improvements">Accuracy improvements</h1>
<p>As I mentioned when the <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a> data set was <a href="/2017/12/05/acl_paragon/">announced</a>, some exotic clips brought to the surface some unusual accuracy issues. These were all investigated and they generally fell into one or both of these categories:</p>
<ul>
<li>Very small and very large scale coupled with very large translation caused unacceptable accuracy loss when using <em>affine matrices</em> to calculate the error</li>
<li>Very large translations in a long bone chain can lead to significant accuracy loss</li>
</ul>
<p>In order to fix the first issue, how we handle the error metric was refactored to better allow a game engine to supply their own. This is documented <a href="https://github.com/nfrechette/acl/blob/develop/docs/error_metrics.md">here</a>. Ultimately what is most important about the error metric is that it closely approximates how the error will look in the host game engine. Some game engines use <em>affine matrices</em> to convert the local space bone transform into object or world space while others use <em>Vector-Quaternion-Vector</em> (VQV). ACL now supports both ways to calculate the error and the default we will be using for all of our statistics is the later as it more closely matches what <em>Unreal 4</em> does. This did not measurably impact the compression results but it did improve the accuracy of the more exotic clips and the overall compression time is faster.</p>
<p>However, the problem of large translations in long bone chains has not been addressed. I compared how the error looked in <em>Unreal 4</em> and it does a much better job than ACL for the time being on those few clips. This is because they implement <a href="http://nfrechette.github.io/2016/12/22/anim_compression_error_compensation/">error compensation</a> which is something that ACL has not implemented <a href="https://github.com/nfrechette/acl/issues/69">yet</a>. In the meantime, ACL is perfectly safe for production use and if these rare clips with a visible error do pop up, less aggressive compression settings can be used. Only <strong>3</strong> clips within the <em>Paragon</em> data set suffer from this.</p>
<p>Ultimately a lot of the error introduced for both ACL and <em>Unreal 4</em> comes from the rotation format we use internally: we drop the quaternion <strong>W</strong> component. This works well enough when its value is close to <strong>1.0</strong> as the square-root used to reconstruct it is accurate in that range but it fails spectacularly when the value is very small and close to <strong>0.0</strong>. I already have plans to try two other rotation formats to help resolve this issue: <a href="https://github.com/nfrechette/acl/issues/47">dropping the largest quaternion component</a> and <a href="https://github.com/nfrechette/acl/issues/74">using the quaternion logarithm</a> instead.</p>
<h1 id="updated-stats">Updated stats</h1>
<p>While investigating the accuracy issues and comparing against <em>Unreal 4</em> I noticed that a fix I <a href="/2017/12/05/acl_paragon/">previously made</a> locally was partially incorrect and in rare cases could lead to bad things happening. This has been fixed and the statistics and graphs for <strong>UE 4.15</strong> were updated for <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">CMU</a> and <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">Paragon</a>. The results are very close to what they were before.</p>
<p>The accuracy improvements from this release are a bit more visible on the Paragon data set.</p>
<h1 id="next-steps">Next steps</h1>
<p>At this point, I can pretty confidently say that ACL is ready for production use but many things are still missing for the library to be of production quality. While the performance and accuracy are good enough, <a href="https://github.com/nfrechette/acl/issues/26"><strong>iOS</strong> support</a> is still missing, support for <a href="https://github.com/nfrechette/acl/issues/48">additive animations</a> is missing, as well as lots of unit testing, documentation, and clean up.</p>
<p>The next release will focus on:</p>
<ul>
<li>Cleaning up</li>
<li>Adding lots of unit tests</li>
<li><strong>iOS</strong> support</li>
<li>Better <strong>Android</strong> support</li>
<li><a href="https://github.com/nfrechette/acl/milestone/2">Many other things</a></li>
</ul>
Arithmetic Accuracy and Performance2017-12-29T00:00:00+00:00http://nfrechette.github.io/2017/12/29/acl_research_arithmetic<p>As I mentioned in my <a href="/2017/12/05/acl_paragon/">previous post</a>, <a href="https://github.com/nfrechette/acl">ACL</a> still suffers from some less then ideal accuracy in some exotic situations. Since the next release will have a strong focus on fixing this, I wanted to investigate using <em>float64</em> and <a href="https://en.wikipedia.org/wiki/Fixed-point_arithmetic"><em>fixed point</em> arithmetic</a>. It is general knowledge that <em>float32</em> arithmetic incurs rounding and can lead to severe accuracy loss in some cases. The question I hoped to answer was whether or not this had a significant impact on ACL. Originally ACL performed the compression entirely with <em>float64</em> arithmetic but this was <a href="https://github.com/nfrechette/acl/releases/tag/v0.3.0">removed</a> because it caused more issues than it was worth but I did not publish numbers to back this claim up. Now we revisit it once and for all.</p>
<p>To this end, the first <a href="https://github.com/nfrechette/acl/tree/research/float-vs-double-vs-fixed-point">research branch</a> was created. Research branches will play an important role in ACL. Their aim is to explore small and large ideas that we might not want to support in their entirety in the main branches while keeping them close. Unless otherwise specified, research branches will not be actively maintained. Once their purpose is complete, they will live on to document what worked and didn’t work and serve as a starting point for anyone hoping to investigate them further.</p>
<h1 id="float64-vs-float32-arithmetic"><em>float64</em> vs <em>float32</em> arithmetic</h1>
<p>In order to fully test <em>float64</em> arithmetic, I templated a few things to abstract the arithmetic used between <em>float32</em> and <em>float64</em>. This allowed easy conversion and support of both with nearly the same code path. The results proved somewhat underwhelming:</p>
<p><img src="/public/acl/arithmetic_float32_float64_summary.png" alt="Float32 VS Float64 Stat Summary" /></p>
<p>As it turns out, the small accuracy loss from <em>float32</em> arithmetic has a barely measurable impact on the memory footprint for CMU and a <strong>0.6%</strong> reduction for Paragon. However, the compression (and decompression) time is much faster.</p>
<p>With <em>float64</em>, the max error for CMU and Paragon is slightly improved for the majority of the clips but not by a significant margin and <strong>4</strong> exotic Paragon clips end up with a worse error.</p>
<p><img src="/public/acl/arithmetic_max_error_distribution.png" alt="Float32 VS Float64 Max Error Distribution" /></p>
<p>Consequently, it is my opinion that <em>float32</em> is the superior choice between the two. The small increase in accuracy and reduction in memory footprint is not significant enough to outweigh the degradation of the performance. Even though the <em>float64</em> code path isn’t as optimized, it will remain slower due to the individual instructions being slower and the increased number of registers needed. It’s possible the performance might be improved considerably with AVX and so this is something we’ll keep in mind going forward.</p>
<h1 id="fixed-point-arithmetic">Fixed point arithmetic</h1>
<p>Another popular alternative to <em>floating point</em> arithmetic is <em>fixed point</em> arithmetic. Depending on the situation it can yield higher accuracy and depending on the hardware it can also be faster. Prior to this, I had never worked with <em>fixed point</em> arithmetic. There was a bit of a learning curve but it proved intuitive soon enough.</p>
<p>I will not explain in great detail how it works but intuitively, it is the same as <em>floating point</em> arithmetic minus the exponent part. For our purposes, during the decompression (and part of the compression), most of the values we work with are normalized and unsigned. This means that the range is known ahead of time and fixed which makes it a good candidate for <em>fixed point</em> arithmetic.</p>
<p>Sadly, it differs so much from <em>floating point</em> arithmetic that I could not as easily support it in parallel with the other two. Instead, I created an <a href="https://github.com/nfrechette/acl/tree/research/float-vs-double-vs-fixed-point/tools/arithmetic_playground">arithmetic_playground</a> and tried a whole bunch of things within.</p>
<p>I focused on reproducing the decompression logic as close as possible. The original high level logic to decompress a single value is simple enough to include here:</p>
<script src="https://gist.github.com/nfrechette/2fe5be8ea2de50353e327164a3d3c15c.js"></script>
<h2 id="not-quite-10">Not quite 1.0</h2>
<p>The first obstacle to using <em>fixed point</em> arithmetic is the fact that our quantized values do not map <strong>1:1</strong>. Many engines dequantize with code that looks like this (including Unreal 4 and ACL):</p>
<script src="https://gist.github.com/nfrechette/28a61b389e3483c224da53333662ccd0.js"></script>
<p>This is great in that it allows us to exactly represent both <strong>0.0</strong> and <strong>1.0</strong>, we can support the full range we care about: <strong>[0.0 .. 1.0]</strong>. A case could be made to use a multiplication instead but it doesn’t matter all that much for the present discussion. With <em>fixed point</em> arithmetic we want to use all of our bits to represent the fractional part between those two values. This means the range of values we support is: <strong>[0.0 … 1.0)</strong>. This is because both <strong>0.0</strong> and <strong>1.0</strong> have the same fractional value of <strong>0</strong> and as such we cannot tell them apart without an extra bit to represent the integral part.</p>
<script src="https://gist.github.com/nfrechette/d25f334afa7da3a6650ab19d3e8dec17.js"></script>
<p>In order to properly support our full range of values, we must remap it with a multiplication.</p>
<script src="https://gist.github.com/nfrechette/8c47551ebb5a2047d0d8584879a502d8.js"></script>
<h2 id="fast-coercion-to-float32">Fast coercion to <em>float32</em></h2>
<p>The next hurdle I faced was how to convert the <em>fixed point</em> number into a <em>float32</em> value efficiently. I independently found a simple, fast, and elegant way and of course it turned out to be very popular for those very reasons.</p>
<script src="https://gist.github.com/nfrechette/eadfee26cc88ca4676c1cc8e96f53415.js"></script>
<p>For all of our possible values, we know their bit width and a shift can trivially be calculated to align it with the <em>float32</em> mantissa. All that remains is <strong>or-ing</strong> the exponent and the sign. In our case, our values are between <strong>[0.0 … 1.0[</strong> and thus by using a hex value of <strong>0x3F800000</strong> for <code class="language-plaintext highlighter-rouge">exponent_sign</code>, we end up with a <em>float32</em> in the range of <strong>[1.0 … 2.0[</strong>. A final subtraction yields us the range we want.</p>
<p>Using this trick with the <em>float32</em> implementation gives us the following code:</p>
<script src="https://gist.github.com/nfrechette/d230108509c27bb648a89a4bc333c8db.js"></script>
<p>It does lose out a tiny bit of accuracy but it is barely measurable. In order to be sure, I tried exhaustively all possible sample and segment range values up to a bit rate of <strong>16</strong> bits per component. The up side is obvious, it is <strong>14.9%</strong> faster!</p>
<h2 id="32-bit-vs-64-bit-variants"><em>32</em> bit vs <em>64</em> bit variants</h2>
<p>Many variants were implemented: some performed the segment range expansion with <em>fixed point</em> arithmetic and the clip range expansion with <em>float32</em> arithmetic and others do everything with <em>fixed point</em>. A mix of <em>32</em> bit and <em>64</em> bit arithmetic was also tried to compare the accuracy and performance tradeoff.</p>
<p>Generally, the <em>32</em> bit variants had a much higher loss of accuracy by <strong>1-2</strong> orders of magnitude. It isn’t clear how much this would impact the overall memory footprint on CMU and Paragon. The <em>64</em> bit variants had comparable accuracy to <em>float32</em> arithmetic but ended up using more registers and more instructions. This often degraded the performance to the point of making them entirely uncompetitive in this synthetic test. Only a single variant came close to the original <em>float32</em> performance but it could never beat the fast coercion derivative.</p>
<p>The fastest <em>32</em> bit variant is as follow:</p>
<script src="https://gist.github.com/nfrechette/6f17bba9035968ba47451c24b7519dc4.js"></script>
<p>Despite being <strong>3</strong> instructions shorter and using faster instructions, it was <strong>14.4%</strong> slower than the fast coercion <em>float32</em> variant. This is likely a result of pipelining not working out as well. It is entirely possible that in the real decompression code things could end up pipelining better making this a faster variant. Other processors such as those used in consoles and mobile devices also might perform differently and proper measuring will be required to get a definitive answer.</p>
<p>The general consensus seems to be that <em>fixed point</em> arithmetic can yield higher accuracy and performance but it is highly dependent on the data, the algorithm, and the processor it runs on. I can corroborate this and conclude that it might not help out all that much for animation compression and decompression.</p>
<p><img src="/public/acl/arithmetic_fixed_point_perf.png" alt="Fixed Point Performance" /></p>
<h1 id="next-steps">Next steps</h1>
<p>All of this work was performed in a branch that will <strong>NOT</strong> be merged into <em>develop</em>. However, some changes will be cherry picked by hand. In the short term, the conclusions reached here will not be integrated just yet into the main branches. The primary reason for this is that while I have extensive scripts and tools to track the accuracy, memory footprint, and compression performance; I do not have robust tooling in place to track decompression performance on the various platforms that are important to us.</p>
<p>Once we are ready, the fast coercion variant will land first as it appears to be an obvious drop-in replacement and some <em>fixed point</em> variants will also be tried on various platforms.</p>
<p>The accuracy issues will have to be fixed some other way and I already have some good ideas how: <a href="https://github.com/nfrechette/acl/issues/19">idea 1</a>, <a href="https://github.com/nfrechette/acl/issues/20">idea 2</a>, <a href="https://github.com/nfrechette/acl/issues/47">idea 3</a>, <a href="https://github.com/nfrechette/acl/issues/50">idea 4</a>, <a href="https://github.com/nfrechette/acl/issues/51">idea 5</a>.</p>
Animation Compression Library: Paragon Results2017-12-05T00:00:00+00:00http://nfrechette.github.io/2017/12/05/acl_paragon<p>While working for Epic to improve Unreal 4’s own animation compression and decompression, I asked for permission to use the Paragon animations for research purposes and they generously agreed. Today I have the pleasure to report the findings from that new data set!</p>
<p>This is significant for two reasons:</p>
<ul>
<li>It allows for extensive stress testing with new data</li>
<li>Paragon is a real game with high animation quality</li>
</ul>
<p><a href="https://www.youtube.com/watch?v=3OCJCZJWA68" title="Paragon Official Trailer"><img src="https://i.ytimg.com/vi/3OCJCZJWA68/hqdefault.jpg" alt="Paragon Official Trailer" /></a></p>
<h1 id="carnegie-mellon-university">Carnegie-Mellon University</h1>
<p>Thus far, the <a href="http://mocap.cs.cmu.edu/">Carnegie-Mellon University</a> data set has been the performance benchmark.</p>
<p>The data set contains <strong>2534</strong> clips. Each clip contains an animated character with <strong>44</strong> bones. The version of the data that I found comes from the <a href="https://www.assetstore.unity3d.com/en/#!/content/19991">Unity store</a> where it is distributed in FBX form but sampled at <strong>24</strong> FPS. The total duration of the database is <strong>09h 49m 37.58s</strong>. It does not contain any 3D scale and its raw size is <strong>1429.38 MB</strong>. It exclusively contains motion capture animation data. It is publicly available and well known within the animation compression research community.</p>
<p>While the database is valuable, it is not entirely representative of all the animation assets that a AAA game might use for a few reasons:</p>
<ul>
<li>Most AAA games today have well over <strong>100</strong> bones per character and sometimes as high as <a href="/2017/10/05/acl_in_ue4/"><strong>500</strong></a></li>
<li>The sample rate is lower than the <strong>30</strong> FPS typically used in games</li>
<li>Motion capture data is often very noisy</li>
<li>Games often animate things other than characters such as cloth, objects, destruction, etc.</li>
<li>Many games make use of 3D scale</li>
</ul>
<p>For these reasons, this data set is wonderful for unit testing and establishing a baseline for comparison but it falls a bit short with what I would ideally like.</p>
<p>You can see how Unreal and ACL compare against it <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">here</a>.</p>
<h1 id="paragon">Paragon</h1>
<p>The Paragon data set contains <strong>6558</strong> clips for a total duration of <strong>07h 00m 45.27s</strong> and a raw size of <strong>4276.11 MB</strong>. As you can see, despite being shorter than CMU, it is about <strong>3x</strong> larger in size.</p>
<p>The data set contains among other things:</p>
<ul>
<li>Lots of characters with varying number of bones</li>
<li>Animated objects of various shape and form</li>
<li>Very short and very long clips</li>
<li>Clips with unusual sample rate (as low as <strong>2</strong> FPS!)</li>
<li>World space clips</li>
<li>Lots of 3D scale</li>
<li>Lots of other exotic clips</li>
</ul>
<p>This is great to stress test any compression algorithm and the results will be very representative of what could be expected in a AAA game.</p>
<p>To extract the animation clips, I used the Unreal 4 animation recompression commandlet and modified it to skip clips that ACL does not yet support (e.g. additive animations). I did my best to retain as many clips as possible. Every clip was saved in the <a href="https://github.com/nfrechette/acl/blob/develop/docs/the_acl_file_format.md">ACL file format</a> allowing a binary exact representation.</p>
<p>Sadly, I am not at liberty to share this data set as I am only allowed to use it under a non-disclosure agreement. All hope is not lost though, Epic has expressed interest in perhaps making a small subset of the data publicly available for research purposes. Stay tuned!</p>
<h1 id="bugs">Bugs!</h1>
<p>The value of undertaking this quickly became obvious when an exotic clip from the data set highlighted a bug in the variable bit rate selection that ACL used. A fix was made and the results were breathtaking: CMU reduced in size by <strong>19%</strong> (and Paragon reduced by <strong>20%</strong>)! You can read about it <a href="/2017/11/23/acl_v0.5.0/">here</a> in my previous blog post.</p>
<p>Three clips stress tested the accuracy of ACL and ended up with an unacceptable error as a result. This will be made evident by the graphs and numbers below. I am hoping to fix a number of accuracy issues in the next ACL release now that I have new data to validate against.</p>
<p>The bugs I found were not exclusively within ACL: two were found and still present in the latest Unreal 4 version. Thankfully, I was able to get in touch with Epic and these should be fixed in a future release.</p>
<p>In order to make the comparison as fair as possible, I had to locally disable the down-sampling variants within the Unreal 4 automatic compression method. One of the two bugs caused these variants to sometime crash. While down-sampling isn’t often selected by the algorithm as the optimal choice for any given clip, disabling it means that compression is faster and possibly a bit larger as a result. Out of the <strong>600</strong> clips I managed to compress before finding the bug, only <strong>3</strong> ended up down-sampled. There are <strong>9</strong> down-sampled variants out of <strong>27</strong> in total (<strong>33%</strong>).</p>
<h1 id="bottom-line">Bottom line</h1>
<p>UE 4.15 took <strong>19h 56m 50.37s</strong> single threaded to compress. It yielded a compressed size of <strong>496.24 MB</strong> for a compression ratio of <strong>8.62 : 1</strong>. The max error is <strong>0.8619cm</strong>.</p>
<p>ACL 0.5 took <strong>19h 04m 25.11s</strong> single threaded to compress (<strong>01h 53m 42.84s</strong> with <strong>11</strong> threads). It yielded a compressed size of <strong>205.69 MB</strong> for a compression ratio of <strong>20.79 : 1</strong>. The max error is <strong>9.7920cm</strong>.</p>
<p>On the surface, the compression time remains faster with ACL even with a significant portion of the variants disabled in the Unreal automatic compression. However, the memory footprint is dramatically smaller, a whooping <strong>58.6%</strong> smaller! As will be made apparent in the graphs below, once again the maximum error proves to be a poor metric of the true performance: <strong>3</strong> clips have an error above <strong>0.8cm</strong> with ACL.</p>
<h1 id="the-results-in-images">The results in images</h1>
<p>All the results and many more images are also on GitHub <a href="https://github.com/nfrechette/acl/blob/develop/docs/paragon_performance.md">here</a> for Paragon just like they are for CMU <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">here</a>. I will only show a few selected images in this post for brevity.</p>
<p><img src="/public/acl/acl_paragon_v050_compression_ratio_distribution.png" alt="Compression ratio distribution" /></p>
<p>As expected, ACL outperforms Unreal by a significant margin. Some clips on the right are truncated with unusually high compression ratios as high as <strong>900 : 1</strong> for some exotic clips but those are likely very long with little to no animated data and aren’t too interesting or representative.</p>
<p><img src="/public/acl/acl_paragon_v050_max_error_distribution.png" alt="Max error distribution" /></p>
<p>Here again ACL outperforms Unreal over the overwhelming majority of the data set. On the right there are a small number of clips that perform somewhat poorly with both compression methods: a total of <strong>101</strong> clips have an error above <strong>0.1cm</strong> with ACL and <strong>153</strong> clips for Unreal.</p>
<p><img src="/public/acl/acl_paragon_v050_exhaustive_error.png" alt="Distribution of the error for every bone at every key frame" /></p>
<p>As I have <a href="/2017/09/10/acl_v0.4.0/">previously</a> mentioned, the max clip error is a poor measure of accuracy. Once again the full picture is much better and tells a different story.</p>
<p>ACL continues to shine, crossing the <strong>0.01cm</strong> threshold at the <strong>99.23th</strong> percentile. Unreal crosses the same threshold at the <strong>89th</strong> percentile.</p>
<p>Despite having a maximum error that is entirely unacceptable, it turns out that only <strong>0.77%</strong> of the compressed samples (out of <strong>112 million</strong>) exceed a sub-millimeter threshold. Aside from the <strong>3</strong> worst offending clips, everything else is cinematic and production quality. Not bad!</p>
<h1 id="conclusion">Conclusion</h1>
<p>As is apparent now, ACL performs admirably in a myriad of scenarios and continues to improve month after month. Real world data now confirms it. Half the memory footprint of Unreal is not insignificant even for a PC or PS4 game: less data to load into memory means faster streaming, less data to transfer means faster game download and installation times, and it can correlate with faster decompression performance too. For many PS4 and XB1 games, <strong>200 MB</strong> is perhaps small enough to load them all into memory up front and never stream them from disk afterwards.</p>
<p>As I continue to improve ACL, I will update the graphs and numbers with the latest significant releases. I also expect the improvements that I made to Unreal’s own animation compression over the last few months to be part of a future release and when that happens I will again update everything.</p>
<p>Special thanks to <a href="https://keybase.io/visualphoenix">Raymond Barbiero</a> for his very valuable feedback and to the continued support of many others!</p>
Animation Compression Library: Release 0.5.02017-11-23T00:00:00+00:00http://nfrechette.github.io/2017/11/23/acl_v0.5.0<p>Today marks the release of <a href="https://github.com/nfrechette/acl/releases/tag/v0.5.0">ACL v0.5</a>. Once again, <a href="https://github.com/nfrechette/acl/blob/develop/CHANGELOG.md">lots of great things</a> were included in this release but three things stand out:</p>
<ul>
<li>Full 3D scale support</li>
<li>Android support (tested within Unreal Engine 4.15)</li>
<li>A fix to the variable quantization optimization algorithm</li>
</ul>
<p>The third point in particular needs explaining. Initially, I did not intend to make significant changes in this release to the way compression was done beyond the scale support and whatever fixes Android required.
However, while investigating accuracy issues within an exotic clip, I noticed a bug. Upon fixing it (and very unexpectedly), everything kicked into overdrive.</p>
<h1 id="performance-results">Performance results</h1>
<p>On the <a href="http://mocap.cs.cmu.edu/">Carnegie-Mellon University (CMU)</a> data set, the memory footprint reduced by <strong>18.4%</strong> with little to no change to the accuracy of the overwhelming majority of clips and a slight accuracy increase to some of them!
Sadly, the compression speed suffered a bit as a result and it is now about <strong>1.5x</strong> slower than v0.4. In my opinion, this is an entirely acceptable trade-off!</p>
<p>Compared to UE 4.15, ACL now stands <strong>37.8%</strong> smaller and <strong>2.82x</strong> faster (single threaded) to compress on CMU. No small feat!</p>
<p>In light of these new numbers, all the charts have been updated and can be found <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">here</a>. Here are the most interesting:</p>
<p><img src="/public/acl/acl_cmu_v050_compression_ratio_distribution.png" alt="Compression ratio distribution" /></p>
<p><img src="/public/acl/acl_cmu_v050_max_error_distribution.png" alt="Max error distribution" /></p>
<p><img src="/public/acl/acl_cmu_v050_exhaustive_error.png" alt="Distribution of the error for every bone at every key frame" /></p>
<p>I also extracted two new charts: the distribution of clip durations within CMU and the distribution of which bit rates ended up selected by the algorithm. A bit rate of <strong>6</strong> means that <strong>6 bits</strong> per component are used. Every track (rotation, translation, and scale) sample has <strong>3</strong> components (X, Y, Z) which means <strong>18 bits</strong> per sample.</p>
<p><img src="/public/acl/acl_cmu_v050_clip_durations.png" alt="Clip duration distribution" /></p>
<p><img src="/public/acl/acl_cmu_v050_bit_rates.png" alt="Bit rate distribution" /></p>
<h1 id="next-steps">Next steps</h1>
<p>The focus of the next few months will be more platform support (Linux, OS X, and iOS in that order) as well as improving the accuracy. A new data set I got my hands on showed edge cases that are not too uncommon from real video games where the accuracy is not good enough. Part of the accuracy loss comes from storing the segment range on 8 bits per component and the fact that we use 32 bit floats to perform the decompression arithmetic. As such, a new research branch will be created to investigate using 64 bit floats to perform the arithmetic and a fixed point represetation as well. A separate blog post will be written with the conclusion of this research.</p>
Animation Compression Library: Unreal 4 Integration2017-10-05T00:00:00+00:00http://nfrechette.github.io/2017/10/05/acl_in_ue4<p>As mentioned in my <a href="/2017/09/10/acl_v0.4.0/">previous post</a>, I started working on integrating <a href="https://github.com/nfrechette/acl">ACL</a> into Unreal 4.15 locally. Today I can finally confirm that not only does it work but it rocks!</p>
<h1 id="matinee-fight-scene">Matinee fight scene</h1>
<p>In order to stress test ACL in a real game engine with real content, I set out to test it on the <a href="https://www.unrealengine.com/en-US/blog/matinee-fight-scene-released-on-marketplace">Matinee fight scene</a> that can be found on the Unreal 4 Marketplace.</p>
<iframe width="854" height="480" src="https://www.youtube.com/embed/EO0k92iVMjE" frameborder="0" allowfullscreen=""></iframe>
<h2 id="acl-in-action">ACL in action</h2>
<p>This is a very complex sequence with fast movements and <strong>LOTS</strong> of data. The main character (the white trooper) has over <strong>540 bones</strong> because the whole cloth motion is baked. The sequence lasts about <strong>66 seconds</strong>. The secondary characters move in and out of view and overall spend the overwhelming majority of the sequence completely idle and off screen.</p>
<p>Here is a short extract of the sequence using ACL for every character. This marks the first visual test and confirmation that ACL works.</p>
<video width="854" height="480" controls="">
<source src="/public/acl/matinee_fight_acl_cut.mp4" type="video/mp4" />
Your browser does not support this video.
</video>
<h2 id="the-data">The data</h2>
<p>The video isn’t too interesting but once again the numbers tell a story of their own. <strong>Packaged As Is</strong> is the default settings used when you first open it up in the editor, as packaged on the marketplace. For ACL, the integration is somewhat dirty for now and uses the same settings as for the <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">CMU database</a>: the error is measured <strong>3cm</strong> away from the bones, the error threshold is <strong>0.1mm</strong>, and the segments are <strong>16 frames</strong> long.</p>
<p><img src="/public/acl/matinee_fight_stats.png" alt="Matinee fight scene stats" /></p>
<p>ACL completely surpassed my own expectations here. The whole sequence is <strong>59.5% smaller</strong>! The main trooper is a whopping <strong>64.5% smaller</strong>! That’s nearly <strong>3x</strong> smaller! Compression time is also entirely reasonable sitting at just over <strong>1 minute</strong>. While the packaged settings are decent here sitting at around <strong>5 minutes</strong>, the automatic compression setting is not practical with almost <strong>3 hours</strong>. The error shown is what Unreal 4 reports in the dialog box after compression, it thus uses the Unreal 4 error metric and here again we can see that ACL is superior.</p>
<p>However, ACL does not perform as good on the secondary characters and ends up significantly larger. This is because they are mostly idle. Idle bones compress extremely well with linear key reduction but because ACL uses short segments, it is forced to retain at least a single key per segment. With some sort of automatic segment partitioning the memory footprint could reduce quite a bit here or even by simply using larger segments.</p>
<h1 id="what-happens-now">What happens now?</h1>
<p>The integration that I have made will not be public or published for quite some time. Until we reach version <strong>1.0</strong>, I wouldn’t want to support actual games while I am still potentially making large changes to the library. Once ACL is production ready and robust, I will see with Epic how we can go about making ACL a first-class citizen in their engine. In the meantime, I will maintain it locally and use it to test and validate ACL on the other platforms supported by Unreal.</p>
<p>For the time being, all hope is not lost! For the past 2 months, I have been working with Epic on improving the stock Unreal 4 animation compression. Our primary focus has been to improve decompression speed and reduce the compression time without compromising the already excellent memory footprint and accuracy. If all goes well these changes should make it in the next release and once that happens, I will update the relevant charts and graphs published here as well as in the ACL documentation.</p>
Animation Compression Library: Release 0.4.02017-09-10T00:00:00+00:00http://nfrechette.github.io/2017/09/10/acl_v0.4.0<p>This marks the fourth release of ACL. It contains a lot of <a href="https://github.com/nfrechette/acl/blob/develop/CHANGELOG.md">good stuff</a> but most notable is the addition of <a href="http://nfrechette.github.io/2016/11/10/anim_compression_uniform_segmenting/">segmenting support</a>. I have not had the chance to play with the settings much yet but using segments of <strong>16</strong> key frames reduces the memory footprint by about <strong>13%</strong> with variable quantization under uniform sampling. Adding range reduction on top of it (per segment), further reduces the memory footprint by another <strong>10%</strong>. This is very significant!</p>
<p>Some optimizations also made it in to the compression time, reducing it by <strong>4.3x</strong> with no compromise to quality.</p>
<p>You can see the latest numbers <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">here</a> as well as how they compare against the previous releases <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance_history.md">here</a>. Note that the documentation contains more graphs than I will share here.</p>
<p>This also represents the first release where graphs have been generated allowing us an unprecedented view into how the ACL and Unreal algorithms perform. As such, I will detail what is note-worthy and thus this blog post will be a bit long. Grab a coffee and buckle up!</p>
<p><strong>TL;DR:</strong></p>
<ul>
<li>ACL compresses better than Unreal for nearly every clip in the CMU database.</li>
<li>ACL is much smaller than Unreal (<strong>23.4%</strong>), is more accurate (<strong>2x+</strong>), and compresses much faster (<strong>4.68x</strong>).</li>
<li>ACL performs as expected and optimizes properly for the error threshold used, validating our assumptions.</li>
<li>A threshold of <strong>0.1cm</strong> is good enough for production use in Unreal as the overwhelming majority (<strong>98.15%</strong>) of the samples have an error smaller than <strong>0.02cm</strong>.</li>
</ul>
<h1 id="why-compare-against-unreal">Why compare against Unreal?</h1>
<p><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/">As I have previously mentioned</a>, Unreal 4 has a very solid error metric and good implementations of common animation compression techniques. It most definitely is well representative of the state of animation compression in game engines everywhere.</p>
<p><strong>NOTE: In the images that follow, the results for an error threshold of UE4 @ 1.0cm were nearly identical to 0.1cm and were thus omitted for brevity</strong></p>
<h1 id="performance-results">Performance results</h1>
<p>ACL 0.4 compresses the <a href="http://mocap.cs.cmu.edu/">CMU</a> database down to <strong>82.25mb</strong> in <strong>50 minutes</strong> single-threaded and <strong>5 minutes</strong> multi-threaded with a maximum error of <strong>0.0635cm</strong>. Unreal 4.15 compresses it down to <strong>107.94mb</strong> in <strong>3 hours and 54 minutes</strong> single-threaded with a maximum error of <strong>0.0850cm</strong> (<strong>1.0cm</strong> threshold used). Importantly, this is achieved with no compromise to decompression speed (although not yet measured, is estimated to be faster or just as fast with ACL).</p>
<p><img src="/public/acl/acl_cmu_v040_compression_ratio_vs_max_error.png" alt="Compression ratio VS max error per clip" /></p>
<p>As can be seen on the above image, ACL performs quite well here. The error is very low and the compression quite high in comparison to Unreal.</p>
<p><img src="/public/acl/acl_cmu_v040_compression_ratio_distribution.png" alt="Compression ratio distribution" /></p>
<p>Here we see the full distribution of the compression ratio over the CMU database. <strong>UE4 @ 0.01cm</strong> fails to do better than dropping the quaternion W and storing everything as full precision most of the time which is why the compression ratio is so consistent. <strong>UE4 @ 0.1cm</strong> performs similarly in that key reduction fails very often on this database and as a result simple quantization is most often selected.</p>
<p><img src="/public/acl/acl_cmu_v040_compression_ratio_distribution_bottom_10.png" alt="Compression ratio distribution (bottom 10%)" /></p>
<p>Here is a snapshot of the bottom <strong>10%</strong> (<strong>10th</strong> percentile and lower). We can see some similarities in shape at the bottom and top 10%.</p>
<p><img src="/public/acl/acl_cmu_v040_compression_ratio_by_duration.png" alt="Compression ratio by clip duration" /></p>
<p>We can see on the above image that Unreal performs consistently regardless of the animation clip duration but ACL performs slightly better the longer the clip is. This is most likely a direct result of using range reduction twice: once per clip, and once per segment.</p>
<p><img src="/public/acl/acl_cmu_v040_compression_ratio_by_duration_shortest_100.png" alt="Compression ratio by clip duration (shortest 100)" /></p>
<p>Both algorithms perform similarly for the shortest clips.</p>
<h2 id="how-accurate-are-we">How accurate are we?</h2>
<p><img src="/public/acl/acl_cmu_v040_max_error_distribution.png" alt="Max error distribution" /></p>
<p>The above image gives a good view of how accurate the algorithms are. We can see <strong>ACL @ 0.01cm</strong> and <strong>UE4 @ 0.01cm</strong> quickly reach the error threshold and only about <strong>10%</strong> of the clips exceed it. <strong>UE4 @ 0.1cm</strong> is less accurate but still pretty good overall.</p>
<p>The biggest source of error in both ACL and Unreal comes from the usage of the simple quaternion format consisting of <a href="http://nfrechette.github.io/2016/10/27/anim_compression_data/">dropping the <strong>W</strong> component</a> to later reconstruct it at runtime. As it turns out, this is terribly inaccurate when that component is very small. Better formats exist and will be implemented later.</p>
<p>ACL performs worse on a larger number of clips likely as a result of range reduction sometimes causing a precision loss for some clips. At some point ACL should be able to detect this and turn it off if it isn’t needed.</p>
<p><img src="/public/acl/acl_cmu_v040_max_clip_error_by_duration.png" alt="Max error by clip duration" /></p>
<p>There does not appear to be any correlation between the max error in a clip and its duration, as expected. One thing stands out though, the longer a clip is, the noisier the error appears to be. This is because the longer a clip is the more likely it is to contain a bad quaternion W that fails to reconstruct properly.</p>
<p>Over the years, I’ve read my fair share of animation compression papers and posts. And while they all measure the error differently the one thing they have in common is that they only talk about the worst error within a clip (or whole set of clips). <a href="http://nfrechette.github.io/2016/11/01/anim_compression_accuracy/">As I have previously mentioned</a>, how you measure the error is very important and must be done carefully but that is not all. Using the worst error within a given clip does not give a full picture. What about the other bones in the clip? What about the other key frames? Do I have a single bone on a single key frame that violates my threshold or do I have many?</p>
<p>In order to get a full and clear picture, I dumped the error of every bone at every key frame in the original clips. This represents over <strong>37 million</strong> samples for the CMU database.</p>
<p><img src="/public/acl/acl_cmu_v040_exhaustive_error.png" alt="Distribution of the error for every bone at every key frame" /></p>
<p>The above image is amazing!</p>
<p><img src="/public/acl/acl_cmu_v040_exhaustive_error_top_10.png" alt="Distribution of the error for every bone at every key frame (top 10%)" /></p>
<p>The above two images clearly show how terrible the max clip error is at giving insight into the true error. Here are some numbers visible only in the exhaustive graphs:</p>
<ul>
<li><strong>ACL</strong> crosses the <strong>0.01cm</strong> error threshold at the <strong>99.85th</strong> percentile (only <strong>0.15%</strong> of our values exceed the threshold!)</li>
<li><strong>UE4 @ 0.01cm</strong> crosses <strong>0.01cm</strong> at the <strong>99.57th</strong> percentile, almost just as good</li>
<li><strong>UE4 @ 0.1cm</strong> crosses <strong>0.01cm</strong> at the <strong>49.8th</strong> percentile</li>
<li><strong>UE4 @ 0.1cm</strong> crosses <strong>0.02cm</strong> at the <strong>98.15th</strong> percentile</li>
</ul>
<p>This clearly shows why <strong>0.1cm</strong> might be good enough for production use in Unreal: half our values remain at or below <strong>0.01cm</strong> and <strong>98%</strong> of the values are below <strong>0.02cm</strong>.</p>
<p>The previous images also clearly show how aggressive ACL is at reducing the memory footprint and at maximizing the error up to the error threshold. Therefore, the error threshold must be very conservative, much more so than for Unreal.</p>
<h1 id="why-acl-is-re-inventing-the-wheel">Why ACL is re-inventing the wheel</h1>
<p>As some have <a href="https://twitter.com/bkaradzic/status/879585542538534913">commented in the past</a>, ACL is largely re-inventing the wheel here. As such I will detail the rational for it a bit further.</p>
<p>Writing a whole animation blending middleware such as <a href="http://www.radgametools.com/granny.html">Granny</a> or <a href="http://www.naturalmotion.com/middleware/morpheme">Morpheme</a> would not have been practical. Just to match production quality implementations out there would have taken 1+ year part time. Even assuming I could have managed to implement something compelling, the cost of switching to a new animation runtime for a game team is very very high. Animators need to learn new tools and workflow, the engine integration might be tightly coupled, and there is no clear way to migrate old assets to the new format. Middlewares are also getting deprecated increasingly frequently. In that regard, the market has largely spoken: most games released today do so either with one of the major engines (Unreal, Unity, Lumberyard, Stingray, etc.) or large studios such as Activision, Electronic Arts, and Ubisoft routinely have in-house custom engines with their own custom animation runtime. Regardless of the quality or feature set, it would have been highly unlikely that it would ever have been used for something significant.</p>
<p>On the other hand, animation compression is a much smaller problem. Integration is easy: everything is pure C++ headers and most engines out there already support more than one animation compression algorithm. This makes migrating existing assets a trivial task providing the few required features are supported (e.g. 3D scale). Any engine or middleware could integrate ACL with few to no issues to be expected once it is production ready.</p>
<p>Animation compression is also a wheel that <strong>NEEDS</strong> re-inventing. Of all my blog posts, a single post receives the overwhelming majority of my traffic: <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/">animation compression in Unity</a>. Why is it so popular? Because as I mention in said post, accuracy issues will be common in Unity and the memory footprint large for high accuracy settings as a direct result of their error metric. Unity is also not alone, Stingray and Lumberyard both use the same metric. It is a <strong>VERY</strong> common error metric and it is terrible. Academic papers on this topic are often using different and poor error metrics and show very little to no data to back their results and claims. This makes evaluating these papers for real world usage in games very problematic.</p>
<p>Take <a href="https://www.cs.ubc.ca/~van/papers/2007-gi-compression.pdf">this paper</a> for example. They use the CMU database as well. Their error metric uses the leaf bone positions in object/world space as a measure of accuracy. This entirely ignores the rotational error of the leaf bone. They show a single graph of their results and two short tables. They do not detail the data further. Compare this with the wealth of information I was able to pull out and publish here. Even though ACL is much stricter when measuring the error, it is obvious that wavelets fail terribly to compete at the same level of accuracy (which barely makes it in their published findings). Note that they make no mention of what is an acceptable quality level that one might be able to realistically use.</p>
<p>Here is <a href="http://dl.acm.org/citation.cfm?id=3102236">another recent paper</a> published by someone I have met and have great respect for. The paper does not mention which error metric was used to compared against what they had prior nor does it mention how competitive their previous implementation was. It does not publish any concrete data either and only claims that the memory footprint reduces by 65% on average against their previous in-house techniques. It does provide a supplemental video which shows a small curated list of clips along with some statistics but without further information, it is impossible to objectively evaluate how it performs and where it lies on the spectrum of published techniques. Despite these shortcomings, it looks very promising (David knows his stuff!) and I am certainly looking forward to implementing this within ACL.</p>
<p>ACL does not only strive to improve on existing techniques; it will also establish a much-needed baseline to compare against and set a standard for how animation compression should be measured.</p>
<h1 id="next-steps">Next steps</h1>
<p>The results so far clearly show that ACL is one step closer to being production ready. The next few months will focus on bridging that gap towards reaching <strong>v1.0.0</strong>. In the coming releases, scale support will be added as well as support for other leading platforms. This will be done through a rudimentary Unreal 4 integration to make sure it is tested in a real engine and thus real world settings.</p>
<p>No further effort on my part will be made towards improving the above results until our first production release is made. However, <a href="https://github.com/CodyDWJones">Cody Jones</a> is working on integrating curve key reduction in the meantime.</p>
<p>Special thanks to Cody and <a href="https://github.com/tirpidz">Martin Turcotte</a> for their constant feedback and contributions!</p>
Math accuracy: Normalizing quaternions2017-08-30T00:00:00+00:00http://nfrechette.github.io/2017/08/30/math_accuracy_quat<p>While investigating precision issues with <a href="https://github.com/nfrechette/acl">ACL</a>, I ran into two problems that I hadn’t seen documented elsewhere and that slightly surprised me.</p>
<h1 id="dot-product">Dot product</h1>
<p>Calculating the dot product between two vectors is a very common operation used for all sorts of things. In an animation compression library, it’s primary use is normalizing <a href="https://en.wikipedia.org/wiki/Quaternion">quaternions</a>. Due to the nature of the code, accuracy is very important as it can impact the final compressed size as well as the resulting decompression error.</p>
<p><a href="https://en.wikipedia.org/wiki/SSE4">SSE 4</a> introduced a dot product instruction: <code class="language-plaintext highlighter-rouge">DPPS</code>. It allows the generated code to be more concise and compact by using fewer registers and instructions. I won’t speak to its performance here but sadly; its accuracy is not good enough for us by a tiny, yet important, sliver.</p>
<p>For the purpose of this blog post, we will use the following nearly normalized quaternion as an example: <code class="language-plaintext highlighter-rouge">{ X, Y, Z, W } = { -0.6767403483, 0.7361232042, 0.0120376134, -0.0006215832 }</code>. This is a real quaternion from a real clip of the <a href="http://mocap.cs.cmu.edu/">Carnegie-Mellon University (CMU) motion capture database</a> that proved to be problematic. With doubles, the dot product is <strong>1.0000001612809224</strong>.</p>
<p>Using plain C++ yields the following code and assembly (compiled with AVX support under Visual Studio 2015 with an x64 target):
<script src="https://gist.github.com/nfrechette/745866253ad7a664bd97af1e008060a2.js"></script></p>
<ul>
<li>The result is: <strong>1.00000024</strong>. Not quite the same but close.</li>
</ul>
<p>Using the SSE 4 dot product instruction yields the following code and assembly:
<script src="https://gist.github.com/nfrechette/22cc41715d593f1cd5fc60aaab1c37ef.js"></script></p>
<ul>
<li>The result is: <strong>1.00000024</strong>.</li>
</ul>
<p>Using a pure SSE 2 implementation yields the following assembly:
<script src="https://gist.github.com/nfrechette/e957d56c693228dd6c6f299ca68936e3.js"></script></p>
<ul>
<li>The result is: <strong>1.00000012</strong>.</li>
</ul>
<p>These are all nice but it isn’t immediately obvious how big the impact can be. Let’s see how they perform after taking the square root (note that the SSE 2 <code class="language-plaintext highlighter-rouge">SQRT</code> instruction is used here):</p>
<ul>
<li>C++: <strong>1.00000012</strong></li>
<li>SSE 4: <strong>1.00000012</strong></li>
<li>SSE 2: <strong>1.00000000</strong></li>
</ul>
<p>Again, these are all pretty much the same. What happens when we take the square root reciprocal after 2 <a href="https://en.wikipedia.org/wiki/Newton%27s_method">iterations of Newton-Raphson</a>?
<script src="https://gist.github.com/nfrechette/42c139a2ebac76976804d8bad1ff7e27.js"></script></p>
<ul>
<li>C++: <strong>0.999999881</strong></li>
<li>SSE 4: <strong>0.999999881</strong></li>
<li>SSE 2: <strong>0.999999940</strong></li>
</ul>
<p>With this square root reciprocal, here is how our quaternions look after being multiplied to normalize them and their associated dot product.</p>
<ul>
<li>C++: <code class="language-plaintext highlighter-rouge">{ -0.676740289, 0.736123145, 0.0120376116, -0.000621583138 }</code> = <strong>0.999999940</strong></li>
<li>SSE 4: <code class="language-plaintext highlighter-rouge">{ -0.676740289, 0.736123145, 0.0120376116, -0.000621583138 }</code> = <strong>1.00000000</strong></li>
<li>SSE 2: <code class="language-plaintext highlighter-rouge">{ -0.676740289, 0.736123145, 0.0120376125, -0.000621583138 }</code> = <strong>0.999999940</strong></li>
</ul>
<p>Here is the dot product calculated with doubles:</p>
<ul>
<li>C++: <strong>0.99999999381912441</strong></li>
<li>SSE 4: <strong>0.99999999381912441</strong></li>
<li>SSE 2: <strong>0.99999999384079208</strong></li>
</ul>
<p>And the new square root:</p>
<ul>
<li>C++: <strong>0.999999940</strong></li>
<li>SSE 4: <strong>1.00000000</strong></li>
<li>SSE 2: <strong>0.999999940</strong></li>
</ul>
<p>Now the new reciprocal square root:</p>
<ul>
<li>C++: <strong>1.00000000</strong></li>
<li>SSE 4: <strong>1.00000000</strong></li>
<li>SSE 2: <strong>1.00000000</strong></li>
</ul>
<p>After all of this, our delta from a true length of <strong>1.0</strong> before (as calculated with doubles) was <strong>1.612809224e-7</strong> before normalization. Here is how they fare afterwards:</p>
<ul>
<li>C++: <strong>6.18087559e-9</strong></li>
<li>SSE 4: <strong>6.18087559e-9</strong></li>
<li>SSE 2: <strong>6.15920792e-9</strong></li>
</ul>
<p>And thus, the difference between using SSE 4 and SSE 2 is just <strong>2.166767e-11</strong>.</p>
<p>As it turns out, the SSE 2 implementation appears the most accurate one and yields the lowest decompression error as well as a smaller memory footprint (by a tiny bit).</p>
<h1 id="normalizing-a-quaternion">Normalizing a quaternion</h1>
<p>There are two mathematically equivalent ways to normalize a quaternion: taking the dot product, calculating the square root, and dividing the quaternion with the result, or taking the dot product, calculating the reciprocal square root, and multiplying the quaternion with the result.</p>
<p>Are the two methods equivalent with floating point mathematics? Again, we will not discuss the performance implications as we are only concerned with accuracy here. Using the previous example quaternion and using the SSE 2 dot product yields the following result with the first method:</p>
<ul>
<li>Dot product: <strong>1.00000012</strong></li>
<li>Length: <code class="language-plaintext highlighter-rouge">sqrt(1.00000012)</code> = <strong>1.00000000</strong></li>
<li>Normalized quaternion using division: <code class="language-plaintext highlighter-rouge">{ -0.6767403483, 0.7361232042, 0.0120376134, -0.0006215832 }</code></li>
<li>New dot product: <strong>1.00000012</strong></li>
<li>New length: <strong>1.00000000</strong></li>
</ul>
<p>And now using the reciprocal square root with 2 Newton-Raphson iterations:</p>
<ul>
<li>Dot product: <strong>1.00000012</strong></li>
<li>Reciprocal square root: <strong>0.999999940</strong></li>
<li>Normalized quaternion using multiplication: <code class="language-plaintext highlighter-rouge">{ -0.676740289, 0.736123145, 0.0120376125, -0.000621583138 }</code></li>
<li>New dot product: <strong>0.999999940</strong></li>
<li>New length: <strong>0.999999940</strong></li>
<li>New reciprocal square root: <strong>1.00000000</strong></li>
</ul>
<p>By using the division, normalization fails to yield us a more accurate quaternion because of square root is <strong>1.0</strong>. The reciprocal square root instead allows us to get a more accurate quaternion as demonstrated in the previous section.</p>
<h1 id="conclusion">Conclusion</h1>
<p>It is hard to see if the numerical difference is meaningful but over the entire CMU database, both tricks together help reduce the memory footprint by <strong>200 KB</strong> and lower our error by a tiny bit.</p>
<p>For most game purposes, the accuracy implication of these methods does not matter all that much and rarely have a measurable impact. Picking whichever method is fastest to execute might just be good enough.</p>
<p>But when accuracy is of a particular concern, special care must be taken to ensure every bit of precision is retained. This is one of the motivating reasons for ACL having its own internal math library: granular control over performance and accuracy.</p>
Animation Compression Library: Release 0.3.02017-07-29T00:00:00+00:00http://nfrechette.github.io/2017/07/29/acl_v0.3.0<p>This release marks an important milestone. It now supports a fully variable bit rate and it performs admirably so far. <a href="https://github.com/nfrechette/acl/blob/develop/docs/cmu_performance.md">The numbers don’t lie.</a> Without using any form of key reduction, we match the compression ratio of Unreal 4 (which uses <a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/">a mix of linear key reduction with a form of variable quantization</a>) and many more tricks will follow to push this even further. It is worth noting that this new variable bit rate algorithm is entirely different from the one I presented at the <a href="http://nfrechette.github.io/2017/03/08/anim_compression_gdc2017/">GDC 2017</a> and it should outperform it. In due time, more stats and graphs will be published to outline how the data looks across the whole dataset.</p>
<p>While <a href="https://github.com/nfrechette/acl/releases/tag/v0.3.0">v0.3.0</a> remains a pre-release, we are quickly approaching a production ready state. Already for the vast majority of clips the error introduced is invisible to the naked eye and the performance is there to match. Major features missing to reach the production ready state are: scale support (sadly the Carnegie-Mellon data set does not contain any scale as such testing this will be problematic), and proper multi-platform support (iOS, OS X, android, clang, gcc, etc.). Both of these things are easily solved problems which is why they were deferred into future releases.</p>
<p>Version 0.4.0 will aim to introduce <a href="http://nfrechette.github.io/2016/11/10/anim_compression_uniform_segmenting/">clip segmenting</a> and hopefully <a href="http://nfrechette.github.io/2016/12/10/anim_compression_curve_fitting/">curve based key reduction</a>. Segmenting should improve our accuracy further and at the same time allow us to reduce the memory footprint even further. Curve key reduction will of course allow us to reduce the memory footprint further as well, perhaps dramatically so. Stay tuned!</p>
Introducing ACL2017-06-25T00:00:00+00:00http://nfrechette.github.io/2017/06/25/introducing_acl<p>Over the years, I’ve had my fare share of discussions about animation compression and two things became obvious over time:
we were all (re-)doing similar things and none of us had access to a state of art implementation to compare against.
This lead to rampant speculation about which algorithm was superior or inferior.
Having implemented a few algorithms in the past, I have finally decided to redo all that work once more and in the open this time.
Say ‘Hello’ to the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> (<strong>ACL</strong> for short).</p>
<p>To quote the readme:</p>
<blockquote>
<p>This library has two primary goals:</p>
<ul>
<li>Implement state of the art and production ready animation compression algorithms</li>
<li>Serve as a benchmark to compare various techniques against one another</li>
</ul>
</blockquote>
<p>Over the next few months, I hope to implement state of the art versions of common algorithms and to surpass what current game engines currently offer.
It is my hope that this library can serve as the foundation for an industry standard so that together we may be able to move forward and well past the foot sliding issues of yester-year!</p>
Optimizing 4x4 matrix multiplication2017-04-13T00:00:00+00:00http://nfrechette.github.io/2017/04/13/modern_simd_matrix_multiplication<p>In modern video games, the 4x4 matrix multiplication is an important cornerstone. It is used for a very long list of things: moving individual character joints, physics simulation, rendering, etc. To generate a single video game image (and we typically generate between 25 and 60 per second), several thousand matrix multiplications will take place. Today, we will take an in-depth look at such a fundamental piece of code.</p>
<p>As I will show, we can improve on some of the most common implementations out in the wild. I will use <a href="https://github.com/Microsoft/DirectXMath">DirectX Math</a> as a reference here but I have seen identical implementations in many state of the art game engines.</p>
<p>This blog post will be broken down into four sections:</p>
<ul>
<li><a href="#test_cases">The test cases that will guide our optimization</a></li>
<li><a href="#function_signatures">Various ways we can write our function signature and their implications</a></li>
<li><a href="#function_implementations">Six different implementations</a></li>
<li><a href="#results">The results</a></li>
</ul>
<p>Note that all the code for this can be found <a href="https://github.com/nfrechette/DirectXMathOptimizations">here</a>. Feel free to toy with it and replicate the results or contribute your own.</p>
<h1 id="our-test-cases"><a name="test_cases"></a>Our test cases</h1>
<p>In order to keep our observations grounded in reality, we will use three test cases that represent common heavy usage of 4x4 matrix multiplication. These are very synthetic in nature but they will make profiling and measuring immensely easier. Modern game engines do many things with many threads which can make profiling on PC somewhat a bit more complicated especially for something as short and simple as matrix multiplication.</p>
<h2 id="test-case-1">Test case #1</h2>
<p>Our first test case applies a constant matrix to an array of 64 matrices. Aside from our constant matrix, each input will be read from memory (here everything fits in our processor cache but it doesn’t matter much) and each output will be written back to memory. This code is meant to simulate the common operation of transforming an array of object space matrices into world space matrices, perhaps for skinning purposes.</p>
<script src="https://gist.github.com/nfrechette/e9c4d07695568480117c76848f24a0a8.js"></script>
<h2 id="test-case-2">Test case #2</h2>
<p>Our second test case transforms an array of 64 local space matrices into an array of 64 object space matrices. To perform this operation, each local space matrix is multiplied by the parent object space matrix. The root matrix is trivial and equal in both local and object space as it has no parent. In this contrived example the parent matrix is always the previous entry in the array but in practice it would be some arbitrary index previously transformed. This operation is common at the end of the animation runtime where the pose generated will typically be in local space.</p>
<script src="https://gist.github.com/nfrechette/4d54d507fa05018e7ec2a14d8ebdce4c.js"></script>
<h2 id="test-case-3">Test case #3</h2>
<p>Our third test case takes two constant matrices and writes the result to a static array. The array is made static to prevent the compiler from stripping the code. This code is synthetic and meant to try and profile one off multiplications that happen everywhere in gameplay code. We perform the operation 64 times to help us measure the impact since the code is very fast to begin with.</p>
<script src="https://gist.github.com/nfrechette/6059d406eb4e522d0cb484ab907b6692.js"></script>
<h1 id="function-signature-variations"><a name="function_signatures"></a>Function signature variations</h1>
<p>Our reference implementation taken from DirectX Math has the following signature:</p>
<script src="https://gist.github.com/nfrechette/d54f3d5bdb2bd78a2288d3ca5006d511.js"></script>
<p>There a few things that are noteworthy here. The function is marked inline but due to its considerable size, the function is generally never inlined. It also uses the <a href="https://msdn.microsoft.com/en-us/library/dn375768.aspx">__vectorcall</a> calling convention with the macro <code class="language-plaintext highlighter-rouge">XM_CALLCONV</code>. This allows up to 6 SIMD input arguments to be passed by register (the default calling convention passes them by value on the stack, unlike with PowerPC) and the return value can also be up to 4 SIMD outputs passed by register. This also works for aggregate types such as <code class="language-plaintext highlighter-rouge">XMMatrix</code>. The function takes 2 arguments: <strong>M1</strong> is passed by register with the help of <code class="language-plaintext highlighter-rouge">FXMMATRIX</code> and <strong>M2</strong> is passed by <code class="language-plaintext highlighter-rouge">const &</code> with the help of <code class="language-plaintext highlighter-rouge">CXMMATRIX</code>.</p>
<p>This function signature will be called in our data: <strong>reg</strong></p>
<p>We can vary the function signature in a number of ways and it will be interesting to compare the results. I came up with a number of variations. They are as follow.</p>
<h2 id="force-inline">Force inline</h2>
<p>As mentioned, since our function is very large, inlining will typically fail to happen. However in very hot code, it still makes sense to inline the function.</p>
<script src="https://gist.github.com/nfrechette/66c6e507c14c0d992eed5818a070343b.js"></script>
<p>This function signature will be called: <strong>inl</strong></p>
<h2 id="pass-everything-from-memory">Pass everything from memory</h2>
<p>An obvious change we can make is to pass both input arguments as <code class="language-plaintext highlighter-rouge">const &</code>. In many cases our matrices might not be cached in local registers to begin with and we have to load them from memory anyway (such as in test case #2).</p>
<script src="https://gist.github.com/nfrechette/d7160fb843b8e7f75078179f70545f65.js"></script>
<p>This function signature will be called: <strong>mem</strong></p>
<h2 id="flip-our-matrix-arguments">Flip our matrix arguments</h2>
<p>In our matrix implementation, the rows of <strong>M2</strong> are multiplied whole with each matrix element from <strong>M1</strong>. The code ends up looking like this:</p>
<script src="https://gist.github.com/nfrechette/782818d62e8e7d34587de66dab557e00.js"></script>
<p>This repeats 4 times for each row of <strong>M1</strong>. It is obvious that we can cache our 4 values from <strong>M2</strong> and indeed the compiler typically does so for us in our reference implementation. Each of those 4 rows will be needed again and again but the same cannot be said of the 4 rows of <strong>M1</strong> which are only needed temporarily. It would thus make sense to pass the matrix arguments in the opposite order: <strong>M2</strong> first by register and <strong>M1</strong> second by <code class="language-plaintext highlighter-rouge">const &</code>.</p>
<script src="https://gist.github.com/nfrechette/bdf686b37b8dd564aeb5e47520954766.js"></script>
<p>Note that we use a macro to perform the flip cleanly. I would have preferred a force inlined function but the compiler was not generating clean assembly from it.</p>
<p>This function signature will be called: <strong>flip</strong></p>
<h2 id="expanded-matrix-argument">Expanded matrix argument</h2>
<p>Even though the __vectorcall calling convention conveniently passes our matrix in 4 registers, it might help the compiler make different decisions if we are explicit about our intentions.</p>
<script src="https://gist.github.com/nfrechette/773243b03b533a22086823679deeae38.js"></script>
<p>Our expanded variant will always use the flipped argument ordering. Measuring the non-flipped ordering is left as an exercise to the reader.</p>
<p>This function signature will be called: <strong>exp</strong></p>
<h2 id="return-value-by-argument">Return value by argument</h2>
<p>Another thing that is very common for a matrix multiplication implementation is to have the return value as a pointer or reference in a third argument.</p>
<script src="https://gist.github.com/nfrechette/30824a485577af1c8d3bb2e60e879926.js"></script>
<p>Again this might help the compiler make different optimization choices. Note as well that implementations with this variant must explicitly cache the rows of <strong>M2</strong> in order to have the correct result in case where the result is written to <strong>M2</strong>. It also improves the generated assembly as otherwise the output matrix would alias the arguments causing the compiler to not perform the caching automatically for you.</p>
<p>This function signature can be applied to all our variants and it will add the suffix <strong>2</strong></p>
<h2 id="permute-all-the-things">Permute all the things!</h2>
<p>Taking all of this together and permuting everything yields 12 variants as follow:</p>
<script src="https://gist.github.com/nfrechette/8fbbd76a6551a4eca24bac721e4c2c36.js"></script>
<p>In our data, they are called:</p>
<ul>
<li>reg</li>
<li>reg2</li>
<li>reg_flip</li>
<li>reg_flip2</li>
<li>reg_exp</li>
<li>reg_exp2</li>
<li>mem</li>
<li>mem2</li>
<li>inl</li>
<li>inl2</li>
<li>inlexp</li>
<li>inlexp2</li>
</ul>
<p>Hopefully this covers a large extent of common and sensible variations.</p>
<h1 id="our-competing-implementations"><a name="function_implementations"></a>Our competing implementations</h1>
<p>I was able to come up with six distinct implementations of the matrix multiplication, including the original reference. Note that I did not attempt to make the fastest implementation possible, there are other things we could try to make them faster. I also made sure that each version gave a result that was a exactly the same as the reference implementation, down to the last bit (binary exact).</p>
<h2 id="reference">Reference</h2>
<p>The reference implementation is quite large as such I will not include the full source here but the code can be found <a href="https://github.com/nfrechette/DirectXMathOptimizations/blob/master/Inc/MatrixMultiply/MatrixMultiply_Ref.h">here</a>.</p>
<p>The reference <strong>regexp2</strong> variant uses 10 XMM registers and totals 70 instructions.</p>
<h2 id="broadcast">Broadcast</h2>
<p>In our reference implementation, an important part can be tweaked a bit.</p>
<script src="https://gist.github.com/nfrechette/e697f4451e67f2568861a9cd5f14a7f8.js"></script>
<p>We load a row from <strong>M1</strong>, extract each component, and replicate it into the 4 lanes of our SIMD register. This will compile down to 1 load instruction followed by 4 shuffle instructions. This was very common on older consoles: loads from memory were very expensive and none of the other instructions could work directly from memory. However, on SSE and in particular with AVX, we can do a bit better. We can use the <a href="https://software.intel.com/en-us/node/524098"><strong>_mm_broadcast_ss</strong></a> instruction. It takes as input a pointer to a scalar floating point value and it will output the replicated value over our 4 SIMD lanes. We thus save and avoid our load instruction.</p>
<script src="https://gist.github.com/nfrechette/a5343e01b4820d63ae5d9a0a791033e1.js"></script>
<p>The code for this variant can be found <a href="https://github.com/nfrechette/DirectXMathOptimizations/blob/master/Inc/MatrixMultiply/MatrixMultiply_V0.h">here</a>.</p>
<p>The version 0 <strong>regexp2</strong> variant uses 7 XMM registers and totals 58 instructions.</p>
<h2 id="looping">Looping</h2>
<p>Another change we can perform is inspired from <a href="http://stackoverflow.com/questions/18499971/efficient-4x4-matrix-multiplication-c-vs-assembly">this post</a> on StackOverflow. I rewrote the assembly into C++ code that uses intrinsics to try and keep it comparable.</p>
<p>Two versions were written: version 2 uses load/shuffle (<a href="https://github.com/nfrechette/DirectXMathOptimizations/blob/master/Inc/MatrixMultiply/MatrixMultiply_V2.h">code here</a>) and version 1 uses broadcast (<a href="https://github.com/nfrechette/DirectXMathOptimizations/blob/master/Inc/MatrixMultiply/MatrixMultiply_V1.h">code here</a>).</p>
<p>Branching was notoriously slow on the old consoles, it will be interesting to see how newer hardware performs.</p>
<p>The version 1 <strong>regexp2</strong> variant uses 7 XMM registers and totals 23 instructions.
The version 2 <strong>regexp2</strong> variant uses 10 XMM registers and totals 37 instructions.</p>
<h2 id="handwritten-assembly">Handwritten assembly</h2>
<p>Similar to our looping versions, I also kept the hand written assembly version referenced. I made a few tweaks to make sure the results were binary exact. Sadly, the tweaks required the usage of one extra register. Having run out of volatile registers, I elected to load the first row of <strong>M2</strong> directly from memory with the multiply instruction during every iteration.</p>
<p>Only two variants were implemented: <strong>regexp2</strong> and <strong>mem2</strong>.</p>
<p>Two versions were written: version 3 uses load/shuffle (<a href="https://github.com/nfrechette/DirectXMathOptimizations/blob/master/Src/MatrixMultiply_V3.asm">code here</a>) and version 4 uses broadcast (<a href="https://github.com/nfrechette/DirectXMathOptimizations/blob/master/Src/MatrixMultiply_V4.asm">code here</a>).</p>
<p>The version 3 <strong>regexp2</strong> variant uses 5 XMM registers and totals 21 instructions.
The version 4 <strong>regexp2</strong> variant uses 5 XMM registers and totals 17 instructions.</p>
<h1 id="the-results"><a name="results"></a>The results</h1>
<p>For our purposes, each test will be run <strong>1000000</strong> times and the cumulative time will be considered to be <strong>1</strong> sample. We will repeat this to gather <strong>100</strong> samples. To avoid skewing in our data that might result from various external sources (CPU frequency changes, other OS work, etc.), we will retain and use the <strong>80th</strong> percentile from our dataset. Due to the simple nature of the code, this should be good enough for us to draw meaningful conclusions. All measurements are in <strong>milliseconds</strong>.</p>
<p>All of my raw results are parsed with a simple python script to extract the desired percentile and format it in a table form. The script can be found <a href="https://github.com/nfrechette/DirectXMathOptimizations/blob/master/parse_stats.py">here</a>.</p>
<p>I ran everything on my desktop computer which has an <a href="https://ark.intel.com/products/94188/Intel-Core-i7-6850K-Processor-15M-Cache-up-to-3_80-GHz">Intel i7-6850K processor</a>. While my CPU differs from what the Xbox One and PlayStation 4 use, they both support AVX and could benefit from the same changes.</p>
<p>I also measured running the test cases with multithreading enabled to see if the results were consistent. Since the results were indeed consistent, we will only talk about the single threaded case but all the data for the test results can be found here:</p>
<ul>
<li>Single threaded: <a href="/public/matrix_multiplication/results_sync_raw.csv">raw data</a>, <a href="/public/matrix_multiplication/results_sync.csv">parsed data</a>, <a href="/public/matrix_multiplication/results_sync.xlsx">charts</a></li>
<li>One thread per physical core: <a href="/public/matrix_multiplication/results_percore_raw.csv">raw data</a>, <a href="/public/matrix_multiplication/results_percore.csv">parsed data</a>, <a href="/public/matrix_multiplication/results_percore.xlsx">charts</a></li>
<li>One thread per logical core: <a href="/public/matrix_multiplication/results_saturate_raw.csv">raw data</a>, <a href="/public/matrix_multiplication/results_saturate.csv">parsed data</a>, <a href="/public/matrix_multiplication/results_saturate.xlsx">charts</a></li>
</ul>
<h2 id="test-case-results">Test case results</h2>
<p>Here are the results for our 3 test cases:</p>
<p><img src="/public/matrix_multiplication/testcase1_all.png" alt="Test Case #1 Results" />
<img src="/public/matrix_multiplication/testcase2_all.png" alt="Test Case #2 Results" />
<img src="/public/matrix_multiplication/testcase3_all.png" alt="Test Case #3 Results" /></p>
<p>A few things are immediately obvious:</p>
<ul>
<li>Version 1 and 2, the looping intrinsic versions, are terribly slow. I moved them to the right so we can focus on the left part.</li>
<li>Version 0 is consistently faster than our reference implementation.</li>
</ul>
<p>Here are the same results but only considering the best 3 variants (<strong>regexp2</strong>, <strong>mem2</strong>, and <strong>inlexp2</strong>) and the best 4 versions (reference, version 0, version 3, and version 4).</p>
<p><img src="/public/matrix_multiplication/best_results.png" alt="Best Results" /></p>
<h2 id="loadshuffle-versus-broadcast">Load/shuffle versus broadcast</h2>
<p><img src="/public/matrix_multiplication/load_shuffle_vs_broadcast.png" alt="Load/Shuffle VS Broadcast" /></p>
<p>Overwhelmingly, we can see that the versions that use broadcast are faster than their counterpart that uses load/shuffle. This is not too surprising: we use fewer registers, as a result fewer registers spill on the stack, and fewer instructions are used. This is more significant when the function isn’t force inlined since in our test cases, whatever we spill on the stack ends up hoisted outside of our loops when inlined.</p>
<p>The fact that we use fewer registers and instructions also has other side effects, namely it can help the compiler to inline functions. In particular, this is the case for version 1 and 2: version 1 uses broadcast and gets inlined automatically while version 2 uses load/shuffle and does not get inlined.</p>
<h2 id="output-in-registers-versus-memory">Output in registers versus memory</h2>
<p><img src="/public/matrix_multiplication/output_register_vs_memory.png" alt="Output: Register VS Memory" /></p>
<p>For test cases #1 and #3, passing our return value as an argument is a net win when there is no inlining. This remains true to a lesser extent even when the functions are force inlined which means it helps the compiler make better choices.</p>
<p>However, for test case #2, it can sometimes be a bit slower. It seems that the assembly generated at the call site isn’t as clean as it could be. It’s possible that by tweaking the test case code a bit, performance could be improved.</p>
<h2 id="flipped-versus-expanded">Flipped versus expanded</h2>
<p><img src="/public/matrix_multiplication/flip_vs_expanded.png" alt="Flipped VS Expanded" /></p>
<p>Looking only at version 0, the behaviour seems to differ depending when the result is passed as an argument or by register. In the <strong>regflip</strong> and <strong>regexp</strong> variants, performance can be faster (test case #2), the same (test case #1), or slower (test case #3). It seems there is high variability with what the compiler chooses to do. On the other hand, with the <strong>regflip2</strong> and <strong>regexp2</strong> variants, performance is generally faster. Test case #2 has about equal performance but as we have seen, that test case seems to favour results being returned by register.</p>
<h2 id="inlining">Inlining</h2>
<p><img src="/public/matrix_multiplication/inlining_on_off.png" alt="Inlining On/Off" /></p>
<p>As it turns out, inlining sometimes gives a massive performance gain and sometimes it comes down to about the same. In general, it is best to let the compiler make inlining decisions but sometimes in very hot code, it is desirable to manually force the inlining for performance reasons. It thus makes sense to provide at least 2 versions of matrix multiplication: with and without force inlining.</p>
<h2 id="looping-1">Looping</h2>
<p><img src="/public/matrix_multiplication/looping_vs_unrolled.png" alt="Looping VS Unrolled" /></p>
<p>The looping versions are quite interesting. The 2 versions that use intrinsics perform absolutely terribly. They are worse by far, generally breaking out of the charts above. Strangely, they seem to benefit massively from passing the result as an argument (not shown on the graph above). Even with the handwritten assembly versions, we can see that they are generally slower than our unrolled intrinsic version 0. As it turns out, branching is still not a great idea in hot code even with modern hardware.</p>
<h2 id="is-handwritten-assembly-worth-it">Is handwritten assembly worth it?</h2>
<p><img src="/public/matrix_multiplication/handwritten_assembly_vs_intrinsics.png" alt="Handwritten Assembly VS Intrinsics" /></p>
<p>Looking at our looping versions, it is obvious that carefully crafting the assembly by hand can still give significant results. However, we must be careful when doing so. <a href="http://stackoverflow.com/questions/41208105/making-assembly-function-inline-in-x64-visual-studio">In particular, with Visual Studio, hand written assembly functions will never be inlined in x64, even by the linker</a>. Something to keep in mind.</p>
<h2 id="best-of-the-best">Best of the best</h2>
<p><img src="/public/matrix_multiplication/reference_vs_best.png" alt="Reference VS The Best" /></p>
<p>In our results, a clear winner stands above all others: version 0 <strong>inlexp2</strong>:</p>
<ul>
<li>In test case #1, it is <strong>34%</strong> faster then the reference implementation</li>
<li>In test case #2, it is <strong>16%</strong> faster then the reference implementation</li>
<li>In test case #3, it is <strong>31%</strong> faster then the reference implementation</li>
</ul>
<p>Even when it isn’t the fastest implementation, it is within measuring error of the leading alternative. And that leading alternative is always a variant of version 0.</p>
<h1 id="conclusion">Conclusion</h1>
<p>As demonstrated by our data, even a hyper optimized piece of code such as matrix multiplication can sometimes be improved by new hardware features such as the AVX broadcast instruction. In particular, the broadcast instruction allows us to reduce register pressure which avoids spilling registers on the stack and saves on the corresponding instructions that do so. On a platform such as x64, register pressure is a real and important problem that must be taken into account for complex and hot code.</p>
<p>From our results, it seems to make sense to provide 2 implementations for matrix multiplication:</p>
<ul>
<li>One of <strong>regflip2</strong>, <strong>regexp2</strong>, or <strong>mem2</strong> that does not force inlining, suitable for everyday usage</li>
<li><strong>inlexp2</strong> that forces inlining, perfect for that piece of hot code that needs to save every cycle</li>
</ul>
<p>This keeps things simple for the user: all variants return the result in a 3rd argument. Macros can be used to keep things clean and fast.</p>
<p>As always with optimizations, it is important to measure often and to never blindly make a change without measuring first.</p>
<p><em>DirectX Math commit <a href="https://github.com/Microsoft/DirectXMath/commit/d1aa00372040715cebdd5bf599314ef6a4fbe713">d1aa003</a></em></p>
Modern SIMD Programming in the SSE Era2017-04-11T00:00:00+00:00http://nfrechette.github.io/2017/04/11/modern_simd_hardware<p>For a very long time now with every new gaming console generation came a new set of hardware. New hardware meant new constraints which meant we had to revisit old optimization tricks, rules, and popular beliefs.</p>
<p>With the latest generation, both leading consoles (<a href="https://en.wikipedia.org/wiki/Xbox_One">Xbox One</a> and <a href="https://en.wikipedia.org/wiki/PlayStation_4">PlayStation 4</a>) have opted for <a href="https://en.wikipedia.org/wiki/X86-64">x64</a> and <a href="https://en.wikipedia.org/wiki/Advanced_Micro_Devices">AMD CPUs</a>. These are indeed significantly different from the old <a href="https://en.wikipedia.org/wiki/PowerPC">PowerPC CPUs</a> used by <a href="https://en.wikipedia.org/wiki/Xbox_360">Xbox 360</a> and <a href="https://en.wikipedia.org/wiki/PlayStation_3">PlayStation 3</a>.</p>
<p>I spent a significant amount of time last year optimizing cloth simulation code that had been written in mind for the previous generation and with a few tweaks I was able to get decent gains on the latest hardware. Unfortunately that code was proprietary and as such I cannot use it to show the changes and lessons I learned. Instead, I will take the decent and well respected <a href="https://github.com/Microsoft/DirectXMath">DirectX Math</a> public library and highlight as well as optimize specific examples.</p>
<p>Due to the large amount of code, charts, and information required to properly explain everything, this topic will be split into several parts:</p>
<ul>
<li><a href="#hardware_highlights">Hardware highlights</a></li>
<li><a href="/2017/04/13/modern_simd_matrix_multiplication/">Matrix multiplication</a></li>
<li>And much more to come…</li>
</ul>
<h1 id="hardware-highlights"><a name="hardware_highlights"></a>Hardware highlights</h1>
<h2 id="the-powerpc-era">The PowerPC era</h2>
<p>In the Xbox 360 and PlayStation 3 generation, both CPUs were quite different, in particular the PlayStation 3 CPU was a significant departure from modern trends at the time. We will focus on it today only because it has been more publicly documented than its counterpart and the lessons we’ll learn from it applied to both consoles.</p>
<p>The PlayStation 3 console sported a fancy <a href="https://en.wikipedia.org/wiki/Cell_(microprocessor)">Cell microprocessor</a>. We will not focus on the multi-threading aspects in this post series and instead take a deeper look at the execution of a single thread at the assembly level. One important characteristic of PowerPC chips is that they are based on the <a href="https://en.wikipedia.org/wiki/Reduced_instruction_set_computing">RISC</a> model. RISC hardware is generally known to have more user addressable registers than <a href="https://en.wikipedia.org/wiki/Complex_instruction_set_computing">CISC</a>, the variant used by Intel and AMD for modern x64 CPUs. In fact, PowerPC CPUs have 32 general purpose registers, 32 floating point registers, and 32 <a href="https://en.wikipedia.org/wiki/AltiVec">AltiVec</a> SIMD registers (note that the Cell SPEs had 128 registers!). Both consoles also had cache lines that were 128 bytes wide and their CPUs did not support <a href="https://en.wikipedia.org/wiki/Out-of-order_execution">out-of-order execution</a>.</p>
<p>This gave rise to two common techniques to optimize code at the assembly level that we will revisit in this series.</p>
<p>First, the large amount of registers meant that we could leverage them easily to mitigate the fact that loading values from memory was slow and could not be easily hidden due to the in-order nature of the execution. This lead to register packing: a technique where values are loaded first in bulk as part of a vector value and specific components are extracted on demand.</p>
<p>Second, because the register sets for floating point and SIMD math were different, moving a value from one set to another involved storing the value in memory (generally the stack) and reloading it. This led to what is commonly known as <a href="http://assemblyrequired.crashworks.org/load-hit-stores-and-the-__restrict-keyword/">load-hit-store</a> stalls. To mitigate this, whenever SIMD math was mixed with scalar math it was generally best to treat simple scalar floating point math as if it were SIMD: the same scalar was replicated to every SIMD lane.</p>
<p>It is also worth noting that the calling convention used on those consoles favours passing many arguments by register, including vector arguments. There are also many other peculiarities that affected how code should best be written for the hardware such as avoiding to stress the poor branch prediction but we will not cover these at this time.</p>
<h2 id="the-x64-sse-era">The x64 SSE era</h2>
<p>The modern Xbox One and PlayStation 4 consoles opted for x64 AMD CPUs. These are state of the art processors with out-of-order execution, powerful branch prediction, and a large set of instructions from SSE and AVX. These CPUs depart from the previous generation significantly in the number of registers they have: 16 general purpose registers and 16 SIMD registers. The cache lines are also standard in size: 64 bytes wide.</p>
<p>The greatly reduced number of registers means that x64 code is much more prone to register pressure issues. Whenever the number of registers needed goes above 16, registers will spill on the stack, degrading performance. In practice, it isn’t a huge issue in large part because x64 supports instructions that directly work from memory, avoiding the need for a separate load instructions (unlike PowerPC CPUs). For example, in the expression <code class="language-plaintext highlighter-rouge">C = add(A, B)</code>, <strong>A</strong> and <strong>B</strong> can both be residing in registers or <strong>B</strong> could optionally be a memory address. Internally the CPU has much more than 16 registers and thanks to register renaming and compound CISC instructions, our <code class="language-plaintext highlighter-rouge">add</code> instruction will end up performing a load for us behind the scenes. Leveraging this can be very important in hot code as we will see in this series.</p>
<p>Another important characteristic of x64 is that scalar floating point math uses the SSE registers. This means that unlike with PowerPC, converting a floating point value into a SIMD value is very cheap (it could be free if you hand write assembly but compilers will generally generate a shuffle instruction regardless of whether or not you need all components). On the other hand, converting a SIMD value into a floating point value is entirely free and no instruction is generated by modern compilers. This is an important factor to take into account and as we will see later in this series, it can be leveraged to great effect.</p>
<h1 id="up-next">Up next</h1>
<p>In the next post we will take a deep look at how 4x4 matrix multiplication is performed in DirectX Math and how we can speed it up in various ways.</p>
Animation Compression: Advanced Quantization2017-03-12T00:00:00+00:00http://nfrechette.github.io/2017/03/12/anim_compression_advanced_quantization<p>I first spoke about this technique at the <a href="http://www.gdconf.com/">Game Developers Conference (GDC)</a> in 2017 and the slides for that presentation can be found <a href="/public/simple_and_powerful_animation_compression_gdc2017.pdf">here</a>. The idea is to take everything that is good about <a href="/2016/11/15/anim_compression_quantization/">simple quantization</a> and super charge it. Since simple quantization is generally a foundation for the <a href="/2016/11/13/anim_compression_families/">other algorithms</a>, this new variant can be used just as well for the same purpose.</p>
<p>It is worth noting that the algorithm I presented used <a href="/2016/11/10/anim_compression_uniform_segmenting/">uniform segmenting</a> as well as <a href="/2016/11/09/anim_compression_range_reduction/">range reduction</a> both per clip and per block.</p>
<p>The key insight is that with simple quantization, using a hard coded bit rate is very naive and overly simplistic. We can do better, here’s how!</p>
<h1 id="variable-bit-rate">Variable Bit Rate</h1>
<p>In the real world, the nature of the data can change quite dramatically from clip to clip, track to track, or even within a track through time. It is thus important to have a solution that can properly adapt to ever changing environmental conditions. A variable bit rate solves this problem for us.</p>
<p>It is ideal for our hierarchical data. As <a href="/2016/10/27/anim_compression_data/">previously mentioned</a>, our bone track data is stored in local space of their parent bone. This means that each bone incurs a small amount of error as a result of the lossy compression and this error will accumulate as we go down the hierarchy. The error accumulates because we need our final bone transforms in world space to perform skinning and thus the final error is visible on our visual mesh. This makes it obvious that bones higher up in the hierarchy such as the root and pelvis will contribute more error and thus require more bits to retain higher accuracy while on the other hand, bones further down such as finger tips or leaf bones do not require as many bits to maintain the same accuracy. A variable bit rate nicely solves this for us.</p>
<p>It is not unusual for some tracks within a clip to be exotic or to vastly differ from the others. Those tracks might require many more bits to retain sufficient accuracy or perhaps they might need far fewer. Again a variable bit rate is ideal for thus.</p>
<p>Some tracks can vary quite a bit through time as well. For example, during a cinematic it is quite common for all characters to be animated simultaneously. However not all characters might be visible on the screen at the same time. Characters that are not visible would typically remain animated but simply not move and thus their animation data remains constant during this time. A variable bit rate again allows us to exploit temporal coherence.</p>
<h1 id="how-it-works">How It Works</h1>
<p>As we saw with simple quantization, it is very common for the bit rate to be hardcoded. Instead, we want our bit rate to vary and adapt and it thus makes sense to calculate it in some way.</p>
<h2 id="per-clip">Per Clip</h2>
<p>One morning, I realized that it might not be all that hard to try and brute force all bit rates within a certain range and find the smallest value that met some defined accuracy threshold. This allowed fast clips to use more bits when they were needed and slower clips to use fewer while still maintaining high accuracy.</p>
<p>A bit rate was considered superior to another if it was lower (yielding a lower memory footprint) and the accuracy as measured with our <a href="/2016/11/01/anim_compression_accuracy/">error function</a> remained within an acceptable threshold.</p>
<h2 id="per-track-type">Per Track Type</h2>
<p>Not long after, I got thinking and realized that each track type is very different. Their units are different and the data also behaves very differently. It thus made sense to attempt and brute force every bit rate for each of our three track types: rotation, translation, and scale. I thus ended up with three bit rates per clip instead of a single value. It later became obvious that if I can do this per clip, I can also do it per block trivially and thus ended up with three bit rates per block. The added overhead was very small compared to the huge memory gains.</p>
<h2 id="per-track">Per Track</h2>
<p>It soon dawned on me that ideally, we wanted to find the best bit rate for each track within each block. The holy grail of our variable rate solution. Unfortunately, the search space for this solution is massive and it is no longer practical to perform a brute force search. A common character clip might easily have 50 to 80 tracks that are animated and each track can have one of several bit rates.</p>
<p>To solve this problem, we needed to find a smart heuristic.</p>
<h1 id="the-algorithm">The Algorithm</h1>
<p>To trim the search space, I opted for a two phase approach: the first phase finds an approximate solution while the second phase refines it to a local minimum. This is an important distinction, the following algorithm does not find the globally optimal solution, instead it settles on an acceptable local minimum. It perhaps could be further improved.</p>
<h2 id="the-search-space">The Search Space</h2>
<p>In order to keep things simple, I settled on 16 possible bit rates that we used internally: 0, 3, 4, …, 16, and 23. These are the number of bits per track key component (e.g. per quaternion component). We need a higher value than 16 in some uncommon cases such as world space cinematic clips where the accuracy needs of the root are very high. As I mention in my presentation, 23 bits might have been chosen prematurely. The idea was to use the same number of bits as the floating point mantissa minus the sign bit but as it turns out, in order to de-quantize, we must be able to accurately and uniquely represent our quantized integers as 32 bit floating point numbers. Sadly, they can only accurately and uniquely represent integers up to 6 significant digits. The end result is thus that we have some rounding happening. I do not know wether or not this is a real issue or not. Perhaps 19 bits might be a more sensible choice. More work is required here to find the optimal value and evaluate the impact of the rounding.</p>
<p>We do have one edge case: when a track has 0 bits per component. This means our track is constant within our evaluation range and when this happens, we no longer need the track range information within that particular block since our range extent is now 0. We instead store our repeating constant key value within the 48 bits we had available for our range. This allows us to use 16 bits per component for our constant key.</p>
<h2 id="initial-state">Initial State</h2>
<p>In order to start our algorithm, we first initialize all our tracks to use the highest bit rate possible. The goal of the algorithm being to lower these values as much as possible while remaining within our accuracy threshold.</p>
<p><img src="/public/advanced_quantization_initial.jpg" alt="Algorithm Initial State" /></p>
<h2 id="first-phase">First Phase</h2>
<p>The first phase will lower the bit rate of every track in lock step as much as possible until we cannot do so without exceeding our error threshold. This allows us to quickly trim the search space by finding the best bit rate globally for a given block. However, there is one exception, during the duration of the first phase, the root tracks remain locked to the highest bit rate and thus the highest accuracy. This is done because some clips, such as a world space cinematic, have an unusually high accuracy requirement on the root tracks and if we process them as we do the others, they will negatively bias the rest of the tracks.</p>
<p><img src="/public/advanced_quantization_phase1.gif" alt="Algorithm Phase 1" /></p>
<h2 id="second-phase">Second Phase</h2>
<p>The second phase iterates over all of our tracks and aims to individually lower their bit rate as much as possible up until we reach our error threshold. The algorithm terminates once all tracks have been processed.</p>
<p><img src="/public/advanced_quantization_phase2.gif" alt="Algorithm Phase 2" /></p>
<p>The order in which the tracks are processed matters. I tried two different orderings: starting at the root bone and going down the hierarchy towards the children and the opposite ordering, starting at the leaf bones and going back up towards the root. As it turns out, the distribution of which bit rate was selected by our algorithm changed quite a bit. Note that in the image, the right-most bit rates are very uncommon but they do happen (they are too rare to show up).</p>
<p><img src="/public/advanced_quantization_bitrate_distribution.jpg" alt="Bit Rate Distribution" /></p>
<p>Starting at the leaf bones, a higher incidence of lower bit rates were selected. This intuitively makes sense since we have more children than we have parents in our hierarchy. On the other hand, we also have a higher incidence of higher bit rates. Again this makes sense since by lowering the children first, we force the parents to retain more bits in order to preserve the accuracy.</p>
<p>Overall, in the data that I had on hand, starting at the leaf bones and going towards the root resulted in a memory footprint that was roughly <strong>2%</strong> smaller without impacting the compression time, accuracy, and our decompression time.</p>
<h1 id="performance">Performance</h1>
<p>The compression speed remains acceptable and it can easily be optimized by evaluating all blocks to compress in parallel.</p>
<p>On the decompression side, everything we do is very simple and fast. Since our data is uniform, all of our data is linearly contiguous in memory and a full key frame typically fits within a handful of cache lines (2-5). This is extremely dense and contributes to our very fast decompression. Sadly due to our variable bit rates, we can easily end up performing unaligned reads but their performance remains entirely acceptable on x64 and even with SSE. In fact, with minimal work, everything can be done in SSE and we could even use the streaming read (non-temporal read) intrinsics if we wanted to.</p>
<p><a href="/2016/11/17/anim_compression_sub_sampling/">Up next: Sub-sampling</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: GDC 2017 Presentation2017-03-08T00:00:00+00:00http://nfrechette.github.io/2017/03/08/anim_compression_gdc2017<p>In early 2016 I had the opportunity to write a novel animation compression algorithm for <a href="https://eidosmontreal.com/en">Eidos Montreal</a> as part of the game engine previously used by <a href="https://en.wikipedia.org/wiki/Rise_of_the_Tomb_Raider">Rise of the Tomb Raider (2015)</a>. Later in February 2017, I had the pleasure to present it at the <a href="http://www.gdconf.com/">Game Developers Conference (GDC)</a>.</p>
<p>The slides for the presentation can be found <a href="/public/simple_and_powerful_animation_compression_gdc2017.pdf">here</a> and a blog post detailing the technique can be found <a href="/2017/03/12/anim_compression_advanced_quantization/">here</a>.</p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Unity 52017-01-30T00:00:00+00:00http://nfrechette.github.io/2017/01/30/anim_compression_unity5<p><a href="https://www.youtube.com/watch?v=AJ6Mkx1KEns" title="Unity 5 Official Trailer"><img src="https://img.youtube.com/vi/AJ6Mkx1KEns/0.jpg" alt="Unity 5 Official Trailer" /></a></p>
<p><a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity 5</a> is a very popular video game engine on mobile devices and other platforms. Being a state of the art game engine, it supports everything you might need when it comes to character animation including compression.</p>
<p>The relevant <a href="https://docs.unity3d.com/Manual/FBXImporter-Animations.html">FBX Importer</a> and <a href="https://docs.unity3d.com/Manual/class-AnimationClip.html">Animation Clip</a> documentation is very sparse. It’s worth mentioning that <a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity 5</a> is a closed source software and as such, there is some amount of uncertainty and speculation. However, I was able to get in touch with an old colleague working at <a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity</a> to clarify what happens under the hood.</p>
<p>Before we dig into what each compression setting does we must first briefly cover the data representations that <a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity 5</a> uses internally.</p>
<h1 id="track-data-encoding">Track Data Encoding</h1>
<p>The engine uses one of three encodings to represent an animation track regardless of the track data type (quaternion, vector, float, etc.):</p>
<ul>
<li><a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#legacy_curve">Legacy Curve</a></li>
<li><a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#streaming_curve">Streaming Curve</a></li>
<li><a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#dense_curve">Dense Curve</a></li>
</ul>
<p>Rotation tracks are always encoded as four curves to represent a full quaternion (one curve per component). An obvious win here could be to instead encode rotations as quaternion logarithms or by dropping the quaternion <strong>W</strong> component or the largest component. This would of course immediately reduce the memory footprint for rotation tracks by <strong>25%</strong> at the expense of a few instructions to reconstruct the original quaternion.</p>
<h2 id="legacy-curve">Legacy Curve</h2>
<p>Legacy curves are a strange beast. The source data is sampled uniformly at a fixed interval such as <strong>30 FPS</strong> and is kept unsorted in full precision. Using discreet samples during decompression a <a href="https://en.wikipedia.org/wiki/Cubic_Hermite_spline">Hermite curve</a> is constructed on the fly and interpolated. It is unclear to me how this format emerged but it has since been superseded by the other two and it is not frequently used.</p>
<p>It must have been quite slow to decompress and should probably be avoided.</p>
<h2 id="streaming-curve">Streaming Curve</h2>
<p>Streaming curves are proper curves that use <a href="https://en.wikipedia.org/wiki/Cubic_Hermite_spline">Hermite coefficients</a>. A track is split into intervals and each interval is encoded as a distinct spline. This allows discontinuities between intervals. For example, a camera cut or teleporting the root in a cinematic. Each interval has a small header of <strong>8</strong> bytes and each control point is stored in full floating point precision plus an index. This is likely overkill. Full floating point precision is typically far too much for encoding rotations and using <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a> to store them on 16 bits per component or less could provide significant memory savings.</p>
<p>The resulting control points are sorted by time followed by track to render them as cache as efficient as possible which is a very good thing. At decompression, a cursor or cache is used to avoid repeatedly searching for our control points when playback is continuous and predictable. For these two reasons streaming curves are very fast to decompress in the average use case.</p>
<h2 id="dense-curve">Dense Curve</h2>
<p>What <a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity 5</a> calls a dense curve I would call a raw format. The original source data is sampled at a fixed interval such as <strong>30 FPS</strong> and nothing more is done to it as far as I am aware. The data is sorted to make it cache efficient by time and track. No <a href="http://nfrechette.github.io/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> is performed or attempted. The sampled values are not quantized and are simply stored with full precision.</p>
<p>Dense curves will typically have a smaller memory footprint than <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#streaming_curve">streaming curves</a> only for very short tracks or for tracks where the data is very noisy such as a motion capture. For this reason, they are unlikely to be used in practice.</p>
<p>Overall their implementation is simple but perhaps a bit naive. Using <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a> would give significant memory gains here without degrading decompression performance and might even speed it up! On the upside decompression speed is very likely to be faster than with streaming curves.</p>
<h1 id="compression-settings">Compression Settings</h1>
<p>At the time of writing, <a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity 5</a> supports three compression settings:</p>
<ul>
<li><a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#no_compression">No Compression</a></li>
<li><a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#keyframe_reduction">Keyframe Reduction</a></li>
<li><a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#optimal">Optimal</a></li>
</ul>
<h2 id="no-compression">No Compression</h2>
<p>The most detailed quote from the documentation about what this setting does is:</p>
<blockquote>
<p>Disables animation compression. This means that Unity doesn’t reduce keyframe count on import, which leads to the highest precision animations, but slower performance and bigger file and runtime memory size. It is generally not advisable to use this option - if you need higher precision animation, you should enable keyframe reduction and lower allowed Animation Compression Error values instead.</p>
</blockquote>
<p>From what I could gather the originally imported clip is sampled uniformly (e.g. <strong>30 FPS</strong>) and each track is converted into a <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#streaming_curve">streaming curve</a>. This ensures everything remains smooth and accurate but the overhead can be very significant since all samples are retained. To make things worse nothing is done for constant tracks with this setting.</p>
<h2 id="keyframe-reduction">Keyframe Reduction</h2>
<p>The most detailed quote from the documentation is:</p>
<blockquote>
<p>Removes redundant keyframes.</p>
</blockquote>
<p>When this setting is used constant tracks will be collapsed to a single value and redundant control points in animated tracks which will be removed within the specified error threshold. This setting uses <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#streaming_curve">streaming curves</a> to represent track data.</p>
<p>Only three thresholds are exposed for this compression setting. One for each track type: rotation, translation, and scale. This is very likely to lead to the problems discussed in my post on <a href="http://nfrechette.github.io/2016/11/01/anim_compression_accuracy/">measuring accuracy</a>. And indeed, a quick search yields <a href="https://forum.unity3d.com/threads/animation-cost-key-reducer.42841/">this gem</a>. Even though it dates from <a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity 3(2010)</a>, I doubt the animation compression has changed much. Unfortunately, the problems it raises are both very telling and common with local space error metric functions. Here are some relevant excerpts:</p>
<blockquote>
<p>Now, you may be asking yourself, why would this guy turn off the key reducer in the first place? The answer is simple. The key reducer sucks. Here’s why.</p>
<p>Every animation I have completed for this project uses planted keys to anchor the feet (and sometimes hands) to the floor. This allows me to grab any part of the body and animate it, knowing that the feet will not move. When I export the FBX, the keys stay intact. I can bring the animation back into Max or into Maya using the keyframe reducer for either software, and the feet remain anchored. When I bring the same FBX into Unity the feet slide around. Often quite noticably. The only way to stop the feet from sliding is to turn off the key reducer.</p>
</blockquote>
<p>This is a very common problem with local space error functions. Tweaking them is hard! The end result is that very often a weaker compression or no compression at all is used when issues are found on a clip by clip basis. I have seen this exact behavior from animators working on <a href="https://en.wikipedia.org/wiki/Unreal_Engine#Unreal_Engine_3">Unreal 3</a> back in the day and very recently in a proprietary AAA game engine. Even though from the user’s perspective the issue is the animation compression algorithm, in reality, the issue is almost entirely due to the error function.</p>
<blockquote>
<p>What I would really like to see is some options within Unity’s animation importer. A couple ideas:</p>
<p>1) Max’s FBX keyframe reduction has several precision threshold settings that dictate how accurate the keyframe reduction should be. In Unity, it’s all or nothing. I would love the ability to adjust the threshold in which a keyframe gets added. I could turn up the sensitivity on some animations to reduce sliding and possibly turn it down on areas that need fewer keys than are given by the current value.</p>
<p>2) I’m not sure if this is possible, but it would be great to set keyframe reductions on some bones and not others. That way I can keep the arm chain in the proper location without having to bloat the keyframes of every bone in the whole skeleton.</p>
</blockquote>
<p>Exposing a single error threshold per track type is very common and provides a source of frustration for animators. They often know which bones need higher accuracy but are unable to properly tweak per bone thresholds. Sadly, when this feature is present the settings often end up being copy & pasted with overly conservative values which yield a higher than needed memory footprint. Nobody has time to tweak a large number of thresholds repeatedly.</p>
<blockquote>
<p>Unity 3 actually corrects the problem by giving us direct control over the keyframe reduction vs allowable error. If you find your animation is sliding to much, dial down the Position Error and/or Rotation Error settings in the animation import settings.</p>
<p>Unfortunately, I didn’t find any satisfying setup :/</p>
<p>I got some characters that move super fast in their anim, and I need the player to see the move correctly for gameplay / balance reasons.</p>
<p>So it can works for some anims, but not for others (making them feel like they are teleporting).</p>
<p>And under a certain reduction threshold, the memory size benefit is too small to resolve loading times problem :/</p>
<p>In fact, the only reduction setting I found that didn’t caused teleportations was :</p>
<p>Position : 0.1<br />
Rotation : 0.1<br />
Scale : 0 (as there is never any animated scale)</p>
<p>But this is still causing huge file sizes :(</p>
</blockquote>
<p>A single error threshold per track type also means that the error threshold has to be as low as your most sensitive bone requires. This will, in turn, retain higher accuracy that might otherwise be needed yielding again a higher memory footprint that is often unacceptable.</p>
<h2 id="optimal">Optimal</h2>
<p>The most detailed quote from the documentation is:</p>
<blockquote>
<p>Let unity decide how to compress. Either by keyframe reduction or by using dense format. Unity will pick the most optimal of the two.</p>
</blockquote>
<p>This is very vague and judging from <a href="http://answers.unity3d.com/questions/822418/what-are-the-differences-between-animation-compres.html">the results</a> of a <a href="http://answers.unity3d.com/questions/1180871/what-did-unitys-animation-compression-options-opti.html">quick search</a>, a lot of people are curious.</p>
<p>Thankfully, I was able to get some answers!</p>
<p>If a track is very short or very noisy (which could happen with motion capture clips or baked simulations), the key reduction algorithm might not give appreciable gains and it is possible that a <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#dense_curve">dense curve</a> might end up having a smaller memory footprint than a <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#streaming_curve">streaming curve</a>. When this happens for a particular track it will use the curve with the smallest memory footprint. As such, within a single clip, we can have a mix of <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#dense_curve">dense</a> and <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#streaming_curve">streaming</a> curves.</p>
<h1 id="conclusion">Conclusion</h1>
<p>The <a href="https://en.wikipedia.org/wiki/Unity_(game_engine)">Unity 5</a> documentation is sparse and at times unclear. It leads to rampant speculation as to what might be going on under the hood and a lot of confusing results.</p>
<p>Its error function is poor, exposing a single value per track type. This leads to classic issues such as turning compression off to retain accuracy and using an overly conservative threshold to retain accuracy at the expense of the memory footprint. It perpetuates the stigma that animation compression can be painful to work with and can easily butcher an animator’s work without manual tweaking. Fixing the error function could be a reasonably simple task.</p>
<p>The optimal compression setting seems to be a very reasonable default value but it is not clear why the other two are exposed at all. Users are very likely to use one of the other settings instead of tweaking the error function thresholds which is probably a bad idea.</p>
<p>All curve types encode the data in full precision with <strong>32-bit</strong> floating point numbers. This is likely overkill in a very large number of scenarios and implementing some form of <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a> could provide huge memory gains with little work and little effort. Due to the reduced memory footprint, decompression timings might even improve.</p>
<p>Furthermore, rotation tracks could be encoded in a better format than a full quaternion further reducing the memory footprint for minimal work.</p>
<p>From what I could find nobody seemed to complain about animation decompression performance at runtime. This is mostly likely a direct result of the cache friendly data format and the usage of a cursor for <a href="http://nfrechette.github.io/2017/01/30/anim_compression_unity5/#streaming_curve">streaming curves</a>.</p>
<p><a href="/2017/03/08/anim_compression_gdc2017/">Up next: GDC 2017 Presentation</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Unreal 42017-01-11T00:00:00+00:00http://nfrechette.github.io/2017/01/11/anim_compression_unreal4<p><a href="https://www.youtube.com/watch?v=WC6Xx_jLXmg" title="Unreal 4 2017"><img src="https://img.youtube.com/vi/WC6Xx_jLXmg/0.jpg" alt="Unreal 4 2017" /></a></p>
<p><a href="https://en.wikipedia.org/wiki/Unreal_Engine#Unreal_Engine_4">Unreal 4</a> offers a wide range of <a href="https://docs.unrealengine.com/latest/INT/Engine/Animation/Sequences/">documented</a> character animation compression settings.</p>
<p>It offers the following compression algorithms:</p>
<ul>
<li><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#least_destructive">Least Destructive</a></li>
<li><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#second_key">Remove Every Second Key</a></li>
<li><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#trivial_keys">Remove Trivial Keys</a></li>
<li><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#bitwise_only">Bitwise Compress Only</a></li>
<li><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#remove_linear">Remove Linear Keys</a></li>
<li><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#compress_independently">Compress each track independently</a></li>
<li><a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#automatic_setting">Automatic</a></li>
</ul>
<p><em>Note: The following review was done with Unreal 4.15</em></p>
<h2 id="least-destructive"><a name="least_destructive"></a>Least Destructive</h2>
<blockquote>
<p>Reverts any animation compression, restoring the animation to the raw data.</p>
</blockquote>
<p>This setting ensures we use raw data. It sets all tracks to use no compression which means they will have full floating point precision where <strong>None</strong> is used for translation and <strong>Float 96</strong> for rotation, explained <a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#bitwise_only">below</a>. Interestingly it does not appear to revert the scale to a sensible default raw compression setting which is likely a bug.</p>
<h2 id="remove-every-second-key"><a name="second_key"></a>Remove Every Second Key</h2>
<blockquote>
<p>Keyframe reduction algorithm that simply removes every second key.</p>
</blockquote>
<p>This setting does exactly what it claims; it removes every other key. It is a variation of <a href="http://nfrechette.github.io/2016/11/17/anim_compression_sub_sampling/">sub-sampling</a>. Each remaining key is further <a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#bitwise_only">bitwise compressed</a>.</p>
<p>Note that this compression setting also removes the <a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#trivial_keys">trivial keys</a>.</p>
<h2 id="remove-trivial-keys"><a name="trivial_keys"></a>Remove Trivial Keys</h2>
<blockquote>
<p>Removes trivial frames of tracks when position or orientation is constant over the entire animation from the raw animation data.</p>
</blockquote>
<p>This takes advantage of the fact that very often <a href="http://nfrechette.github.io/2016/11/03/anim_compression_constant_tracks/">tracks are constant</a> and when it happens a single key is kept and the rest is discarded.</p>
<h2 id="bitwise-compress-only"><a name="bitwise_only"></a>Bitwise Compress Only</h2>
<blockquote>
<p>Bitwise animation compression only; performs no key reduction.</p>
</blockquote>
<p>This compression setting aims to retain every single key and encodes them with various variations. It is a custom flavor of <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a> and will use the same format for every track in the clip. You can select one format per track type: rotation, translation, and scale.</p>
<p>These are the possible formats:</p>
<ul>
<li><strong>None:</strong> Full precision.</li>
<li><strong>Float 96:</strong> Full precision is used for translation and scale. For rotations, a quaternion component is dropped and the rest are stored with full precision.</li>
<li><strong>Fixed 48:</strong> For rotations, a quaternion component is dropped and the rest is stored on <strong>16</strong> bits per component.</li>
<li><strong>Interval Fixed 32:</strong> Stores the value on <strong>11-11-10</strong> bits per component with range reduction. For rotations, a quaternion component is dropped.</li>
<li><strong>Fixed 32:</strong> For rotations, a quaternion component is dropped and the rest is stored on <strong>11-11-10</strong> bits per component.</li>
<li><strong>Float 32:</strong> This appears to be a semi-deprecated format where a quaternion component is dropped and the rest is encoded onto a custom float format with <strong>6</strong> or <strong>7</strong> bits for the mantissa and <strong>3</strong> bits for the exponent.</li>
<li><strong>Identity:</strong> The identity quaternion is always returned for the rotation track.</li>
</ul>
<p>Only three formats are supported for translation and scale: <strong>None, Interval Fixed 32</strong>, and <strong>Float 96</strong>. All formats are supported for rotations.</p>
<p>If a track has a single key it is always stored with full precision.</p>
<p>When a component is dropped from a rotation quaternion, <strong>W</strong> is always selected. This is safe and simple since it can be reconstructed with a square root as long as our quaternion is normalized. The sign is forced to be positive during compression by negating the quaternion if <strong>W</strong> was originally negative. A quaternion and its negated opposite <a href="https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation#The_hypersphere_of_rotations">represent the same rotation on the hypersphere</a>.</p>
<p>Sadly, the selection of which format to use is done manually and no heuristic is used to perform some automatic selection with this setting.</p>
<p>Note that this compression setting also removes the <a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#trivial_keys">trivial keys</a>.</p>
<h2 id="remove-linear-keys"><a name="remove_linear"></a>Remove Linear Keys</h2>
<blockquote>
<p>Keyframe reduction algorithm that simply removes keys which are linear interpolations of surrounding keys.</p>
</blockquote>
<p>This is a classic variation on <a href="http://nfrechette.github.io/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> with a few interesting twists.</p>
<p>They use a <a href="http://nfrechette.github.io/2016/11/01/anim_compression_accuracy/">local space error metric coupled with a partial virtual vertex error metric</a>. The local space metric is used classically to reject key removal beyond some error threshold. On the other hand, they check the impacted end effector (e.g. leaf) bones that are children of a given track. To do so a virtual vertex is used but no attempt is made to ensure it isn’t co-linear with the rotation axis. Bones in between the two bones (the one being modified and the leaf) are not checked at all and should they end up with unacceptable error, it will be invisible and not accounted for by the metric.</p>
<p>There is also a bias applied on tracks to attempt to remove similar keys on children and their parent bone tracks. I’m not entirely certain why they would opt to do this but I doubt it can harm the results.</p>
<p>On the decompression side no cursor or acceleration structure is used; meaning that each time we sample the clip we will need to search for which keys to interpolate between and we do so per track.</p>
<p>Note that this compression setting also removes the <a href="http://nfrechette.github.io/2017/01/11/anim_compression_unreal4/#trivial_keys">trivial keys</a>.</p>
<h2 id="compress-each-track-independently"><a name="compress_independently"></a>Compress each track independently</h2>
<blockquote>
<p>Keyframe reduction algorithm that removes keys which are linear interpolations of surrounding keys, as well as choosing the best bitwise compression for each track independently.</p>
</blockquote>
<p>This compression setting combines the <a href="http://nfrechette.github.io/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> algorithm along with the <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a> bitwise compression algorithm.</p>
<p>It uses various custom <a href="http://nfrechette.github.io/2016/11/01/anim_compression_accuracy/">error metrics</a> which appear local in nature with some adaptive bias. Each encoding format will be tested for each track and the best result will be used based on the desired error threshold.</p>
<h2 id="automatic"><a name="automatic_setting"></a>Automatic</h2>
<blockquote>
<p>Animation compression algorithm that is just a shell for trying the range of other compression schemes and picking the smallest result within a configurable error threshold.</p>
</blockquote>
<p>A large mix of the previously mentioned algorithms are tested internally. In particular, various mixes of <a href="http://nfrechette.github.io/2016/11/17/anim_compression_sub_sampling/">sub-sampling</a> are tested along with <a href="http://nfrechette.github.io/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> and <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a>. The best result is selected given a specific error threshold. This is likely to be somewhat slow due to the large number of variations tested (<strong>35</strong> at the time of writing!).</p>
<h1 id="conclusion">Conclusion</h1>
<p>In memory, every compression setting has a layout that is naive with the data sorted by tracks followed by keys. This means that during decompression each track will incur a cache miss to read the compressed value.</p>
<p>Performance wise, decompression times are likely to be high in <a href="https://en.wikipedia.org/wiki/Unreal_Engine#Unreal_Engine_4">Unreal 4</a> in large part due to the memory layout not being cache friendly and the lack of an acceleration structure such as a cursor when linear key reduction is used; and it is likely used often with the <em>Automatic</em> setting. This is an obvious area where it could improve.</p>
<p>Memory wise, the linear key reduction algorithm should be fairly conservative but it could be minimally improved by ditching the local space <a href="http://nfrechette.github.io/2016/11/01/anim_compression_accuracy/">error metric</a>. Coupled with the bitwise compression it should yield a reasonable memory footprint; although it could be further improved by using more <a href="/2017/03/12/anim_compression_advanced_quantization/">advanced quantization</a> techniques.</p>
<p><a href="/2017/01/30/anim_compression_unity5/">Up next: Unity 5</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Case Studies2016-12-23T00:00:00+00:00http://nfrechette.github.io/2016/12/23/anim_compression_case_studies<p>In order to keep the material concrete, we will take a deeper look at some popular video game engines. We will highlight the techniques that they use for character animation compression and where they shine or come short.</p>
<ul>
<li><a href="/2017/01/11/anim_compression_unreal4/">Unreal 4</a></li>
<li><a href="/2017/01/30/anim_compression_unity5/">Unity 5</a></li>
</ul>
<p><a href="/2017/01/11/anim_compression_unreal4/">Up next: Unreal 4</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Error Compensation2016-12-22T00:00:00+00:00http://nfrechette.github.io/2016/12/22/anim_compression_error_compensation<p>Unless you use the curves directly authored by the animator for your animation clip, the animation compression you use will end up being lossy in nature and some amount of fidelity will be lost (possibly visible to the naked eye or not). To combat the loss of accuracy, three error compensation techniques emerged over the years to help push the memory footprint further down while keeping the visual results acceptable: in-place correction, additive correction, and <a href="https://en.wikipedia.org/wiki/Inverse_kinematics">inverse kinematic</a> correction.</p>
<h1 id="in-place-correction">In-place Correction</h1>
<p>This technique has already been detailed in <a href="http://www.amazon.com/Game-Programming-Gems-Series/dp/1584505273">Game Programming Gems 7</a>. As such, I won’t go into implementation details unless there is significant interest.</p>
<p>As we have <a href="/2016/10/27/anim_compression_data/">seen previously</a>, our animated bone data is stored in local space which means that any error on a parent bone will propagate to its children. Fundamentally this technique aims to compensate our compression error by applying a small correction to each track to help stop the error propagating in our hierarchy. For example, if we have two parented bones, a small error in the parent bone will offset the child in object space. To account for this and compensate, we can apply a small offset on our child in local space such that the end result in object space is as close to the original animation as possible.</p>
<h2 id="up-sides">Up Sides</h2>
<p>The single most important up side of this technique is that it adds little to no overhead to the runtime decompression. The bulk of the overhead remains entirely on the offline process of compression. We only add overhead on the decompression if we elect to introduce tracks to compensate for the error.</p>
<h2 id="down-sides">Down Sides</h2>
<p>There are four issues with this technique.</p>
<h3 id="missing-tracks">Missing Tracks</h3>
<p>We can only apply a correction on a particular child bone if it is animated. If it is not animated, adding our correction will end up adding more data to compress and we might end up increasing the memory footprint. If the translation track is not animated, our correction will be partial in that with rotation alone we will not be able to match the exact object space position to reach in order to mirror the original clip.</p>
<h3 id="noise">Noise</h3>
<p>Adding a correction for every key frame will tend to yield noisy tracks. Each key frame will end up with a micro-correction which in turn will need somewhat higher accuracy to keep it reliable. For this reason, tracks with correction in them will tend to compress a bit more poorly.</p>
<h3 id="compression-overhead">Compression Overhead</h3>
<p>To properly calculate the correction to apply for every bone track, we must calculate the object space transforms of our bones. This adds extra overhead to our compression time. It may or may not end up being a huge deal, your mileage may vary.</p>
<h3 id="track-ranges">Track Ranges</h3>
<p>Due to the fact that we add corrections to our animated tracks, it is possible and probable that our track ranges might change as we compress and correct our data. This needs to be properly taken into account, further adding more complexity if range reduction is used.</p>
<h1 id="additive-correction">Additive Correction</h1>
<p>Additive correction is very similar to the in-place correction mentioned above. Instead of modifying our track data in-place by incorporating the correction, we can instead store our correction separately as extra additive tracks and combine it during the runtime decompression.</p>
<p>This variation offers a number of interesting trade-offs which are worth considering:</p>
<ul>
<li>Our compressed tracks do not change and will not become noisy nor will their range change</li>
<li>Missing tracks are not an issue since we always add separate additive tracks</li>
<li>Adding the correction at runtime is very fast and simple</li>
<li>Additive tracks are compressed separately and can benefit from a different accuracy threshold</li>
</ul>
<p>However by its nature, the memory overhead will most likely end up being higher than with the in-place variant.</p>
<h1 id="inverse-kinematic-correction">Inverse Kinematic Correction</h1>
<p>The last form of error compensation leverages <a href="https://en.wikipedia.org/wiki/Inverse_kinematics">inverse kinematics</a>. The idea is to store extra object space translation tracks for certain high accuracy bones such as feet and hands. Bones that come into contact with things such as the environment tend to make compression inaccuracy very obvious. Using these high accuracy tracks, we run our <a href="https://en.wikipedia.org/wiki/Inverse_kinematics">inverse kinematic algorithm</a> to calculate the desired transforms of a few parent bones to match our desired pose. This will tend to spread the error of our parent bones, making it less obvious while keeping our point of contact fixed and accurate.</p>
<p>Besides allowing our compression to be more aggressive, this technique does not have a lot of up sides. It does have a number of down sides though:</p>
<ul>
<li>Extra tracks means extra data to compress and decompress</li>
<li>Even a simple <a href="http://mrl.nyu.edu/~perlin/gdc/ik/ik.java.html">2-bone inverse kinematic algorithm</a> will end up slowing down our decompression since we need to calculate object space transforms for our bones involved</li>
<li>By its very nature, our parent bones will no longer closely match the original clip, only the general feel might remain depending on how far the <a href="https://en.wikipedia.org/wiki/Inverse_kinematics">inverse kinematic</a> correction ends up putting us.</li>
</ul>
<h1 id="conclusion">Conclusion</h1>
<p>All three forms of error correction can be used with any compression algorithm but they all have a number of important down sides. For this reason, unless you need the compression to be very aggressive, I would advise against using these techniques. If you choose to do so, the first two appear to be the most appropriate due to their reduced runtime overhead. Note that if you really wanted to, all three techniques could be used simultaneously but that would most likely be very extreme.</p>
<p>Note: I profiled with and without error compensation in Unreal Engine 4 and the results were underwhelming, see <a href="/2019/07/31/pitfalls_linear_reduction_part4/">here</a>.</p>
<p><a href="/2016/12/23/anim_compression_case_studies/">Up next: Case Studies</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Signal Processing2016-12-19T00:00:00+00:00http://nfrechette.github.io/2016/12/19/anim_compression_signal_processing<p><a href="https://en.wikipedia.org/wiki/Signal_processing">Signal processing</a> algorithm variants come in many forms but the most common and popular approach is to use <a href="https://en.wikipedia.org/wiki/Wavelet">Wavelets</a>. Having utilized this method for all character animation clips on all supported platforms for <a href="https://en.wikipedia.org/wiki/Thief_(2014_video_game)">Thief (2014)</a> I have a fair amount to share with you.</p>
<p>Other signal processing algorithm variants include <a href="https://en.wikipedia.org/wiki/Discrete_cosine_transform">Discrete Cosine Transform</a>, <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">Principal Component Analysis</a>, and <a href="https://en.wikipedia.org/wiki/Principal_geodesic_analysis">Principal Geodesic Analysis</a>.</p>
<p>The latter two variants are commonly used alongside clustering and database approaches which I’ll explore if enough interest is expressed but I’ll be focusing on <a href="https://en.wikipedia.org/wiki/Wavelet">Wavelets</a> here.</p>
<h1 id="how-it-works">How It Works</h1>
<p>At their core implementations that leverage wavelets for compression will be split into four distinct steps:</p>
<ul>
<li>Pre-processing</li>
<li>The wavelet transform</li>
<li>Quantization</li>
<li>Entropy coding</li>
</ul>
<p>The flow of information can be illustrated like this:</p>
<p><img src="/public/wavelet_compression_process.jpg" alt="Wavelet Algorithm Layout" /></p>
<p>The most important step is, of course, the wavelet function, around which everything is centered. Covering the wavelet function first will help clarify the purpose of every other step.</p>
<p>Aside from quantization all of the steps involved are effectively lossless and only suffer from minor floating point rounding. By altering how many bits we use for the quantization step we can control how aggressive we want to be with compression.</p>
<p>The decompression is simply the same process of steps performed in reverse order.</p>
<h1 id="wavelet-basics">Wavelet Basics</h1>
<p>We will avoid going too in depth on this topic in this series instead we will focus on discussing the wavelet properties and what they mean for us with respect to character animation and compression in general. A good starting point for the curious would be the <a href="https://en.wikipedia.org/wiki/Haar_wavelet">Haar wavelet</a> which is the simplest of wavelet functions, however, it’s generally avoided for compression.</p>
<p>By definition wavelet functions are recursive. Each application of the function is referred to as a sub-band and will output an equal number of scale and coefficient values each exactly half the original input size. In turn, we can recursively apply the function on the resulting scale values of the previous sub-band. The end result is a single scale value and <code class="language-plaintext highlighter-rouge">N - 1</code> coefficients where <code class="language-plaintext highlighter-rouge">N</code> is the input size.</p>
<p><img src="/public/wavelet_recursive_1d.png" alt="Recursive Wavelet" /></p>
<p><a href="https://en.wikipedia.org/wiki/Haar_wavelet">Haar wavelet</a> scale is simply the sum of two input values and the coefficient represents their difference. As far as I know most wavelet functions function similarly which yield coefficients that are as close to zero as possible and exactly zero for a constant input signal.</p>
<p>The reason the <a href="https://en.wikipedia.org/wiki/Haar_wavelet">Haar wavelet</a> is not suitable for compression because it has a single vanishing moment. This means input data is processed in pairs, each outputting a single scale and a single coefficient. The pairs never overlap which means that if there is a discontinuity in between two pairs it will not be taken into account and yield undesirable artifacts if the coefficients are not accurate. A decent alternative is to use a <a href="https://en.wikipedia.org/wiki/Daubechies_wavelet">Daubechies D4 wavelet</a>. This is the function I used on <a href="https://en.wikipedia.org/wiki/Thief_(2014_video_game)">Thief (2014)</a> and it turned out quite decently for our purposes.</p>
<p>The wavelet transform can be entirely lossless by using an integer variant but in practice, an ordinary floating point variant is appropriate since compression is lossy by nature and the rounding will not measurably impact the results.</p>
<p>Since wavelet function decomposes a signal on an <a href="https://en.wikipedia.org/wiki/Orthonormal_basis">orthonormal basis</a> we will be able to achieve the highest compression by considering as much of the signal as possible not unlike <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">principal component analysis</a>. Simply concatenate all tracks together into a single 1D signal. The upside of this is that by considering all data as a whole we can find a single <a href="https://en.wikipedia.org/wiki/Orthonormal_basis">orthonormal basis</a> which will allow us to quantize more aggressively but by having a larger signal to transform the decompression speed will suffer. To keep the process reasonably fast in practice on modern hardware each track would likely be processed independently in a small power of two, such as 16 keys at a time. For <a href="https://en.wikipedia.org/wiki/Thief_(2014_video_game)">Thief (2014)</a>, all rotation tracks and translation tracks were aggregated independently up to a maximum segment size of 64 KB. We ran the wavelet transform once for rotation tracks, and once for translation tracks.</p>
<h1 id="pre-processing">Pre-processing</h1>
<p>Because wavelet functions are recursive the size of the input data needs to be a power of two. If our size doesn’t match we will need to introduce some form of padding:</p>
<ul>
<li>Pad with zeroes</li>
<li>Repeat the last value</li>
<li>Mirror the signal</li>
<li>Loop the signal</li>
<li>Something even more creative?</li>
</ul>
<p>Which padding approach you choose is likely to have a fairly minimal impact on compression. Your guess is as good as mine regarding which is best. In practice, it’s best to avoid padding as much as possible by keeping input sizes fairly small and processing the input data in blocks or segments.</p>
<p>The scale of output coefficients is a function of the scale and smoothness of our input values. As such it makes sense to perform <a href="http://nfrechette.github.io/2016/11/09/anim_compression_range_reduction/">range reduction</a> and to normalize our input values.</p>
<h1 id="quantization">Quantization</h1>
<p>After applying the wavelet transform the number of output values will match the number of input values. No compression has happened yet.</p>
<p>As mentioned previously our output values will be partitioned into sub-bands, and a single scale value somewhat centered around zero — both positive and negative. Each sub-band will end up with a different range of values. Larger sub-bands resulting from the first applications of the wavelet function will be filled with high-frequency information while the smaller sub-bands will comprise the low-frequency information. This is important. It means that a single low-frequency coefficient will impact a larger range of values after performing the inverse wavelet transform. Because of this low-frequency coefficients need higher accuracy than high-frequency coefficients.</p>
<p>To achieve compression we will quantize our coefficients into a reduced number of bits while keeping the single scale value with full precision. Due to the nature of the data, we will perform range reduction per sub-band and normalize our values between <code class="language-plaintext highlighter-rouge">[-1.0, 1.0]</code>. We only need to keep the range extent for reconstruction and simply assume that the range is centered around zero. Quantization might not make sense for the lower frequency sub-bands with 1, 2, 4 coefficients due to the extra overhead of the range extent. Once our values are normalized we can quantize them. To choose how many bits to use per coefficient we can simply hard code a high number such as 16 bits, 12 bits, or alternatively experiment with values in an attempt to optimize a solution to meet an error threshold. Quantization could also be performed globally to reduce the range of information overhead instead of per sub-band depending on the number of input values being processed. For example processing 16 keys at a time.</p>
<h1 id="entropy-coding">Entropy Coding</h1>
<p>In order to be competitive with other techniques, we need to push compression further using <a href="https://en.wikipedia.org/wiki/Entropy_encoding">entropy coding</a> which is an entirely lossless compression step.</p>
<p>After quantization we obtain a number of integer values all centered around zero and a single scale. The most obvious thing that we can compress now is the fact that we have very few large values. To leverage this we apply a <a href="https://wiki.multimedia.cx/index.php/Zigzag_Reordering">zigzag transform</a> on our data, mapping negative integers to positive unsigned integers such that values closest to zero remain closest to zero. This transforms our data in such a way that we still end up with very few large values which are significant because it means that most of our values represented in memory now have many leading zeroes.</p>
<p>For example suppose we quantize everything onto 16 bit signed integers: <code class="language-plaintext highlighter-rouge">-50, 50, 32760</code>. In memory these values are represented with <a href="https://en.wikipedia.org/wiki/Two's_complement">twos complement</a>: <code class="language-plaintext highlighter-rouge">0xFFCE, 0x0032, 0x7FF8</code>. This is not great and how to compress this further is not immediately obvious. If we apply the <a href="https://wiki.multimedia.cx/index.php/Zigzag_Reordering">zigzag transform</a> and map our signed integers into unsigned integers: <code class="language-plaintext highlighter-rouge">100, 99, 65519</code>. In memory these unsigned integers are now represented as: <code class="language-plaintext highlighter-rouge">0x0064, 0x0063, 0xFFEF</code>. An easily predictable pattern emerges with smaller values with a lot of leading zeroes which will compress well.</p>
<p>At this point, a generic entropy coding algorithm is used like <a href="https://en.wikipedia.org/wiki/Zlib">zlib</a>, <a href="https://en.wikipedia.org/wiki/Huffman_coding">Huffman</a>, or some custom <a href="https://en.wikipedia.org/wiki/Arithmetic_coding">arithmetic coding algorithm</a>. Luke Mamacos gives a <a href="https://worldoffries.wordpress.com/2015/08/25/simple-entropy-coder/">decent example</a> of a wavelet arithmetic encoder that takes advantage of leading zeros.</p>
<p>It’s worth noting that if you process a very large input in a single block you will likely end up with lots of padding at the end. This typically ends up as all zero values after the quantization step and it can be beneficial to use <a href="https://en.wikipedia.org/wiki/Run-length_encoding">run length encoding</a> to compress those before the entropy coding phase.</p>
<h1 id="in-the-wild">In The Wild</h1>
<p>Signal processing algorithms tend to be the most complex to understand while requiring the most code. This makes maintenance a challenge which is represented by a decreased use in the wild.</p>
<p>While these compression methods can be used competitively if the right entropy coding algorithm is used, they tend to be far too slow to decompress, too complex to implement, and too challenging to maintain for the results that they yield.</p>
<p>Due to its popularity at the time I introduced wavelet compression to <a href="https://en.wikipedia.org/wiki/Thief_(2014_video_game)">Thief (2014)</a> to replace the <a href="http://nfrechette.github.io/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> algorithm used in <a href="https://en.wikipedia.org/wiki/Unreal_Engine#Unreal_Engine_3">Unreal 3</a>. <a href="http://nfrechette.github.io/2016/12/07/anim_compression_key_reduction/">Linear key reduction</a> was very hard to tweak properly due to a naive error function it used resulting in a large memory footprint or inaccurate animation clips. The wavelet implementation ended up being faster to compress with and yielded a smaller memory footprint with good accuracy.</p>
<h1 id="performance">Performance</h1>
<p>Fundamentally the wavelet decomposition allows us to exploit temporal coherence in our animation clip data, but this comes at a price. In order to sample a single keyframe, we must reverse everything. Meaning, if we process 16 keys at a time we must decompress our 16 keys to sample a single one of them (or two if we linearly interpolate as we normally would when sampling our clip). For this reason, wavelet implementations are terribly slow to decompress and speeds end up not being competitive at all which only gets worse as you process a larger input signal. On <a href="https://en.wikipedia.org/wiki/Thief_(2014_video_game)">Thief (2014)</a> full decompression on the <a href="https://en.wikipedia.org/wiki/PlayStation_3">Play Station 3</a> <a href="https://en.wikipedia.org/wiki/Cell_(microprocessor)#Synergistic_Processing_Elements_.28SPE.29">SPU</a> took between <strong>800us</strong> and <strong>1300us</strong> for blocks of data up to 64 KB.</p>
<p>Obviously, this is entirely unacceptable with other techniques in the range of <strong>30us</strong> and <strong>200us</strong>. To mitigate this and keep it competitive an intermediate cache is necessary.</p>
<p>The idea of the cache is to perform the expensive decompression once for a block of data (e.g. 16 keys) and re-use it in the future. At 30 FPS our 16 keys will be usable for roughly 0.5 seconds. This, of course, comes with a cost as we now need to implement and maintain an entirely new layer of complexity. We must first decompress into the cache and then interpolate our keys from it. The decompression can typically be launched early to avoid stalls when interpolating but it is not always possible. This is particularly problematic on the first frame of gameplay where a large number of animations will start to play at the same time while our cache is empty or stale. For similar reasons, the same issue happens when a cinematic moment starts or any moment in gameplay with major or abrupt change.</p>
<p>On the upside, as we decompress only once into the cache we can also take a bit of time to swizzle our data and sort it by key and bone such that our data per key frame is now contiguous. Sampling from our cache then becomes more or less equivalent to sampling with <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a>. For this reason sampling from the cache is extremely fast and competitive (as fast as <a href="http://nfrechette.github.io/2016/11/15/anim_compression_quantization/">simple quantization</a>).</p>
<p>Our small cache for <a href="https://en.wikipedia.org/wiki/Thief_(2014_video_game)">Thief (2014)</a> was held in main memory while our wavelet compressed data was held in video memory on the <a href="https://en.wikipedia.org/wiki/PlayStation_3">Play Station 3</a>. This played very well in our favor with the rare decompressions not impacting the rendering bandwidth as much and keeping interpolation fast. This also contributed to slower decompression times but it was still faster than it was on the <a href="https://en.wikipedia.org/wiki/Xbox_360">Xbox 360</a>.</p>
<p>In conclusion signal processing algorithms should be avoided in favor of simpler algorithms that are easier to implement, maintain, and end up just as competitive when properly implemented.</p>
<p><a href="/2016/12/22/anim_compression_error_compensation/">Up next: Error Compensation</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Curve Fitting2016-12-10T00:00:00+00:00http://nfrechette.github.io/2016/12/10/anim_compression_curve_fitting<p><a href="https://en.wikipedia.org/wiki/Curve_fitting">Curve fitting</a> builds on what we last saw with <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a>. With it, we saw that we leveraged linear interpolation to remove keys that could easily be predicted. Curve fitting archives the same feat by using a different interpolation method: a spline function.</p>
<h1 id="how-it-works">How It Works</h1>
<p>The algorithm is already fairly well described by <a href="https://engineering.riotgames.com/news/compressing-skeletal-animation-data">Riot Games</a> and Bitsquid (now called Stingray) in <a href="http://bitsquid.blogspot.ca/2009/11/bitsquid-low-level-animation-system.html">part 1</a> and <a href="http://bitsquid.blogspot.ca/2011/10/low-level-animation-part-2.html">part 2</a>, and as such I will not go further into details at this time.</p>
<p><a href="https://en.wikipedia.org/wiki/Cubic_Hermite_spline#Catmull.E2.80.93Rom_spline">Catmull-Rom</a> splines are a fairly common and a solid choice to represent our curves.</p>
<h1 id="in-the-wild">In The Wild</h1>
<p>This algorithm is again fairly popular and is used by most animation authoring software and many game engines. Sadly, I never had the chance to get my hands on a state of the art implementation of this algorithm and as such I can’t quite go as far in depth as I would otherwise like to do.</p>
<h1 id="performance">Performance</h1>
<p>Most character animated tracks move very smoothly and approximating them with a curve is a very natural choice. In fact, clips that are authored by hand are often encoded and manipulated as a curve in Maya (or 3DS Max). If the original curves are available, we can use them as-is. This also makes the information very dense and compact. The memory footprint of curve fitting should be considerably lower than with <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> but I do not have access to competitive implementations of both algorithms to make a fair comparison.</p>
<p>For example, take this screen capture from some animation curves in Unity:</p>
<p><img src="/public/unity_curves.jpg" alt="Animation Curves" /></p>
<p>We can easily see that each track has <strong>five</strong> control points but with a total clip duration of <strong>2.5 seconds</strong> (note that the image uses a sample rate of 25 FPS which makes the numbering a bit quirky) we would need <code class="language-plaintext highlighter-rouge">2.5 seconds * 30 frames/second = 75 frames</code> to represent the same data. Even after using <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a>, the number of keys would remain higher than <strong>five</strong>.</p>
<p>As with <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a>, our spline control points will have time markers and most of what was mentioned previously will apply to curve fitting as well: we need to search for our neighbour control points, we need to sort our data to be cache efficient, etc.</p>
<p>One important distinction is that while <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> only needed two keys per track to reconstruct our desired value at a particular time <code class="language-plaintext highlighter-rouge">T</code>, with curve fitting we might need more. For example, <a href="https://en.wikipedia.org/wiki/Cubic_Hermite_spline#Catmull.E2.80.93Rom_spline">Catmull-Rom</a> splines require four control points. This makes it more likely to increase the amount of cache lines we need to read when we sample our clip. For this reason, and the fact that a spline interpolation function is more expensive to execute, decompression should be slower than with <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> but without access to a solid implementation, the fact that it might be slower is only an educated guess at this point.</p>
<h1 id="additional-reading">Additional Reading</h1>
<ul>
<li><a href="http://jamie-wong.com/post/bezier-curves/">Bezier Curves</a></li>
</ul>
<p><a href="/2016/12/19/anim_compression_signal_processing/">Up next: Signal Processing</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Linear Key Reduction2016-12-07T00:00:00+00:00http://nfrechette.github.io/2016/12/07/anim_compression_key_reduction<p>With <a href="/2016/11/15/anim_compression_quantization/">simple key quantization</a>, if we needed to sample a certain time <code class="language-plaintext highlighter-rouge">T</code> for which we did not have a key (e.g. in between two existing keys), we linearly interpolated between the two.</p>
<p>A natural extension of this is of course to remove keys or key frames which can be entirely linearly interpolated from their neighbour keys as long as we introduce minimal or no visible error.</p>
<h2 id="how-it-works">How It Works</h2>
<p>The process to remove keys is fairly straight forward:</p>
<ul>
<li>Pick a key</li>
<li>Calculate the value it would have if we linearly interpolated it from its neighbours</li>
<li>If the resulting track error is acceptable, remove it</li>
</ul>
<p>The above algorithm continues until nothing further can be removed. How you pick keys may or may not impact significantly the results. I personally only ever came across implementations that iterated on all keys linearly forward in time. However, in theory you could iterate in any number of ways: random, key with smallest error first, etc. It would be interesting to try various iteration methods.</p>
<p>It is worth pointing out that you need to check the error at a higher level than the individual key you are removing since it might impact other removed keys by changing the neighbour used to remove them. As such you need to look at your error metric and not just the key value delta.</p>
<p>Removing keys is not without side effects: now that our data is no longer uniform, calculating the proper interpolation alpha to reconstruct our value at time <code class="language-plaintext highlighter-rouge">T</code> is no longer trivial. To be able to calculate it, we must introduce in our data a time marker per remaining key (or key frame). This marker of course adds overhead to our animation data and while in the general case it is a win, memory wise, it can increase the overall size if the data is very noisy and no or very few keys can be removed.</p>
<p>A simple formula is then used to reconstruct the proper interpolation alpha:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TP = Time of Previous key
TN = Time of Next key
Interpolation Alpha = (Sample Time - TP) / (TN - TP)
</code></pre></div></div>
<p>Another important side effect in introducing time markers is that when we sample a certain time <code class="language-plaintext highlighter-rouge">T</code>, we must now search to find between which two keys we must interpolate. This of course adds some overhead to our decompression speed.</p>
<p>The removal is typically done in one of two ways:</p>
<ul>
<li>Removal of whole key frames that can be linearly interpolated</li>
<li>Removal of independent keys that can be linearly interpolated</li>
</ul>
<p>While the first is less aggressive and will generally yield a higher memory footprint, the decompression speed will be faster due to needing to search only once to calculate our interpolation alpha.</p>
<p>For example, suppose we have the following track and keys:
<img src="/public/key_reduction.png" alt="Some Keys" /></p>
<p>The key <strong>#3</strong> is of particular interest:
<img src="/public/key_reduction_3.png" alt="Key #3" /></p>
<p>As we can see, we can easily recover the interpolation alpha from its neighbours: <code class="language-plaintext highlighter-rouge">alpha = (3 - 2) / (4 - 2) = 0.5</code>. With it, we can perfectly reconstruct the missing key: <code class="language-plaintext highlighter-rouge">value = lerp(0.35, 0.85, alpha) = 0.6</code>.</p>
<p>Another interesting key is <strong>#4</strong>:
<img src="/public/key_reduction_4.png" alt="Key #4" /></p>
<p>It lies somewhat close to the value we could linearly interpolate from its neighbours: <code class="language-plaintext highlighter-rouge">value = lerp(0.6, 0.96, 0.5) = 0.78</code>. Whether the error introduced by removing it is acceptable or not is determined by our error metric function.</p>
<h2 id="in-the-wild">In The Wild</h2>
<p>This algorithm is perhaps the most common and popular out there. Both Unreal 4 and Unity 5 as well as many popular game engines support this format. They all use slight variations mostly in their error metric function but the principle remains the same. Sadly most implementations out there tend to use a poorly implemented error metric which tends to yield bad results in many instances. This typically stems from using a local error metric where each track type has a single error threshold. Of course the problem with this is that due to the hierarchical nature of our bone data, some bones need higher accuracy (e.g. pelvis, root). Some engines mitigate this by allowing a threshold per track or per bone but this requires some amount of tweaking to get right which is often undesirable and sub-optimal.</p>
<p>Twice in my career I had to implement a new animation compression algorithm and both times were to replace bad linear key reduction implementations.</p>
<p>From the implementations I have seen in the wild, it seems more popular to remove individual keys as opposed to removing whole key frames.</p>
<h2 id="performance">Performance</h2>
<p>Sadly due to the loss of data uniformity, the cache locality of the data we need suffers. Unlike for <a href="/2016/11/15/anim_compression_quantization/">simple key quantization</a>, we can no longer simply sort by key frame if we remove individual keys (you still can if you remove whole key frames though) to keep things cache efficient.</p>
<p>Although I have not personally witnessed it, I suspect it should be possible to use a variation of a technique used by <a href="/2016/12/10/anim_compression_curve_fitting/">curve fitting</a> to sort our data in a cache friendly way. It is well described <a href="http://bitsquid.blogspot.ca/2011/10/low-level-animation-part-2.html">here</a> and we’ll come back to it when we cover <a href="/2016/12/10/anim_compression_curve_fitting/">curve fitting</a>.</p>
<p>The need to constantly search for which neighbour keys to use when interpolating quickly adds up since it scales poorly. The longer our clip is, the wider the range we need to search and the more tracks we have also increases the amount of searching that needs to happen. I have seen two ways to mitigate this: partitioning our clip or by using a cursor.</p>
<p>Partitioning our clip data as we discussed with <a href="/2016/11/10/anim_compression_uniform_segmenting/">uniform segmenting</a> helps reduce the range to search in as our clip length increases. If the number of keys per block is sufficiently small, searching can be made very efficient with a sorting network or similar strategy. The use of blocks will also decrease the need for precision in our time markers by using a similar form of <a href="/2016/11/09/anim_compression_range_reduction/">range reduction</a> which allows us to use fewer bits to store them.</p>
<p>Using a cursor is conceptually very simple. Most clips play linearly and predictably (either forward or backward in time). We can leverage this fact to speed up our search by caching which time we sampled last and which neighbour keys were used to kickstart our search. The cursor overhead is very low if we remove whole key frames but the overhead is a function of the number of animated tracks if we remove individual keys.</p>
<p>Note that it is also quite possible that by using the above sorting trick that it could speed up the search but I cannot speak to the accuracy of this statement at this time.</p>
<p>Even though we can reach a smaller memory footprint with linear key reduction compared to <a href="/2016/11/15/anim_compression_quantization/">simple key quantization</a>, the amount of cache lines we’ll need to touch when decompressing is most likely going to be higher. Along with the need to search for key neighbours, these facts makes it slower to decompress using this algorithm. It remains popular due to the reduced memory footprint which was very important on older consoles (e.g. PS2 and PS3 era) as well as due to its obvious simplicity.</p>
<p>See the following posts for more details:</p>
<ul>
<li><a href="/2019/07/23/pitfalls_linear_reduction_part1/">Pitfalls of linear sample reduction: Part 1</a></li>
<li><a href="/2019/07/25/pitfalls_linear_reduction_part2/">Pitfalls of linear sample reduction: Part 2</a></li>
<li><a href="/2019/07/29/pitfalls_linear_reduction_part3/">Pitfalls of linear sample reduction: Part 3</a></li>
<li><a href="/2019/07/31/pitfalls_linear_reduction_part4/">Pitfalls of linear sample reduction: Part 4</a></li>
</ul>
<p><a href="/2016/12/10/anim_compression_curve_fitting/">Up next: Curve Fitting</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Sub-sampling2016-11-17T00:00:00+00:00http://nfrechette.github.io/2016/11/17/anim_compression_sub_sampling<p>Once upon a time, sub-sampling was a very common compression technique but it is now mostly relegated to history books (and my boring blog!).</p>
<h1 id="how-it-works">How It Works</h1>
<p>It is conceptually very simple:</p>
<ul>
<li>Take your source data, either from Maya (3DS Max, etc.) or already sampled data at some sample rate</li>
<li>Sample (or re-sample) your data at a lower sample rate</li>
</ul>
<p>Traditionally, character animations have a sample rate of 30 FPS. This means that for any given animated track, we end up with 30 keys per second of animation.</p>
<p>Sub-sampling works because in practice, most animations don’t move all that fast and a lower sampling rate is just fine and thus 15-20 FPS is generally good enough.</p>
<h1 id="edge-cases">Edge Cases</h1>
<p>Now of course, this fails miserably if this assumption does not hold true or if a particular key is very important. It can often be the case that an important key is removed with this technique and there is sadly not much that can be done to avoid this issue short of selecting another sampling rate.</p>
<p>It is also worth mentioning that not all sampling rates are necessarily equal. If your source data is already discretized at some original sample rate, sampling rates that will retain whole keys are generally superior to sample rates that will force the generation of new keys by interpolating their neightbours.</p>
<p>For example, if my source animation track is sampled at 30 FPS, I have a key every <code class="language-plaintext highlighter-rouge">1s / 30 = 0.033s</code>. If I sub-sample it at 18 FPS, I have a key every <code class="language-plaintext highlighter-rouge">1s / 18 = 0.055s</code>. This means every key I need is not in sync with my original data and thus new keys must be generated. This will yield some loss of accuracy.</p>
<p>On the other hand, if I sub-sample at 15 FPS, I have a key every <code class="language-plaintext highlighter-rouge">1s / 15 = 0.066s</code>. This means every other key in my original data can be discarded and the remaining keys are identical to my original keys.</p>
<p>Another good example is sub-sampling at 20 FPS which will yield a key every <code class="language-plaintext highlighter-rouge">1s / 20 = 0.05s</code>. This means every 3rd key will be retained from the original data (<strong>0.0</strong> … 0.033 … 0.066 … <strong>0.1</strong> … 0.133 … 0.166 … <strong>0.2</strong> …). The other keys do not line up and will be artificially generated from our original neighbour keys.</p>
<p>The problem of keys not lining up is of course absent if your source animation data is not already discretized. If you have access to the Maya (or 3DS Max) curves, the sub-sampling will retain higher accuracy.</p>
<h1 id="in-the-wild">In The Wild</h1>
<p>In the wild, this used to be a very popular technique on older generation hardware such as the Xbox 360 and the PlayStation 3 (and older). It was very common to keep most of your main character animations with a high sample rate of say 30 FPS, while keeping most of your NPC animations at a lower sample rate of say 15 FPS. Any specific animation that required high accuracy would not be sub-sampled and this selection process was done by hand, making its usage somewhat error prone.</p>
<p>Due to its simplicity, it is also commonly used alongside other compression techniques (e.g. <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a>) to further reduce the memory footprint.</p>
<p>However, nowadays this technique is seldom used in large part because we aren’t as constrained by the memory footprint as we used to be and in part because we strive to push the animation quality ever higher.</p>
<p><a href="/2016/12/07/anim_compression_key_reduction/">Up next: Linear Key Reduction</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Simple Quantization2016-11-15T00:00:00+00:00http://nfrechette.github.io/2016/11/15/anim_compression_quantization<p>Simple quantization is perhaps the simplest compression technique out there. It is used for pretty much everything, not just for character animation compression.</p>
<h1 id="how-it-works">How It Works</h1>
<p>It is fundamentally very simple:</p>
<ul>
<li>Take some floating point value</li>
<li>Normalize it within some range</li>
<li>Scale it to the range of a <strong>N</strong> bit integer</li>
<li>Round and convert to a <strong>N</strong> bit integer</li>
</ul>
<p>That’s it!</p>
<p>For example, suppose we have some rotation component value within the range <code class="language-plaintext highlighter-rouge">[-PI, PI]</code> and we wish to represent it as a signed <strong>16</strong> bit integer:</p>
<ul>
<li>Our value is: <code class="language-plaintext highlighter-rouge">0.25</code></li>
<li>Our normalized value is thus: <code class="language-plaintext highlighter-rouge">0.25 / PI = 0.08</code></li>
<li>Our scaled value is thus: <code class="language-plaintext highlighter-rouge">0.08 * 32767.0 = 2607.51</code></li>
<li>We perform arithmetic rounding: <code class="language-plaintext highlighter-rouge">Floor(2607.51 + 0.5) = 2608</code></li>
</ul>
<p>Reconstruction is the reverse process and is again very simple:</p>
<ul>
<li>Our normalized value becomes: <code class="language-plaintext highlighter-rouge">2608.0 / 32767.0 = 0.08</code></li>
<li>Our de-quantized value is: <code class="language-plaintext highlighter-rouge">0.08 * PI = 0.25</code></li>
</ul>
<p>If we need to sample a certain time <code class="language-plaintext highlighter-rouge">T</code> for which we do not have a key (e.g. in between two existing keys), we typically linearly interpolate between the two. This is just fine because key frames are supposed to be fairly close in time and the interpolation is unlikely to introduce any visible error.</p>
<h1 id="edge-cases">Edge Cases</h1>
<p>There is of course some subtlety to consider. For one, proper rounding needs to be considered. Not all <a href="http://number-none.com/product/Scalar%20Quantization/">rounding modes are equivalent</a> and there are <a href="http://www.eetimes.com/document.asp?doc_id=1274485">so many</a>!</p>
<p>The safest bet and a good default for compression purposes, is to use symmetrical arithmetic rounding. This is particularly important when quantizing to a signed integer value but if you only use unsigned integers, asymmetrical arithmetic rounding is just fine.</p>
<p>Another subtlety is whether to use signed integers or unsigned integers. In practice, due to the way two’s complement works and the need for us to represent the 0 value accurately, when using signed integers, the positive half and the negative half do not have the same size. For example, with <strong>16</strong> bits, the negative half ranges from <code class="language-plaintext highlighter-rouge">[-32768, 0]</code> while the positive half ranges from <code class="language-plaintext highlighter-rouge">[0, 32767]</code>. This asymmetry is generally undesirable and as such using unsigned integers is often favoured. The conversion is fairly simple and only requires doubling our range (<code class="language-plaintext highlighter-rouge">2 * PI</code> in the example above) and offsetting it by the signed ranged (<code class="language-plaintext highlighter-rouge">PI</code> in the example above).</p>
<ul>
<li>Our normalized value is thus: <code class="language-plaintext highlighter-rouge">(0.25 + PI) / (2.0 * PI) = 0.54</code></li>
<li>Our scaled value is thus: <code class="language-plaintext highlighter-rouge">0.54 * 65535.0 = 35375.05</code></li>
<li>With arithmetic rounding: <code class="language-plaintext highlighter-rouge">Floor(35375.05 + 0.5) = 35375</code></li>
</ul>
<p>Another point to consider is: what is the maximum number of bits we can use with this implementation? A single precision floating point value can only accurately represent up to 6-9 significant digits as such the upper bound appears to be around 19 bits for an unsigned integer. Higher than this and the quantization method will need to change to an exponential scale, similar to how <a href="http://aras-p.info/blog/2009/07/30/encoding-floats-to-rgba-the-final/">depth buffers are often encoded in RGB textures</a>. Further research is needed on the nuances of both approaches.</p>
<h1 id="in-the-wild">In The Wild</h1>
<p>The main question remaining is how many bits to use for our integers. Most standalone implementations in the wild use the simple quantization to replace a purely raw floating point format. Tracks will often be encoded on a hardcoded number of bits, typically <strong>16</strong> which is generally enough for rotation tracks as well as range reduced translation and scale tracks.</p>
<p>Most implementations that don’t exclusively rely on simple quantization but use it for extra memory gains again typically use a hardcoded number of bits. Here again <strong>16</strong> bits is a popular choice but sometimes as low as <strong>12</strong> bits is used. This is common for <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> and <a href="/2016/12/10/anim_compression_curve_fitting/">curve fitting</a>. Wavelet based implementations will typically use anywhere between <strong>8</strong> and <strong>16</strong> bits per coefficient quantized and these will vary per sub-band as we will see when we cover this topic.</p>
<p>The resulting memory footprint and the error introduced are both a function of the number of bits used by the integer representation.</p>
<p>This technique is also very commonly used alongside other compression techniques. For example, the remaining keys after <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> will generally be quantized as will curve control points after <a href="/2016/12/10/anim_compression_curve_fitting/">curve fitting</a>.</p>
<h1 id="performance">Performance</h1>
<p>This algorithm is very fast to decompress. Only two key frames need to be decompressed and interpolated. Each individual track is trivially decompressed and interpolated. It is also terribly easy to make the data very processor cache friendly: all we need to do is sort our data by key frame, followed by sorting it by track. With such a layout, all our data for a single key frame will be contiguous and densely packed and the next key frame we will interpolate will follow as well contiguously in memory. This makes it very easy for prefetching to happen either within the hardware or manually in software.</p>
<p>For example, suppose we have <strong>50</strong> animated tracks (a reasonable number for a normal AAA character), each with <strong>3</strong> components, stored on 16 bits (<strong>2</strong> bytes) per component. We end up with <code class="language-plaintext highlighter-rouge">50 * 3 * 2 = 300 bytes</code> for a single key frame. This yields a small number of cache lines on a modern processor: <code class="language-plaintext highlighter-rouge">ceil(300 / 64) = 5</code>. Twice that amount needs to be read to sample our clip.</p>
<p>Due in large part to the cache friendly nature of this algorithm, it is quite possibly the fastest to decompress.</p>
<p><em>My GDC presentation goes in further depth on this topic, its content will find its way here in due time.</em></p>
<p><a href="/2017/03/12/anim_compression_advanced_quantization/">Up next: Advanced Quantization</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Main Compression Families2016-11-13T00:00:00+00:00http://nfrechette.github.io/2016/11/13/anim_compression_families<p>Over the years, a number of techniques have emerged to address the problem of character animation compression. These can be roughly broken down into a handful of general families:</p>
<ul>
<li><a href="/2016/11/15/anim_compression_quantization/">Simple Quantization</a>: Simply storing our key values on fewer bits</li>
<li><a href="/2017/03/12/anim_compression_advanced_quantization/">Advanced Quantization</a>: Super charge our simple quantization with a variable bit rate</li>
<li><a href="/2016/11/17/anim_compression_sub_sampling/">Sub-sampling</a>: Varying the sampling rate to reduce the number of key frames</li>
<li><a href="/2016/12/07/anim_compression_key_reduction/">Linear Key Reduction</a>: Removing keys that can be linearly interpolated from their neighbours</li>
<li><a href="/2016/12/10/anim_compression_curve_fitting/">Curve Fitting</a>: Calculating a curve that approximates our keys</li>
<li><a href="/2016/12/19/anim_compression_signal_processing/">Signal Processing</a>: Using <a href="https://en.wikipedia.org/wiki/Signal_processing">signal processing mathematical tools</a> such as <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">PCA</a> and <a href="https://en.wikipedia.org/wiki/Wavelet">Wavelets</a></li>
</ul>
<p>Note that <a href="/2016/11/15/anim_compression_quantization/">simple quantization</a> and <a href="/2016/11/17/anim_compression_sub_sampling/">sub-sampling</a> can be and often are used in tandem with the other three compression families although they can be used entirely on their own.</p>
<p><a href="/2016/11/15/anim_compression_quantization/">Up next: Simple Quantization</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Uniform Segmenting2016-11-10T00:00:00+00:00http://nfrechette.github.io/2016/11/10/anim_compression_uniform_segmenting<p>Uniform segmenting is performed by splitting a clip into a number of blocks of equal or approximately equal size. This is done for a number of reasons.</p>
<h1 id="range-reduction">Range Reduction</h1>
<p>Splitting large clips (such as cinematics) into smaller blocks allows us to use <a href="/2016/11/09/anim_compression_range_reduction/">range reduction</a> on each track within a block. This can often help reach higher levels of compression especially on longer clips. This is typically done on top of the clip range reduction to further increase our compression accuracy.</p>
<h1 id="easier-seeking">Easier Seeking</h1>
<p>For some compression algorithms, seeking is slow because it involves searching for the keys or control points surrounding the particular time <code class="language-plaintext highlighter-rouge">T</code> we are trying to sample. Techniques exist to speed this up by using optimal sorting but it adds complexity. With blocks sufficiently small (e.g. 16 key frames), optimal sorting might not be required nor the usage of a cursor (cursors are used by these algorithms to keep track of the last sampling performed to accelerate the search of the next sampling). See the posts about <a href="/2016/12/07/anim_compression_key_reduction/">linear key reduction</a> and <a href="/2016/12/10/anim_compression_curve_fitting/">curve fitting</a> for details.</p>
<p>With uniform segmenting, finding which blocks we need is very fast in large part because there are typically few blocks for most clips.</p>
<h1 id="streaming-and-tlb-friendly">Streaming and <a href="https://en.wikipedia.org/wiki/Translation_lookaside_buffer">TLB</a> Friendly</h1>
<p>The easier seeking also means the clips are much easier to stream. Everything you need to sample a number of key frames is bundled together into a single contiguous block. This makes it easy to cache them or prefetch them during playback.</p>
<p>For similar reasons, this makes our clip data more TLB friendly. All our relevant data needed for a number of key frames will use the same contiguous memory pages.</p>
<h1 id="required-for-wavelets">Required For Wavelets</h1>
<p>Wavelet functions typically work on data whose’s size is a power of two. Some form of padding is used to reach a power of two when the size doesn’t match. To avoid excessive padding, clips are split into uniform blocks. See the post about wavelet compression for details.</p>
<h1 id="downsides">Downsides</h1>
<p>Segmenting a clip into blocks does add some amount of memory overhead:</p>
<ul>
<li>Each clip will now need to store a mapping of which block is where in memory.</li>
<li>If <a href="/2016/11/09/anim_compression_range_reduction/">range reduction</a> is done per block as well, each block now includes range information.</li>
<li>Blocks might need a header with some flags or other relevant information.</li>
<li>And for some compression algorithms such as <a href="/2016/12/10/anim_compression_curve_fitting/">curve fitting</a>, it might force us to insert extra control points or to retain more key frames.</li>
</ul>
<p>However, the upsides are almost aways worth the hassle and overhead.</p>
<p><a href="/2016/11/13/anim_compression_families/">Up next: Main Compression Families</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Range Reduction2016-11-09T00:00:00+00:00http://nfrechette.github.io/2016/11/09/anim_compression_range_reduction<p>Range reduction is all about exploiting the fact that the values we compress typically have a much smaller range in practice than in theory.</p>
<p>For example, in theory our translation track range is infinite but in practice, for any given clip, the range is fixed and generally small. This generalizes to rotation and scale tracks as well.</p>
<h1 id="how-it-works">How It Works</h1>
<p>First we start with some animated translation track.</p>
<p><img src="/public/range_reduction_track.png" alt="Animated Track" /></p>
<p>Using this track as an example, we first find the minimum and maximum value that will bound our range: <code class="language-plaintext highlighter-rouge">[16, 40]</code></p>
<p>Using these, we can calculate the range extent: <code class="language-plaintext highlighter-rouge">40 - 16 = 24</code></p>
<p>Now, we have everything we need to re-normalize our track. For every key: <code class="language-plaintext highlighter-rouge">normalized value = (input value - range minimum) / range extent</code></p>
<p><img src="/public/range_reduction_normalized_track.png" alt="Normalized Track" /></p>
<p>Reconstructing our input value becomes trivial: <code class="language-plaintext highlighter-rouge">input value = (normalized value * range extent) + range minimum</code></p>
<p>We thus represent our range as a tuple: <code class="language-plaintext highlighter-rouge">(range minimum, range extent) = (16, 24)</code></p>
<p>This representation has the advantage that reconstructing our input value is very efficient and can use a single fused multiply-add instruction when it is available (and it is on modern CPUs that support FMA as well as modern GPUs).</p>
<p>This increases the information we need to keep around by an extra tuple for every track. For example, a translation track would require a pair of Vector3 values (6 floats) to encode our 3D range.</p>
<p>This extra information allows us to increase our accuracy (sometimes dramatically). While both representations are 100% equivalent mathematically, this only holds true if we have infinite precision. In practice, single precision floating point values only have 6 to 9 significant decimal digits of precision. Typically, our original input range will be much larger than our normalized range and as such its precision will be lower.</p>
<p>This is most easily visible when our track needs to be quantized on a fixed number of bits. For example, our example translation track would typically be bounded by a large hardcoded number such as 100 centimetres. Our hardcoded theoretical track range thus becomes: <code class="language-plaintext highlighter-rouge">[-100, 100]</code>. Tracks with values outside of this range can’t be represented and might need a different base range or a raw un-quantized format. The above range is thus represented by the tuple: <code class="language-plaintext highlighter-rouge">(-100, 200)</code>. If we quantized this range on 16 bits, our input range of 200cm is evenly divided by 65536 which gives us a precision of: <code class="language-plaintext highlighter-rouge">200cm / 65536 = 0.003cm</code>. However, because we know the range is much smaller, using the same number of bits yields a precision of: <code class="language-plaintext highlighter-rouge">1.0 / 65536 = 0.000015 * 24cm = 0.00037cm</code>. Our accuracy has increased by 8x!</p>
<p>In practice, rotation tracks are generally bounded by <code class="language-plaintext highlighter-rouge">[-PI, PI]</code> but animated bones will only use a very small portion of that range on any given clip. Similarly, translation tracks are typically bounded by some hardcoded range but only a small portion is ever used by most bones. This translates naturally to scale tracks as well.</p>
<p>A side effect of range reduction is that all tracks end up being normalized. This is important for the simple quantization compression algorithms as well as when wavelets are used on an aggregate of tracks (as opposed to per track). Future posts will go further into detail.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Range reduction allows us to focus our precision on the range of values that actually matters to us. This increased accuracy very often allows us to be more aggressive in our compression, reducing our overall size despite any extra overhead.</p>
<p>Do note that the extra range information does add up and for very short clips (e.g. 4 key frames) this extra overhead might yield a higher overall memory footprint. These are typically uncommon enough that it is simpler to accept that occasional loss in exchange for simpler decompression that can assume that all tracks use range reduction.</p>
<p><a href="/2016/11/10/anim_compression_uniform_segmenting/">Up next: Uniform Segmenting</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Constant Tracks2016-11-03T00:00:00+00:00http://nfrechette.github.io/2016/11/03/anim_compression_constant_tracks<p>One of the primary advantage of storing bone transforms in local space as opposed to object space is that it reduces dramatically the amount of data that changes. For example, if we have two bones connected, and they have their local rotation animated, we end up with two animated tracks in local space, but three tracks in object space. This is true because the rotation changes from the root bone will cause a translation offset on the second bone.</p>
<p>It is also the case that many bones are never animated by a majority of clips. For example, inverse kinematic (IK) bones are only ever animated when they are required (e.g. object pick up) and facial bones are typically used by cinematic clips or overlay clips. When bones aren’t animated, they will be by definition constant in our raw data.</p>
<p>Consequently, it is often the case that tracks are constant. In general, character animations end up animating many more rotation tracks than they do for translation or scale. As such, translation and scale tracks often end up entirely constant in a clip. Taking advantage of this can be done in one of two ways:</p>
<ul>
<li>Implicitly: linear key reduction will typically reduce such tracks to two keys since everything else can be interpolated (curve fitting and wavelet benefit similarly from this)</li>
<li>Explicitly: by using a bit set that marks tracks as constant or variable</li>
</ul>
<p>Typically, using a bit set and explicitly dropping constant tracks is the superior method to reduce the memory footprint but it does add a bit more complexity to the decompression algorithm. It is entirely worth it in my opinion.</p>
<p>Constant tracks will come in two forms:</p>
<ul>
<li>Constant and equal to the default track value (typically equal to the bind pose or identity)</li>
<li>Constant and equal to some arbitrary track value</li>
</ul>
<p>For obvious reasons, these tracks compress very well. In the first case, we only need to store a single bit per track. If a track is constant, we easily know which value it should have based on the track type (rotation, translation, or scale). In the second case, in addition to our single bit per track, we also need to store our single repeated key value. This single key can be stored raw since its memory footprint overall will remain small regardless or you can quantize it on fewer bits.</p>
<p><em>My GDC presentation will include real numbers on the frequency of constant bones, constant tracks, and constant track components from a modern untitled AAA video game but I cannot include them here just yet.</em></p>
<p>In my experience, the number of constant tracks is closely correlated by the number of constant track components (e.g. a constant scale track is equivalent to three constant scale track components). As such any extra gains we might have from using a bit set per track component (instead of per track) is likely to be offset by the larger bit set size and the extra complexity will likely make decompression slower.</p>
<p><a href="{ post_url 2020-08-09-animation_data_numbers }">See also Animation Data in Numbers</a></p>
<p><a href="/2016/11/09/anim_compression_range_reduction/">Up next: Range Reduction</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Measuring Accuracy2016-11-01T00:00:00+00:00http://nfrechette.github.io/2016/11/01/anim_compression_accuracy<p>Every compression algorithm covered herein is lossy in nature, so how we measure accuracy is critically important. Measured deviation from source data needs to be representative of the visual differences observed.</p>
<p>It is also important to compare algorithms against each other. It is surprising to learn that many academic papers on this topic use varying error metrics making meaningful comparison quite difficult.</p>
<p>Error measuring needs to meet a number of important criteria:</p>
<ul>
<li>Account for the hierarchical nature the data since errors accumulate down the hierarchy when using local space transforms</li>
<li>Account for errors in all three tracks for every bone</li>
<li>Account for the fact that visual mesh almost never overlaps with its skeleton and the further away a vertex is from its bone the larger the error will end up being</li>
</ul>
<h1 id="object-space-vs-local-space">Object Space vs Local Space</h1>
<p>For compression reasons, transforms and blending poses are usually stored in local space. However, due to the hierarchical nature, a small error on a parent bone could end up causing a much larger error on a child’s bone further down. For this reason, errors are best measured in object space.</p>
<p>Surprisingly, it is quite common for compression algorithms to use local space instead. Error is measured using linear key reduction and curve fitting algorithms.</p>
<h1 id="approximating-the-visual-error">Approximating the Visual Error</h1>
<p>Since a skeleton is generally never directly visible errors are visible on the visual mesh instead. This means the most accurate way to measure errors is to perform skinning for every keyframe and comparing vertices before and after compression.</p>
<p>Sadly, this would be terribly slow even with help from the GPU in many instances.</p>
<p>Instead, we can use virtual or fake vertices at a fixed distance from each bone. This is both intuitive and easy to achieve: for every object space bone transform we simply apply the transform to a local vertex position. For example, a vertex in local space of a bone at a fixed offset of 10cm will end up in object space with the object space bone transform applied. This step is done with both the source animation and the compressed animation so we can compare the distance between the two transformed vertices to measure error.</p>
<p>For this to work we need to use two virtual vertices per bone and they must be orthogonal to each other. Otherwise, if our single vertex happens to be co-linear with the bone’s rotation axis, the error could end up being less visible or entirely invisible. For this reason, two vertices are used with the maximum error delta. In practice, we would use something like <code class="language-plaintext highlighter-rouge">vec3(0, 0, distance)</code> and <code class="language-plaintext highlighter-rouge">vec3(0, distance, 0)</code>.</p>
<p>This properly takes into account all three bone tracks and it approximately takes into account the fact that the visual mesh is always some distance away from the skeleton.</p>
<p>Which virtual distance you use is important. A virtual distance much smaller than the distance of real vertices the error might end up being visible since error increases with distance for rotation and scale. If the distance is much larger we might end up retaining more accuracy than we otherwise need.</p>
<p>For most character bones a distance in the range of 2-10cm makes sense. Special bones like the root and camera might require higher accuracy and as such should probably use a distance closer to 1-10m.</p>
<p>Large animated objects also occasionally benefit from vertex to bone distances of 1-10m and either need extra bones added to reduce maximum distance, or the virtual distance needs to be adjusted correspondingly.</p>
<p>It’s also worth mentioning that we could probably extract the furthest skinned vertex per bone from the visual mesh and use that instead but keep in mind some bones might not have skinned vertices such as the root and camera bones.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Accuracy is modified using a single intuitive number, the virtual vertex distance and is measured with a single value, object space distance.</p>
<p>Both Unity and Unreal measure accuracy in ways that are sub-optimal and fail to take into account all three properties outlined above. From what I have seen their techniques are also representative of a lot of game engines out there. Future posts will explore the solutions those game engines provide.</p>
<p>A number of compression algorithms use the error metric function to attempt to converge on a solution like linear key reduction, for example. If metrics are imprecise or poorly represent true error it is very likely that will be reflected in the end result. Many linear key reduction algorithms use a local space error metric which sometimes leads to important keys being removed from distant children of the point of error. For example, a pelvis key being removed can translate into a large error at the fingertip. The typical way to combat this side-effect is to use overly conservative error thresholds but this translates directly in the memory footprint increasing.</p>
<p>See also: <a href="/2023/02/26/dominance_based_error_estimation/">Dominance based error estimation</a></p>
<p><a href="/2016/11/03/anim_compression_constant_tracks/">Up next: Constant Tracks</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Animation Data2016-10-27T00:00:00+00:00http://nfrechette.github.io/2016/10/27/anim_compression_data<p>Animation clips more or less always boil down to the same thing: a series of animated tracks. There might also be other pre-processed data (such as the total root displacement, etc.) but we won’t concern ourselves with this since their impact on compression is usually limited. Instead, we will focus on bone transform tracks: rotation, translation, and scale.</p>
<p>While we can represent all three tracks within a single affine 4x4 matrix, for compression purposes it is usually best to treat them separately. For one thing, a full column of the matrix doesn’t need to be compressed as it never changes and we can typically encode the rotation in a form much more compact than a 3x3 matrix.</p>
<h1 id="rotation">Rotation</h1>
<p>Rotations in general can be represented by many forms and choosing an appropriate representation is important for reducing our memory footprint while maintaining high accuracy.</p>
<p>Common representations include:</p>
<ul>
<li>3x3 matrix (9 float values)
<ul>
<li>wasteful, don’t use this for compression…</li>
</ul>
</li>
<li>quaternion (4 float values)
<ul>
<li>high accuracy, fast decompression, a bit larger than the others</li>
</ul>
</li>
<li>quaternion with dropped W component (3 float values)
<ul>
<li>W can be reconstructed with a square root, the quaternion can be flipped to keep it in the positive hemisphere</li>
</ul>
</li>
<li>quaternion with dropped largest component (3 float values + 2 bit)
<ul>
<li>similar to dropping W, but offers higher accuracy</li>
</ul>
</li>
<li>quaternion logarithm (3 float values)
<ul>
<li>this is equivalent to the rotation axis multiplied by the rotation angle around our axis</li>
</ul>
</li>
<li>axis & angle (4 float values)
<ul>
<li>a bit larger than most alternatives</li>
</ul>
</li>
<li>polar axis & angle (3 float values)
<ul>
<li>high accuracy but slower decompression</li>
</ul>
</li>
<li>Euler angles (3 float values)
<ul>
<li>best avoided due to <a href="https://en.wikipedia.org/wiki/Gimbal_lock">Gimbal lock</a> issues near the poles if quaternions are expected at runtime</li>
</ul>
</li>
</ul>
<p>Which representation is best depends in large part on what the animation runtime uses internally for blending clips. There are three common formats used at runtime:</p>
<ul>
<li>4x4 affine matrix</li>
<li>quaternion</li>
<li>quaternion logarithm</li>
</ul>
<p>If the compressed format doesn’t match the runtime format, a conversion will be required and it has an associated cost (typically sin & cos are involved).</p>
<p>Some formats offer higher accuracy than others at an equal number of bits. A separate post will go further in depth about their various error/size metrics.</p>
<p>Generally speaking, the formats with 3 float values are a good starting point to build compression on. Which of them you use might not have a very dramatic impact on the resulting memory footprint but I could be wrong. Obviously, using a format with 4 floats or more will result in an increased memory footprint but beyond that, it might not matter all that much.</p>
<p>Rotations are often animated but their range of motion is typically fairly small in any given clip and the total range is bounded by the unit sphere (-PI/PI for angles, and -1/+1 for axis/quaternion components).</p>
<p>Because rotations work on the unit sphere, the error needs to be measured on the circumference traced by the associated children. For example, if I have an error of 1 degree on my parent bone rotation, the error at the position of a short child bone will be less than that of a longer child bone: the arc length formed by a 1 degree rotation depends on the sphere radius. This needs to be taken into account by our error measuring function as we will see later in a separate post.</p>
<h1 id="translation">Translation</h1>
<p>Translations are typically encoded as 3 floats. Translations are generally not animated except for a few bones such as the pelvis. They tend to be constant and often match the bind pose translation. Sadly, they are typically not bounded as some bones can be animated in world space (e.g. root bone in a cinematic) or they can move very far from their parent bone (e.g. camera).</p>
<p>They could also be encoded as an axis and a length (4 floats) and in its polar equivalent (3 floats). Whether the polar form is a better tradeoff or not, I cannot say at this time without measuring the impact on the accuracy, memory footprint, and decompression speed. It might not work out so great for very large translation values (e.g. world space).</p>
<h1 id="scale">Scale</h1>
<p>Scale is not common nor is it typical in character animation but it does turn up. For our purposes we will only consider non-uniform scaling with 3 floats but uniform scaling with a single float would be a natural extension.</p>
<p>Scale is very rarely animated and as such it will generally match the bind pose scale. Unfortunately, the range of possible values is unbounded with the minor exception that a scale of 0.0 is generally invalid.</p>
<p>Because the scale affects the rotation part of a 4x4 affine matrix, the same issues pertaining to the error are present.</p>
<p>Similar to translations, it might be possible to encode it in axis/length form but whether or not this is a good idea remains unclear to me at this time.</p>
<h1 id="the-skeleton-hierarchy">The skeleton hierarchy</h1>
<p>Due to its hierarchical nature, we can infer a number of properties about our data:</p>
<ul>
<li>In local space, bones higher up in the hierarchy contribute more to the overall error</li>
<li>In local space, a parent bone and its children can be animated independently, leading to improved compression but reduced accuracy since any error will accumulate down the hierarchy</li>
<li>In object space, if a parent is animated, all children will in turn be animated as well but this also means their error does not propagate</li>
</ul>
<p>As we will see in a later post, these properties can be leveraged to reduce our overall error.</p>
<h1 id="bone-velocity">Bone velocity</h1>
<p>Generally, we will only compress sampled keys from our original source sequence and as such the velocity information is partially lost. However, it remains relevant for a number of reasons:</p>
<ul>
<li>High velocity sequences (or moments) will typically compress poorly and these are quite common when they come from an offline simulation (e.g. cloth or hair simulation) or motion capture (e.g. facial animation).</li>
<li>Smooth and low velocity clips will generally compress best and thankfully these are also the most common. Most hand authored sequences will fall in this category.</li>
<li>Most character animation sequences have low velocity motion for most bones (they move slowly and smoothly)</li>
<li>Many bone tracks have no velocity due to the fact that they are constant and it is also common for bones to be animated only for certain parts of a clip</li>
</ul>
<p>To properly adapt to these conditions, our compression algorithm needs to be adaptive in nature: it needs to use more bits when they are needed for high velocity moments, and use fewer bits when they aren’t needed. As we will see later, there are various techniques to tackle this.</p>
<p><a href="{ post_url 2020-08-09-animation_data_numbers }">See also Animation Data in Numbers</a></p>
<p><a href="/2016/11/01/anim_compression_accuracy/">Up next: Measuring Accuracy</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Terminology2016-10-26T00:00:00+00:00http://nfrechette.github.io/2016/10/26/anim_compression_terminology<h1 id="animation-sequence-or-clip">Animation Sequence or Clip</h1>
<p>Central to character animation is our animation sequence. These are also commonly called animation clips and I will use both terms interchangeably.</p>
<p>A clip is composed of several bones and standalone tracks. This post will not focus on standalone tracks since they can typically be represented as fake bones and regardless, all the techniques we will cover could be used to compress them as well as a natural extension.</p>
<h1 id="skeleton">Skeleton</h1>
<p>Our character animation sequences will always play back over a rigid skeleton. The skeleton defines a number of bones and their hierarchical relationship (parent/child).</p>
<p>A skeleton has a well defined bind pose. The bind pose represents the default reference pose of the skeleton. For example, the pelvis bone would have a bind pose with a fixed translation offset of 60cm above the ground. The first spine bone would have a small offset above the parent pelvis bone, etc.</p>
<p>A skeleton has a root bone. The root bone is the bone highest in the hierarchy on which all other bones are parented. This is typically the bone that is animated to move characters about the world when the motion is animation driven.</p>
<p><img src="/public/unity_skeleton.jpg" alt="Hierarchical Skeleton" /></p>
<h1 id="bone">Bone</h1>
<p>A bone is commonly composed of at least 2 tracks: rotation and translation. The next post will go into further details about how these are represented in the data.</p>
<p>Also very common is for bones to have a scale value associated. When it is present, bones have a 3rd scale track.</p>
<p>All bones have exactly one parent bone (except the root bone which has none) and optional children.</p>
<h1 id="bone-transform">Bone transform</h1>
<p>A bone transform can be represented in a number of ways but for our intents and purposes, we can assume it is a 4x4 <a href="https://en.wikipedia.org/wiki/Affine_transformation">affine matrix</a>. These support rotation, translation, and non-uniform scale.</p>
<p>A bone transform can be either in object space or local space.</p>
<h1 id="track">Track</h1>
<p>A track is composed of 1+ keys. All tracks in a raw sequence will have the same number of keys. A sequence with a single key represents a static pose and consequently has an effective length in time of 0 seconds.</p>
<h1 id="track-key">Track key</h1>
<p>A track key is composed of 1+ components. For example, a translation track would generally have 3 components: X, Y, and Z.</p>
<h1 id="key-component">Key component</h1>
<p>A key component is a 32 bit float value in its raw form. In practice, standalone tracks could have any value or type (bool, int, etc.) but we will mostly focus on rotation, translation, and scale in this series and these will all use floats.</p>
<h1 id="key-interpolation">Key interpolation</h1>
<p>Key interpolation is the act of taking two neighbour keys and interpolating a new key at some time T in between. For example, if I have a key at time 0.0s and another at time 0.5s, I can trivially reconstruct a key at time 0.25s by interpolating my two keys.</p>
<p>Because video games are realtime, it is very rare for the game to sync up perfectly with animation sequences, as such nearly every frame will end up interpolating between 2 keys. Consequently, when we sample our clips for playback, we will typically always sample 2 keys to interpolate in between.</p>
<p>Generally speaking, the interpolation will always be linear even for rotations. This is typically safe since keys are assumed to be somewhat close to one another such that linear interpolation will correspond approximately to the shortest path.</p>
<p>Note that some algorithms perform this interpolation implicitly when you sample them: linear key reduction, curve fitting, etc.</p>
<h1 id="object-space">Object space</h1>
<p>Bone transforms in object space are relative to the whole skeleton. For certain animation clips such as cinematics, they are sometimes animated in world space in which case object space will correspond to world space. When this distinction is relevant, we will explicitly mention it.</p>
<h1 id="local-space">Local space</h1>
<p>Bone transforms in local space are relative to their parent bone. For the root bone, the local space is equivalent to the object space since it has no parent.</p>
<p>Converting from local space to object space entails multiplying the object space transform of our parent bone by the local space transform of our current bone. Converting from object space to local space is the opposite and requires multiplying the inverse object space transform of our parent bone by the object space transform of our current bone.</p>
<h1 id="rotation">Rotation</h1>
<p>A rotation represents an orientation at a given point in time. It can be represented in many ways but the two most common are 3x3 matrices and quaternions. Other format exist and will be covered to some extent in future posts.</p>
<h1 id="translation">Translation</h1>
<p>A translation represents an offset at a given point in time. It is typically represented at a vector with 3 components.</p>
<h1 id="scale">Scale</h1>
<p>A scale is composed of either a single scalar value or a vector with 3 components. The later is used to support non-uniform scaling in which skew and shear are possible. For our purposes, scale will always be a vector with 3 components.</p>
<h1 id="frame">Frame</h1>
<p>A frame represents a unit of time. Most games are played at 30 frames per second (FPS) as such every frame has a length of time equal to 1.0s / 30.0 = 0.03333s.</p>
<p>A clip with 2 keys consequently has 1 unit of time that elapses in between. If the first key is at 0.0s and the second key is at 0.5s, we can reconstruct the key position at any time in between by linearly interpolating our keys. A clip with 11 keys, has 10 frames, etc.</p>
<p>The frame rate of the game does need to match the sample rate of animation clips.</p>
<p>Note that in the literature, the term ‘frame’ is sometimes used to mean a transform (e.g. frame of reference).</p>
<h1 id="sample-rate">Sample rate</h1>
<p>The sample rate dictates the frequency at which the keys are sampled in the original raw clip. A sample rate of 30 frames per second translates in 1 second of time being divided into 30 individual frames. For a sequence with a length of 1 second, the resulting clip will have 30 frames and 31 keys.</p>
<p>If our game runs at a higher frame rate than our sequence sample rate (e.g. 60 FPS), we will end up interpolating between 2 keys twice as much. If in turn our games runs slower than our sample rate (e.g. 15 FPS), some keys will be skipped during playback.</p>
<p>Common sample rates are: 15 FPS, 20 FPS, and 30 FPS.
Most games run at a frame rate of 30 FPS or 60 FPS.</p>
<h1 id="affine-transform">Affine transform</h1>
<p>An <a href="https://en.wikipedia.org/wiki/Affine_transformation">affine transform</a> is a 4x4 matrix that can represent simultaneously rotation, translation, and non-uniform scaling.</p>
<h1 id="quaternion">Quaternion</h1>
<p>A <a href="https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation">quaternion</a> is a 4D complex number that can efficiently and conveniently represent a rotation in 3D space. We will not go further in depth but I will do my best to explain the relevant bits as we encounter them. Since we will mostly use them for storage, we will do very little math involving them.</p>
<p>The most important thing to know about them is that for a quaternion to represent a rotation, it must be normalized (unit length). And obviously, it is made up of 4 scalar float numbers.</p>
<p><a href="/2016/10/27/anim_compression_data/">Up next: Animation Data</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Preface2016-10-25T00:00:00+00:00http://nfrechette.github.io/2016/10/25/anim_compression_preface<p>This post is meant as a preface to explain the overall context of what is character animation, how we use it, and why it needs special consideration.</p>
<p>This series of posts will only cover the compression aspect of animation data, we will not look into everything else that goes into writing an animation runtime library such as Morpheme.</p>
<h1 id="modern-character-animation">Modern character animation</h1>
<p>For the purpose of this series, we will only refer to character animation sequences that play back over a rigid skeleton (also called a rig). In Maya (and other similar products), an animator will typically author the animation sequence from a set of curves by manipulating the skeleton to achieve the desired motion.</p>
<p><img src="/public/unity_curves.jpg" alt="Unity Animation Curves" /></p>
<p>This information is used at runtime to animate the rigid skeleton which in turn deforms the visual mesh bound to it. Note that animation sequences can be authored in 3 main ways: by hand, from motion capture, or procedurally (for example, a cloth simulation).</p>
<p>Central to all this is our rigid skeleton. The skeleton is composed of a number of bones, typically in the range of 100-200 for main characters (but this number can be much larger, up to 1000 in some cases). Some of these will almost always be animated (such as the pelvis), some are animated procedurally at runtime and thus never authored, while others are only animated in a small subset of animations such as facial bones.</p>
<p><img src="/public/unity_skeleton.jpg" alt="Hierarchical Skeleton" height="500" /></p>
<p>The actual process to generate the final skeleton pose will vary but it generally involves blending a number of animation sequences together. The number of poses blended will vary greatly at runtime, and for a main character it can often range from 6 to 20 poses. Although an interesting topic, we will not go into further detail.</p>
<p>The final visual mesh deformation process is commonly referred to as ‘skinning’. Depending on the video game engine, skinning is typically done either by interpolating 4x4 matrices or dual quaternions. The former is the most common but can yield some artifacts while the later approach is the mathematically correct way to do skinning. Both have their pros and cons but we will not go further into detail since it is not immediately relevant to our topic at hand.</p>
<p><img src="/public/unity_mesh.jpg" alt="Skeleton & Visual Mesh" height="500" /></p>
<h1 id="how-is-it-used-in-aaa-video-games">How is it used in AAA video games?</h1>
<p>For a long time now (at least since the original PlayStation), 3D character animation has been used by a large array of video game titles. Over time the techniques used to author the animation sequences have evolved and the amount of content has gradually crept up. Today, it isn’t unusual for a AAA video game to have several thousands of individual character animations.</p>
<p>Animation sequences are used for all sort of things beyond just animating characters: they can animate cameras, vegetation, animals or critters, objects (e.g. door, chest), etc.</p>
<p>As a result of this, it is not unusual to end up having between 20-200MB worth of animations at runtime. On disk, the size is of course even larger.</p>
<h1 id="its-unique-requirements">Its unique requirements</h1>
<p>To generate a single image (or frame), it is not uncommon to end up sampling over 100 animation sequences. As such, it is critically important for this operation to be very fast. Typically on a modern console, we are targeting 50-100us or lower to sample a single sequence.</p>
<p>Due to the large amount of animation sequences, it is also very important for their size to remain small. This is both important on disk, which impacts our streaming speed, and in memory, which impacts the maximum amount of sequences we can hold.</p>
<p>As we will see later, not all animation sequences are created equal, as some will require higher accuracy than others.</p>
<p>These things must all be taken into account when designing or selecting a compression algorithm. The compression algorithms we will be discussing will all be lossy in nature as that is the most common industry practice.</p>
<p><em>Note: All images taken from Unity 5</em></p>
<p><a href="/2016/10/26/anim_compression_terminology/">Up next: Terminology</a></p>
<p><a href="/2016/10/21/anim_compression_toc/"><strong>Back to table of contents</strong></a></p>
Animation Compression: Table of Contents2016-10-21T00:00:00+00:00http://nfrechette.github.io/2016/10/21/anim_compression_toc<p>Character animation compression is an exotic topic that seems to need to be re-approached only about once per decade. Over the course of 2016 I’ve been developing a novel twist to an older technique and I was lucky enough to be invited to talk about it at the <a href="http://www.gdconf.com/">2017 Game Developers Conference (GDC)</a>. This gave me the sufficient motivation to write this series.</p>
<p>The amount of material available regarding animation compression is surprisingly thin and often poorly detailed. My hope for this series is to serve as a reference for anyone looking to improve upon or implement efficient character animation compression techniques.</p>
<p>Each post is self-contained and detailed enough to be approachable to a wide audience.</p>
<h2 id="table-of-contents">Table of Contents</h2>
<ul>
<li><a href="/2016/10/25/anim_compression_preface/">Preface</a></li>
<li><a href="/2016/10/26/anim_compression_terminology/">Terminology</a></li>
<li><a href="/2016/10/27/anim_compression_data/">Animation Data</a>
<ul>
<li><a href="/2020/08/09/animation_data_numbers/">Animation Data in Numbers</a></li>
</ul>
</li>
<li>Measuring Compression Accuracy
<ul>
<li><a href="/2016/11/01/anim_compression_accuracy/">Simulating mesh skinning</a></li>
<li><a href="/2023/02/26/dominance_based_error_estimation/">Dominance based error estimation</a></li>
</ul>
</li>
<li><a href="/2016/11/03/anim_compression_constant_tracks/">Constant Tracks</a></li>
<li><a href="/2016/11/09/anim_compression_range_reduction/">Range Reduction</a></li>
<li><a href="/2016/11/10/anim_compression_uniform_segmenting/">Uniform Segmenting</a></li>
<li><a href="/2016/11/13/anim_compression_families/">Main Compression Families</a>
<ul>
<li><a href="/2016/11/15/anim_compression_quantization/">Simple Quantization</a></li>
<li><a href="/2017/03/12/anim_compression_advanced_quantization/">Advanced Quantization</a></li>
<li><a href="/2016/11/17/anim_compression_sub_sampling/">Sub-sampling</a></li>
<li><a href="/2016/12/07/anim_compression_key_reduction/">Linear Sample Reduction</a>
<ul>
<li><a href="/2019/07/23/pitfalls_linear_reduction_part1/">Pitfalls of linear sample reduction: Part 1</a></li>
<li><a href="/2019/07/25/pitfalls_linear_reduction_part2/">Pitfalls of linear sample reduction: Part 2</a></li>
<li><a href="/2019/07/29/pitfalls_linear_reduction_part3/">Pitfalls of linear sample reduction: Part 3</a></li>
<li><a href="/2019/07/31/pitfalls_linear_reduction_part4/">Pitfalls of linear sample reduction: Part 4</a></li>
</ul>
</li>
<li><a href="/2016/12/10/anim_compression_curve_fitting/">Curve Fitting</a></li>
<li><a href="/2016/12/19/anim_compression_signal_processing/">Signal Processing</a></li>
</ul>
</li>
<li><a href="/2016/12/22/anim_compression_error_compensation/">Error Compensation</a></li>
<li>Bind Pose Optimizations
<ul>
<li><a href="/2018/05/08/anim_compression_additive_bind/">Bind Pose Additive</a></li>
<li><a href="/2022/01/23/anim_compression_bind_pose_stripping/">Bind Pose Stripping</a></li>
</ul>
</li>
<li><a href="/2020/08/11/clip_metadata_packing/">Clip metadata packing</a></li>
<li><a href="/2020/05/04/morph_target_compresion/">Morph Target Animation Compression</a></li>
<li><a href="/2021/01/17/progressive_animation_streaming/">Progressive Animation Streaming</a></li>
<li><a href="/2022/04/03/anim_compression_looping/">Compressing looping animations</a></li>
<li><a href="/2022/06/05/anim_compression_rounding_time/">Manipulating the sampling time for fun and profit</a></li>
<li><a href="/2016/12/23/anim_compression_case_studies/">Case Studies</a>
<ul>
<li><a href="/2017/01/11/anim_compression_unreal4/">Unreal 4</a></li>
<li><a href="/2017/01/30/anim_compression_unity5/">Unity 5</a></li>
</ul>
</li>
<li><a href="/2017/03/08/anim_compression_gdc2017/">GDC 2017 Presentation</a></li>
</ul>
<h2 id="acl-an-open-source-solution">ACL: An open source solution</h2>
<p>My answer to the lack of open source implementations of the above algorithms and topics has prompted me to start my own. See all the code and much more over on GitHub for the <a href="https://github.com/nfrechette/acl">Animation Compression Library</a> as well as its <em>Unreal Engine</em> <a href="https://github.com/nfrechette/acl-ue4-plugin">plugin</a>!</p>
Memory Allocators: Table of Contents2016-10-18T00:00:00+00:00http://nfrechette.github.io/2016/10/18/memory_allocators_toc<p>It is perhaps a bit late but I am starting to have quite a few posts on the topic of memory allocators and navigation was beginning to be a bit complex.</p>
<p>I thought it would be a good idea to have a central place that links everything together.</p>
<p>All the relevant code for this lives under <a href="/2015/05/03/introducing_gin/">Gin</a> (<a href="https://github.com/nfrechette/gin">code</a>) on GitHub.</p>
<h1 id="table-of-content">Table of Content</h1>
<ul>
<li>Linear allocators
<ul>
<li><a href="/2015/05/21/linear_allocator/">Classic linear allocator</a></li>
<li><a href="/2015/06/11/vmem_linear_allocator/">Virtual memory aware linear allocator</a></li>
<li>Thread safe linear allocator</li>
</ul>
</li>
<li><a href="/2016/05/08/stack_frame_allocators/">Stack frame allocators</a>
<ul>
<li><a href="/2016/05/09/greedy_stack_frame_allocator/">Greedy stack allocator</a></li>
<li><a href="/2016/10/17/vmem_stack_frame_allocator/">Virtual memory aware stack allocator</a></li>
</ul>
</li>
<li>Circular allocators
<ul>
<li>Classic circular allocator</li>
<li>Circular frame allocator</li>
</ul>
</li>
<li>Miscellaneous
<ul>
<li><a href="/2015/06/25/out_of_memory/">Dealing with being out of memory</a></li>
</ul>
</li>
</ul>
Virtual Memory Aware Stack Frame Allocator2016-10-17T00:00:00+00:00http://nfrechette.github.io/2016/10/17/vmem_stack_frame_allocator<p>Today we cover the second variant: the virtual memory aware stack frame allocator (<a href="https://github.com/nfrechette/gin/blob/master/include/gin/vmem_stack_frame_allocator.h">code</a>). This is a fairly popular incarnation of the <a href="/2016/05/08/stack_frame_allocators/">stack frame allocator pattern</a> and I have seen quite a few implementations in the wild in line with this toy implementation. Similar in spirit to the <a href="/2015/06/11/vmem_linear_allocator/">virtual memory aware linear allocator</a>, here again we gain in simplicity and performance by leveraging the virtual memory system.</p>
<h3 id="how-it-works">How it works</h3>
<p>Not a whole lot differs from the previous implementation aside from the fact that we are no longer segmented. In AAA video games, it is quite common to have a known upper bound (or it is easily approximated) for these allocators and as such we can simplify the implementation quite a bit by having a single segment. To avoid wasting memory, we simply reserve a virtual address range, and commit/decommit memory as needed.</p>
<p>With a single segment, a lot of logic is removed and simplified. When we allocate, we will commit more memory gradually in blocks of our page size. In a real implementation, this size would likely be larger depending on the page size, 64KB would be a decent size to use.</p>
<p>Popping is very easy and fast and it never causes memory to decommit. This is done to avoid repeated commit/decommit calls and managing some hysteresis. In practice, these allocators often have a known fixed lifetime. For example, we might have an instance of the allocator for main thread allocations during a single frame. As such, it is easy at the end of the frame to decommit once and retain some amount of slack. It is also very common to have 2 or 3 for rendering purposes and while they might be double (or triple) buffered, the principle remains.</p>
<h3 id="what-can-we-use-it-for">What can we use it for</h3>
<p>Similar to the classic segmented implementation, this variant is almost a drop in replacement. It is best suited when the upper bound on the usage is known. It would not be unusual for this implementation to live alongside the segmented implementation: during development, the segmented version is used such that it never fails, and closer to releasing the product, the upper bound is calculated and the virtual memory aware variant is used instead.</p>
<h3 id="what-we-cant-use-it-for">What we can’t use it for</h3>
<p>For obvious reasons, this variant is not ideal when the maximum memory footprint is not known or if it needs to grow unbounded. While on a 64 bit system you could technically reserve 4TB of address space and it would likely be enough, in practice another variant might be a better fit.</p>
<h3 id="edge-cases">Edge cases</h3>
<p>Unlike the classic greedy variant, popping will not decommit memory, as such if memory pressure is high, special care needs to be taken to avoid issues. Under some circumstances, if an out of memory situation arises with some other allocator, decommitting the slack might allow you to retry the allocation.</p>
<h3 id="performance">Performance</h3>
<p>The performance of this allocator is very good. Allocation is O(1) and deallocation is O(1). Both operations are very cheap if no call to the kernel is required to commit memory.</p>
<h3 id="conclusion">Conclusion</h3>
<p>This is our second usable allocator and its usage is quite common. Single segment implementations might not always be virtual memory aware, but they remain the same in nature. Many AAA video game titles have been released with very similar implementations to great success.</p>
<p>Next up, we will revisit the linear memory allocator family to cover an important addition: the thread safe linear allocator. This will be our first thread safe implementation and it is a variant that is very common in multithreaded rendering code.</p>
<p><a href="https://www.reddit.com/r/programming/comments/57y0iv/memory_allocators_explained_the_virtual_memory/">Reddit thread</a></p>
<p><a href="/2016/10/18/memory_allocators_toc/"><strong>Back to table of contents</strong></a></p>
Greedy Stack Frame Allocator2016-05-09T00:00:00+00:00http://nfrechette.github.io/2016/05/09/greedy_stack_frame_allocator<p>Today we cover the first stack frame allocator variant: the greedy stack frame allocator (<a href="https://github.com/nfrechette/gin/blob/master/include/gin/stack_frame_allocator.h">code</a>). It takes the term greedy from how it manages free segments after popping a frame. To better understand how this allocator works, you should first read up on the <a href="/2016/05/08/stack_frame_allocators/">stack frame allocator family</a>.</p>
<h3 id="how-it-works">How it works</h3>
<p>This allocator uses memory segments chained together. When an allocation is made, we find a free segment that can accommodate our allocation request. The first segment we check is the current live segment (the segment last used). Should it be full or the remaining space too small, we then look in the free segment list and pick (or allocate) a suitable candidate. Finally, we update our current live segment to the new used segment.</p>
<p>A segment behaves much like our <a href="/2015/05/21/linear_allocator/">linear allocator</a> previously seen (also commonly called a memory region). Allocation is very cheap since it only involves bumping a pointer forward.</p>
<p>When we pop a frame, all segments that were used and are no longer required or alive are appended to a free list. This is where the allocator is greedy: it will only free the segments once the allocator is destroyed. The upside of this is that we will rarely require to allocate new segments which can involve an expensive malloc/kernel call.</p>
<p>Supporting <code class="language-plaintext highlighter-rouge">realloc</code> is trivial and works as expected.</p>
<p>The allocator also supports supplying it with pre-allocated segments that are externally managed. This is handy if you want to prime it with a larger segment which will thus become the minimum managed size. For example, you might know that the allocator is usually on average around 2MB, instead of letting it allocate segments of 64KB on demand, you register a single 2MB segment at initialization and later if it needs more memory, it will allocate 64KB segments on demand.</p>
<h3 id="what-can-we-use-it-for">What can we use it for</h3>
<p>The greedy nature of the algorithm makes it ideal when the upper bound on memory usage is known or at the very least manageable and we can afford to keep the memory allocated. This is suitable for many applications and is indeed used in a similar form in Unreal 3 under the name of <code class="language-plaintext highlighter-rouge">FMemStack</code>. Many AAA games have shipped with this making extensive use of it.</p>
<h3 id="what-we-cant-use-it-for">What we can’t use it for</h3>
<p>The greedy nature makes it far less ideal when the upper bound isn’t known ahead of time or is possibly unbounded.</p>
<h3 id="edge-cases">Edge cases</h3>
<p>Again, the usual edge cases here involve overflow caused by the size or alignment requested and must be carefully dealt with. Another edge case specific to this allocator remains: because of the linear allocation nature of the algorithm, live segments might not be 100% used and will naturally keep some unused slack. Generally speaking this isn’t such a big deal but it can trivially be tracked if required.</p>
<h3 id="potential-optimizations">Potential optimizations</h3>
<p>Here again we can omit the overflow checks. In practice they are rarely required and could be left to assert instead. This is generally the approach taken in AAA game implementations.</p>
<p>Reallocate support is optional here as well and could be stripped if needed.</p>
<p>The implementation uses a <code class="language-plaintext highlighter-rouge">FrameDescription</code> in order to keep track of the order in which the frames are pushed to ensure we pop them back in reverse order. This is entirely for safety and in practice it could be omitted in a shipping build.</p>
<h3 id="performance">Performance</h3>
<p>The performance of this allocator is very good. Allocation is <em>O(1)</em> and deallocation is a noop. Frame pushing is <em>O(1)</em> and popping is <em>O(n)</em> where <em>n</em> is the number of live segments freed. All of these operations are very cheap.</p>
<p>Frames are very compact and their footprint is split between <code class="language-plaintext highlighter-rouge">AllocatorFrame</code> and a frame description allocated internally.</p>
<p>Segments have a small header (at most 24 bytes) required to chain them as well as to keep track of the allocated size, segment size, and some flags.</p>
<p>Generally, a stack frame allocator will often manage less than 4GB of memory as such it can be templated to use <code class="language-plaintext highlighter-rouge">uint32_t</code> internally. This yields a footprint of 40 bytes on 64 bit platforms for the allocator (a single cache line). This ensures that when we allocate, only 2 cache lines will usually be touched (the allocator instance and the live segment header).</p>
<h3 id="conclusion">Conclusion</h3>
<p>This is our first usable allocator and it is a powerful one. On older, slower hardware where taking locks and using a general malloc/free was expensive, I used a thread local stack allocator to great success in the performance sensitive portions of the code. Its simplicity and flexibility makes it ideal to replace allocators of containers that end up on the stack which might often reallocate memory.</p>
<p>Next up, we will cover a <a href="/2016/10/17/vmem_stack_frame_allocator/">virtual memory aware variant</a> better suited for high performance AAA video games where the upper bound is generally known or can be reliably approximated. This second variant will be the last variant we will cover in this allocator family.</p>
<p>Note that it is often desirable to avoid the allocator to grow unbounded in memory in the presence of spikes and a common way to deal with this is to free some segments when we pop frames by keeping a free list with a small number of entries (slack). This is easily achieved with an hysteresis constraint. Alternatively, at a specific point in the code, you could call a function on the allocator to free some of its slack (e.g: at the start or end of a frame).</p>
<p><a href="https://www.reddit.com/r/programming/comments/4ihr38/memory_allocators_explained_the_greedy_stack/">Reddit thread</a></p>
<p><a href="/2016/10/18/memory_allocators_toc/"><strong>Back to table of contents</strong></a></p>
Stack Frame Allocators2016-05-08T00:00:00+00:00http://nfrechette.github.io/2016/05/08/stack_frame_allocators<p>This allocator family is quite possibly one of the most common memory management technique.</p>
<p>Note that there are many variants possible of this allocator. As such we will only cover a few to give a general overview of what it can look like. Actual production versions in the wild will often be custom and tailored to the specific application needs. Unlike our <a href="/2015/05/21/linear_allocator/">linear allocator</a> previously covered, this allocation technique is very common and in use in many applications.</p>
<h3 id="how-it-works">How it works</h3>
<p>A topic that comes very early when introducing many programming languages is the call stack. It is central to C, C++, and many other languages. The execution stack used for this mechanic operates as a stack frame based memory management algorithm which is handled by the compiler and the hardware (on x86 and x64).</p>
<p>Every C++ programmer is familiar with the stack and all it has to offer. Each function call pushes a new frame on the stack when it enters and later pops it when it exits. In this context, a frame is a section of memory that holds our local function variables that cannot be stored in registers.</p>
<p>Our stack frame allocator works in much the same way except that pushing and popping of frames is made explicit. This makes it a very powerful and familiar tool.</p>
<p>Our frame based allocators will use a common <a href="https://github.com/nfrechette/gin/blob/master/include/gin/allocator_frame.h">AllocatorFrame</a> class and the usage is simple:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">AllocatorFrame</span> <span class="nf">frame</span><span class="p">(</span><span class="n">allocatorInstance</span><span class="p">);</span>
<span class="c1">// Allocate here with ‘allocatorInstance’</span></code></pre></figure>
<p>Popping of the frame is automatic when the destructor is called or it can be popped manually with <code class="language-plaintext highlighter-rouge">AllocatorFrame::Pop()</code>.</p>
<p>In a frame based allocator such as this, when the frame is popped, all allocations made within our frame can be freed and the memory reclaimed. Different allocator variants will handle this differently as we will later see. The bottom line is that you should not keep pointers to that memory since it is no longer valid. On the plus side, freeing memory is very fast as a bulk operation.</p>
<h3 id="what-can-we-use-it-for">What can we use it for</h3>
<p>It allows us to allocate memory dynamically and return it from a function. This is very common and handy when we do not know the maximum amount of potential results returned (maybe we have a rare worst case scenario) and we want to avoid increasing the stack.</p>
<p>It supports <code class="language-plaintext highlighter-rouge">realloc</code> for the topmost allocation, useful when appending to vectors.</p>
<p>When using a thread local allocator, we can avoid expensive heap accesses since we offer better performance due to reduced cache and TLB usage. Execution stack memory is often forced to use 4KB pages and this is often controlled by the kernel but our parallel stack can use any page size it wants (when such control is available to user space).</p>
<p>Moving large allocations (such as strings) off the call stack reduces the risks of a malicious user causing the return address to be overwritten and abused in the presence of buffer overflow bugs. It can trivially replace <code class="language-plaintext highlighter-rouge">alloca</code> and everything it is used for.</p>
<p>Depending on the variant, it can access all system memory unlike our linear allocator.</p>
<p>Because it is often faster than a generalized heap allocator (potentially shared between multiple threads), if our allocated data is thread local (e.g: a vector on the stack), we can trivially migrate the allocations to use our frame allocator and save on allocation, deallocation, and reallocation.</p>
<h3 id="what-we-cant-use-it-for">What we can’t use it for</h3>
<p>Much like our linear allocator, stack frame allocators do not support generalized deallocation of memory. This is largely mitigated by the fact that freeing is still supported but now happens in a last in/first out order due to our stack frames.</p>
<h3 id="performance">Performance</h3>
<p>Performance of stack frame based allocators is generally very good due in large part to their simplicity when allocating and the bulk free operation. It is very common to have a stack frame allocator per thread which can also avoids a lock operation since they generally do not require to be shared between threads.</p>
<p>Generally speaking, the performance is enhanced compared to a generalized heap allocator because by design we use a smaller virtual memory range or we guaranty that we can re-use a previously used range keeping our TLB usage optimal. This also helps the CPU cache.</p>
<h3 id="conclusion">Conclusion</h3>
<p>Up next we will cover a simple variant: <a href="/2016/05/09/greedy_stack_frame_allocator/">the greedy stack frame allocator</a>.</p>
<h3 id="alternate-names">Alternate names</h3>
<p>This memory allocator is often abbreviated to simply ‘Stack Allocator’. There are so many variants that are possible that there are probably just as many names.</p>
<p><em>Note that if you know a better name or alternate names for this allocator, feel free to contact me.</em></p>
<p><a href="https://www.reddit.com/r/programming/comments/4ie1xv/memory_allocators_explained_the_stack_frame/">Reddit thread</a></p>
<p><a href="/2016/10/18/memory_allocators_toc/"><strong>Back to table of contents</strong></a></p>
GPGPU woes part 42016-04-14T00:00:00+00:00http://nfrechette.github.io/2016/04/14/gpgpu_woes_part_4<p>After fixing the <a href="/2015/07/11/gpgpu_woes_part_3/">previous</a> addressing issue, we move on to the next.</p>
<h3 id="inconsistencies-and-limitations">Inconsistencies and limitations</h3>
<p>Here are the outputs for the various OpenCL driver values on my laptop:</p>
<h4 id="windows-81-x64">Windows 8.1 x64</h4>
<ul>
<li>
<p>Device: Intel(R) Iris(TM) Graphics 5100</p>
<ul>
<li>Max workgroup size: 512</li>
<li>Num compute units: 40</li>
<li>Local mem size: 65536</li>
<li>Global mem size: 1837105152</li>
<li>Is memory unified: true</li>
<li>Cache line size: 64</li>
<li>Global cache size: 2097152</li>
<li>Address space size: 64</li>
<li>Kernel local workgroup size: 512</li>
<li>Kernel local mem size: 0</li>
</ul>
</li>
<li>
<p>Device: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz</p>
<ul>
<li>Max workgroup size: 1024</li>
<li>Num compute units: 4</li>
<li>Local mem size: 32768</li>
<li>Global mem size: 1837105152</li>
<li>Is memory unified: true</li>
<li>Cache line size: 64</li>
<li>Global cache size: 262144</li>
<li>Address space size: 64</li>
<li>Kernel local workgroup size: 1024</li>
<li>Kernel local mem size: 12288</li>
</ul>
</li>
</ul>
<h4 id="os-x-x64">OS X x64</h4>
<ul>
<li>
<p>Device: Iris</p>
<ul>
<li>Max workgroup size: 512</li>
<li>Num compute units: 40</li>
<li>Local mem size: 65536</li>
<li>Global mem size: 1610612736</li>
<li>Is memory unified: true</li>
<li>Cache line size: 0</li>
<li>Global cache size: 0</li>
<li>Address space size: 64</li>
<li>Kernel local workgroup size: 512</li>
<li>Kernel local mem size: 30720</li>
</ul>
</li>
<li>
<p>Device: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz</p>
<ul>
<li>Max workgroup size: 1024</li>
<li>Num compute units: 4</li>
<li>Local mem size: 32768</li>
<li>Global mem size: 0</li>
<li>Is memory unified: true</li>
<li>Cache line size: 3145728</li>
<li>Global cache size: 64</li>
<li>Address space size: 64</li>
<li>Kernel local workgroup size: 1</li>
<li>Kernel local mem size: 0</li>
</ul>
</li>
</ul>
<p>As we can see, many things differs and both seems to report bad values for a few things.</p>
<p>An important difference between the CPU and GPU is that CPU local storage is limited to 32KB. But Why? Couldn’t the CPU variant simply consider local storage to be the same as normal memory and thus only be bounded by virtual (or physical) memory?</p>
<p>This seemingly arbitrary limitation makes a lot of sense when you consider the fact that local storage is supposed to be used to avoid touching main memory by keeping intermediate results in fast memory. The <a href="http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html">CUDA programming guide</a> states that they <a href="http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability-2-x">expose functions</a> to let you tweak how much of GPU L1 is split between real L1 usage and local storage and L1 is fast, really fast.</p>
<p>Sadly, on the CPU we usually have no such control with how the L1 is used or controlled and it is also quite small: usually 32KB for code and 32KB for data. Note that some exotic platforms do allow you some minimal control over the CPU caches such as the Wii cache line locking instructions. However, generally speaking, it is very hard to control effectively by hand.</p>
<p>With most CPUs (including the one I am using) having very small L1 caches, it makes sense to limit local storage to the size of the L1. After all, if you design an algorithm around the use of very fast memory, performance could be adversely affected should you have less available in reality.</p>
GPGPU woes part 32015-07-11T00:00:00+00:00http://nfrechette.github.io/2015/07/11/gpgpu_woes_part_3<p>After fixing the <a href="/2015/07/07/gpgpu_woes_part_2/">previous</a> mind bending issue, we move on to the next issue.</p>
<h3 id="on-the-gpu-the-address-space-size-can-differ">On the GPU, the address space size can differ</h3>
<p>Virtually every programmer out there knows that the address space size on the CPU can differ based on the hardware, executable, and kernel. The two most common sizes being 32 bits and 64 bits (partially true since in practice only 48 bits are used for now). Some older programmers will also remember the times of the 16 bits address space.</p>
<p>What few programmers realize is that other chips in their computer might have a different address space size and as such sharing binary structures with them that contain pointers is generally unsafe.</p>
<h3 id="case-in-point-my-gpu-has-a-64-bits-address-space">Case in point: my GPU has a 64 bits address space.</h3>
<p>What this means is that if I run a i386 executable in either OS X or Windows, the pointer size will be 4 bytes on the CPU but the GPU will use 8 bytes for pointers. This means that structures like this that are shared will not work:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">struct</span> <span class="nc">dom_node</span>
<span class="p">{</span>
<span class="k">struct</span> <span class="nc">dom_node</span> <span class="o">*</span><span class="n">parent</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">id</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">tag_name</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">class_count</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">first_class</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">style</span><span class="p">[</span><span class="n">MAX_STYLE_PROPERTIES</span><span class="p">];</span>
<span class="p">};</span></code></pre></figure>
<p>The above code was causing the reading and writing of memory far past the buffer that contained them when running on the GPU. This caused the output to differ from the expected result of the CPU version.</p>
<p>In particular, it ended up corrupting read-only buffers that the kernel was using (the read-only stylesheet) causing further confusion since the output differed from run to run. I was lucky enough that I didn’t end up corrupting anything else more critical as it could have made debugging the issue much harder (e.g: driver crash).</p>
<p>The fix is to ensure there is proper padding by wrapping the pointer in a union:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">struct</span> <span class="nc">dom_node</span>
<span class="p">{</span>
<span class="k">union</span>
<span class="p">{</span>
<span class="k">struct</span> <span class="nc">dom_node</span> <span class="o">*</span><span class="n">parent</span><span class="p">;</span>
<span class="n">cl_ulong</span> <span class="n">pad_parent</span><span class="p">;</span>
<span class="p">};</span>
<span class="n">cl_int</span> <span class="n">id</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">tag_name</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">class_count</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">first_class</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">style</span><span class="p">[</span><span class="n">MAX_STYLE_PROPERTIES</span><span class="p">];</span>
<span class="p">};</span></code></pre></figure>
<p>OpenCL has integer types with the <code class="language-plaintext highlighter-rouge">cl_</code> prefix in order to ensure their size does not differ between the CPU and GPU but sadly no such trick is possible with pointers: we have to resort to our manual union hack. In practice we could probably introduce a macro to wrap this and make it cleaner.</p>
<p>Ideally sharing structures that contain pointers with the GPU isn’t such a good idea. Unless memory is unified, the GPU will not be able to access that memory and as such it is wasteful. Even if the memory is unified, typically the GPU accessible memory must be allocated with particular flags and depending on the platform it might only be possible through the driver.</p>
GPGPU woes part 22015-07-07T00:00:00+00:00http://nfrechette.github.io/2015/07/07/gpgpu_woes_part_2<p>After fixing the <a href="/2015/07/02/gpgpu_woes_part_1/">previous</a> painful issue, we move on to the next strange issue (and much worse!).</p>
<h3 id="on-the-gpu-a-null-pointer-isnt-always-null">On the GPU, a NULL pointer isn’t always NULL</h3>
<p>Despite the code being very simple, the CPU and GPU versions produced very different outputs for me in OS X (not verified in Windows 8.1). For the life of me I could not figure it out as it proved even stranger than the previous issue.</p>
<p>Ultimately, the issue was here:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="c1">// Inlined the CSS_CUCKOO_HASH_FIND macro and minor formatting cleanup
</span>
<span class="n">__global</span> <span class="k">const</span> <span class="k">struct</span> <span class="nc">css_rule</span> <span class="o">*</span> <span class="n">rule</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">hash_</span><span class="o">-></span><span class="n">left</span><span class="p">[</span><span class="n">left_index_</span><span class="p">].</span><span class="n">type</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">hash_</span><span class="o">-></span><span class="n">left</span><span class="p">[</span><span class="n">left_index_</span><span class="p">].</span><span class="n">value</span> <span class="o">==</span> <span class="n">value_</span><span class="p">)</span> <span class="p">{</span>
<span class="n">rule</span> <span class="o">=</span> <span class="o">&</span><span class="n">hash_</span><span class="o">-></span><span class="n">left</span><span class="p">[</span><span class="n">left_index_</span><span class="p">];</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">hash_</span><span class="o">-></span><span class="n">right</span><span class="p">[</span><span class="n">right_index_</span><span class="p">].</span><span class="n">type</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">hash_</span><span class="o">-></span><span class="n">right</span><span class="p">[</span><span class="n">right_index_</span><span class="p">].</span><span class="n">value</span> <span class="o">==</span> <span class="n">value_</span><span class="p">)</span> <span class="p">{</span>
<span class="n">rule</span> <span class="o">=</span> <span class="o">&</span><span class="n">hash_</span><span class="o">-></span><span class="n">right</span><span class="p">[</span><span class="n">right_index_</span><span class="p">];</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">rule_set</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// code
</span>
<span class="p">}</span></code></pre></figure>
<p>Can you see it? If you can’t, I don’t think anybody could blame you.</p>
<p>After painfully narrowing it down to this precise piece of code, it turns out that the <code class="language-plaintext highlighter-rouge">rule_set != 0</code> check was failing when no rule was found (neither <code class="language-plaintext highlighter-rouge">if</code> statement was taken) on the GPU (and was obviously working as expected on the CPU).</p>
<p>This is mere speculation but I have the gut feeling that this issue might be caused because internally some bits in the memory addresses must be used to tell global memory from local memory on the GPU.</p>
<p>It is entirely possible that a NULL pointer for global memory might not equal a NULL pointer for local memory (or constant memory). In such a scenario, comparing both would return <code class="language-plaintext highlighter-rouge">false</code>. Perhaps the memory qualifier information was lost and the optimizer was left to perform a comparison with the literal <em>0</em> value.</p>
<p>The only other alternative would be a compiler bug. Sadly I could not take a look at the generated assembly to double check what was happening.</p>
<p>The fix was simply to introduce a boolean/integer variable that we could safely compare against. Again note that I was more concerned with getting the code to work while keeping it as close to the original than making it fast. It is also possible that force casting the <em>0</em> literal to the pointer type might have worked. This is left as an exercise to the reader.</p>
GPGPU woes part 12015-07-02T00:00:00+00:00http://nfrechette.github.io/2015/07/02/gpgpu_woes_part_1<p>Curiosity about the <a href="https://github.com/servo/servo">Servo</a> project from Mozilla finally got the best of me and I looked around to see if I could contribute somehow. A particular exploratory task caught my eye that involved running the CSS rule matching on the GPU. I forked the <a href="https://github.com/nfrechette/selectron">original sample code</a> and got to work.</p>
<p>Little did I know I would hit some very weird and peculiar issues related to OpenCL on my Intel Iris 5100 inside my MacBook Pro. These issues are so exotic and rarely discussed that I figure they warranted their own blog posts.</p>
<p>Just getting the sample to work reliably on OS X and Windows 8.1 while attempting to get identical results in x86, x64, CPU, and GPU versions proved to take considerable time due to various issues.</p>
<h3 id="moving-pieces">Moving pieces</h3>
<p>GPGPU has a lot of moving pieces that can easily cause havoc. Code is typically written in a C like dialect (OpenCL, CUDA) and it is easy to make mistakes if you are not careful. If the old adage of blowing your foot off with C is true on the CPU, writing C on the GPU is probably akin to blowing up your whole neighbourhood along with your foot.</p>
<p>The first moving piece is the driver you use. Different platforms have different drivers, they also differ by hardware and how they update also differs. Bugs in the drivers are not unheard of and quite frequent since the hardware is still rapidly evolving and the range of supported devices grows everyday. For example, OS X provides the drivers as opposed to the manufacturer providing them like they do on Windows. This means the update process is much slower.</p>
<p>The second moving piece is the hardware itself. Even from a single manufacturer, there is considerable variation. From the number of compute units, the size of local storage, the size of the address space, all the way to whether memory is unified or not with the CPU.</p>
<p>This brings us to our first issue.</p>
<h3 id="on-the-gpu-there-is-no-stack">On the GPU, there is no stack</h3>
<p>The first exotic issue I hit was that the original code would not run for me. On Windows 8.1, the GPU would hit 100% utilization and cause the driver to time out (sometimes forcing me to power cycle). On OS X, the kernel would return after 5 or 10 seconds of runtime and attempting to run it a second time would cause the program to crash (after modifying it to run the kernel more than once to gather average timings).</p>
<p>After hunting for several hours, I finally found the culprit: a C style stack array with 16 elements. The total size of this array was 3 integers times 16 or 192 bytes. This seems fairly small but it fails to take into account how the GPU and generated assembly handle C style stack arrays.</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">struct</span> <span class="nc">css_matched_property</span>
<span class="p">{</span>
<span class="n">cl_int</span> <span class="n">specificity</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">property_index</span><span class="p">;</span>
<span class="n">cl_int</span> <span class="n">property_count</span><span class="p">;</span>
<span class="p">};</span>
<span class="c1">// Later inside the kernel function
</span>
<span class="k">struct</span> <span class="nc">css_matched_property</span> <span class="n">matched_properties</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span></code></pre></figure>
<p>On the GPU, there is no stack. From past experiences, the generated assembly will attempt to keep everything in registers instead of putting it in local or global storage (since local and global storage usage require explicit keywords). Because of this reason, it also will fail to spill in local storage or global memory if we run out of registers. In practice the driver could probably spill to memory when this happens but the performance would be terrible.</p>
<p>According to <a href="https://software.intel.com/sites/default/files/managed/f3/13/Compute_Architecture_of_Intel_Processor_Graphics_Gen7dot5_Aug2014.pdf">the hardware specifications</a> of my GPU, each thread has 128 registers that each store 32 bytes (SIMD 8 elements of 32 bits). The above array requires 48 such registers if the data is not coalesced into fewer registers. Since we use a struct with 3 individual integers, this is a reasonable assumption. Along with everything else going on in the function, my kernel would in all likelihood (due to the nature of the crash, I failed to get exact measurements for the number of registers used) exhaust all available registers.</p>
<p>This marks the second time I see a GPU crash caused by the driver attempting to run a kernel that requires more registers than are available.</p>
<p>These sort of issues are nasty since the same kernel would work on hardware with more registers. It also looks like clean and simple code if you aren’t aware of what happens being the curtains.</p>
<p>The fix, as is probably obvious now, was to keep the array in shared local memory. This ensures we can calculate how much actual memory we require for this and based on the amount available on the given hardware, it caps the maximum number of threads we can execute in a group to avoid running out.</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">const</span> <span class="n">cl_int</span> <span class="n">MAX_NUM_WORKITEMS_PER_GROUP</span> <span class="o">=</span> <span class="mi">320</span><span class="p">;</span>
<span class="n">__local</span> <span class="k">struct</span> <span class="nc">css_matched_property</span> <span class="n">matched_properties</span><span class="p">[</span><span class="mi">16</span> <span class="o">*</span> <span class="n">MAX_NUM_WORKITEMS_PER_GROUP</span><span class="p">];</span>
<span class="n">cl_int</span> <span class="n">matched_properties_base_offset</span> <span class="o">=</span> <span class="mi">16</span> <span class="o">*</span> <span class="nf">get_local_id</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span></code></pre></figure>
<p>Keep in mind that at this stage of development, I was more concerned with getting the code running correctly than I was in getting it to run fast. The is no point in having fast code that does not do what you want.</p>
Out of Memory2015-06-25T00:00:00+00:00http://nfrechette.github.io/2015/06/25/out_of_memory<p>I have been playing <em>Game of War: Fire Age</em> for some time now and a peculiar issue keeps recurring which prompted this post: the game often runs out of memory.</p>
<p>This is an often rarely discussed issue and the solutions to it are seldom discussed as well. This post is an attempt to document various causes and solutions to this and help offer some insight into this.</p>
<h3 id="the-causes">The causes</h3>
<p>In video games, the memory workload is typically very predictable: we have hard limits on many features (e.g: max number of players) and generally fixed data. These two things coupled together imply that out of memory situations are generally quite rare. The typical causes are as follow:</p>
<blockquote>
<p>An excessively large memory size is requested and cannot be serviced.</p>
</blockquote>
<p>I mean by this that 2GB or more might be requested on a device with very little memory (e.g: 256MB). This is generally caused by attempting to allocate memory with a negative size (size_t is typically used in C++ to represent this and it is unsigned however depending on the warning level and the compiler, automatic (or sometimes programmer forced) coercion can happen). This is generally an error due to unforeseen circumstances which rarely happens in a released title but that happens from time to time during development.</p>
<blockquote>
<p>More memory is allocated than the system allows.</p>
</blockquote>
<p>Again, due to the predictable memory footprint of things in video games, it generally happens during development and very rarely in a released title.</p>
<blockquote>
<p>Over time, due to memory leaking, you run out of memory.</p>
</blockquote>
<p>This can happen in released titles and more than a few have shipped with memory leaks.</p>
<blockquote>
<p>Memory fragmentation.</p>
</blockquote>
<p>If the memory pressure is high and fragmentation is present, even though free memory might exist to service a particular allocation request, the system might fail due to fragmentation (either in user space or due to physical memory fragmentation on some embedded devices). Fragmentation is a real and painful issue to deal with when it creeps up. It will often remain hidden during development until very late primarily due to two things: final content often comes very late in production and the game will often not run for more than one hour until late in production as well. Out of memory situations can happen in released titles on memory constrained devices. I see it at least twice a day in Game of War on my android tablet.</p>
<h3 id="how-to-deal-with-it">How to deal with it</h3>
<p>On memory constrained devices, if you have memory fragmentation it is a fact of life that you will hit out of memory situations. This is even more likely if your software might run on devices below your minimum specifications (e.g: mobile android devices). When this happens, there are a few ways to deal with this. Here are the ones that come to mind, in increasing order of complexity:</p>
<blockquote>
<p>Do nothing and let it crash and burn.</p>
</blockquote>
<p>Many games go this route and it costs literally nothing to adopt this strategy (if you can call it that). Sadly, not all crashes will be equal in impact. It is quite common for save games to generate quite a few memory allocations and crashing while the game is saving can often result in corrupted save games. For obvious reasons, this is very bad for the user experience.</p>
<blockquote>
<p>Let it crash but do so in a controller manner.</p>
</blockquote>
<p>Games that realize that they might crash and instead opt to handle this with the least amount of effort will typically poll how much free memory remains and crash when it passes a threshold in a controlled and safe manner. Typically this implies doing this when the game isn’t doing anything important (such as saving the game) and presenting some kind of fatal error message to the user. As far I as I can recall, a version of <em>Gears of War</em> running on <em>Unreal 3</em> simply displayed a dirty disk error. This is generally considered acceptable since while it isn’t ideal for the user, at least nothing of value will be lost and ultimately it will remain a minor annoyance (depending on the frequency of course).</p>
<blockquote>
<p>Deal with it in a clever way.</p>
</blockquote>
<p><em>Game of War</em> is a good example of this. When the game runs out of memory, it sends you back to your home city screen and flashes some colours briefly. (I do not have access to the source code to confirm this but it appears to me to be the cause of this peculiar behaviour.) This can happen almost anywhere except when in the city screen. This is likely because the city screen has a low or very predictable memory footprint. This is superior to the previous approaches since while it remains a minor annoyance, at least you remain within the game and presumably you can continue playing for a little while longer.</p>
<blockquote>
<p>Fix the underlying issues.</p>
</blockquote>
<p>This often requires the largest time investment. Not only does it require extensive testing to make sure even under all your imposed hard limits you do not exceed the maximum memory allowed (e.g: 16 vs 16 players) but it often requires dealing with memory fragmentation and making sure that there is none or very little.</p>
<h3 id="case-study-aaa-title-2014">Case study: <em>AAA title (2014)</em></h3>
<p>During the development of <em>AAA title (2014)</em>, our final content for most maps came in very late in development and it all came at once. This made testing everything very hard. We knew very early on that a single platform would struggle with memory pressure and that prediction proved very accurate: our <em>PlayStation 3</em> title suffered from rampant out of memory situations.</p>
<p>A number of factors lead to this:</p>
<ul>
<li>64KB page sizes meant that our memory allocator had to deal properly with virtual memory to avoid fragmentation.</li>
<li>The PS3 has ~213MB of usable main memory and ~256MB of usable video memory. While you can use video memory as general purpose memory, accessing it from the CPU is very slow and is generally not recommended. This makes it the platform with the least amount of general purpose memory.</li>
<li>With high memory pressure comes memory fragmentation issues.</li>
</ul>
<p>While we made sure to perform memory optimizations throughout development to reduce our footprint, ultimately it proved not to be sufficient when we neared our release date. The final major memory optimizations (both code and data) came in about 6 months before we released our title. Around that time, memory fragmentation reared its ugly head and the battle began.</p>
<p>Fighting memory fragmentation is hard and painful. Even though I had knowledge of how it happens prior to facing it, I had never had actual experience dealing with it.</p>
<p>The battle raged on for 6 months before we finally eradicated it for good. We ultimately released the game with much more free memory than we anticipated: our efforts finally paid off.</p>
<p>But that is not the whole story. Dealing with memory fragmentation is complicated and is best left to a future blog post. I will however discuss our plan to deal with our worst case scenario: failure to fix our memory fragmentation issues.</p>
<p>Few people on <em>AAA title (2014)</em> really knew how bad it really got. At one point I had over 100 separate bug reports of out of memory issues: so many that whenever I would make an improvement, all I could do was claim everything as fixed and see what came back. It became a running gag that I had over half the bug reports assigned to me.</p>
<p>At some point, about 2 months before our release date, we could play the game for about an hour or two before running out of memory due to memory fragmentation. It was bad and we weren’t sure if we were going to be able to fix the issue in time for the release or even in time for our first patch. To prepare for this scenario, we took a similar approach to <em>Game of War</em> and when we detected a low memory situation, we would force a map transition into the <em>Player Hub</em> level. This was a small level that you would return to in between story arcs. This made it the perfect place. It was expected that the map transition would always succeed (at least before the user got tired!) due to the fact that whatever was unloading was larger than what we loaded. The map transition would also save your progress ensuring that your save game would never corrupt due to this.</p>
<p>It was a horrible hack, it was ugly, but it was necessary. With this, we knew that it was better than crashing and that if the user continued to play after this, and fragmentation became really bad, at worst he would not be able to leave that level without reloading into it and they would eventually get the message and restart the title. A necessary evil.</p>
<p>Ultimately, only 2 or 3 bug reports ever spoke of this weird behaviour and by the time we released our game, our memory fragmentation issues were fixed and it became unnecessary. In the end, we removed the hack from the final product since it was only for a single platform and now unnecessary. I personally played the game for over 4 consecutive hours on the night prior to our release and made sure our free memory would never dip below our acceptable threshold: 10MB. Most maps ended up with 15-20MB of free memory with our biggest maps closer to 10MB.</p>
<p>These hacks are a poor substitute for a real fix but with the pressures of the real world, they are often a necessary and realistic option. Do you have a similar war story?</p>
<p><a href="/2016/10/18/memory_allocators_toc/"><strong>Back to table of contents</strong></a></p>
Virtual Memory Aware Linear Allocator2015-06-11T00:00:00+00:00http://nfrechette.github.io/2015/06/11/vmem_linear_allocator<p>This allocator is a variant of the <a href="/2015/05/21/linear_allocator/">linear allocator</a> we covered last time and again it serves to introduce a few important concepts to allocators. Today we cover the virtual memory aware linear memory allocator (<a href="https://github.com/nfrechette/gin/blob/master/include/gin/vmem_linear_allocator.h">code</a>).</p>
<h3 id="how-it-works">How it works</h3>
<p>The internal logic is nearly identical to the linear allocator with a few important tweaks:</p>
<ul>
<li>We <code class="language-plaintext highlighter-rouge">Initialize</code> the allocator with a buffer size. The allocator will use this size to reserve virtual memory but it will not commit any physical memory to it until it is needed.</li>
<li><code class="language-plaintext highlighter-rouge">Allocate</code> will commit physical memory on demand as we allocate.</li>
<li><code class="language-plaintext highlighter-rouge">Deallocate</code> remains unchanged and does nothing.</li>
<li><code class="language-plaintext highlighter-rouge">Reallocate</code> remains largely unchanged and like <code class="language-plaintext highlighter-rouge">Allocate</code> it will commit physical memory when needed.</li>
<li><code class="language-plaintext highlighter-rouge">Reset</code> remains largely the same but it now decommits all previously committed physical memory.</li>
</ul>
<p>Much like the vanilla linear allocator, the buffer is not modified by the allocator and there is no memory overhead per allocation.</p>
<p>There are two things currently missing from the implementation. The first is that we do not specify how eagerly we commit physical memory. There is a cost associated with committing (and decommitting) physical memory since we must call into the kernel to update the TLB page tables (and invalidate the relevant TLB entries). Depending on the usage scenarios, this can have an important cost. Currently we commit memory with a granularity of 4KB (the default page size on most platforms). In practice, even if the system uses pages of 4KB, we could use any multiple higher than this to commit memory with (e.g: commit in blocks of 128KB). The second missing implementation detail is that we simply decommit everything when we <code class="language-plaintext highlighter-rouge">Reset</code>. In practice, some sort of policy would be used here in order to manage slack. This policy would be required at initialization as well as when we reset. For similar reasons as stated prior, committing and decommitting have costs and reducing that overhead is important. In many cases it would make sense to keep some amount of the memory always committed. While these two important details are not necessarily important in this allocator variant, in many other allocators it is of critical importance. Decommitting memory is often very important to fight fragmentation and is critical when multiple allocators must work along side each other.</p>
<h3 id="what-can-we-use-it-for">What can we use it for</h3>
<p>Unlike the vanilla linear allocator, because we commit physical memory on demand, this allocator is better suited for large buffers or for buffers where the used size is not known ahead of time. The only requirement is that we know an upper bound.</p>
<p>In the past, I have used something very similar to manage video game checkpoints. It is often known or enforced by the platform that a save game or checkpoint does not have a size larger than a known constant. However, it is very rare that we know the exact size it will have when the buffer must be created. To create your save game, you can simply employ this linear allocator variant with an upper bound, use it behind a stream type class to facilitate serialization and be done with it. You can sleep well at night knowing that no realloc will happen and no memory will be needlessly copied.</p>
<h3 id="what-we-cant-use-it-for">What we can’t use it for</h3>
<p>Much like the vanilla linear allocator, this allocator is ill suited if freeing memory at the pointer granularity is required.</p>
<h3 id="edge-cases">Edge cases</h3>
<p>The edge cases of this allocator are identical to the vanilla linear allocator with the exception of two new ones we add to the list.</p>
<p>When we first initialize the allocator, virtual memory must be reserved. Virtual memory is a finite resource and can run out like everything else. On 32 bit systems, this value is typically 2 to 4GB. On 64 bit systems, this value is very large since typically 40 bits are used to represent it.</p>
<p>When we commit physical memory, we might end up running out. In this scenario, we still have free reserved virtual memory but we are out of physical memory. This can happen if the physical memory becomes fragmented and mixed page sizes are used by the application. For example, consider an allocator <em>A</em> using pages of 2MB while another allocator <em>B</em> uses pages of 4KB. It might be possible for the system to end up with holes that are smaller than 2MB. Since the TLB must refer to a contiguous region of physical memory when large pages are used, this is bad. On many platforms, the kernel will defragment physical memory if this happens by copying memory around and remapping TLB entries. However, not all platforms will do this and some will simply bail out on you.</p>
<h3 id="potential-optimizations">Potential optimizations</h3>
<p>Much like the vanilla linear allocator, all observes optimization opportunities are also available here. Also, as previously discussed, depending how greedy we are with committing and how much slack we keep when decommitting, we can tune the performance quite effectively.</p>
<p>One notable other optimization avenue that is not for the faint of heart is that we can remove the check to commit memory inside the <code class="language-plaintext highlighter-rouge">Allocate</code> function and instead let the system produce an invalid access fault. By modifying the handler function and registering our allocator, we could commit memory when this happens and retry the faulting instruction. Depending on the granularity of the pages used to commit memory, this could reduce somewhat the overhead required for allocation by removing the branch required for this check.</p>
<p>While this implementation uses a variable to keep track of how much memory is committed, depending on the actual policy used, it could potentially be dropped as well. It was added for simplicity not out of necessity since in the current implementation, the allocated size could be rounded up to a multiple of the page size used for the purpose of tracking the committed memory.</p>
<h3 id="performance">Performance</h3>
<p>Due to its simplicity, it offers great performance. All allocation and deallocation operations are <em>O(1)</em> and only amount to a few instructions. However, resetting and destroying the allocator now have the added cost of decommitting physical memory and will thus be linearly dependant on the amount of committed memory.</p>
<p>On most 32 bit platforms, the size of an instance should be 28 bytes if <code class="language-plaintext highlighter-rouge">size_t</code> is used. On 64 bit platforms, the size should be 56 bytes with <code class="language-plaintext highlighter-rouge">size_t</code>. Both versions can be made even smaller with smaller integral types such as <code class="language-plaintext highlighter-rouge">uint32_t</code> or by stripping support for <code class="language-plaintext highlighter-rouge">Reallocate</code>. As such, either version will comfortably fit inside a single typical cache line of 64 bytes.</p>
<h3 id="conclusion">Conclusion</h3>
<p>Once again, this is a very simply and perhaps even toy allocator. However it serves as an important building block to discuss the various implications of committing and decommitting memory as well as the general implementation details surrounding these things. Manual virtual memory management is a classic and important tool of modern memory allocators and seeing it put to use in a simple context will serve as a learning example.</p>
<p>Fundamentally, linear allocators are a variation of a much more interesting and important allocator: the stack frame allocator. In essence, linear allocators are stack frame allocators where only a single frame is supported. Pushing of the frame happens at initialization and popping happens when we reset or at destruction.</p>
<p>Next up, we will cover the ever so useful: <a href="/2016/05/08/stack_frame_allocators/">stack frame allocators</a>.</p>
<h3 id="alternate-names">Alternate names</h3>
<p>To my knowledge, there is no alternate name for this allocator since it isn’t really a real allocator one would see in the wild.</p>
<p><em>Note that if you know a better name or alternate names for this allocator, feel free to contact me.</em></p>
<p><a href="http://www.reddit.com/r/programming/comments/39gl0d/memory_allocators_explained_the_virtual_memory/">Reddit thread</a></p>
<p><a href="/2016/10/18/memory_allocators_toc/"><strong>Back to table of contents</strong></a></p>
How to legally generate the Windows ISO from a Mac or Linux PC2015-05-27T00:00:00+00:00http://nfrechette.github.io/2015/05/27/generate_windows_iso_mac<p>Surprisingly, it turns out that getting your hands on a legal Windows ISO after purchasing it is not as easy as it might seem. After purchasing a digital copy of Windows in the Microsoft store, you will be able to download a Windows application that downloads the ISO for you after entering your product key.</p>
<p>Therein lies the issue: to create a Windows install media, you need access to a PC with Windows already installed. If like me all you have on hand is a Mac or Linux PC without easy or quick access to a PC with Windows (or one that you can trust with your product key), you are left in a tight spot. Even the Microsoft support will not be able to help you and will simply point out that it can’t be that hard to find a PC with Windows.</p>
<p>Worse still is that not all versions of Windows are supported and for example, all versions that run on the Free Amazon Web Services are not supported so this avenue is not open to us either (at least at the time of writing).</p>
<p>The only viable alternative appears to be to download the ISO from an untrusted torrent site. If like me this leaves you uneasy, read on.</p>
<p>As it turns out, an unrelated tool that Microsoft releases will allow us to execute that ISO generating executable in a safe and legal way.</p>
<h3 id="step-1">Step 1</h3>
<p>Install <a href="https://www.virtualbox.org/">Virtual Box</a>. Virtual Box is a virtual machine application from Oracle that allows you to run an operating system in a virtualized environment as a guest.</p>
<h3 id="step-2">Step 2</h3>
<p>Download a Virtual Box image from the <a href="https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/">Internet Explorer Developer website</a>. These images are legal versions of Windows provided by Microsoft to allow developers to test various Internet Explorer versions on various Windows versions. They expire after 90 days and cannot be activated but that doesn’t matter to us since we’ll only really use it for as long as the download of the ISO takes.</p>
<p>Simply grab the image for the platform that you have (Mac or Linux). Note that most images are 32 bit which will only allow you to later download the 32 bit ISO. At the time of writing, the Windows 10 image is 64 bit but requires <a href="https://stackoverflow.com/questions/30212542/the-ie11-windows-10-vm-for-virtualbox-on-osx-doesnt-start">the following fix</a> to work properly.</p>
<h3 id="step-3">Step 3</h3>
<p>Inside Virtual Box, import the Windows image downloaded at the previous step. The documentation recommends you set at least 2GB of RAM for the virtual machine to use. This is fine since we’ll only run it for a short amount of time anyway.</p>
<h3 id="step-4">Step 4</h3>
<p>Use Virtual Box to <a href="https://www.virtualbox.org/manual/ch04.html#sharedfolders">share a directory</a> with the virtual machine with write access and make sure it automatically mounts (easier for us). This is the directory where we will copy the ISO file into. At the time of writing it does not work with the Windows 10 image yet and as such you will need to plug in a USB key and share it with the virtual machine from the settings or share a network folder (I had more luck sharing the folder inside the Windows guest virtual machine and accessing it from my Mac).</p>
<h3 id="step-5">Step 5</h3>
<p>Launch the virtual machine instance and use it to run the executable from the Windows store. If you do not have access to it, I have not tested but presume that you might be able to use <a href="http://windows.microsoft.com/en-us/windows-8/create-reset-refresh-media">this utility</a> as well.</p>
<h3 id="step-6">Step 6</h3>
<p>Follow the instructions and download the ISO to some directory (e.g: the desktop). For some reason Virtual Box did not allow me to download directly to the shared directory we setup in <em>Step 4</em>. Either way, once the download terminates, simply copy the ISO to the shared directory.</p>
<h3 id="complications">Complications</h3>
<p>Sadly, the Windows 10 image is a bit finicky. You can’t shutdown properly and install the updates or it won’t boot again and you’ll have to start over. I could not manage to get my USB stick to work either and as such I had to create a shared directory inside the Windows guest. In order to be able to access the shared directory, I had to hot swap the virtual machine network card from NAT to Host bridge (you will need to add a host adapter in the Virtual Box preferences). I also needed to add a default gateway in the IPv4 settings of the Windows guest for my mac to be able to access it and I also needed to allow Guest access in the network settings.</p>
<p>Next, due to our previous hack to change the time to allow Windows to boot, the executable will refuse to download the ISO and complain it can’t connect to the internet. To resolve this I had to change the time inside the guest manually to the current date. I also needed to hot swap the network card back to the NAT setting and removed the default gateway.</p>
<p>Last but not least in order to copy the ISO out, I had to hot swap the network card once again to Host bridge and add back the default gateway.</p>
<h3 id="profit">Profit!</h3>
<p>That’s it! You are done and you can now get rid of Virtual Box if you wish to. You can now continue the steps to create your bootable DVD or USB stick from the ISO.</p>
<p><a href="http://www.tomshardware.com/answers/id-1800781/windows-iso-file.html">There</a> <a href="http://kb.parallels.com/en/121009">are</a> <a href="http://log.maniacalrage.net/post/78047230047/how-to-install-windows-8-1-on-a-new-mac-pro">so</a> <a href="http://forums.whirlpool.net.au/archive/2155844">many</a> threads out there asking how to do this that I hope this will be able to help someone avoid the hassle I went through to find this. You would think Microsoft would make it easier for you to install their operating system…</p>
Linear Allocator2015-05-21T00:00:00+00:00http://nfrechette.github.io/2015/05/21/linear_allocator<p>The first memory allocator we will cover is by far the simplest and serves as an introduction to the material. Today we cover the linear memory allocator (<a href="https://github.com/nfrechette/gin/blob/master/include/gin/linear_allocator.h">code</a>).</p>
<h3 id="how-it-works">How it works</h3>
<p>The internal logic is fairly simple:</p>
<ul>
<li>We initialize the allocator with a pre-allocated memory buffer and its size.</li>
<li><code class="language-plaintext highlighter-rouge">Allocate</code> simply increments a value indicating the current buffer offset.</li>
<li><code class="language-plaintext highlighter-rouge">Deallocate</code> does nothing</li>
<li><code class="language-plaintext highlighter-rouge">Reallocate</code> first checks if the allocation being resized is the last performed allocation and if it is, we return the same pointer and update our current allocation offset.</li>
<li><code class="language-plaintext highlighter-rouge">Reset</code> is used to free all allocated memory and return the allocator to its original initialized state.</li>
</ul>
<p>There is very little work to do for all these functions which makes it very fast. <code class="language-plaintext highlighter-rouge">Deallocate</code> does nothing because in practice, it makes little sense to support it and other allocators are often better suited when freeing memory at the pointer level is required. The only sane implementation we could do is similar to how <code class="language-plaintext highlighter-rouge">Reallocate</code> works by checking if the memory being freed is the last allocation (last in, first out). Because we work with a pre-allocated memory buffer, <code class="language-plaintext highlighter-rouge">Reallocate</code> does not need to perform a copy if the last allocation is being resized and there is enough free space left regardless of whether it grows or shrinks.</p>
<p>Interestingly, you can almost free memory at the pointer level by using <code class="language-plaintext highlighter-rouge">Reallocate</code> and using a new size of 0 but in practice, the alignment used for the allocation would remain lost forever (if it was originally mis-aligned).</p>
<p>Critical to this allocator is that it adds no overhead per allocation and it does not modify the pre-allocated memory buffer. Not all allocators will have these properties and I will always mention this important bit. This makes it ideal for low level systems or for working with read-only memory.</p>
<h3 id="what-can-we-use-it-for">What can we use it for</h3>
<p>Despite being a very simple allocator, it has a few uses. I have used it in the past with success to clean up code dealing with a lot of pointer arithmetic. The general idea is that if you have a memory buffer representing a custom binary format with most fields having a variable size and requiring alignment, you will end up with a lot of logic to take your raw buffer and split it into the various internal bits.</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="kt">uintptr_t</span> <span class="n">buffer</span><span class="p">;</span> <span class="c1">// user supplied
</span>
<span class="kt">size_t</span> <span class="n">bufferOffset</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">uint32_t</span><span class="o">*</span> <span class="n">numValue1</span> <span class="o">=</span> <span class="k">reinterpret_cast</span><span class="o"><</span><span class="kt">uint32_t</span><span class="o">*></span><span class="p">(</span><span class="n">buffer</span><span class="p">);</span> <span class="c1">// assume buffer is properly aligned for first value
</span>
<span class="n">bufferOffset</span> <span class="o">=</span> <span class="n">AlignTo</span><span class="p">(</span><span class="n">bufferOffset</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">*</span> <span class="nf">sizeof</span><span class="p">(</span><span class="kt">uint32_t</span><span class="p">),</span> <span class="k">alignof</span><span class="p">(</span><span class="kt">uint8_t</span><span class="p">));</span>
<span class="kt">uint8_t</span><span class="o">*</span> <span class="n">values1</span> <span class="o">=</span> <span class="k">reinterpret_cast</span><span class="o"><</span><span class="kt">uint8_t</span><span class="o">*></span><span class="p">(</span><span class="n">buffer</span> <span class="o">+</span> <span class="n">bufferOffset</span><span class="p">);</span>
<span class="n">bufferOffset</span> <span class="o">=</span> <span class="n">AlignTo</span><span class="p">(</span><span class="n">bufferOffset</span> <span class="o">+</span> <span class="o">*</span><span class="n">numValue1</span> <span class="o">*</span> <span class="nf">sizeof</span><span class="p">(</span><span class="kt">uint8_t</span><span class="p">),</span> <span class="k">alignof</span><span class="p">(</span><span class="kt">uint16_t</span><span class="p">));</span>
<span class="kt">uint16_t</span><span class="o">*</span> <span class="n">numValue2</span> <span class="o">=</span> <span class="k">reinterpret_cast</span><span class="o"><</span><span class="kt">uint16_t</span><span class="o">*></span><span class="p">(</span><span class="n">buffer</span> <span class="o">+</span> <span class="n">bufferOffset</span><span class="p">);</span>
<span class="n">bufferOffset</span> <span class="o">=</span> <span class="n">AlignTo</span><span class="p">(</span><span class="n">bufferOffset</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">*</span> <span class="nf">sizeof</span><span class="p">(</span><span class="kt">uint16_t</span><span class="p">),</span> <span class="k">alignof</span><span class="p">(</span><span class="kt">float</span><span class="p">));</span>
<span class="kt">float</span><span class="o">*</span> <span class="n">values2</span> <span class="o">=</span> <span class="k">reinterpret_cast</span><span class="o"><</span><span class="kt">float</span><span class="o">*></span><span class="p">(</span><span class="n">buffer</span> <span class="o">+</span> <span class="n">bufferOffset</span><span class="p">);</span></code></pre></figure>
<p>Using a linear allocator to wrap the buffer allows you to manipulate it in an elegant and efficient manner.</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">LinearAllocator</span> <span class="nf">buffer</span><span class="p">(</span><span class="cm">/* user supplied */</span><span class="p">);</span>
<span class="kt">uint32_t</span><span class="o">*</span> <span class="n">numValue1</span> <span class="o">=</span> <span class="k">new</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="kt">uint32_t</span><span class="p">;</span>
<span class="kt">uint8_t</span><span class="o">*</span> <span class="n">values1</span> <span class="o">=</span> <span class="k">new</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="kt">uint8_t</span><span class="p">[</span><span class="o">*</span><span class="n">numValue1</span><span class="p">];</span>
<span class="kt">uint16_t</span><span class="o">*</span> <span class="n">numValue2</span> <span class="o">=</span> <span class="k">new</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="kt">uint16_t</span><span class="p">;</span>
<span class="kt">float</span><span class="o">*</span> <span class="n">values2</span> <span class="o">=</span> <span class="k">new</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="kt">float</span><span class="p">[</span><span class="o">*</span><span class="n">numValue2</span><span class="p">];</span></code></pre></figure>
<p>Note that the above two versions are not 100% equivalent due to the fact C++11 offers no way to access the required alignment of the requested type when implementing the <code class="language-plaintext highlighter-rouge">new</code> operator. However, with macro support, the original intent above can be supported and be just as clear while supporting alignment properly. I plan to cover this important bit in a later post.</p>
<p>I wrote earlier that this allocator can be used with read-only memory which is a strange property for a memory allocator. Indeed, since it mostly abstracts away pointer arithmetic when partitioning a raw memory region without modifying it, we can use it to do just that over read-only memory. In the example above, this means that we can easily use it for writing and reading our custom binary format.</p>
<h3 id="what-we-cant-use-it-for">What we can’t use it for</h3>
<p>Due to the fact that we do not add per allocation overhead, we cannot properly support freeing memory at the pointer level while supporting variable alignment. When support for freeing memory is needed, other allocators are better suited.</p>
<p>This allocator is also generally a poor fit for very large memory buffers. Due to the fact that we need to pre-allocate it up front, we bear the full cost regardless of how much actual memory we allocate internally.</p>
<h3 id="edge-cases">Edge cases</h3>
<p>There are two important edge cases with this allocator and they are shared by all allocators: overflow and out of memory conditions.</p>
<p>We can cause arithmetic overflow in two ways in this allocator: first by supplying a large alignment value, and second by attempting to allocate a large amount of memory. This is fairly simple to test but there is one small bit we must be careful with: if the alignment provided is very large, we can overflow in such a way that our resulting pointer would end up in our buffer either at a lower memory address if the allocation size is very large as well or at a higher memory address if the allocation size is small. The proper way to deal with this is to check overflow after taking into account the alignment and again after adjusting for the new allocation size.</p>
<p>We eventually run out of memory if we attempt to allocate more than our pre-allocated buffer owns.</p>
<h3 id="potential-optimizations">Potential optimizations</h3>
<p>While the implementation I provide aims for safety first, in practice a linear allocator should never run out of memory nor should it overflow if the logic using it is correct. This is because by providing a pre-allocated buffer, either we assume we do not need more memory or our assumption is wrong. In the later case, it is highly likely that we do not check the return value of allocations which is of little help even if our allocator is safe. Usage scenarios for this allocator are also generally simple in logic with few unknown variables to cause havoc.</p>
<p>In this light, a number of improvements can be made by stripping the overflow and out of memory checks and simply keeping asserts around. I may end up providing a way to do just that with template arguments or macros in the future.</p>
<p>Another option is to remove the branches added by the overflow and out of memory checks and simply ensure the internal state does not change instead. There is very little logic and as such an early out branch could very well save very little in the rare case it is taken and at the same time, since it is rarely taken, we always end up performing most of the logic anyway.</p>
<p>Last but not least, <code class="language-plaintext highlighter-rouge">Reallocate</code> support is often not required and could trivially be stripped as well.</p>
<h3 id="performance">Performance</h3>
<p>Due to its simplicity, it offers great performance. All operations are <em>O(1)</em> and only amount to a few instructions.</p>
<p>On most 32 bit platforms, the size of an instance should be 24 bytes if <code class="language-plaintext highlighter-rouge">size_t</code> is used. On 64 bit platforms, the size should be 48 bytes with <code class="language-plaintext highlighter-rouge">size_t</code>. Both versions can be made even smaller with smaller integral types such as <code class="language-plaintext highlighter-rouge">uint16_t</code> (which would be very appropriate since this allocator is predominantly used with small buffers) or by stripping support for <code class="language-plaintext highlighter-rouge">Reallocate</code>. As such, either version will comfortably fit inside a single typical cache line of 64 bytes.</p>
<h3 id="conclusion">Conclusion</h3>
<p>Despite its simple internals, the linear allocator is an important building block. It serves as an ideal example for a number of sibling allocators we will see in the next few posts which involve similar internal logic and edge cases.</p>
<p>Next up, we will cover a variation: <a href="/2015/06/11/vmem_linear_allocator/">the virtual memory aware linear allocator</a>.</p>
<h3 id="alternate-names">Alternate names</h3>
<p>The most common alternate name for this allocator is called the <em>arena</em> allocator. However, that name is overloaded and is also often associated with a number of other allocators which manage a fixed memory region.</p>
<p><em>Note that if you know a better name or alternate names for this allocator, feel free to contact me.</em></p>
<p><a href="/2016/10/18/memory_allocators_toc/"><strong>Back to table of contents</strong></a></p>
Virtual memory explained2015-05-12T00:00:00+00:00http://nfrechette.github.io/2015/05/12/virtual_memory_explained<p>Virtual memory is present on most hardware platforms and it is surprisingly simple. However, it is often poorly understood. It works so well and seamlessly that few inquire about its true nature.</p>
<p>Today we will take an in-depth look at why do we have virtual memory and how it works under the hood.</p>
<h3 id="physical-memory-a-scarce-ressource">Physical memory: a scarce ressource</h3>
<p>In the earlier days of computing, there was no virtual memory. The computing needs were simple and single process operating systems were common (batch processing was the norm). As the demand grew for computers, so did the complexity of the software running on them.</p>
<p>Eventually the need to run multiple processes concurrently appeared and soon became mainstream. The problem with multiple processes in regards to memory is three-fold:</p>
<ul>
<li>Physical memory must be shared somehow</li>
<li>We must be able to access more than 64KB of memory using 16 bit registers</li>
<li>We must protect the memory to prevent tampering (malicious or accidental)</li>
</ul>
<h3 id="memory-segments">Memory segments</h3>
<p><a href="http://en.wikipedia.org/wiki/X86_memory_segmentation">On x86</a>, the first and second point were addressed first with <em>real mode</em> (1MB range) and later, memory protection was introduced with <em>protected mode</em> (16MB range). Virtual memory was born but it was not in the form most commonly seen today: early x86 used a segment based virtual memory model.</p>
<p>Segments were complex to manage but served their original purpose. Coupled with secondary storage, the operating system was now able to share the physical memory by swapping entire segments in and out of memory. Under 16 bit processors, segments had a fixed size of 64KB but later, when 32 bit processors emerged, segments grew to a variable and maximum size of 16MB. This later development had an unfortunate side-effect: due to the variable size of segments, physical memory fragmentation could now occur.</p>
<p>To address this, paging was introduced. Paging, like earlier segments, divides physical memory into fixed sized blocks. They also introduce an added indirection. With paging, segments were now further divided into pages and each segment further contained a page table to resolve the mapping between segment relative addresses and real effective physical addresses. By their nature, pages do not need to be contiguous within a given segment allowing them to resolve the memory fragmentation issues.</p>
<p>At this point in time, memory accesses now contain two indirections: first we must construct the segment relative address using a 16 bit segment index, reading the 32 bit segment base address associated and adding a 32 bit segment offset (386 CPUs shifted from 24 bit base addresses and offsets to 32 bit at the same time it introduced paging). This yields us a memory address that we must now look up in the segment page table to ultimately find the physical memory page (often called frame) that contains what we are looking for.</p>
<h3 id="modern-virtual-memory">Modern virtual memory</h3>
<p>Things now look much closer to modern virtual memory. Now that we have 32 bit processors and that they are common enough, there is no longer a need for segments and paging alone can be used. The x86 hardware of the time already used 32 bit segment base addresses and 32 bit segment offsets. Memory becomes far easier to manage if we assume it is a single memory segment with paging and it allows us to drop one level of indirection.</p>
<p>Most of this memory address translation logic now happens inside the MMU (memory management unit) and is helped by internal caches. This cache is now more commonly called the Translation Look-aside Buffer, or TLB for short.</p>
<p><a href="http://www.rcollins.org/ddj/May96/">Earlier x86 processors</a> only supported 4KB memory pages. To accommodate this, a virtual memory address was split into three parts:</p>
<ul>
<li>The first 10 bits represent an index in a page directory that is used to look up a page table.</li>
<li>The following 10 bits represent an index in the previously found page table that is used to look up a physical page frame.</li>
<li>The remaining 12 least significant bits are the final offset into the physical page frame leading to the desired memory address.</li>
</ul>
<p>A dedicated register points to a page directory in memory. Both page directory entries and page table entries are 32 bits and contain flags to indicate memory protection and other attributes (cacheable, write combine, etc.). Another dedicated register holds the current process identifier which is used to tell which TLB entries are valid or not (and avoids the need for flushing the entire TLB when context switching between processes).</p>
<p>As memory grew, 4KB pages had difficulty scaling. Servicing large allocation requests requires mapping large numbers of 4KB pages putting more and more pressure on the TLB. The bookkeeping overhead also grows along with the number of pages used. Eventually, <a href="http://en.wikipedia.org/wiki/Page_Size_Extension">larger pages (4MB)</a> were introduced to address these issues.</p>
<p>And then memory sizes grew even further. Soon enough, being confined to an address space of 4GB with 32 bit pointers became too small and <a href="http://en.wikipedia.org/wiki/Physical_Address_Extension">physical address extension</a> was introduced and extended the addressable range to 64GB. This forced larger pages to reduce to a size of 2MB as now some bits were needed for another indirection level in the page table walk.</p>
<h3 id="virtual-memory-today">Virtual memory today</h3>
<p>Today, x64 hardware most commonly supports pages of the following sizes: 4KB, 2MB, and sometimes 1GB. Typically, only 48 bits are used limiting the addressable memory to 256TB.</p>
<p>Another important aspect of virtual memory today is how it interacts with virtualization (when present). Because physical memory must be shared between the running operating systems, page directory entries and page table entries now point into virtual memory instead of physical memory and require further TLB look-ups to resolve the complete effective physical address. Much like process identifiers, a virtualization instance identifier has been introduced for the same purpose: avoiding full TLB flushes when context switching.</p>
<p>Modern hardware will generally have separate caches per page size which means that to get the most performance, <a href="http://lwn.net/Articles/379748/">which page sizes are used for what data must be carefully planned</a>. For example, in certain embedded platforms, it is not uncommon to have 4KB pages be used for code segments, data segments, and the stack while encouraging programmers to use 2MB pages inside their programs. It is also not uncommon to have virtual pages introduced: a virtual page will be composed of a smaller number of pages. For example, you might request allocations to use 64KB or 4MB pages even though the underlying hardware only supports 4KB and 2MB pages. The distinction is mostly important for the kernel since managing larger pages implies lower bookkeeping overhead and faster servicing.</p>
<p>An important point bears mentioning, when pages larger than 4KB are used, the kernel must find contiguous physical memory to allocate them. This can be a problem if page sizes are mixed, it opens the door to fragmentation to rear its ugly head. When the kernel fails to find enough contiguous space but it knows enough space would otherwise remain, it has two choices:</p>
<ul>
<li>Bail out and return stating that you have run out of memory.</li>
<li>Defragment the physical memory by copying memory around and remapping the pages.</li>
</ul>
<p>Generally speaking, if mixed page sizes are used, it is generally recommended to allocate large pages as early as possible in the process’s life to remedy the above problem.</p>
<h3 id="virtual-memory-secondary-storage">Virtual memory secondary storage</h3>
<p>The fact that modern hardware allows virtual memory to be mapped in your process without mapping all the required pages to physical memory enables modern kernels to spill memory onto secondary storage to artificially increase the amount of memory available up to the limits of that secondary storage.</p>
<p>As is common knowledge, the most common form of secondary storage is the swap file on your hard drive.</p>
<p>However, the kernel is free to place that memory anywhere and implementations exist where the memory will be distributed in a network or some other medium (e.g: memory mapped files).</p>
<h3 id="conclusion">Conclusion</h3>
<p>This concludes our introduction to virtual memory. In later blog posts we will explore the TLB into further details and the implications virtual memory has on the CPU cache. Until then, here are some additional links:</p>
<ul>
<li><a href="http://www.cs.princeton.edu/courses/archive/fall11/cos318/lectures/L15_VMPaging.pdf">http://www.cs.princeton.edu/courses/archive/fall11/cos318/lectures/L15_VMPaging.pdf</a></li>
<li><a href="http://www.it.uu.se/edu/course/homepage/dark/ht11/dark15-vm2.pdf">http://www.it.uu.se/edu/course/homepage/dark/ht11/dark15-vm2.pdf</a></li>
<li><a href="http://www.informit.com/articles/article.aspx?p=29961&seqNum=4">http://www.informit.com/articles/article.aspx?p=29961&seqNum=4</a></li>
<li><a href="http://lwn.net/Articles/250967/">http://lwn.net/Articles/250967/</a></li>
</ul>
A memory allocator interface2015-05-11T00:00:00+00:00http://nfrechette.github.io/2015/05/11/memory_allocator_interface<p><a href="/2015/05/01/cpp_stl_container_deficiencies/">As previously discussed</a>, the memory allocator integration into C++ STL containers is far from ideal.</p>
<p><a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2554.pdf">Attempts to improve on it</a> have been made but I have yet to meet anyone satisfied with the current state of things.</p>
<p>Memory in large applications will typically come from two majors places: new/malloc calls and container allocations. Today we will only discuss the later.</p>
<h3 id="where-are-we-at">Where are we at?</h3>
<p>There appears to be two main approaches when it comes to memory allocator interfaces:</p>
<ul>
<li>Templating the container with the allocator used (C++ STL, Unreal 3)</li>
<li>Implementing an interface and perform a virtual function call (Bitsquid)</li>
</ul>
<p>The first, as previously discussed, has a number of downsides but on the up side, it is the fastest since all allocation calls are likely to either be inline or at least be a static branch.</p>
<p>The second, while much more flexible, introduces indirection and as <a href="/2015/05/05/caches_everywhere/">previously discussed</a>, is slower and generally less performant. Not only do we introduce an extra cache miss (and likely TLB miss) but we introduce an indirect branch which the CPU will not be able to prefetch.</p>
<p>Can we do better?</p>
<h3 id="the-compromise">The compromise</h3>
<p>It seems that the cost of added flexibility is to use an indirect branch. This is unavoidable. However, we can remove the extra cache and TLB miss by defining a partial inline virtual function dispatch table in the allocator interface.</p>
<p>The idea is simple:</p>
<ul>
<li>There are typically few allocator instances (on the order of <100 instances isn’t unusual)</li>
<li>Allocator instances are often small in memory footprint and often will fit within one or two cache lines (even when they manage a lot of memory, the footprint will generally lie somewhere else in memory and not inline with the allocator)</li>
<li>Most container allocations can be implemented with a single function: <code class="language-plaintext highlighter-rouge">realloc</code></li>
</ul>
<p>We can thus conclude that it is a viable alternative to add a function pointer to a <code class="language-plaintext highlighter-rouge">realloc</code> function inline within the allocator and call this for all our needs.</p>
<p>Here is the code (<a href="https://github.com/nfrechette/gin/blob/master/include/gin/allocator.h">which is also on github</a>):</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">class</span> <span class="nc">Allocator</span>
<span class="p">{</span>
<span class="nl">protected:</span>
<span class="k">typedef</span> <span class="kt">void</span><span class="o">*</span> <span class="p">(</span><span class="o">*</span><span class="n">ReallocateFun</span><span class="p">)(</span><span class="n">Allocator</span><span class="o">*</span><span class="p">,</span> <span class="kt">void</span><span class="o">*</span><span class="p">,</span> <span class="kt">size_t</span><span class="p">,</span> <span class="kt">size_t</span><span class="p">,</span> <span class="kt">size_t</span><span class="p">);</span>
<span class="kr">inline</span> <span class="n">Allocator</span><span class="p">(</span><span class="n">ReallocateFun</span> <span class="n">reallocateFun</span><span class="p">);</span>
<span class="nl">public:</span>
<span class="k">virtual</span> <span class="o">~</span><span class="n">Allocator</span><span class="p">()</span> <span class="p">{}</span>
<span class="k">virtual</span> <span class="kt">void</span><span class="o">*</span> <span class="n">Allocate</span><span class="p">(</span><span class="kt">size_t</span> <span class="n">size</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">alignment</span><span class="p">)</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">virtual</span> <span class="kt">void</span> <span class="n">Deallocate</span><span class="p">(</span><span class="kt">void</span><span class="o">*</span> <span class="n">ptr</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">size</span><span class="p">)</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kr">inline</span> <span class="kt">void</span><span class="o">*</span> <span class="n">Reallocate</span><span class="p">(</span><span class="kt">void</span><span class="o">*</span> <span class="n">oldPtr</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">oldSize</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">newSize</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">alignment</span><span class="p">);</span>
<span class="k">virtual</span> <span class="kt">bool</span> <span class="n">IsOwnerOf</span><span class="p">(</span><span class="kt">void</span><span class="o">*</span> <span class="n">ptr</span><span class="p">)</span> <span class="k">const</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nl">protected:</span>
<span class="n">ReallocateFun</span> <span class="n">m_reallocateFun</span><span class="p">;</span>
<span class="p">};</span>
<span class="kt">void</span><span class="o">*</span> <span class="n">Allocator</span><span class="o">::</span><span class="n">Reallocate</span><span class="p">(</span><span class="kt">void</span><span class="o">*</span> <span class="n">oldPtr</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">oldSize</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">newSize</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">alignment</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">assert</span><span class="p">(</span><span class="n">m_reallocateFun</span><span class="p">);</span>
<span class="k">return</span> <span class="p">(</span><span class="o">*</span><span class="n">m_reallocateFun</span><span class="p">)(</span><span class="k">this</span><span class="p">,</span> <span class="n">oldPtr</span><span class="p">,</span> <span class="n">oldSize</span><span class="p">,</span> <span class="n">newSize</span><span class="p">,</span> <span class="n">alignment</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>A number of things stand out and are important:</p>
<ul>
<li>This is not a proper interface but instead an abstract base class.</li>
<li>We provide some virtual functions for common things and more will come later (debugging features, etc.)</li>
<li>Reallocate is inline and <em>NOT</em> virtual</li>
<li>Deallocate/Reallocate follow the latest C++ standard and include the <em>size</em> used when the original allocation was made to allow further optimizations within allocator implementations.</li>
<li>Alignment must be provided which is important for AAA video games, and more generally with SIMD code.</li>
<li>The first argument to the <code class="language-plaintext highlighter-rouge">ReallocateFun</code> is an instance of the base class itself. This is important because it points to a static free standing function as such when we call <code class="language-plaintext highlighter-rouge">Reallocate</code>, the implicit <code class="language-plaintext highlighter-rouge">this</code> present as first argument will simply be forwarded as is with no extra register shuffling (at least on x64) even if it ends up not inlined and allows the implementation to call a member function without shuffling registers as well.</li>
</ul>
<p>Usage is very simple:</p>
<ul>
<li>An allocator simply derives from this base class, implement the necessary function and initializes the base class with a function pointer to a suitable reallocate function.</li>
<li>Containers simply call <code class="language-plaintext highlighter-rouge">Reallocate(nullptr, 0, size, alignment)</code> to allocate, <code class="language-plaintext highlighter-rouge">Reallocate(ptr, size, 0, alignment)</code> to deallocate and simply supply all arguments to reallocate.</li>
</ul>
<p>It is often the case that containers will simply reallocate (e.g: vector<..> with POD) and as such this interface is ideally suited for this. With this implementation, when a container performs an allocation or deallocation, the cache miss to access the reallocate function pointer will load relevant data in its cache line leading to less wasted cycles compared to the virtual function dispatch approach while maintaining all the flexibility needed by the indirection and the interface.</p>
<p>In the coming blog posts, I will introduce a number of memory allocators and containers that will use this new interface.</p>
Caches everywhere2015-05-05T00:00:00+00:00http://nfrechette.github.io/2015/05/05/caches_everywhere<p>The modern computer is littered with caches to help improve performance. Each cache has its own unique role. A solid understanding of these caches is critical in writing high performance software in any programming language. I will describe the most important ones present on modern hardware but it is not meant to be an exhaustive list.</p>
<p>The modern computer is heavily based on the <a href="http://en.wikipedia.org/wiki/Von_Neumann_architecture">Von Neumann architecture</a>. This basically boils down to the fact that we need to get data into a memory of sorts before our CPU can process it.</p>
<p>This data will generally come from one of these sources: a chip (such as ROM), a hard drive or a network and this data must make it somehow all the way to the processor. Note that other external devices can DMA data into memory as well but the above three are the most common and relevant to today’s discussion.</p>
<p>I will not dive into the implications for all the networking stack and the caches involved but I will mention that there are a few and understanding them can yield <a href="http://blog.superpat.com/2010/06/01/zero-copy-in-linux-with-sendfile-and-splice/">good gains</a>.</p>
<h3 id="the-kernel-page-cache">The kernel page cache</h3>
<p>Hard drives are <a href="https://gist.github.com/jboner/2841832">notoriously slow</a> compared to memory. To speed things up, a number of caches lay in between your application and the file on disk you are accessing. From the disk up to your application, the caches are in order: <a href="http://en.wikipedia.org/wiki/Disk_buffer">disk embedded cache</a>, <a href="http://en.wikipedia.org/wiki/Disk_array_controller">disk controller cache</a>, <a href="http://en.wikipedia.org/wiki/Page_cache">kernel page cache</a>, and last but not least, whatever <a href="http://docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html">intermediate buffers</a> you might have in your user space application.</p>
<p>For the vasty majority of applications dealing with large files, the single most important optimization you can perform at this level is to use MMAP to access your files. You can find an excellent write up about it <a href="http://duartes.org/gustavo/blog/post/page-cache-the-affair-between-memory-and-files/">here</a>. I will not repeat what he says but add that this is a massive win not only because it allows avoiding needless copying of memory but also by mentioning that you can <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/hh780543%28v=vs.85%29.aspx">prefetch easily</a>, something not so easily achieved with <code class="language-plaintext highlighter-rouge">fread</code> and the likes.</p>
<p>If you are writing an application that does a lot of IO or where it needs to happen as fast as possible (such as console/mobile games), you should ideally be using MMAP where it is supported (<a href="http://developer.android.com/reference/java/nio/channels/FileChannel.html">android supports it</a>, <a href="http://stackoverflow.com/questions/13425558/why-does-mmap-fail-on-ios">iOS supports it but with 700MB virtual memory limit on 32bit processors</a>, and both Windows and Linux obviously support it). I am not 100% certain, but in all likely hood, the newest game consoles (XboxOne and PlayStation 4) should support it as well. However, note that not all these platforms might have a kernel page cache but that it is a safe bet that going forward, newer devices are likely to support it.</p>
<h3 id="the-cpu-cache">The CPU cache</h3>
<p>The next level of caches lays much closer to the CPU and take the form of the popular <a href="http://en.wikipedia.org/wiki/CPU_cache">L1, L2, and L3 caches</a>. Not all processors will have these and when they do, they might only have one or two levels. Higher levels are generally inclusive in regards to the lower levels but <a href="http://stackoverflow.com/questions/19775173/inclusive-or-exclusive-l1-l2-cache-in-intel-core-ivybridge-processor">not always</a>. For example, while the L3 will generally be inclusive of the L2, the L2 might not be inclusive of the L1. This is so because some instructions such as <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0438d/BABJCGGE.html">prefetching</a> will prefetch in the L2 but not the L1 and thus potentially cause eviction of cache lines that are in the L1.</p>
<p>The CPU cache is always moving data in and out of memory in units called a <a href="http://stackoverflow.com/questions/3928995/how-do-cache-lines-work">cache line</a> (they will always be aligned in memory to a multiple of the cache line size). Cache line sizes vary depending on the hardware. Popular values are: 32 bytes on some older mobile processors, 64 bytes on most modern mobile, desktop and game console processors, and 128 bytes on some PowerPC processors (notably the Xbox360 and the PlayStation 3).</p>
<p>This cache level will contain code that is executing, data being processed, and translation look-aside buffer entries (for both code and data, see next section). They might be dedicated to either code or data (e.g: L1) or be inclusive of both (e.g: L2/L3).</p>
<p>The CPU cache is usually N-way set associative. You can find a good write up about this <a href="http://www.hardwaresecrets.com/article/How-The-Memory-Cache-Works/481/8">here</a>. Note that depending on the cache level, where the cache line index and tag comes from might differ. For example, the <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0360f/CHDEJHGD.html">code L1 might use the physical memory address to calculate both the index and the tag</a> while the <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0360f/CBAHAAGB.html">data L1 might use the virtual memory address for both the index and the tag</a>. The L2 is also free to choose whatever arrangement it pleases and note that in theory, they could be mixed (tag from virtual, index from physical).</p>
<p>When the CPU requests for a memory address, it is said to <em>hit</em> the cache if the desired cache line is inside a particular cache level and a <em>miss</em> if it is not and we must look either in a higher cache level or worse, main memory.</p>
<p>This level of caching is the primary reason why packing things as tight as possible in memory is important for high performance: fetching a cache line from main memory is slow and when you do, you want most of that cache line to contain useful information to reduce waste. The larger the cache line, the more waste will generally be present. Things can be pushed further by aligning your data along cache lines when you know a cache miss will happen for a particular data element and to group relevant members next to it. This should not be done carelessly, as it can easily degrade performance if you are not careful.</p>
<p>This level of caching is also an important reason while calling a virtual function is often painted as slow: not only must you incur a potential cache miss to read the pointer to the virtual function table but you must also incur a potential cache miss for the pointer to the actual function to call afterwards. It is also generally the case that these two memory accesses will not be on the same cache line as they will generally live in different regions in memory.</p>
<p>This level of caching is also partly responsible for explaining why reading and writing unaligned values is slower (when the CPU supports it). Not only must it split or reconstruct the value from potentially two different cache lines but each access is potentially a separate cache miss.</p>
<p>Generally speaking, all cache levels discussed up to here are shared between processes that are currently executed by the operating system.</p>
<h3 id="the-translation-look-aside-buffer">The translation look-aside buffer</h3>
<p>The next level of caching is the <a href="http://en.wikipedia.org/wiki/Translation_lookaside_buffer">translation look-aside buffer</a>, or TLB for short.</p>
<p>The TLB is responsible for caching results of the translation of virtual memory addresses into physical memory addresses. Translation happens with a granularity of a fixed page size and modern processors generally support two or more page sizes with the most popular on x64 hardware being 4KB and 2MB. Modern CPUs will often support 1GB pages but using anything but 4KB pages in a user space application is sometimes not trivial. However, on game console hardware, it is common for the kernel to expose this and using this important tool is important for high performance.</p>
<p>The TLB will generally have separate caches for the different page sizes and it often has several levels of caching as well (L1 and L2, note that these are separate from the CPU caches mentioned above). When the CPU requests the translation of a virtual memory address, since it does not yet know the page size used, it will look in all TLB L1 caches for all page sizes and attempt to find a match. If it fails, it will look in all L2 caches. These operations are often done in parallel at every step. If it fails to find a cached result, a table walk will generally follow or a callback into the kernel happens. This step is potentially <a href="https://www.kernel.org/doc/gorman/html/understand/understand006.html">very expensive</a>! On x64 with 4KB pages, typically four memory accesses will need to happen to find which physical memory frame (frame is generally the word used to refer to the unit of memory management the hardware and kernel use) contains the data pointed to and a fifth memory access to finally load that data into the CPU cache. Using 2MB pages will remove one memory access reducing the total from five to four. Note that each time a TLB entry is accessed in memory, it will be cached in the CPU cache like any other code or data.</p>
<p>This sounds very scary but in practice, things are not as dire as they may seem. Since all the memory accesses are cached, a TLB miss does not generally result in five cache misses. Pages cover a large range of memory addresses and typically the top levels of the address used will cover a large range of virtual memory and as such, the least virtual memory touched, the less data the TLB needs to manage. However, a TLB entry miss at a level N will generally guaranty that all lower TLB entry accesses will result in cache misses as well. In essence, not all TLB misses are equal.</p>
<p>As must be obvious now, every time I mentioned the potential for CPU cache misses in the previous section, is now potentially even worse if it results in a TLB miss as well. For example, as previously mentioned, virtual tables require an extra memory access. This extra memory access, by virtue of being in a separate memory region (the virtual table itself is read only and compiled by the linker while the pointers to said virtual table could be on the heap, stack, or part of the data segment), will typically require a separate cache line for it and separate TLB entries. It is clear that the indirection has a very real cost, not only in terms of the branch the CPU must take that it cannot predict but also in the extra pressure on the CPU and TLB caches.</p>
<p>Note that when a hypervisor is used with multiple operating systems running concurrently, since the physical memory is shared between all of these, generally speaking when looking up a TLB entry, an additional virtual memory address translation might need to take place to find the true location. Depending on the security restrictions of the hypervisor, it might elect to share the TLB entries between guests and the hypervisor or not.</p>
<h3 id="the-micro-tlb">The micro TLB</h3>
<p>Yet another popular cache that is not present (as far as I know) in x64 hardware but common on PowerPC and ARM processors is a higher level cache for the TLB. ARM calls this the micro TLB while PowerPC calls it ERAT.</p>
<p>This cache level, like the previous one, caches the result of translation of virtual addresses into physical addresses but it uses a different granularity from the TLB. For example, while the <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0535b/CACCAFHH.html">ARM processors</a> will generally support pages of 4KB, 64KB, 1MB, and sometimes 16MB; the micro TLB will generally have a granularity of 4KB or 1MB. What this means is that the micro TLB will miss more often but will often hit in the TLB afterwards if a larger page is used.</p>
<p>This cache level will generally be split for code and data memory accesses but will generally contain mixed page sizes and it is often fully associative due to its reduced size.</p>
<p>The CPU, TLB, and micro TLB caches are not only shared by currently running processes of the current operating system but when running in a virtualized environment, they are also shared between all the other operating systems. When the hardware does not support the sharing by means of registers holding a process identifier and virtual environment identifier, generally these caches must be flushed or cleared when a switch happens.</p>
<h3 id="the-cpu-register">The CPU register</h3>
<p>The last important cache level in a computer I will discuss today is the CPU register. The register is the basic unit modern processors use to manipulate data. As has been outlined so far, getting data here was not easy and the journey was long. It is no surprise that at this level, everything is now very fast and as such packing information in a register can yield good performance gains.</p>
<p>Values are loaded in and out of registers directly into the L1 assisted by the TLB. Register sizes keep growing over the years: 16bit is a thing of the past, 32bit is still very common on mobile processors, 64bit is now the norm on newer mobile and desktop processors, and processors with multimedia or vector processing capability will often have 128bit or even 256bit registers. In this later case, this means that we only need two 256bit registers to hold an entire cache line (generally 64 bytes on these processors).</p>
<h3 id="conclusion">Conclusion</h3>
<p>This last points hammers in the overall high performance mantra: don’t touch what you don’t need and do the most work you can with as little memory as possible.</p>
<p>This means loading as little as possible from a hard drive or the network when such accesses are time sensitive or look into compression: it is not unusual that simple compression or packing algorithms will improve throughput significantly.</p>
<p>Use as little virtual memory as you can and ideally use large pages if you can to reduce TLB pressure. Data used together should ideally resides close by in memory. Note that virtual memory exists primarily to help reduce physical memory fragmentation but the larger the pages you use, the less it helps you.</p>
<p>Pack as much relevant data as possible together to help ensure that it ends up on the same cache line or better yet, align it explicitly instead of leaving it to chance.</p>
<p>Pack as much relevant data as possible in register wide values and manipulate them as an aggregate to avoid individual memory accesses (bit packing).</p>
<p>Ultimately, each cache level is like an instrument in an orchestra: they must play in concert to sound good. Each has their part and purpose. You can tune individually all you like but if the overall order is not present, it will not sound good. It is thus not about the mastery of any one topic but in understanding how to make them work well together.</p>
<p>This post is merely an overview of the various caches and their individual impacts. A lot of information is incomplete and missing to keep this post concise (I tried..). I hope to revisit each of these topics in separate posts in the future when time will allow.</p>
Introducing the 'gin' library2015-05-03T00:00:00+00:00http://nfrechette.github.io/2015/05/03/introducing_gin<p><code class="language-plaintext highlighter-rouge">gin</code> aims to be a playground for high performance C++ ideas all under the MIT license. <code class="language-plaintext highlighter-rouge">gin</code> is currently empty but it will not remain so for long!</p>
<p>Here is an overview of the things I intend to cover in the coming posts and that will be included in the library:</p>
<p>A convenient memory allocator interface suitable for high performance and containers
Several high performance memory allocators (linear, frame, small block, heap, external, etc.)
A number of containers using the above technology (all sorts of array variants, bit sets, etc.)
And much more!</p>
<p>I will use C++11 features where relevant and keep things in headers as much as I can for easy integration as well as <a href="http://bitsquid.blogspot.ca/2012/11/bitsquid-foundation-library.html">all the popular reasons</a>.</p>
<p>I choose the name <code class="language-plaintext highlighter-rouge">gin</code> for a number of reasons: it is a refreshing drink for the upcoming summer months, there is no associated programming material with that word, it is short, and if I ever need a side project, I can cleverly name it <code class="language-plaintext highlighter-rouge">tonic</code>.</p>
<p>The code can be found <a href="https://github.com/nfrechette/gin">here</a> on github.</p>
C++ STL container deficiencies2015-05-01T00:00:00+00:00http://nfrechette.github.io/2015/05/01/cpp_stl_container_deficiencies<p>In the AAA video game community, it is common to reimplement containers to fix a number of deficiencies. EA STL being a very good example of this. You can find a document <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2271.html">here</a> that explains in great detail the reasoning behind it. Some of the points are a bit dated and have been fixed with C++11 or will be fixed in C++14. I will not repeat them here but I will add some of my own.</p>
<h3 id="some-function-names-are-un-intuitive">Some function names are un-intuitive</h3>
<p><code class="language-plaintext highlighter-rouge">deque</code> and <code class="language-plaintext highlighter-rouge">list</code> have <code class="language-plaintext highlighter-rouge">push/pop</code> which are symmetric and this is good. But <code class="language-plaintext highlighter-rouge">insert/erase</code> seems a poor choice of words and we have the <code class="language-plaintext highlighter-rouge">emplace</code> variants which overlap the two functionalities.
<code class="language-plaintext highlighter-rouge">vector</code> appears to have a mix of queue/stack like functions (<code class="language-plaintext highlighter-rouge">push/pop</code> and <code class="language-plaintext highlighter-rouge">front/back</code>) but this choice of words is a poor fit when talking about arrays.</p>
<p>Naming things is always sensitive subject and in the interest of discussion I give the following two examples: Java uses <code class="language-plaintext highlighter-rouge">add/remove</code> for all container operations while C# uses <code class="language-plaintext highlighter-rouge">Add/Remove</code> and <code class="language-plaintext highlighter-rouge">Insert/RemoveAt</code>.</p>
<h3 id="vector-is-a-poor-container-name">Vector is a poor container name</h3>
<p>In video games, and mathematical applications, vectors are N dimensional. This reflects the STL container name but in video games vectors are almost always 2D, 3D, or 4D. Ultimately, this can yield some confusion. Even in the <code class="language-plaintext highlighter-rouge">vector</code> documentation, it is explained in terms of arrays while array is reserved for an array of fixed size that also happens to be inline. In practice, there are many useful variants of array implementations and adding a simple qualifier to the name makes everything much more clear: <code class="language-plaintext highlighter-rouge">FixedArray</code>, <code class="language-plaintext highlighter-rouge">InlineArray</code>, <code class="language-plaintext highlighter-rouge">DynamicArray</code>, <code class="language-plaintext highlighter-rouge">HybridArray</code>, etc.</p>
<h3 id="the-meaning-of-the-word-size-is-overloaded-and-confusing">The meaning of the word ‘size’ is overloaded and confusing</h3>
<p>Size in C++ means many things. <code class="language-plaintext highlighter-rouge">size_t</code> is defined as the type of the result of the <code class="language-plaintext highlighter-rouge">sizeof</code> family of operators. On most systems it will be the same size as the largest integral register (32 or 64 bits) and will typically match <code class="language-plaintext highlighter-rouge">uintptr_t</code> for this reason. The <code class="language-plaintext highlighter-rouge">sizeof</code> operators also return a size and this size is measured in bytes (which can be 8 or more bits depending on the platform). For the containers, <code class="language-plaintext highlighter-rouge">size()</code> returns the number of elements.</p>
<p>Size is thus a type, a number of bytes and a number of elements depending on the context.</p>
<h3 id="templating-the-allocator-on-the-container-type-is-a-bad-idea">Templating the allocator on the container type is a bad idea</h3>
<p>EA STL touches on most of the issues related with the poor STL memory allocator support but it forgets an important point: the speed gain of templating the allocator type renders refactoring code much harder. It is not unusual to write a function that takes a container as argument and at a later time, the need to change the allocator type arises. This forces the user to change every site where the allocator type appears which can end up being many functions. I have witnessed optimizations that could not be performed because the amount of work would be too great due to this. It is also often desirable to write functions that will require allocation/deallocation that are allocator agnostic without templating the whole function on the allocator type.</p>
<p>A common fix for this is to use an allocator interface with virtual functions which is simple and clean. The bitsquid engine <a href="http://bitsquid.blogspot.ca/2010/09/custom-memory-allocation-in-c.html">does this</a>. It is not without its faults but it is ultimately much more flexible. I plan to discuss this in further detail in later posts.</p>
<h3 id="containers-are-often-larger-than-they-need-to-be">Containers are often larger than they need to be</h3>
<p>It is not unusual to end up with hundred of thousands or even millions of array containers and the likes. In this scenario, every byte counts and this can end up bloating objects that hold them. Generally, <code class="language-plaintext highlighter-rouge">size_t</code> is far too large to keep track of sizes and in fact, many video game engines will simply hardcode a <code class="language-plaintext highlighter-rouge">uint32_t</code> instead (only the most recent game consoles now have more than 4GB of memory, and just barely). However, even this is often wasteful as the norm is to have tiny arrays and not larger ones. It would be much more flexible is we had a base container templated on the type used to track the sizes and define all popular variants: <code class="language-plaintext highlighter-rouge">Vector8</code>, <code class="language-plaintext highlighter-rouge">Vector16</code>, <code class="language-plaintext highlighter-rouge">Vector32</code>, <code class="language-plaintext highlighter-rouge">Vector64</code>, etc. The container should then check for overflow and assert/fail if it does so. We could have a default <code class="language-plaintext highlighter-rouge">Vector</code> without a size specified that defaults to a sensible configurable value (e.g: <code class="language-plaintext highlighter-rouge">Vector32</code> or <code class="language-plaintext highlighter-rouge">Vector64</code>).</p>
<p>Another source of waste is the allocator contained inside the containers. For reasons discussed above, it is desirable to store a pointer to the allocator inside containers. Since a container that requires allocation will already have a pointer to point to the allocated memory, and since we can rely on the fact that the container has two states (unallocated and allocated), we can store that pointer inside the data pointer in the former case (and use either a flag in the LSB or the capacity of the container to tell which state it is in) and simply append it at the front or back of the allocated memory in the later case. This will save a few bytes in the container object itself reducing the size of the holding object. I will discuss this again in a future blog post.</p>
<h3 id="removing-items-from-an-unsorted-array">Removing items from an unsorted array</h3>
<p>It is generally the norm that array containers will hold their contents unsorted. When this is the case, upon removing an item, we can simply overwrite it with the last item of the array instead of shifting all the elements left. This ensures a much more deterministic runtime cost for this operation.</p>
<h3 id="containers-hide-everything">Containers hide everything</h3>
<p>Sometimes, having a C style API with a raw view is beneficial. <a href="http://bitsquid.blogspot.ca/2012/11/bitsquid-foundation-library.html">The bitsquid foundation library</a> explains and uses this. However, at the same time, in the vast majority of cases, it does not make sense to expose programmers directly to this as it can be error prone (it becomes easy to make a container out of sync or corrupted). For example, bit sets often come in two variants: an inline array of bits and a dynamically allocated array of bits. It is often the case that the code will be duplicated to support both cases. However, at times a third cases arises: when you need to make your dynamic bit array inline as part of a larger memory block (e.g.: imagine a dynamic array where for each slot we also store a bit, with the STL containers, we would have two dynamic allocations when one would suffice). In the later scenario, none of the bit set functions can be reused and they must again be duplicated. In practice, all three cases can be implemented by reusing functions that take as input a raw view of the bit set along with the number of bits.</p>
<p>I plan to look at a number of these topics more in depth in future blog posts, stay tuned!</p>
Virtual function dispatching2015-04-30T00:00:00+00:00http://nfrechette.github.io/2015/04/30/virtual_function_dispatching<p>Very often, <a href="http://programmers.stackexchange.com/questions/80591/why-is-no-c-interview-complete-if-it-does-not-have-vtable-questions">C++ interviews ask about the virtual table</a>: how it works, how is it implemented and the tricky bits around constructors & destructors. However, rarely are the following questions asked: why is it the way it is and are there alternatives?</p>
<p>Seeing how the C++ standard does not mandate how virtual function dispatching should be implemented, it is interesting that most compilers ended up converging on the current table approach. The <a href="http://en.wikipedia.org/wiki/Virtual_method_table">wikipedia article</a> does a fairly good job to explain how it works and gives a good idea of the problem it is trying to solve. However, it fails to address the assumptions and performance constraints that likely shaped the current design.</p>
<p>For simplicity’s sake, we will only consider the case of single inheritance but keep in mind all methods discussed here can be extended to support multiple inheritance.</p>
<p>Let us go back to the drawing board and take a closer look at the problem we are trying to solve, what our tools are and our performance constraints.</p>
<h3 id="the-problem">The problem:</h3>
<p>It is often desirable to derive from a base class and override part of its behaviour. This is best achieved by implementing a specialized function that the base class would call (or for that matter, any code) when present or call the base implementation when our only knowledge is the type of the base class at a particular function call site.</p>
<h3 id="the-tools">The tools:</h3>
<p>All classes and functions are known at compile time and ideally we want to leverage that as much as possible.</p>
<h3 id="the-performance-constraints">The performance constraints:</h3>
<p>Memory and CPUs are slow and scarce and we want to have this mechanism be as fast as possible and at the same time be as compact in memory as possible. We can safely assume that for some classes we could have a large number of such virtual functions and we can assume as well that for some classes we might have a large number of instances.</p>
<h3 id="the-inline-solution">The inline solution:</h3>
<p>The simplest solution would simply be to store a function pointer for every such virtual function inside the object itself. At object construction, we can simply set these pointers and later when we need to invoke them, we simply call the correct function pointer.</p>
<p>The upside of such a simple method is that the information we need is right there inside the object and we can access it directly. It will also end up with the rest of the adjacent object data in whatever <a href="http://en.wikipedia.org/wiki/CPU_cache">cache line</a> is loaded to access the function pointer. By identifying which data is used by which function, we could move relevant data close to that function pointer and improve locality. Calling such a virtual function would thus incur a single data cache miss (to access the pointer).</p>
<p>The downside is that it does not scale. The more virtual functions we add, the larger our objects will end up being. This is made even worse on platforms with 64 bit pointers. The cost to initialize those data members also increases with each new virtual function. Finally, if enough virtual functions are added, our cache line might end up being completely filled by them such that no other immediately useful information ends up being loaded anyway.</p>
<p>Ultimately, this method will not degrade gracefully when either the number of object instances increases or the number of virtual functions rises.</p>
<p>This method is ideal when:</p>
<ul>
<li>the number of virtual functions per class is low</li>
<li>meaningful data can be moved close to the virtual function pointer (eg: a getter and the returned value)</li>
<li>the number of class instances is low</li>
</ul>
<h3 id="the-dispatch-function">The dispatch function:</h3>
<p>A more flexible approach would be for our objects to have a single function pointer to a per class dispatch function. This function would then be called in a similar fashion to syscall(..): an extra argument to indicate which virtual function is expected would be provided. On object construction, similar to the previous approach, we simply set the proper dispatch function.</p>
<p>The upside is that now our cost per object is a single function pointer and we are very flexible. At runtime we could add extra data to change the dispatching behaviour (and indeed a mechanism similar to this is used by many dynamic languages for duck typing and such) and we are also very flexible in our we store the virtual function pointers: we can either store them in an array somewhere along with the identifier that we need to match or we can store it inline in the assembly stream. The latter approach is also interesting because it could end up being more compact than the former on architectures where instruction sizes can vary and that support relative jumps (e.g.: x64). Another point in favour of the later approach is that the code cache generally has less pressure than the data cache by virtue of the facts that code execution is largely linear and more easily prefetched and there is also less code compared to data. Last but not least, a dispatch function for a given class only needs to test the virtual functions it implements and defer resolution to its base class should it not handle the call.</p>
<p>The downside is that to find the correct virtual function pointer to call we need to introduce extra branching which will could end up being poorly predicted. Various methods can mitigate this: sorting the virtual functions in a tree structure to guaranty O(log(n)) access time (it could even be updated at runtime to favour hot functions) or the introduction of a bloom filter as an early out mechanism. Another problem is the calling convention: because for efficiency reasons we need a register to hold the virtual function identifier when we call our dispatch function, that register must be reserved to avoid shuffling all the registers and the stack when the effective virtual function is called. While this approach scales well when the number of object instances increases, it does not scale as well when the number of virtual functions rises.</p>
<p>Compared to the previous approach, we will need to incur one data cache miss for the dispatch function pointer and one (or more) code cache miss for the dispatching code. The compiler could be smart here and position the dispatch code close to the virtual functions of the same class in hope that it ends up being on the same page and thus avoid a <a href="http://en.wikipedia.org/wiki/Translation_lookaside_buffer">TLB</a> miss when the actual function is called.</p>
<p>Much like the previous approach, this method does not degrade gracefully when the number of virtual functions increases.</p>
<p>This method is ideal when:</p>
<ul>
<li>The dispatching behaviour needs to be altered at runtime</li>
<li>The dispatching logic is complicated</li>
<li>The number of virtual functions per class is low</li>
<li>The number of class instances is high</li>
</ul>
<h3 id="the-virtual-table">The virtual table:</h3>
<p>Finally we reach the current popular implementation: a single pointer to an array of virtual function pointers. Similar to previous methods, on object construction we simply set the proper array pointer. For this method to work, the array must contain function pointers for all virtual functions of the current class as well as all base classes. When we perform a virtual function call, knowing which function is called we can infer an offset in the array to use and simply call the correct virtual function pointer.</p>
<p>The upside is that like the previous approach, this scales well to a large number of object instances due to the low per object overhead but unlike the previous approach, it also scales well to a large number of virtual functions since finding the correct one is O(1).</p>
<p>There are no obvious downsides and in the end, this method will degrade gracefully.</p>
<p>However, things are not perfect: the array of virtual functions must be put somewhere. When we call a virtual function, we will thus incur two data cache misses: one for the pointer to the array and another for the actual virtual function pointer used. This second cache miss may very well not contain relevant information in the cache line besides the few bytes we need for the pointer. This will of course depend heavily on the usage but in general, there will be a lot of waste in that cache line. Because we can’t position the array in meaningful position, it is also possible that the page where it resides does not contain other relevant information, causing pressure on the TLB. In practice, other virtual function arrays are likely to end up in that page and a large number of them could fit inside. Thus a program using virtual functions with this method with 16KB worth of virtual function arrays could very easily end up permanently using four 4KB TLB entries (it is typical for most kernels to use 4KB pages for this sort of data). Incidentally, with increased TLB pressure, we also have an increased pressure on higher level caches such as ERAT on PowerPC and the <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0360e/CHDDIJBD.html">micro TLB</a> on ARM chips.</p>
<p>This method is ideal when:</p>
<ul>
<li>The number of class instances is high</li>
<li>The number of virtual functions per class is high</li>
</ul>
<h3 id="the-conclusion">The conclusion:</h3>
<p>Depending on the scenario and usage, each variant could end up being the superior choice but ultimately, if a language does not expose control over which method is used for this, it must make a safe choice that supports its relevant use cases (e.g.: duck typing at runtime) and degrades as gracefully as possible under unknown scenarios.</p>
The need for speed2015-04-26T00:00:00+00:00http://nfrechette.github.io/2015/04/26/the_need_for_speed<p>The art of writing fast software is subtle and ever changing. Fundamentally, it is all about where your data comes from and how you access it.</p>
<p>At a high level, you have the entire field of algorithmic complexity dedicated to touching the least amount of data for a given task. At this level, the actual hardware that executes the algorithm hardly matters since we deal in very large numbers but this is not to say that it is irrelevant, faster hardware will always help.</p>
<p>As the amount of data manipulated shrinks, the gains from an optimal algorithm versus a slightly less optimal one will blur and a new perspective is necessary.</p>
<p>At a lower level, knowing how to leverage the hardware becomes of paramount importance. The operating system and the hardware go to great lengths to cache data in various ways and knowing how they are used is key in squeezing the very last drop of performance.</p>
<p>Last but not least, the modern processor is really a <a href="http://blog.erratasec.com/2015/03/x86-is-high-level-language.html#.VUGC8q3BzGc">tiny virtual computer</a>. At this micro level, the instruction stream ordering and what instructions are used can still make a significant difference on performance.</p>
<p>Each tier has its own optimization specialists but they do not all enjoy the same amount of attention. The reason for this is simple: they are all not equal in value. Out in the real world, most programmers will be impacted largely by the first and second tier, in that order. It isn’t unusual to work with thousands of elements or more and at these scales, algorithmic complexity and good cache usage will always dominate. The last tier is mostly reserved to extreme low level programmers: hardware designers, embedded programmers, compiler programmers and specialized library programmers.</p>
<p>The secret to writing fast software is to make your assumptions explicit and measure early and often. Explicit assumptions are key to making sure and documenting that the choices you make as a result are proper and make sense. Should your assumptions change or prove wrong, bad things could happen or new opportunities might arise.</p>
<blockquote>
<p>It is not unusual for this to happen due to changes in technology. Consider how performance assumptions had to be revised after SSDs were introduced or when C++11 made the addition of move semantics.</p>
</blockquote>
<p>However, high performance software is slightly different. Much like race cars, <a href="http://hacksoflife.blogspot.ca/2015/01/high-performance-code-is-designed-not.html">performance is built in</a> from the ground up, it is intrinsic to its DNA. A Toyota Camry is a good car but fitting in a Formula One engine into it and dropping the car on a race track will yield disappointing results compared to true breeds. The same applies to software and the same could be said of the <a href="/2015/04/25/all_about_data/">many goals</a> previously discussed (as Adobe Flash found out, security cannot be retro-fitted easily).</p>
All about data2015-04-25T00:00:00+00:00http://nfrechette.github.io/2015/04/25/all_about_data<p>The modern computers of today are an amazing and complex creation. From the smallest cellphones up to your super charged desktop PC, each and everyone of them is in reality a mixture of many smaller specialized computers working in concert.</p>
<p>However, at the end of the day, they all do the same thing: they munch on data. That is their only purpose and from the meaning inscribed in that data comes out the complex behaviours that we see in everyday programs and devices.</p>
<p>Programming is the art of organizing that data in meaningful ways in order to achieve a specific end result. Many criteria exist to take into consideration when designing such systems:</p>
<ul>
<li>User friendly: how do we show the data and how do we allow the data to be manipulated by the user?</li>
<li>Development friendly: how do we organize the data (and code) for it to be easily manipulated in the ways that are required by the product?</li>
<li>Hardware friendly: how do we make sure that the data is layered out in the optimal way for the hardware underneath?</li>
<li>Communication friendly: how do we make sure that the data is easily communicated in between various systems either internally in the computer or externally?</li>
<li>Resilient to damage: how do we make the data safe from hardware failures?</li>
<li>Tamper proof: how do we make the data safe from malicious tampering?</li>
<li>Secure: how do we make sure that data in transit isn’t intercepted by a third party?</li>
</ul>
<p>I am sure there are many more, these a simply a small subset.</p>
<p>What is often less discussed is that very often these goals have conflicting desires. In such scenarios, it is paramount to clearly identify the main goals of the software in order to make and validate our assumptions. These should guide all the important decisions regarding how the software is built and how the data is dealt with.</p>
<blockquote>
<p>For example, in the context of AAA video games, with the hardware being largely fixed and the demand for ever higher & prettier visuals, the two most important criteria are almost always user friendliness and hardware friendliness for single player games. For multiplayer games, the complexity increases and communication & tampering become important. Last but not least, for games with micro transactions, security comes into play. Depending on the platform and the game, the risks will also vary in scale and scope.</p>
<p>In contrast, in the context of mobile games, development friendliness is king to allow churning updates, content and new games as quick as possible. It isn’t unusual to find mobile games with poor user interfaces, bad performance and that are easily tampered or compromised. In the top tier of games, user & hardware friendliness are again very important and show up in very clean games such as Clash of Clans and Candy Crush.</p>
</blockquote>
<p>The single most important thing for a team is to be aligned on these goals and consider them when making every decision. Failure to do this exercise can have disastrous consequences: a change might easily introduce a performance issue or a security issue if one isn’t careful and paying attention to these goals. Ultimately the entire product can become at risk.</p>
<p>This is the reality in software design as much as it is in any other industry where teams must juggle multiple goals.</p>