Mike Acton

Utility: match

Sat, 07 Apr 2007 06:13:23 +0000

Update! If fixed up all the greater-than and less-than symbols in this entry. I didn't make much sense before. I always forget to change those up in the HTML.

I'm just sharing a little utility I use all the time called match.

Usage: ./match [-h]  

For each line in  print the index to the 
first matching line in .

[-h] Print results in 32 bit hexidecimal (default is decimal)

Note: The max line width supported is 4095 characters.
Note: Maximum number of lines supported is (2^32)

If I have a source file of data represented as text (as I often do because it's often easier for me to read binary dumps in a text editor than a special "hex editor"), I use match to create a table of indices to unique lines (often these correspond to 128 bits since that's the size of an SPU register).

I commonly use it like so (given I have a file called "source_file")

sort source_file | uniq > uniq_file
match source_file uniq_file

Now I have a handy table of indices!

Download: match.c

Open Source and Console Games

Wed, 09 Aug 2006 08:45:28 +0000

On August 16, 2006 I participated in a panel discussion on Open Source and media as part of Digital Hollywood's Building Blocks 2006 conference.

Here is the description of the panel [from digitalhollywood.com]

The Open Source movement began during the dot.com rise with young companies developing great tools to deliver applications and services across multiple platforms. The consumer’s appetite for new content driven experiences has expanded to include ways to view, manage, and share content across devices. With the changing landscape around the home, Open Source promises to power a new generation of applications running over today’s high-speed networks and the systems used to create, manage, and distribute that content.

Come join key leaders in the global electronics, online, and media communities to discuss Open Source's definition, and learn how companies will create systems, infrastructure, and applications for the next generation of the Consumer Entertainment Experience.

For those of you who did not attend, I would like to take an opportunity to discuss here my personal opinions on these issues.

Understanding Strict Aliasing

Thu, 01 Jun 2006 09:35:19 +0000

UPDATED! (25 Feb 08) Added note on bitfields. Corrected a typo (Thanks Turly O'Connor!)

UPDATED! (08 Aug 06) More Clarifications! Special thanks to Nicolas Riesch, André de Leiradella and pinskia for their comments and suggestions.

UPDATED! (28 Dec 06) Minor fixes. Special thanks to Kobi Cohen-Arazi and Chris Pickett.

Aliasing

One pointer is said to alias another pointer when both refer to the same location or object. In this example,

  0uint32_t 
  1swap_words( uint32_t arg )
  2{
  3  uint16_t* const sp = (uint16_t*)&arg;
  4  uint16_t        hi = sp[0];
  5  uint16_t        lo = sp[1];
  6  
  7  sp[1] = hi;
  8  sp[0] = lo;
  9
 10  return (arg);
 11}

Using GCC 3.4.1 and above, the above code will generate warning: dereferencing type-punned pointer will break strict-aliasing rules on line 3.

The memory referred to by sp is an alias of arg because they refer to the same address in memory. In C99, it is illegal to create an alias of a different type than the original. This is often refered to as the strict aliasing rule. The rule is enabled by default in GCC at optimization levels at or above O2. Although the above example would compile, the results are undefined. More than likely, arg would be returned unchanged because a pointer to uint16_t cannot be an alias to a pointer to uint32_t when applying the strict aliasing rule.

Dereferencing a cast of a variable from one type of pointer to a different type is usually in violation of the strict aliasing rule.

However, having multiple representations of the same location in memory is often beneficial. Properly balancing the compiler's memory optimizations and the programmer's optimizations based on real-world context and data is a bit of a black art. It requires an understanding of the tradeoffs among what's permitted by the standard, what's the reality of compilers and the value of a particular transformation based on the architecture and the data. It's worth it in the end though when the results speak for themselves.

All of the examples in this article have been tested with various versions of GCC. Although you can expect most of the examples to generate similar results across the major compilers, programmers' expectations should always be validated for the compilers and compiler revisions required.

Read on for details on the strict aliasing rule and some common pitfalls.

Demystifying The Restrict Keyword

Mon, 29 May 2006 08:13:54 +0000

UPDATED! More examples! More detailed explainations!

Contract

The restrict keyword can be considered an extension to the strict aliasing rule. It allows the programmer to declare that pointers which share the same type (or were otherwise validly created) do not alias eachother. By using restrict the programmer can declare that any loads and stores through the qualified pointer (or through another pointer copied either directly or indirectly from the restricted pointer) are the only loads and stores to the same address during the lifetime of the pointer. In other words, the pointer is not aliased by any pointers other than its own copies.

Restrict is a “no data hazards will be generated” contract between the programmer and the compiler. The compiler relies on this information to make optimizations. If the data is, in fact, aliased, the results are undefined and a programmer should not expect the compiler to output a warning. The compiler assumes the programmer is not lying.

THE RESTRICT CONTRACT

I, [insert your name], a PROFESSIONAL or AMATEUR [circle one] programmer recognize that there are limits to what a compiler can do. I certify that, to the best of my knowledge, there are no magic elves or monkeys in the compiler which through the forces of fairy dust can always make code faster. I understand that there are some problems for which there is not enough information to solve. I hereby declare that given the opportunity to provide the compiler with sufficient information, perhaps through some key word, I will gladly use said keyword and not bitch and moan about how "the compiler should be doing this for me."

In this case, I promise that the pointer declared along with the restrict qualifier is not aliased. I certify that writes through this pointer will not effect the values read through any other pointer available in the same context which is also declared as restricted.

* Your agreement to this contract is implied by use of the restrict keyword ;)

Read on for more information on the practical use and benefits to using the restrict keyword...

A Practical GCC Trick To Use During Optimization

Sat, 15 Apr 2006 09:10:13 +0000

Splitting a basic block (by force)

Warning: This is a trick to use during optimization. It is not documented nor gauranteed to work across multiple platforms or different revisions of the compiler. Many programmers will say that this non-portable code should not be used in production, but there is such a huge practical benefit to using it has to be mentioned.

One of the first things that the compiler's scheduler does is organize the code into blocks of code without branches, out-of-line function calls or other optimization barriers. These basic blocks will then be scheduled among eachother based on their dependencies. Toward the end of scheduling individual instructions will be scheduled within basic blocks. Finally, instructions may be moved across block boundaries.

Inline assembly is most definately an optimization barrier and every block of inline assembly is treated as an independent basic block. This is why assembly should be inlined one instruction at a time - to give the compiler the maximum number of options for scheduling.

Why do I mention this?

Well, it turns out that if you have an empty inline assembly statement you can rely on the side-effect of splitting the basic block witout actually adding any instructions.

#define GCC_SPLIT_BLOCK __asm__ ("");

Performance and Good Data Design

Mon, 03 Apr 2006 08:47:20 +0000

Why Is Data Design Important For Games?

All game technology is simply a function to manipulate data. A game can be thought of as a very complicated DSP with controllers, source art and time as inputs and an audio-visual display as output. This is not a radical or revolutionary concept. It is however, widely forgotten or ignored in console game development. Development has changed dramatically over the years and the idea behind this article is to remind game programmers that the only thing we do now of any consequence is transform data. The only thing that really makes games unique is the types and amounts of data programmers must transform in a short, fixed period of time.

Data access is the biggest problem in attaining maximum performance from a game console. This is necessarily true. Modern consoles are made up of deeply pipelined systems – The CPUs and coprocessors are designed for minimum cycle throughput, caches are designed for maximum throughput on sequential access, DSPs and GPUs are designed to maximize performance at the cost of instruction and data space – Any significant change in data access patterns will stall any or all of these pipelines. Of course there are required stalls in any pipelined system, however game systems themselves provide a pipeline from the content creators to the hardware and the best of those maximize the width and speed of those pipelines by minimizing non-required stalls along the way.

Good code follows good data, not the other way around.

Fundamentally game systems can not be built for performance independent of the data. Good code follows good data, not the other way around. It is a common misperception that the products of programmers are programs – code. The truth is that the only people who care about code are programmers and we do not ultimately serve ourselves – The product of programmers is a service – console game programmers provide a mechanism for the content creators to put the content on a particular piece of hardware. Both the content creators and the game players want more and more and if we do not do everything we can within the limits of time and budget simply because the code doesn’t meet our expectations of how code should be designed or because a particular piece of hardware is more difficult to work with than we had hoped, then we are doing a disservice to our ultimate customers.

In order to effectively design and optimize a system for a console game, both the data and how it is used must be known. This is the most obvious, most crucial and most neglected principle in software architecture.

It used to be that programmer's did everything - the design, the art, the testing, not to mention writing the code. Things have changed. There are professional designers, level builders, graphic artists, technical artists, art directors, creative directors, and a QA staff large enough to require their own building.

In today's console game development world, programmers are responsible for only one thing: the data traffic through the console. detailed knowledge of what that content actually is, when it is transferred and how it is constructed it is very difficult to shape that data for best-fit performance.

What follows are some simple rules of thumb that programmers can follow to create a solid pipeline from the content creators to the screen and speakers.