Hardware Report - Thomas D

This is the system level integration practical final report for the Group C of 2008/09. The project was the development of a human interface using Prospeckz and Orient devices together to interact with a fighting video game.

This particular document is a report on the development, and implementation of an application linking Prospeckz devices to a Super Nintendo gaming console. The first part of the report briefly goes over the goal of the project, and the early decisions made. Then the next section puts this part of the project in the context of the overall project. Section 3 goes over the technical specifications. Section 4 describes the possible implementations with their advantages and disadvantages. Section 5 is a technical description of the implementation with the problems encountered, and section 6 is a discussion of the project life cycle.

Goal

The original idea was to create a new human interface for a commercial fighting game using two given technologies: Orients and Prospeckz.

Orients are a set of sensors which output over the radio their coordinates in a 3 dimensional space plus other physical measurements such as acceleration. The sensors are attached to different parts of a human body, the output data is processed in real time, and models a human body and its movements. The model is displayed on a screen, and the data is made available through an API which can be used by third party applications. One of the main benefits of this technology is it lets its users have a model of a human body with real time movements, but without a complicated set of cameras reading infrared spots of visual sensors on the body of an actor. Furthermore, as each Orient device is entirely wireless and powered by a battery, they are much more practical to use.

Prospeckz are small devices which have a micro-controller, memory, I/O ports, and a radio to communicate with other Prospeckz devices. With their large memory, low power consumption, ability to communicate between each other over the radio, and fast micro-processor, they can be used in a large range of passive and active applications. The Prospeckz implement the ZigBee wireless protocol for their communication. ZigBee is suitable for wireless personal area networks (WPANs), and is intended to be simpler and more energy efficient than other solutions such as Bluetooth.

Prospeckz

The Orients, and at least a Prospeckz were to be used in the design. At first, our group discussed and agreed on what we were going to do, as a group, but also as individuals. We looked at different fighting games, from older games such as the Street Fighter series released on Nintendo platforms, to more modern and realistic systems available on the Sony Playstation 3. The consensus was to use Street Fighter 2 Turbo on the Super Nintendo gaming console, as we could access the hardware, but also open source emulators. The 15 year old game offers all the features we wanted to interact with: multiple fighting gestures, basic movements, and AI.

Street Figher II Turbo

It was agreed from the very beginning the project would be divided into a number of parts. One was the classification of gestures recorded from the Orients in real time. Two different classifiers were created, one by Vasilios, and one by Jonathan. Another part was getting a Prospeckz device to interact with the Super Nintendo (SNES), this part was done by myself (Thomas). Other parts were: graphical user interfaces (Stuart), software simulation, and integration (Stuart).

Context

The goal of this specific sub-project was to get at least a Prospeckz device to interact with a Super Nintendo on the Street Fighter 2 Turbo game and a computer. Many different approaches were possible in order to achieve this. The consensus was to use two Prospeckz devices talking to each other using the radio. One connected to a PC receiving instructions from the gestures classifier, and the other one connected to the SNES, transmitting the instructions. This implies using a serial component to communicate with the computer, using the radio to communicate between to the prospeckz devices, and using the digital input/output ports to communicate with the SNES using its own proprietary protocol.

Street Figher II Turbo

The idea was the classifiers would simply connect to the serial port in order to communicate their instructions to one of the prospeck devices, which would in its turn communicate the right button command to the SNES. The graphical user interface would sit directly next to the classifiers to help set them up, but would have little control over the prospeckz. Finally, the software simulator was to be designed in case the group failed to connect to the SNES in hardware over the project period, and implied developping a plugin for one of the emulators available (ZNES or SNES9X).

Specifications

The specification of the project were to use the Orients with a software gesture classifier and send signals to a Prospeck device which would connect to the Super Nintendo. The Prospeckz being able to communicate between each other using their radio, it was suggested we use two Prospeckz in order to have the classifying station physically seperate from the gaming console.

Prospeck A had to communicate with the computer using a generic communicate protocol. Prospeck B had to be able to communicate with the SNES using Nintendo's gamepad protocol, or use another way to communicate button commands to the SNES.

Implementations

For Prospeckz A, which is the device which receives data from the gesture classifier on a PC, a communication protocol had to be used in order to at least send data from the computer to the device. The simplest solution is to use a generic communication protocol using serial or parallel interfaces. The prospeckz devices can implement a number of communication protocols including generic serial receiver/transmitters. Although many different solutions would have been possible, a UART (Universal Asynchronous Receiver Transmitter) is certainly one of the most straight forward ways to communicate between two devices which already have hardware serial interfaces available. For instance, another option could have been to use a prototol such as I2C which is more scalable and robust, but it would have implied using another peripheral attached to the PC in order to communicate with the device. It would also be possible to communicate directly from the PC to the Prospeckz B using the wireless, and not use Prospeckz A at all, but again this would have implied attaching another device to the PC in order to give it ZigBee radio capabilities.

The communication over the radio between two Prospeckz is straight forward as long no other Prospeckz are operating in the area. If that is the case, then it is just required to send data byte per byte, which is reliable. When multiple Prospeckz are used, then it becomes necessary to filter packets as it is not possible to send a packet to a specific device over the radio and make the assumption that another device is not going to read the packet. The same goes when reading packets, when multiple Prospeckz are in use in the same area, it is not possible to directly detect which devices are emitting a packet without wrapping the desired packet with some extra information. In this system, the packets are not wrapped, and the assumption is made that no other Prospeckz are in use while Prospeckz A and B are operating.

example graphic

For Prospeckz B, several implementations were possible: replicate the SNES gamepad hardware on the Prospeckz B, or generate the right signals in software using the onboard microprocessor. The first one seemed like the most complicated as it implied finding what was the hardware used by the Super Nintendo gamepads and replicating their behavior. We agreed that polling for the SNES gamepad signals and changing the output signals accordingly was the simplest solution.

Another solution, much simpler to implement but not as clean would have been to hack a working SNES gamepad and "emulate" button presses. The buttons in a SNES gamepad are made of rubber with a small carbon conducting patch at their base. Rubber has a high electrical resistance, which explains why the base of the rubber buttons are covered with a carbon patch which has a low electrical resistance. The gamepad button sits right on top of two non connected conductive pathways on a printed circuit board. When the button is not pressed, the two conductive pathways remain unconnected. When the button is pressed, the carbon patch under the button connects both conductive pathways, creating a low resistive electrical connection. Without going in the details of the SNES gamepad, when the button is pressed, then the signal under that button is grounded (0v). When the button on the SNES gamepad is not pressed, the signal under that button is not connected to anything, or has a high impedance (High Z). It would have been possible to connect the output pins of the Prospeck B to these electrical signals, and force them into high impedance or ground the signals in order to emulate a button press or release. The advantage of this solution is that it does not implie understanding the SNES gamepad protocol and timing the output signal according to it. The major disadvantage of this solution is it implies modifying directly a SNES gamepad and make it unusable by humans.

Design Methodology

One of the first step was to understand how to communicate with the SNES using one of the gamepad ports. The SNES has two gamepad ports which each have 5 wires. Two are power supply wires: brown (ground) and white (5v). Two of the three remaining wires are used by the SNES to output clock and timing information: yellow (data latch), and orange (data clock).

Summary of the SNES gamepad port wires:
  1. GND - Brown
  2. Not used
  3. Not used
  4. Data - Red
  5. Latch - Yellow
  6. Clock - Orange
  7. VCC - White
example graphic

The SNES sends a 12 us positive impulse on the data latch wire (yellow). This data latch indicates the SNES will start sending 16 data clock periods to the gamepad on the clock wire (orange) which are used by the gamepads to indicate what buttons are pressed. Each of the 16 data clock periods represent a different button, and the SNES will expect the ouput signal to toggle on those data clock periods which indicates a button on the gamepad is pressed.

There are 12 buttons on the SNES gamepad, but the SNES sends 16 data clocks periods. Only the first 12 data clock periods are used, the following 4 are ignored by the SNES. The protocol was first designed for the NES console which had less buttons. Nintendo added new buttons on the SNES but didn't change the communication protocol as it could support more buttons. Potentially, this protocol could support up to 16 different buttons.

SNES timing

The following image is a capture of the logic analyzer screen, which shows a positive data latch sent by the SNES at the top, then the 16 data clock periods also sent by the SNES in second position. The third line is the gamepad output (Prospeckz B). The SNES expects the output signal to be grounded during the data clock period when the button pressed. In this particular picture, we can the see button Right is being pressed on the SNES gamepad.

Logic Analyzer
Data clock periods and corresponding SNES buttons:
  1. B
  2. Y
  3. Select
  4. Start
  5. Up
  6. Down
  7. Left
  8. Right
  9. A
  10. X
  11. L
  12. R
  13. n/a
  14. n/a
  15. n/a
  16. n/a

The data clock has a period of 12us, which is approximatively 83.3 KHz. Knowing the PSOC micro-controller on the Prospeckz devices runs at 20Mhz, a clock cycle should take 0.05us. We can make the assumption that executing an instruction on the Prospeckz devices consumes between 1 and 5 CPU clock cycles, or up to 0.25us. This would mean the micro-controller can execute up to 48 instructions during each data clock period. This is over course an over-simplification of reality.

With this data in mind, we can make the assumption that polling for the data latch signal and data clock periods should be reliable enough, and toggling the output for the desired buttons should be possible with this hardware. The first implementation of the application linking the second Prospeckz device to the SNES used polling, which is a simpler solution than trying to replicate the hardware of the SNES gamepad in the PSOC micro-controller.

In reality, the SNES gamepad controller are entirely hardware. We saw earlier how a mechanical pression could actually be detected eletronically using a rubber button and a carbon patch to connect two conductive pathways. The SNES gamepad is made of two 8 bit shift registers. These are two integrated circuits which output 8 bits in serie. Two are used in order to output 16 bits. Each of the SNES button conductive pathways are connected to one of the 16 bits of the shift registers. The data latch activates the 8 bit shift registers, allowing them to read their 2*8 inputs. The SNES data clock is also sent to the 8 bit shift registers which send the status of their input in due time, as represented in the timing diagram above. Because this is an entirely hardware solution based on integrated circuits, it is very robust, and the response time is negligible.

Tools Used

This project involved testing electrical signals coming in and out of multiple devices, functionning at different voltages, and operating at different frequencies. One of the difficulties was to work with this varied range of devices, and it made the use of numerous tools necessary in order to test and read measures off the hardware directly. It started with a generic multimeter capable of measuring low potential differences (in volts), currents (in amperes), and electrical resistances (in ohms). This is a simple tool which can be used to quickly measure the potential difference between two pins, one of the most common operations when working with electronics. The problem with a multimeter is it becomes impractical to use when the signals change over time. An oscilloscope can then be used, which has a screen which displays a graphical representation proportional to a potential difference over a given period of time. As the SNES sends signals on a few wires which change over time, it is important to be able to visualise change over time. The problem with a regular oscilloscope is that it only displays a graphical reprensation of the potential difference in real time. In the digital world, signals can change much over a short amount of time, which makes it humanly impossible to read signals in real time. A logic analyser is used to memorise an display a digital signal after a defined trigger. Without such a tool, it would have been extremely difficult to test and debug applications using numerical signals.

Lab and tools

A regular power supply was also used in order to avoid using batteries when testing the system. The power supply can output a regulated 5v which is fed in the 5v input of the Prospeckz, offsetting the need of a battery. The Prospeckz have an onboard power regulator which converts the voltage into 3.3v, used by the onboard electronics.

Power supply

All the hardware design and testing was made on a breadboard, which makes it easy to add and remove components until the prototype works as desired. Once the prototype is ready, it can be transfered onto a circuit board which takes less space and is more durable.

Breadboard

To program the Prospeckz device, we used a PSOC programmer. All code was written in C, then compiled (converted) to assembly code, and then assembled into machine code. Machine code is raw data which is programmed in the flash memory of the Prospeckz device. This is transfered using a USB programmer. Once the device is programmed and checked, the device is restarted and automatically starts the newly programmed application. It can be tested using the onboard LED, input/output ports and radio.

PSOC programer

Implementation, testing, and re-usability

Reading the SNES signals

The first issue we ran into was reading the two signals from the SNES: data latch and data clock. We could measure the right potential difference between the brown and white wire, 5 volts as expected, but could not see a data latch nor clock periods on the orange and yellow wires using the oscilloscope and the logic analyzer. After looking closer at the internal design of a malfunctionning SNES gamepad, we found out resistors and a capacitor were used. In general, a capacitor and a resitor are added in serie with the ground (GND) wire somewhere on a long communication wire in order to smooth out noise which normally appears. Adding a capacitor and a resitor will also draw some current. Looking for the data latch and clock periods was possible when a simple resitor and capacitor were added.

Another issue when attempting to read the data clock periods with an oscilloscope was synchronisation. Because the signal only clocks 16 times, then has a long gap before clocking another 16 times, a basic oscilloscope cannot synchronise itself on the signal, which makes it difficult to read it. At that point, we started using the old logic analyzer, which was able to read the numeric signal as required.

Adapting potentials

The SNES functionning with standard on 5v and the Prospeckz devices functionning on 3.3v, it was necessary to test the compatibility of these voltages between the two devices, and if necessary adapt them. It is safe to use the 5v from the SNES two data outputs as inputs for the Prospeckz, but the 3.3v output from the Prospeckz is not registered by the SNES. This means the 3.3v output has to be amplified in order for it to work on the SNES. Amplifying a low power signal in electronics is trivial thanks to the transistor. All is required are two resitors and a generic transitor. The 3.3v output of the Prospeckz device is connected to the base of the transistor via a resistor. The collector of the transistor is connected to the 5v of the SNES also via a resistor, and the emitter is connected directly to the ground. The collector pin of the transistor will be the 5v amplified output signal.

example graphic

First attempt for Prospeckz A

The implementation went through different phases. The first few weeks of the project were dedicated to the implementation and test of the Universal Asynchronous Receiver Transmitter on the first Prospeckz device. The implemenation was quite straight forward as the protocol is generic. All is needed is set up a few parameters, and copy a few lines of code from the samples provided. In order to function properly, a clock has to be generated using a generic counter. The clock is set to 33KHz so the UART can communicate with the PC at 33KBps. It is also possible to set up the parity bits, which does not make much difference as long the settings correspond between the PC and the Prospeckz device. A listing on the code is available in the appendices of this document.

example graphic

In terms of hardware, a serial connection is made of two communication signals: transmit and receive. These are the two wires (blue and white) in the picture above, connected to Port 0(0) and Port 0(1). The orange and black wires are both ground (GND). One is connected to the GND of the power supply, while the other one is connected to the GND of a chip in the serial cable. The red wire is connected to the 5 volts of the power supply. This 5 volts is what powers the entire Prospeckz device, it is later converted to 3.3v by a power regulator. The last wire, orange/white, is a 3.3v output from the prospeckz device (converted from the 5v) which is used to power the serial cable chip. This hardware design wasn't changed, and is final.

example graphic

Basically, the application functions using an infinite loop after setting up the counter and starting the UART device. It listens to the input pin until it receives a character (byte). The byte is then checked and converted to its corresponding SNES button. For debugging purposes, the button is outputted to the UART along with the corresponding SNES button values. A radio packet is created and the converted data byte received from the UART is added to the packet. The packet is then sent over the radio to the other Prospeckz device. In order to visualise the flow of information, the onboard LED can be toggled each time a packet is sent.

The conversion from a command from the PC to a command which can be sent to the SNES is the most complicated process in this particular part. The classifier on the PC sends a data byte to the serial port of the PC. The byte is read by the UART of the Prospeckz. This byte is a representative of the button the classifier wants pressed on the SNES gamepad. There are 13 different options:

  • 'B' - Button B
  • 'Y' - Button Y
  • 'E' - Button Select
  • 'T' - Button Start
  • '8' - Button Up
  • '2' - Button Down
  • '4' - Button Left
  • '6' - Button Right
  • 'A' - Button A
  • 'X' - Button X
  • 'L' - Button L
  • 'R' - Button R
  • Other (0) - Reset

Two approaches were tested. The first approach tried was to convert the byte to a number which represents the clock period we need the output toggled. This could range from 0x01 to 0x0C. This number would be sent to the second Prospeckz via the radio, and the second Prospeck would count the clock periods until it reaches the number it received over the radio, then it would toggle the output for that clock period. This approach worked but had major inconvenients. It made it a lot more difficult to support multiple button press at one given time, and it also made it complicated to keep a button pressed for more than one clock period. Indeed, some of the buttons are usually pressed for longer amounts of time, like the movements buttons. With this implementation, we managed to show it was possible to send instructions from a classifier to the serial port, to the UART, over the radio, and then to the SNES. It worked fine for a proof of concept, but wasn't good enough as we wanted to be able to press multiple buttons, and move our Street Fligher character as with a normal gamepad.

The second approach was developed much later in the project, after the software on the other Prospeck went through multiple cycles. We'll discuss the second implementation of the convertion method later in this document.

First attempt for Prospeckz B: Pure polling

As discussed before, it was decided the first implementation of the second Prospeckz device would be polling for the data latch and clock signals, and toggle the output accordingly. In order to do that, an infinite loop polls for the positive data latch on port 0(0). When the data latch is detected, another loop starts and looks for the 16 data clock periods on port 0(1). The idea is to toggle the output when the counter is equal to the value received over the radio, which represents the clock period number which needs to be toggled.

After some testing and debugging, using the data analyzer, we were able to toggle the output, but the result wasn't very reliable. The data latch polling would sometimes miss a data latch, and we were not able to poll fast enough to count the 16 data clock periods. Tweaking and optimising made it a little bit better, but we were never able to toggle the output precisely and on a constant basis.

Toggling the output not only involved changing its electrical state to 0, but also holding its state to 0 for some time, then returning it to 1, and continue the normal operations. It is important the output is held at 0 for some time, otherwise the SNES cannot register it. In theory, it should be held at 0 for the entire time of the clock cycle: 12 micro seconds. It is complicated to wait for a precise amount of time on a micro-controller, generally the solution is to create a finite loop which uses approximatively 12us to execute.

Logic Analyzer Output

Second attempt for Prospeckz B: GPIO Interrupt & Polling

In order to solve this problem, the idea was to use GPIO interrupts instead of polling. This has the advantage of executing code only when an interrupt is triggered, and avoids missing an interrupt if they are managed correctly. We enabled the GPIO interrupt on the data latch. This means each time there is a positive edge on the data latch wire (yellow), the interrupt handler is called, and a specific piece of code is executed. Clearly, the advantage using this is to execute code when an event happens, not when the event is detected. This is important, because when polling, the data latch can be detected several micro seconds after it was actually triggered by the SNES, delaying the loop supposed to count the data clock period. The impact of this delay was inconsistant amounts of data clock period counted.

Using the GPIO interrupt on the data latch, and then polling for the clock periods, we were able to count more consistently the data clock periods. But again, we were not able to count the 16 of clock periods sent by the SNES. At first, it was thought this would not be too important, as long we could count the first 12 clocks periods which are those representative of a button on the SNES gamepad. The goal was to optimise the code so we could detect consistently at least the first 12 data clock periods after the data latch.

After some trial and error, we were able to precisely toggle the output on the first 9 data clock periods, which meant it wasn't possible to send 'X', 'L', and 'R' reliably to the SNES. All the other buttons would generally work. The problem of having mulitple buttons pressed at the same time, or having buttons pressed for a longer amount of time was still not resolved. Although, now we did not miss data latches, we did optimise the code in order to get longer button presses possible. We updated the way the classifier sends its instructions. One byte would mean press the corresponding button, and the same byte again would now mean release it. This replaced the previous algorithm which said when the classifier sends an instruction, then the button is pressed once. This made long button press possible, and quite reliable, although we still had three buttons not working (including one of the main 4 buttons).

The next problem we wanted to resolve was adding the capability of pressing on multiple buttons at once, which is important as a player wants to be able to move and do another action at the same time. All Street Figher combos techniques use multiple buttons, which is one of the important features of the game. The best way to resolve this problem was to keep track of what button is pressed on Prospeckz B, and keep sending their signal to the SNES until there is an order to release one or all of them. In order to do that, an array with 12 values was created, to track the status of the 12 SNES gamepad buttons. When the array[1] was set to 1, then it would mean clock period 1 was to be toggled. When array[6] was set to 1, it would mean clock period 6 was to be toggled, until array [6] is set to 0, which only happens when the classifier sends the second byte representing this button. This approach worked but had an incovenient, reading the array quickly became a major overhead, and it reduced the number of clock periods would could count reliably. It was then a trade-off between features: multiple buttons, long button presses, and the amount of buttons we could actually press. After trying to optimise the code in order to make it faster, we realise we were not going to be able to get all the features to work together with this approach.

Second attempt for Prospeckz B: GPIO Interrupt Only

After failing to poll for the two signals reliably, and use the GPIO interrupt on the data latch in order to have a consistent number of data clock periods, we decided to try add a interrupt on the clock period directly. This means there would be 17 interrupts each time, one for the data latch, and 16 for the data clock periods. This was supposed to allow us to see all the data clock periods, and finally be able to use all SNES gamepad buttons.

Using multiple interrupts is more complicated than using just one interrupt, as we have to find out in the interrupt handler where the GPIO interrupt is generated from. This involves polling the data latch and and the data clock wires to see which are the one who activated the interrupt. In general this is fine, but in our case, because the interrupts are happening in such a small amount of time, we cannot afford to spend anytime doing trivial software operations. After some testing, we realised we still could not detect all the data clock periods, which did not make much sense. We found out that in some cases, generating an interrupt would implie saving the state of multiple CPU registers, then executing the interrupt code, and then restoring the CPU registers to their original state. This is generally done in order to allow the developper to create complex interrupts routines which use the CPU registers. The problem is this operation takes an awful amount of time compared to what we had available. After some research, we found it is possible to avoid saving the state of the CPU when using simple interrupt routines (fast interrupt). The conditions was the routine must not use some of the CPU registers, and it must not call a method external to the routine (which could potentially change the CPU registers). When those conditions are met, the interrupt handler does not save the CPU state before executing the interrupt routine. This saved us precious micro seconds at each interrupt, and gave us more reliable results.

But now, toggling the output once the interrupt is triggered, reading the array to find out if the current button is pressed just took too much time. Once again, we were able to detect more data clock periods but unable to toggle the output in time for the SNES to register a button press. This solution was actually worse than the previous one as practically none of the buttons would be registered by the SNES, even if we were able to see more of the data clock periods.

example graphic

Third attempt for Prospeckz B: 8 bits shift register

A couple of weeks were left in order to try another approach. It could have been the hack described earlier in this document, hack a functional gamepad a emulate button presses. We decided to try replicate the hardware of a SNES gamepad in the PSOC using its hardware blocks features. The PSOC micro-controller is flexible and can be used as many different things. The UART is for example a hardware block which is easily set up. The counter is another hardware block. The advantage of a hardware block is it behaves just like any other hardware component, and does not have the inconvenient of being slown down by software. The inconvenient is it isn't as flexible as software. Hardware is designed to do one thing, and does it quickly and properly, nothing else. Luckily, the PSOC has a 8 bit shift register which works just like a regular 8 bit shift register.

With the help of James and Martin, we got the 8 bit shift register to work with the data clock periods. It would output a data byte set in software when it receives the 8 data clocks. What we had to do is repeat the process twice in order to use the 16 data clock periods. This technique involved changing how the data is sent to the SNES. Until now, we had a clock period number we wanted to toggle, but then we needed to output a data byte to the 8 bit shift register.

Change in the conversion from the classfier instruction to the SNES signal

It was necessary to reduce the overhead on the second Prospeckz and do as much as the processing as possible on the first one. We stopped converting the classifier button instruction into a clock period number, and started converting the classifier button instruction into two data byte. The SNES sends 16 data clock cycles, which represent two bytes. These clock cyles can be used by the shift register. The shift register output a data byte for each 8 clock cycles, which means the shift register can send 2 bytes (or 16 bits). It was then clear that we had to convert the classifier instructions into two bytes directly usable by the shift register. The conversion could still be done on the first prospeckz, hosting the UART, before the data is sent over the radio.

Button B is the first clock cycle sent by the SNES out of 16. The easiest way to represent the first bit out of 16 is to use an hexadecimal or binary notation: b1000 0000 0000 0000 or 0x8000. Using this, we would be able to represent all buttons using two bytes:

  • B - 0x8000
  • Y - 0x4000
  • E - 0x2000
  • T - 0x1000
  • 8 - 0x0800
  • 2 - 0x0400
  • 4 - 0x0200
  • 6 - 0x0100
  • A - 0x0080
  • X - 0x0040
  • L - 0x0020
  • R - 0x0010

For practical reasons, the two bytes were kept seperate: B would be 0x80 and 0x00. The classifier now sends a button command which is received by the UART from the serial port of the PC. The button command is converted into two data bytes representing what data period needs to be toggled on the output.

example graphic

We kept the method for switching on and off a button. Sending one button command would have the effect of pressing a button and maintaining it, until the same button command is sent again. For instance, for a simple punch, the classifier would have to send "XX", which presses the button, and then releases it. In order to crouch, and remain crouched, the classifier would have to send "2". When the classifier wants the player to stand, it would send "2" again. Until the second command, the remains crouched.

With this method, it is also extremely simple to support multiple buttons pressed at the same time. To do this, we use byte masks. The two byte masks are initialised to 0x00. When the second Prospeckz receives a packet over the radio with the two bytes, it updates the masks by doing "mask" = "mask" XOR "instruction". If the current mask is 0x0000 and the instruction is 0x0800, then the updated mask becomes 0x0800 (Jump). Then if a new instruction comes in, 0x0020, the new mask becomes 0x0820 (Jump + Long Punch). Then if the player wants to stop jumping, the classifier can simply send another 0x0800, the mask becomes 0x0020. If the classifier send the instruction 0x0020, the mask comes back to its original state: 0x0000. The mask can be reset automatically by sending the command "0".

Using the shift register to send two data bytes, having changed the command convertion method in order to simplify the task of the second prospeckz, we now have a system which can send one or many button commands from a PC to a SNES. All buttons work all the time using this implementation, it is very robust and efficient. A button can be pressed from just 0.12 micro seconds to an infinite amount of time. Multiple buttons can also be pressed at the same time, some can be pressed for a short amount of time while other are held, making it possible to have complex combinations.

This method of using shift registers is also a lot more energy efficient, as it does not poll constantly for signals. Although, in this particular case, energy consumption was not one of the design requirements, it could have been if we found out that we could not use the 5v from the SNES to power the Prospeckz device. In which case, it would have been necessary to power the device using a battery.

Hardware implementation

example graphic

A few weeks before the end of the project, we transfered the prototype from the breadboard on a stripped circuit board. We made sure the board could be plugged in directly the ports of the prospeckz. The board carries a connector to the SNES gamepad port. It has the capacitor and resistors to smooth noise out of the SNES communication and power wires, and it has the transistor plus two resistors which amplify the 3.3 volts output from the Prospeckz into 5 volts for the SNES. Summary: 2 connectors (SNES and Prospeckz) 5 resistors (3 input, 2 output), 1 capacitor (noise filter) and 1 transistor (amplification). We also added pins in order to test the hardware implementation during construction.

example graphic

The hardware implementation is pretty much independant from the software, which makes it possible to change the software implementation without having to change any hardware. Because the hardware has specific functions: output amplification (which is required and does not change) and noise filtering (technically not required but good to have), the PSOC program is what define the functionnalities. The advantage of this design is that a software upgrade is relatively easy to do.

Analysis and Evaluation

Because of the lack of experience in reproducing signals from a given protocol using a PSOC micro-controller, the implementation took a long time. During the project, many ideas arised, some good, and some bad. We made a decision at the beginning of the project which was clearly wrong now we have finished. A lot of time was wasted on polling signals and trying to optimise code in order to output the signals in the restricted time frame. Because this method worked partially, we tried to optimise it more to make it better, by using interrupts for example. This again was a bad choice. At the very end of the project we decided to try replicate the hardware we thought was used in the original SNES gamepads. Although this approach did make is struggle, it worked much better and much faster. If we did take that direction at the very beginning of the project, we could have finished the entire module in just a few weeks rather than a whole semester.

But because we wasted so much time on trying to optimise something which at the end did not work, we learned a lot from that mistake. We now know a lot more on how GPIO interrupts work, and when it is appropriate to use them or not. We also learnt how to measure timings and learned that in some applications, timing constraints can be extremely tight. The experience gained from the time spend trying and testing is valuable from a technical point of view, as it would now be possible to rebuild the whole project from scratch in just a few hours.

Compared to the other parts of the overall project, the issues we had with the hardware were consistent with the issues other people in the group were having. The classifier were also built over time, and required refactoring in order to improve them. Because this particular project is hardware based and reproducing a set of features, we can say with confidence we achieved exactly what we were expecting at the beginning of the project, with a state of the art solution. It is certainly easier to achieve a robust, stable, and clean solution in hardware with the technology constraints imposed, as opposed to software design, which can be improved, refactored, and maintained forever.

We are confident that if we did not get the shift register version to work in time, we could have found a work around in order to get all the features specified in time. But such a solution wouldn't have been as nice as the shift register, as it would have implied hacking a functionnal SNES gamepad in order to connect our input/output signal directly to the circuit.

In terms of communication with the PC, the UART solution is certainly one of the easiest and most straight forward to implement. UART modules are extremely common in electronics, and are used as a standard to communicate between two computing devices. They have the advantage of being relatively fast and easy to deploy. Their limitations did not affect us directly, such as speed, or scale. Other solutions would have been possible, we could have used other types of communications protocols or buses. I2C could have been one, but again would have required some extra hardware, which is not required by the UART. A wireless solution from the PC directly to the second Prospeckz would have been nice, but at the time we worked on the project, PC computers do not generally come with ZigBee radio, and ZigBee is the only standard supported by our selected hardware platform: Prospeckz. ZigBee is an interesting solution because of its cost, and its efficiency in terms of power consumption.

What was the most frustrating during this project was the lack of documentation on the PSOC side. The PSOC technology is absolutely incredible in terms of features and capabilities, but developping on it can be extremely discouraging. The software around it is heavy because it is clear the developers tried to include as many features as possible, and also tried to make the interfaces user friendly. The result is a developper can spend hours trying to solve a minor problem, which hasn't been documentated properly.

If we did have a bit more time at the end of the project, we could have tried to create a second interface for the SNES, and try to have to human players using their own set of Orients. This would have certainly been interesting, and fair compared to having a player physically fighting the AI, or another player using a gamepad.

example graphic

Another source of frustration is the lack of simulation platform. In order to test an application, it seems the software has to be compiled (which gives the developper a chance to find errors), then the device has to be programmed which can take up to a minute, and tested using a LED, the radio, and the input/output ports. When debugging software, a developer can sometimes need to compile, program, and test numerous times by changing minimal amounts of code. This takes a very long time on this platform, which is not cost effective at all.

The Prospeckz and PSOC platform were a requirement, so we did not investigate other solutions. On one hand, they were good because of their extensive abilities, but on another hand they were difficult to use because of the lack of proper documentation.

In terms of future development, the second Prospeckz's software could be improved by getting the shift register to work on interrupts, which we failed to do. The impact of this is an error which happens once in a while at initialisation of the Prospeckz. When this happens, all is required is restart the Prospeckz. This is the perfect example of a minor problem which was not resolved because of the lack of documentation. Getting the shift register to work on interrupts should be a trivial operation. One this is fixed, the Prospeckz and the circuitry could be packaged into a more professional way.

The Prospeckz A could be improved by also packaging it better, and changing the PC interface to something more modern than the serial bus, such as USB. If that is done, then the product could be sold to the public, with a set of Orient device if these were to become more affordable. In this was the case, the possibilities of such technology are limitless. The Prospeckz devices could be attached to any gaming console, and software classifiers could be configured for all sorts of games.

Conclusion

Overall, the project was a success. We did prove it was possible to use both Orient devices and the Prospeckz platform in one system. We demonstrated it was possible to classify in real time fight gestures, and send instructions which would be registered by a gaming console. The system involves AI techniques as well as basic notions of embedded software. The demonstration was also a success, as the public was clearly enjoying seeing a game from their childhood, adapted to work with cutting edge technology. We were also able to work as a small team, by each working on different parts of the system, and then integrating them together successfuly.

Appencides

Videos

Final UART Sample Code

//----------------------------------------------------------------------------
// C main line
//----------------------------------------------------------------------------
#include         // part specific constants and macros
#include "PSoCAPI.h"    // PSoC API definitions for all User Modules
#include "PSoCGPIOInt.h"
#include "stdlib.h"
#include "prospeckz.h"

void conv(char data);
BYTE b1 = 0x00;
BYTE b2 = 0x00;
int i = 0;
char data;

//This is the main function that would be executed after boot.asm is executed.
void main() {
    Packet pkt;
    
    LED_RED_Off;
    LED_GREEN_Off;
    LED_BLUE_Off;
    
    RADIO_Init(0);                                //Init radio reciever
    RADIO_SetTxStrength(32);                     //Set the transmission strength
    RADIO_RecieverOn;
    
    Counter8_WritePeriod(155);                     // Set up baud rate generator
    Counter8_WriteCompareValue(77);
    Counter8_Start();                             // Turn on baud rate generator
    UART_Start(UART_PARITY_NONE);                 // Enable UART
    UART_CPutString("\r\nWelcome!\r\n");

    while(1) {
        
        // get data from serial port
        data = UART_cGetChar();
        
        // convert command into snes gamepad clock number
        conv(data);
        
        // display data
        UART_CPutString("Button ");
        UART_PutChar(data);
        UART_CPutString(": 0x");
        UART_PutSHexByte(b1);
        UART_CPutString(" - 0x");
        UART_PutSHexByte(b2);
        UART_CPutString("\r\n");
        
        // send data over wireless
        pkt.dataLength = 2;
        pkt.data[0] = b1;
        pkt.data[1] = b2;
        RADIO_Send(&pkt);
        
        // set LED color
        if (i) {
            i = 0;
            LED_RED_On;
        } else {
            i = 1;
            LED_RED_Off;
        }
    }
}

void conv(char data) {
	// x,y,a,b,select,start,top,left,bottom,right,L,R
	switch(data) {
		case 'B':
			b1 = 0x80; 
			b2 = 0x00;
			break;
		case 'Y':
			b1 = 0x40; 
			b2 = 0x00;
			break;
		case 'E': // select
			b1 = 0x20; 
			b2 = 0x00;
			break;
		case 'T': // start
			b1 = 0x10; 
			b2 = 0x00;
			break;
		case '8': // up
			b1 = 0x08; 
			b2 = 0x00;
			break;
		case '2': // down
			b1 = 0x04; 
			b2 = 0x00;
			break;
		case '4': // left
			b1 = 0x02; 
			b2 = 0x00;
			break;
		case '6': // right
			b1 = 0x01; 
			b2 = 0x00;
			break;
		case 'A':
			b1 = 0x00; 
			b2 = 0x80;
			break;
		case 'X':
			b1 = 0x00; 
			b2 = 0x40;
			break;
		case 'L':
			b1 = 0x00; 
			b2 = 0x20;		case 'B':
			b1 = 0x80; 
			b2 = 0x00;
			break;
		case 'Y':
			b1 = 0x40; 
			b2 = 0x00;
			break;
		case 'E': // select
			b1 = 0x20; 
			b2 = 0x00;
			break;
		case 'T': // start
			b1 = 0x10; 
			b2 = 0x00;
			break;
		case '8': // up
			b1 = 0x08; 
			b2 = 0x00;
			break;
		case '2': // down
			b1 = 0x04; 
			b2 = 0x00;
			break;
		case '4': // left
			b1 = 0x02; 
			b2 = 0x00;
			break;
		case '6': // right
			b1 = 0x01; 
			b2 = 0x00;
			break;
		case 'A':
			b1 = 0x00; 
			b2 = 0x80;
			break;
		case 'X':
			b1 = 0x00; 
			b2 = 0x40;
			break;
		case 'L':
			b1 = 0x00; 
			b2 = 0x20;
			break;
		case 'R':
			b1 = 0x00; 
			b2 = 0x10;
			break;
		case 'R':
			b1 = 0x00; 
			b2 = 0x10;
			break;
		default:
			b1 = 0x00; 
			b2 = 0x00;
			break;
	}
}

Final Gamepad Emulation Sample Code

//----------------------------------------------------------------------------
// C main line
//----------------------------------------------------------------------------
#include         // part specific constants and macros
#include "PSoCAPI.h"    // PSoC API definitions for all User Modules
#include "PSoCGPIOInt.h"
#include "stdlib.h"
#include "prospeckz.h"

Packet pkt;
BYTE cmd1 = 0x00;
BYTE cmd2 = 0x00;
int i = 0;

void main() {
    
    // init radio reciever
    RADIO_Init(0);    
    RADIO_SetTxStrength(32);
    RADIO_RecieverOn;

    // switch off LED
    LED_RED_Off;
    LED_BLUE_Off;
    LED_GREEN_Off;
    
    // initialise digital inverter
    DigInv_1_Start();
    
    // initialise shift register
    ShiftReg_Start(ShiftReg_SPIS_MODE_2);
    ShiftReg_EnableSS();
    
    while (1) {

        // listen the radio
        if (RADIO_Recieve(&pkt) == SUCCESS) {
            if (pkt.dataLength == 2) {
                
                // set command array
                cmd1 ^= (BYTE)pkt.data[0];
                cmd2 ^= (BYTE)pkt.data[1];
                
                // resend packet
                RADIO_Send(&pkt);                
                
                if ((char)pkt.data[0] == 0x00 && (char)pkt.data[1] == 0x00) {
                    cmd1 = 0x00;
                    cmd2 = 0x00;
                }
            }
        }
    
        // wait for the snes datalatch
        while(DataLatch_Data_ADDR & ~DataLatch_MASK);
    
        // send data to shift register
        while (!(ShiftReg_bReadStatus() & ShiftReg_SPIS_TX_BUFFER_EMPTY));
        ShiftReg_SetupTxData(cmd2);
        while (!(ShiftReg_bReadStatus() & ShiftReg_SPIS_RX_BUFFER_FULL));
        ShiftReg_bReadRxData();
        while (!(ShiftReg_bReadStatus() & ShiftReg_SPIS_TX_BUFFER_EMPTY));
        ShiftReg_SetupTxData(cmd1);
        while (!(ShiftReg_bReadStatus() & ShiftReg_SPIS_RX_BUFFER_FULL));
        ShiftReg_bReadRxData();
    }
}

Files

References