The Alya Project

History

I did this project in 2000. In the meantime (now it's 2004), I'm not using the Alya box anymore. As memory cards grew bigger and bigger, I could store an increasing number of images without changing the card. In 2003, I had a 256 MB CF card - enough for 400 pictures, more than I'd need for one holiday. But recently, I bought a 6 megapixel DSLR with raw image file sizes of 7 MB. This is only 70 images on a 512 MB card! So is Alya back in business? No, my new computer does not run Windows 98 anymore and W2K does not allow you to access the printer port directly. I could add a USB interface... but - what's the point. In 2000, I had Alya running before you could buy such a thing. Now, with all the image tanks, X-drives... available, there is no reason to build just another variation of those.

On the old Nikoma web site, this project was responsible for almost all e-mails I got concerning any of my projects. Of course, digital cameras are en vogue and interfacing to a flash card or IDE harddisk is both interesting and useful. Finally, as many people ask for it, here is the PIC assembler source code and the Altera FPGA code. Still more, here's the schematic:page 1 page 2 page 3

April 2004: I just bought an X-Drive for my DSLR images. 20GB harddisk, five card reader slots, USB 2.0 interface... easy. But:

In the year 2000...

 What are you doing with your digital camera during a long holiday? The pictures are stored in a memory card which has a pretty limited capacity. The pictures of my Nikon Coolpix 950 are 400-500k (JPEG compressed), giving a 32MB compact flash (CF) card a capacity of about 80 pictures. When the card is full, you can either use another card (which is still pretty expensive) or use your laptop computer to copy the pictures to the harddisk. But, apart from the fact that I don't have a laptop, I wouldn't want to need one during my holidays.
I soon had the idea to build a box to copy the contents of the memory card to another medium. Among all media (MO, CD-R, tape, floppy disk, ZIP...) I found that a 2,5" laptop harddisk was best suited for this task. It's small, cheap (I got a 1,8G drive for DM 200), rugged and easy to interface to. The Alya board mainly consists of a PIC16C64 microcontroller and an Altera 10K10 FPGA to handle the data transfer.
 

 How it works

 I wanted to keep the design as simple as possible, for that reason, no complex data handling by the microcontroller is involved. The harddisk space is divided into blocks of 32 MB. The Alya board simply copies the entire compact flash card to a user-selected 32MB slot. There is no interpreting of directory structures, files or images. It's just a simple, 1:1 copy of the memory card contents. The user interface is simple: Initially, the 32MB slots are all marked empty and after power-up, the microcontoller finds the first free block and displays the block number in the LC display. You can change the block number manually with +- keys, which makes it possible to overwrite used slots (dangerous!). When you press + or -, the controller also checks if the slot is empty and displays a "full" or "empty" message. I never needed this feature though - the standard procedure is to switch the box on, wait for initialisation and press the "copy" button. It then copies the CF card contents to the harddisk (taking about three minutes) and switches off the harddisk. The next time the controller simply selects the next slot.
To read the data from the harddisk, the Alya box can be connected to a PC via a parallel port (EPP) interface.

 Electronics

As I have all the development tools for the PIC devices, I chose a PIC16F877 for this board. Later I changed to a PIC16C64, but for developing, the flash device is more convenient. Initially, the main problem was to connect all the devices to the microcontroller: 40-pin IDE, 50-pin CF, 25-pin parallel port, 14-pin LCD module, four keys... just too much for a small 40-pin device! I tried to fit everything together with some latches, registers, bus drivers but finally gave up. Instead, I decided to use an Altera FPGA and just connect all pins to the FPGA. This is a good example how FPGAs simplify a design - instead of having to understand all the details of the devices involved, just connect them directly and implement the details later. This lead to a 208 pin device and, as I intended to use it only as a wiring, register and multiplexer box, the smallest FPGA available, a 10K10A, seemed to be sufficient.

When I had the board soldered together, I decided to start with the EPP interface and use this as development port for the FPGA programming. Initially, the FPGA contents simply consisted of a number of registers that could be written using EPP write accesses. The registers are connected to the FPGA pins and, for read access, all input pins are connected to a multiplexer and can be read via the EPP port. Using this set-up, I implemented and tested the ATA, CF and LCD interfaces.
As there were still lots of resources left in the FPGA, I implemented a hardware sector read/write mechanism that reads data from the ATA/CF port into an internal ram buffer (512 byte=1 sector) or writes it in the other direction. Using this, the CF to HD copying is mainly done in hardware - only the command setup is done by the microcontroller. Later, I also added hardware support for reading the sector buffer via the EPP port - making the read operations pretty fast!
When everything worked via PC control, I simply duplicated the parallel control port and connected the second one to the PIC. All I had to do was to port some of the routines (device identification, copying, reading sectors) into PIC assembler code. Using this scheme, I had almost no debugging phase with the PIC (which could have been quite annoying).

 The ATA interface

 In former times, harddisks needed separate controllers (remember MFM or RLL?) to control the head stepper motor, head select, read/write data etc. As these controllers got more and more compact, the hard disk manufacturers began to integrate the controller into the drive (IDE - integrated drive electronics). The interface to an IDE drive is more or less the AT (ISA) bus - just like if you connected a flat cable to the AT edge connector on the mainboard. It is defined in the ATA (AT Attachment) spec. Simple ISA-Bus IDE interface cards consist just of an address decoder and a bus driver for the data. Nowadays, UDMA66 interfaces are built directly into the mainboard chipset.
All information about the ATA interface can be found in the ATA spec. You can't download the spec itself, but there are countless drafts at www.t13.org that are as well. I'd recommend you download the ATA-2 draft, it has only 90 pages and contains everything you need for programming. The newer ones (ATA-4) are much thicker and have hundreds of pages you have to sift through for relevant information. The ATA command description is actually much clearer in the CF spec (see below).
The minimal ATA interface is pretty simple (direction as seen from the hard disk):
 

D7-0 or D15-0	bidirectional	Data bus
A2-0	input	Address bus
/RESET	input	Device reset
/CS0	input	chip select
/DIOW	input	IO write
/DIOR	input	IO read

All ATA operations are performed by reading or writing eight registers. These are mainly the command, cylinder, head, drive... and data register. For example, to set the drive to sleep mode (power down spindle motor), write the drive number (master or slave) to the C/D/H register (6) and the SLEEP opcode (99) to register 7.
As data transfers, only entire sectors (=512 bytes) can be read or written. A write operation is performed by writing the drive number, sector number and WRITE SECTOR(S) opcode to the command... registers and writing 256 16 bit words to the data register. While the control registers are only 8 bit wide, the data register is 16 bit wide. There is a SET FEATURE command to switch to 8 bit data transfers, but with modern drives, this doesn's seem to work (no wonder, as the manufacturers implement high performance UDMAxxx modes). CF cards do support this, see below. You have to check the status flags (busy, error) periodically, eg after a read command to wait until the data is ready.
The Identify Drive command (EC) is especially useful. You can use this during your experiments as this is the only way of getting meaningful data from the drive without having written it before. It's also a nice way to make sure everything works - after power-up, the microcontroller of the Alya board first issues an identify drive command and displays the model number of harddisk and CF card in the display. You can also get the number of sectors from the drive, which might be interesting for programming. Nowadays, harddisk sectors are addressed using the LBA number, which is simply the sector number starting from 0 to the maximum sector number. This linear addressing scheme is much easier than the older cylinder/head/sector (CHS) addressing.

 The CF interface

 Compact Flash (CF) cards behave more or less like harddisks. They have the same interface, are organized in sectors and the contents are the same (partition table, FAT16 partition...) as common harddisks in a PC with a M$ operating system.
The compact flash interface (download spec from www.compactflash.org) has two main operating modes: The first is the PC-CARD (PCMCIA) compatible mode. There are passive adaptors letting you plug a CF card into a laptop PC-CARD slot. The second in the "true IDE mode" in which the CF card behaves exactly like an ATA device. This makes it possible to connect a CF card directly to your computers' IDE interface (see picture). This is not very practical, though, as neither the interface nor the operating system are made for hot-plugging the memory card.

Anyway, by selecting the true IDE mode (by tying /ATA SEL low), you simply use the same control scheme like for an ATA drive. There are two main differences:
1. CF cards do support 8 bit data transfers. This saves eight bidirectional data pins and makes the interface much more compact (14 pins, not counting reset), which is important for small processors like the PIC devices.
2. CF cards work with 3,3 and 5V. In 5V mode, the levels are not TTL compatible (no problem for PICs though). Read the spec! I evaded this problem by operating the CF card with 3,3V (no problem as the FPGA is also a 3,3V device).

 Mass storage for embedded systems

 If you need to store a large amount of data, compact flash cards or harddisks are an option. The capacity ranges from 16M (CF, DM100) to currently 75G (harddisk, DM 1350).
Most people I spoke to thought the harddisk interface would be pretty complex and interfacing to it some kind of wizardry. It isn't. Actually it's that simple that you'll soon be asking yourself what gimmicks you could build with it!
When you decide to use the ATA interface, you can choose between CF cards and harddisks. CF cards are more compact, need less power when running and have an (optional) 8 bit data port. They are limited in size, though. Harddisks have a good capacity/price ratio and the 2,5" ones (more expensive) are also quiet compact. 3,5" drives need 5 and 12V, while 2,5" drives run with 5V (usually max. 1A during startup and 500mA operating). The nice thing is that you can use (almost) the same interface for accessing a 35GB harddisk as for an 8 MB CF card. You'll need some kind of power management, as both CF and HD need too much power when active. You could use a buffer memory and periodically switch the drive on, copy the contents of the buffer to the disk and switch it off again. A harddisk will need 5-10s before it's ready, my 32MB CF card is ready after about 3s.
The interface itself is simple - you either connect the signals directly to your system's bus or if there isn't one (like when using a PIC), connect them to the port pins. Generating read and write cycles by pulling /cs and /rd or /wr low is easy. You can even use all but the /cs lines for other functions - like for interfacing to an LCD module or to another drive.
In a simple data logger application, the WRITE SECTOR(S) command might be the only ATA command you'll need. A serious problem when reading or writing data is that you always have 512 bytes - especially with the almost ram-less PIC devices. Perhaps a serial EEPROM could be used as low power (and low pin count) buffer memory.

 Getting around the FAT16 file system

 An unformatted drive is basically a large array of sectors you can read or write. To store files and directory structures on a harddisk, you need a file system. On DOS based machines, FAT12 is used for floppy disks (or for small (8MB) CF cards), FAT16 for partitions <2G and FAT32 for larger (several TB) drives. A drive can consist of several (different) partitions. A very good introduction in FAT file systems was printed (in German) in c't 6/2000.
As I wrote above, Alya does not interpret the structure of the CF card, but simply copies the sectors 0..62975 of the CF card to sectors n*65536..n*65536+62975 of the harddisk (n=slot number). Unfortunately, I have to decode the file system because the camera writes its pictures using FAT16 format. I'm decoding the FAT16 structures when reading the harddisk via the parallel port.
For other applications (like a simple data logger), you could get around this. If you only need one linear area for your data, you could do the following:

1. Install the drive (harddisk or CF card) in your computer
2. Format the drive (format f:)
3. Immediately after this, write a large file (dummy.txt) occupying almost all of the free disk space

 On a non-fragmented drive, your file will occupy sectors a...b in an increasing order. Find out (using a disk editor) where sector a is located and you can overwrite this file with your microcontroller system, simply by writing your data to sectors a...b. After re-installing the drive in your computer, you can read the dummy file (dummy.txt) and get the data. Fortunately, there is no checksum of the data! You could also write a specific signature to the file and let the microcontroller find the first sector of it.

 Finally...

 I have to admit that most of the explanations above are very superficial. It's just not possible to reduce a three-week-project into a few lines of text. But I hope it's still a starting point if you intend to use CF cards or harddisks in your design.
After almost four weeks holiday in south france, the Alya box was filled with 700 pictures from the digital camera. Extracting them worked at the first go and none of the pictures was corrupted. As the parallel port transfer is much faster than the camera's serial connection, I also use it for downloading images to the computer.
If I had too much time (...), I'd redesign the electronics to fit in a much more compact case. There is no reason why it should be much bigger than the harddisk itself. The prototype wasn't planned to fit together mechanically (and it didn't, consequently) because I didn't have the time for it. The power supply I got from the junk somewhere was huge and the LC display is not really necessary. A single button and a green LED would do it. Next time...