CMD Native Partitions

Read the latest info on the New Partition Types to be included in Maurice's HD-DOS+ update!

Note:
Most of the data contained herein was taken directly from CMD's own documentation. Some of it is based upon observations made while working with my own equipment. Furthermore, it is possible that I have misinterpreted some of the data collected while preparing this document. I encourage you to cross-check these facts with others who may know more than myself. Todd Elliott has also made significant contributions and corrections to this document. At any rate, the foremost authority on all things CMD is Maurice Randall.

If you discover any errors or omissions, please do contact me ASAP so that I may make the appropriate corrections.

How to interpret values:

• Values in parentheses represent actual bytes contained within the feild
• Hex notation is preceded with "$", otherwise decimal is assumed
• ASCII means PETSCII, thus "A" means CHR$(65) as in CBM BASIC

Definitions:

• PHYSICAL = true geometric location of data on the physical media
• LOGICAL = translated location of the data as seen by the filesystem
• FDC (Floppy Disk Controller) = the part of the DOS responsible for stepping the read/write head, seeking tracks and reading/writing sectors
• IP (Interface Processor) = the part of the DOS responsible for parsing DOS commands and managing the filesystem
• IEC = The proprietary peer-to-peer network protocol used by Commodore's "Serial Bus" devices such as the 1541 and C-64
• DIR = Directory
• Part = Partition
• NATV = CMD Native Mode Part

Overview of CMD patitions

CMD devices store part data in an MBR-like area called the "System Partition". Each part stores its physical media location, part type (Native, Emulation or Foreign) and size there. In a CMD RAMLink, the System Partition is stored in the top of available memory (the system map) and its actual address is dependant upon the amount of memory installed.

Todd Elliott

CMD FD drives format the media into 80 physical tracks per side, numbered 0-79. The track numbers are internally translated by CMD DOS to 1-80. The FD also formats an additional 81'st track to store the System Partition. The Commodore DOS passes to the FD-2000 the track/sector of a disk block it wants to read. The CMD DOS in the FD disk drive has to translate it into a real physical disk block following the disk's geometry to arrive at the correct location and fetch it.

As to CMD HD I do not know where it stores the System Partition. All CMD devices will report a list of all parts contained on the media when issued the following command:

Partition Directory Command

LOAD "$=P",device

It's worth noting the difference between PHYSICAL formatting and LOGICAL LAYOUT. Because disk media comes in many sizes, the physical formatting doesn't directly correspond to the logical layout. A 1581 drive will format a 800k DS/DD floppy into 160 tracks (80 per side) with 10 sectors of 512 bytes each. That is the physical formatting and the FDC (Floppy Disk Controller) works with the disk media on this level.

When the IEC controller (ie C-64) requests a file or block, the IP (Interface Processor) will parse the request and fetch logical blocks via a translation table. The translation table serves to associate a 256 byte logical block with either the high or low segment of a physical sector on the media (ie side 0 or side 1). Please understand that the 2 sided geometry of the 1581 is fairly simple when compared to the complexities of a hard disk which may have several heads and where tracks are divided into cylinders.

All filesystem calls to the drive are processed by the IP. The computer has no need to understand the physical layout of the media. The work of finding files, block chains and seeking tracks and sectors is all done within the DOS. The IP accepts requests for files or logical blocks, does the translation to physical sectors and passes the request to the FDC. The FDC reads the requested physical sectors, then returns the results to the IP. The IP then passes the data back to the computer.

The filesystem is built on top of the LOGICAL LAYOUT. This gives CMD DOS an opportunity to do magic! CMD DOS can partition the disk by associating groups of physical tracks with parts as defined in the System Partition. Thus if part #2 begins on PHYSICAL track #40, a Block-Read to track #1 of part #2 will actually return data from PHYSICAL track #40. Again you should note that the computer neither knows nor cares the true physical location of data on the media.

CMD DOS divides the PHYSICAL media into a number of LOGICAL parts. The CMD HD supports up to 254 parts. The CMD FD and RAMLink are limited to 31 parts, although I don't know whether this restriction is enforced by the DOS or imposed by limited media size. Part types are defined according to either the drive they emulate or to the limitations of the filesystem (whichever comes first). When a 1541 Emulation part is created on a CMD device, its LOGICAL LAYOUT will conform directly to that of a standard 1541 disk.

NATV Part size is user defined at creation time and is measured in 256 block increments; Thus the smallest possible part is 65536 bytes (one track). The first logical track within the part will be identified as Track 1 in keeping with the CBM standard. The largest possible NATV is 255 tracks and this limitation is imposed by the 8 bit width of track pointers of the CBM filesystem. Therefore the largest possible NATV contains 255 tracks X 256 Blocks X 256 bytes for a total of 16,711,680 bytes.

Filesystem is CBM 1541 compatible with extensions to support sub-directories and time/date stamping for devices equipped with RTC. To remain compatible with the CBM filesystem, a track value of 0 in a block link (bytes 0,1 in any block of a file chain) indicates that this is the last block in the chain. In such cases, byte 1 of that block points the last valid byte of the file within this block. Much of this should be very familiar if you are acquainted with the CBM 1541 filesystem.

• Track Range: 1-255 (8 bits, Track 0 is illegal)
• Block Range: 0-255 (8 bits), 256 Blocks per track

Track 1 is contains the VOLUME_BAM and ROOT_DIR. Block 0 is not defined in the CMD documentation and appears to contain garbage. On my RAMLink it is filled with $55.

Todd Elliott

Track 1, Block 0 is reserved for the CBM boot sector. This holds true even for CMD Native parts. Insert special bootsector code there at Track 1, Block 0, say at the CMD HD located at unit 10, issuing the "BOOT ON U10" command from 128's direct mode will then initiate the boot sequence.

Block 1 chains to Block 34 which contains the ROOT_DIR_ENTRIES.

Track 1 Block 1: ROOT_DIR_HEADER Block

Track 1 Block 2: VOLUME_BAM_HEADER Block, BAM for tracks 1-7

Track 1 Blocks 3-33: VOLUME_BAM for tracks 8-255

The VOLUME_BAM is continued in Track 1 Block 3 which contains the BAM for tracks 8-15. Each successive block of Track 1 contains the BAM IMAGES for subsequent sets of 8 tracks each. Thus on a part with 255 tracks, Track 1 Blocks 3-33 are all reserved for the BAM.

Todd Elliott

The BAM entries located at Track 1, Blocks 3-33 should be constant, regardless of the CMD part size. In the case of the FD-2000, a lot of blocks in the BAM would be unused, but they should be reserved and not used by CMD DOS in writing actual disk data. I guess this is due to compatibility as CMD FD DOS is based on CMD HD DOS and not a lot of code has changed in this respect.

The 256 bytes in each BAM block are divided into IMAGES 32 bytes in length where each IMAGE represents the BAM for one track. Thus, each BAM block contains IMAGES for eight tracks in the part (256/32=8). The bits in each IMAGE represent the availability status of the 256 Blocks contained within the track which that IMAGE represents on a 1/1 basis. The LSb of each byte within the IMAGE represents the highest numbered block covered by that byte. Thus, Track 1/Block 2/Byte 32/Bit 0 represents the availability status of Track 1/Block 7. Some definitions may help to clarify this:

BAM Block

8 IMAGES representing 8 tracks

IMAGE

256 bits (32 bytes) representing 1 track

Byte

8 bits (7,6,5,4,3,2,1,0) representing 8 blocks of a track IMAGE

LSb

bit 0 of a byte, least significant numerically but represents highest numbered block covered by that byte

MSb

bit 7 of a byte, most significant numerically but represents lowest numbered block covered by that byte

0

Represented block is ALLOCATED (unavailable)

1

Represented block is UNALLOCATED (free for use)

BAM IMAGE Layout, 32 bytes per IMAGE

Track 1 Block 34: The Root Directory File Entries

Each block of the DIR_ENTRIES Chain contains up to eight FILE_ENTRIES describing the properties of each file in the DIR. Each of the FILE_ENTRIES is thirty bytes long and preceeded by a pair of 0 (null) bytes. The exception is the first FILE_ENTRY in each block of the DIR_ENTRIES chain, which is preceeded by the two chain bytes. Blocks of the DIR_ENTRIES Chain are usually allocated with an interleave of 1/1.

File Layouts

If FILE_TYPE is set to $80 the Type is "DEL" and it will appear in the DIR. Such files may be sequential in nature or may use a custom structure. If FILE_TYPE is set to $00 the file will not appear in the DIR and no blocks will be allocated to it. Such a file is said to be "deleted" although any data blocks within it's chain which have not yet been overwritten can still be recovered.

FILE_TYPE 4 (REL) Layout

FILE_TYPE 6 (DIR) - CMD Native Mode Sub-Directory

The DIR_ENTRIES block layout for Sub-Directories is the same as that of ROOT_DIR_ENTRIES.

Future Partition Types

Maurice is cooking up a new "Extended Native" partition type which will allow NATV parts as large as 4 GigaBytes. I'll try to keep things up to date as new details are announced. For now, here's some quoted info from Maurice's posts on the Homestead List.

The previously mentioned 'extended native' partitions are still being developed. These can be as large as 4 gigs each. The file system is under development and is coming along quite nicely. The design no longer uses a BAM/FAT design as previously mentioned. A BAM is still being used, but the FAT design has been dropped in favor of a 'CAT' for Chain Allocation Table. Each directory and file will have its own CAT. The DOS can look at the CAT block and immediately determine where any byte of the file is located. It can quickly tell if the file is fragmented or contained entirely within consecutive sectors. As few as 8 bytes can describe the entire location of a file within the partition if the file isn't fragmented. If the file is fragmented, then 8 bytes will describe each broken up chunk of the file. If the file is scattered about in 4 pieces, that means that only 32 bytes will describe where any portion of the file is located. By reading these bytes, the DOS can quickly go to any byte within the file. The CAT design also saves a great deal of space and is much faster than the FAT design. The FAT design would require 3 bytes to describe each block of the file. The DOS would still have to chain through the FAT to determine where a particular block of the file is located. With a FAT design, the DOS would have to read through about 2600 blocks of a FAT table to determine where the end of a 650 meg file is located. Those 2600 blocks are 512 byte blocks. That means that 1300K of data would have to be read to find the end of the file. Of course it wouldn't be that bad on smaller files, but you get the point. With my CAT design, the DOS might only have to read one block of the CAT table for the file to find where the end of the file is or any point in between. So, the performance is almost the same no matter how big or small a file is. 4 bytes describe the starting block and 4 bytes describe the ending block, or they describe the starting and ending of a chunk of a broken up file. Chances are, most files won't be broken up in very many pieces. Even on a badly fragmented partition, only 1 CAT block would have to be read and analyzed to be able to index into any portion of a file that is broken up into as many as 63 chunks. Additional CAT blocks will be used for describing files with more than 63 broken chunks.

I'm working on the formatting routines and other aspects so I can get one of these new partitions installed for testing. I've got to also add some new extended block addressing commands. The old B-R, B-W, U1, and U2 commands won't work with an extended native partition. There will be new commands to take their place. They aren't finished yet, but will most likely be called E1 and E2. They will use 3 bytes for addressing and will always read or write a 512 byte block and can be used with any partition type. There will also be a file pointer (F-P??) command similar to the current buffer pointer (B-P) command to use for pointing to any byte within a file. The B-P command can only index into the current block that is loaded into the channel buffer. But the F-P command can point to any single byte within an entire file. To point to the last byte of the file, the programmer wouldn't even have to know the size of the file. The F-P command would be used to point to the maximum size that a file can be and the DOS would automatically adjust the pointer to the byte following the last byte. This allows for files to easily have data added to them.

It makes no difference how big or small an extended native partition is, the BAM routines are fast because of the design. There are the normal BAM blocks that contain a bit for each block to indicate whether or not the block is in use. And there is also a BAM Block Allocation Map (BBAM). The BBAM indicates which BAM blocks might still show some available blocks. Without this design, the DOS would be very, very slow on a big partition. A 4gig partition has 2048 BAM blocks. Imagine the DOS reading through those blocks to find a free block in the partition. With this design, DOS+ can quickly find a free block no matter where it is. The BBAM is only 256 bytes in size and contains 2048 bits, one for each BAM block. It merely has to find a free bit in the BBAM and then go to the indicated BAM block to find a free block. During normal use, both the BBAM and a BAM containing free blocks are already loaded into memory. It's not necessary to load in a different BAM block until 2megs of data have been written to the partition since each BAM block represents 2megs of space.

For the first time, a Commodore disk drive allows a direct access buffer to be more than 256 bytes in size. When opening a channel to the HD, if the current partition is an extended native partition, the channel will be assigned to a 512 byte buffer instead of the usual 256 byte buffer. Fortunately, the CMD-HD is not hurting when it comes to available buffer space.

Also, for the first time ever, a Commodore disk drive now has the ability to not only move forward to the next block in a chain, but it can also move backwards through the chain. In all other drives as well as all other partition types in the HD, the only way an application can move backwards through a file chain is to first read the blocks forward and keep a list of the blocks that make up the chain. Then by referring to that list, the application can call up a block from the chain. The disadvantage is obvious, the file must first be read from start to finish. This is not necessary in an extended native partition. The application can request any block number offset from the start of the chain and DOS+ will load it into a buffer. Any block in a chain can be found and loaded as fast as the first block in the chain. Note: Files and directories are made up of chains of blocks.

Another new thing added to DOS+ in an extended native partition is a new concept with directories. Instead of having a root directory, the partition now has a Master directory. The Master directory is normally hidden under normal disk access methods. Inside the Master directory is one or more Root directories. When the partition is accessed, the default root directory is the one that is made visible. Or the user can switch to a different root directory and work from that one. This is similar to switching partitions, except that you stay in the same partition. A Master directory can also contain files in addition to the root directories. But these files are normally hidden from the user and would mostly be for operating system use.

The advantage to multiple root directories is that different operating systems can each have their own root directory. Also, from the default root directory, the user could launch a small boot program that will boot up an operating system. All of the OS files could be contained in a separate root directory completely hidden from normal use. The OS could also switch the user to a different working root directory if that's how it's configured.

DOS+ also has support for file links. A file link is nothing more than a directory entry. There are primary links and secondary links. A primary link is what we have always been used to. This is a normal directory entry that points to a file. But now we can have a secondary link that can point to the same file data. The user could create secondary links in any root partition or subdirectory and be able to access the same file data wherever that link is placed even from other root directories. Users of other computer platforms have used this method, maybe without even knowing it. In Windows, the icons on the desktop are links to files. Some operating systems might call them 'aliases' or 'shortcuts'. We now have that ability built right into the drive's DOS, but only in the extended native partitions.

You can open a secondary link just as if you were opening the actual file. You can read and write to the file. But if you delete a secondary link, the DOS won't delete the actual file, it will simply remove the secondary link from the directory. On the other hand, if you delete a primary link (the actual file) and if there's secondary links attached to this file, all the secondary links will also disappear. There is actually two ways to do this. One method will report an error if there are any secondary links for the file and the other method will delete every one of them without reporting an error. The first method works the same as if the file were locked and is the default method of deleting a file. This is safe because the file might actually be needed by another operating system or a program that you forgot about.

DOS+ does not support REL files in an extended native partition. That's OK because all current software written for relative files understands the filesize limit of a relative file on all of the existing disk drives and would be of no advantage in these big partitions. Instead, DOS+ has native support for the GEOS type VLIR file structure, only with a bonus. A VLIR file in an extended native partition can have unlimited records. The only limit is the amount of space on the partition. A new application can have fast access to any byte within any chain inside any VLIR record. Any new operating system can use this type of file structure with HD-DOS support.

DOS+ supports 'hidden' files. There is a byte in the directory entry that identifies the file as a hidden file. Normal directory listings will ignore these files unless an option in the command specifies to list them.

DOS+ supports 'trashed' files. These are files that you want to delete but are not really sure. So, you trash them instead of deleting them. Trashed files are also hidden in normal directory listings and will never be overwritten by new files. They can easily be recovered by untrashing them. An operating system can optionally create a trash directory containing secondary links to all the trashed files for easily locating them and untrashing them or completely deleting them.

If you have any questions about this information I'll be happy to answer. Especially I'd like to hear from folks who may be more familiar with the Native Parts and Sub-Directories than I. Got any ideas for other information you'd like to see in the web site?

EMail me