FAQ about the 4GB problem

Why 4GB ?

4GB is not a magic number. It is just the biggest value which can be stored in 32 bits. It's 2 to the power of 32, which is 4,294,967,296. By hard drive manufacturers this is usually called 4.3 GB.

To be exact, the highest number which can be stored is 4,294,967,295. One can store 4,294,967,296 different numbers and 0 is one of them, so numbers from 0 to 4,294,967,295 can be stored.

Why is there a limit at all ?

Numbers (offsets, sizes etc.) have to be stored somehow. And storing numbers means to allocate some memory for them. And in a fixed amount of memory only a limited number of numbers can be stored.

When the Amiga OS was designed in the late 1980's, the Amiga was a very modern machine. It was one of the first computers with a fully 32 bit-cappable processor. Back then and also until only a few years ago, the biggest data type all programming languages could handle out of the box was a 32 bit number. So using a 32 bit number for disk sizes and offsets in 1987 was quite foresighted.

Where is the 4GB limit ?

The limit is created by the interface of the hardware driver. The hardware driver for the internal IDE bus is called scsi.device (yes, the IDE driver is called SCSI, that's for historical and compatibility reasons). All drivers of the ".device" kind are accessed with the same common interface. This interface understands commands like for example "please read x bytes beginning with byte y". X and y are numbers stored in 32 bits as discussed above.

Hard drives are usually divided into blocks of 512 bytes each. So to read the 1,000th block from a harddrive, you ask the driver to "read 512 bytes starting from byte 512,000". Nothing difficult so far. Now let's imagine a big HDD and the application wants to read the 8,388,608th block. So the command is "please read 512 bytes starting at byte 4,294,967,296".

But stop. This is one more than can be stored in 32 bits. Unfortunately the programming language does not check this. Without explicit checking in the program an overflow like this is not recognised. What happens is that the overflowing bits are cut off and only the lower 32 bits are used. For 4,294,967,296 the 33rd bit would be 1 and all other bits are zero. So the command which is sent to the driver actually is "please read 512 bytes starting at byte 0".

And here you see the problem: block 0 is used twice without notice. If you write data to a file which should be stored at block 8,388,608, block 0 is overwritten instead. Block 0 usually holds the partition table, so all data on this HDD is lost now and the computer refuses to boot again, it thinks the HDD is empty.

Do I need a new HDD controller now ?

No. The problem is not the hardware. The hardware can handle hard drives of any size. The 4GB limit is in software and can be fixed by software.

So I just replace scsi.device with a new verson and everything is fine ?

No, not completely. As shown above commands are sent to the driver by applications. The driver can be changed to understand new commands, but the applications have to be changed, too, so that they use these new commands. Existing applications cannot magically be made working by inventing a new interface.

Do all applications have to be replaced ?

No. Usually an application does not ask the driver directly to read blocks. There are a couple of software layers inbetween. An appplication usually asks dos.library to read files. The library then asks the file system and the file system tells the driver to read blocks.

So the most common solution is to replace the file system as well as the hardware driver, then most usual applications can use big harddrives automatically.

But there are some applications which access scsi.device directly. For example defragmentation programs or salvage programs. These have to be changed individually, they cannot be made work just by a new driver.

I don't care about the 4GB limit, I just divide my hardrive into many smaller partitions, then everything works.

STOP !!! This is a big big mistake. The 4GB limit applies to the entire hard drive, not to partitions.

As outlined above, scsi.device is one side of the problem. And scsi.device does not know about partitions. It sees the entire harddrive as one big stream of bytes. It only knows that for example there is a hard drive at unit 0 and a CD-ROM drive at unit 1, but all these drives are seen entirely.

Partitions are made by dos.library. Dos.library mounts a partition like "HD0 starts at cylinder 2 and is 1,000 cylinders big, one cylinder has 1008 blocks and one block has 512 bytes". This information is given to the file system and the file system then calculates byte offsets. For example if the file system decides to read the first block from the second cylinder of HD0, it has to read the 3,024th block from HDD (1,008 * 3) and the command sent to scsi.device is "please read 512 bytes beginning at byte 1,548,288.

So if a partition passes the 4GB border and the file system calculates the byte offsets as 32bit numbers, then parts of the area below the 4GB border are used instead, destroying important data on other partitions or even the partition table itself.

How can scsi.device be replaced ?

Well, it can't. Scsi.device is part of the Kickstart ROM and it cannot be replaced without creating a new ROM version.

But it can be patched by software. However, there is a big caveat of such a patch: when you switch the computer on and the RAM is completely empty, only the ROM version of scsi.device exists. This means that the boot partition which starts the computer and loads the patches has to be accessed by the original version of scsi.device. This means that the boot partition has to be inside the first 4GB of the HDD.

How is the problem solved ? What does a patch for scsi.device do ?

There is no real solution for that. As the patch cannot be applied to the ROM, it does it in RAM. And this means that the patch is removed when you switch off the computer and it is not there when you switch it on. It has to be loaded after the boot process has already be started.

This entails two needs:

1. The boot partition must be in the area which can be accessed by the unpatched scsi.device. Because when the computer is switched on, only the original ROM version of scsi.device is active, the boot partition must be reachable by this version.

2. Because a program which is already running cannot be replaced on-the-fly, the computer has to be reset after the patch is applied. Otherwise the new version of scsi.device would not become active.

So I cannot partition a big harddrive into one big piece ?

Correct. But actually this is not a big problem because historically the Amiga always had more than one partition. Every pre-installed Amiga had one smaller system partition which held Workbench, fonts, libraries etc. and one larger work partition which stored applications, user data and so on.

For backup and safety reasons I would even recommend to have more than one boot partition and more than one work partition.

But is there no other solution which allows to use bigger harddrives without that annoying reset which slows down everything so much ?

There is indeed a solution which works without a patch for scsi.device, but it is not a generic one. It only allows to change the limit from 4GB to ca. 8GB.

The solution is HD_SCSICMD, which allows to send SCSI commands to the driver. This often is also referred to as Direct-SCSI.

SCSI commands use block addresses rather than byte addresses. So while with the normal ".device" commands one could address 4 billion bytes, with SCSI you can theoretically address 4 billion blocks of 512 bytes each. This would increase the limit from 4GB to 2TB.

Unfortunately it does not work like that. On the hardware side, scsi.device still works with the CHS (Cylinder/Head/Sector) addressing scheme to read the capacity. And the limit for this is 16383 cylinders, 16 heads and 63 sectors which results in 16383 * 16 * 63 * 512 = 8,455,200,768 bytes or 7.87 GB.

Well, 8 GB is enough for me, how is it done ?

You can use one of these file systems:

• FFSTD64 (link).

  This package patches the original FastFileSystem to work with 64 bit commands and alternatively with HD_SCSICMD.

• SFS up to version 1.87 (description, download).

  Please note that the cappability to use HD_SCSICMD was removed in later versions of SFS, so you have to use this old version.

• PFS (commercial, support page).

  PFS comes in two version, one which uses TD64 and one which uses HD_SCSICMD.

What does TD64 mean ?

Above we were speaking about commands. "Normal" commands and HD_SCSICMD for example. "Normal" commands are CMD_READ and CMD_WRITE. These two commands are present in every ".device" driver and have a limit of 4GB.

HD_SCSICMD is a different command which is only understood by a few drivers for SCSI or IDE controllers. And it needs a lot more programming effort to use it.

TD64 was invented to remove the 4GB limit. It consists of new commands like TD_READ64 and TD_WRITE64. These new commands use 64bit numbers as offset and thus have a much higher limit than 4GB.

Shortly after TD64 another command set for the same purpose was invented by another party. It is called NSD and contains the commands NSCMD_TD_READ64 and NSCMD_TD_WRITE64. These commands work exactly like the TD64 commands but are not compatible.

So there are three new command sets which all do the same. Which one should I use ?

Actually it does not matter which one you use. You just have to find a combination of .device driver and file system which both support the same command set.

NSD was declared "the standard" when OS 3.5 came out. But it is not supported by every manufacturer of HDD controllers.

NSD? Great! There is NSDPatch and it solves all my problems.

No, it doesn't. Actually NSDPatch is of little help for the 4GB problem.

Yes, NSDPatch makes many .device drivers understand NSD commands. But it does not replace the drivers. The hardware access is still the same. And as we already saw above, scsi.device has an absolute limit of 7.8 GB. This limit can only be removed by a replacement, not by a patch.

So to use a 7.8 GB drive, it's better to use one of the aforementioned file systems instead of NSDPatch which still needs the boot partition inside the first 4GB.

Now stop all this talking and show me where I can download the software I need!

As always there is no solution which is both, easy and cheap. There are easy solutions which can be bought and there are free solutions which are difficult to install.

If you already have an Amiga which fulfills all the requirements, the easiest solution is to buy OS 3.5 or OS 3.9. These operating systems come with everything you need.

For both operating systems, you should first create an emergency boot disk, boot from it and only then partition your big HDD. For OS 3.9 the emergency disk is the first option in the installer menu, for OS 3.5 it is the last one. But in both cases it should be done first.

Don't forget to put the boot partition inside the first 4GB of the drive. The other partitions can be arranged as you like.

I don't want to upgrade my operating system. What can I do instead ?

If you want to stick with OS 3.1, another quite easy solution is IDE-fix'97 (now called IDE-max'97; demo, full version).

To make IDEfix work as a solution for the 4GB problem, after it is installed, you have to edit s:startup-sequence and replace C:IDEfix by C:LoadIDE RESET. And you have to install a file system which supports big harddrives. IDEfix supports all three command sets (HD_SCSICMD, TD64 and NSD), so there is no restriction which file system to use.

Again the boot partition must be inside the first 4GB of the HDD.

Erm, yes, but which file systems support big harddrives ?

These file systems suppprt big harddrives:

• SmartFileSystem up to version 1.87 (description, download)

  Supports HD_SCSICMD, TD64 and NSD

• current version of SmartFileSystem (link)

  Supports only TD64 and NSD

• ProfessionalFileSystem (support page)

  Supports either HD_SCSICMD or TD64, depending on which version you install

• FastFileSystem version 43 (download page)

  Supports only NSD

• FastFileSystem version 44 (result of FFSTD64)

  Supports HD_SCSICMD and TD64

• FastFileSystem version 45 and above (part of the operating system since OS 3.5)

  Supports only NSD

You can see that there are many different file systems which support different command sets, even different between versions of the same file system. As mentioned above you have to choose a file system and a device driver which fit together.

Where is the cheap solution ?

The updates which were later incorporated into OS 3.5 and OS 3.9 could formerly be downloaded as beta versions from the German Commodore Amiga homepage. DJBase has saved the files before the homepage was shut down and now offers them for download on his site: download page

You need FastFileSystem version 43.20 and scsi.device version 43.24.

These files are the original beta versions which contain date checks which have long expired. Therefore when installed they issue a lot of beta warnings before the Amiga starts to boot. There are patches to remove these warnings on Aminet: for scsi.device and for FastFileSystem.

The patch for FFS is for an older version. In version 43.20 the offset has changed from $002A2C to $002A34.

Finally the LoadV43Module program which is supplied with the scsi.device update only works on Kickstart 3.1. If you want to use the patches on older Kickstart versions, you should use Thor's LoadModule instead: Aminet link.

The installation steps are as follows:

• Part 1
  • unpack the scsi.device archive and the date check patch for scsi.device
  • select the right strip file for your Amiga model (Read the docs ! The file names are misleading !)
  • patch away the date check
  • put the resulting file in to the Devs directory of your boot partition
  • unpack the LoadModule archive and put the program into the C directory of your boot partition
  • edit s:startup-sequence and add the line which loads the new scsi.device into memory
• Part 2
  • unpack the FastFileSystem archive and the date check patch for FFS.
  • patch away the date check
  • copy the resulting file into the L directory of your boot partition
  • run HDToolbox, click on Partition Drive, enable Advanced options and click on Add/Update.
  • select FastFileSystem from the list, click on Update and select the new FFS file created before.
  • if FFS was not in the list, add a new entry. Make sure that the DosType/Identifier matches the DosType/Identifier of your partitions.
• Verification
  • reboot the Amiga
  • open a shell window
  • enter version scsi.device
  • it should reply scsi.device 43.24.
  • if the version is still 40 or lower, then the update did not work.
  • now enter version DH0: (replace DH0 by the name of your boot partition)
  • it should reply filesystem 43.20.
  • if the version is still 40 or lower, then the update did not work.

Does the same procedure also work if I have a third-party HDD controller ?

No. Scsi.device is the driver for the internal IDE or SCSI bus of the Commodore Amiga models only. If you have a third-party controller, it has a different driver.

Whether large hard drives can be used with third-party controllers depends on the driver. Some drivers already support large hard drives either through HD_SCSICMD or TD64. Other controllers might need a firmware upgrade. Some controllers just don't support big HDDs and cannot be made to.

If your driver supports HD_SCSICMD or TD64 and you use a file system which supports the same, you can create partitions as you like, even big boot partitions.

If your driver supports HD_SCSICMD or TD64, it can be made to support NSD also by running NSDPatch. But in this case the boot partition(s) need to be inside the first 4GB of the HDD.

I installed everything but am still unsure if it works correctly

Download Check4GB from Aminet and run it. It should tell you if everything is ok.

Aminet link

I installed the new scsi.device but HDToolbox still does not allow to use more than 7.8 GB

Use FixHDDSize from Aminet before you run HDToolbox.

Aminet link

Ok, now the diagram in HDToolbox looks much bigger, but the numbers still go only from 0 to 4095 MB and not higher.

That's the same problem with 32 bits as explained in the beginning. If you increase the partition size slowly you will see that it jumps from 4095 to 0 whenever you cross a 4 GB boundary.

But this time it's only a cosmetic problem of HDToolbox. Internally it does it right. You just have to remember how often it jumped from 4095 to 0 to get the actual partition size.

Partitions seem ok but now I get strange problems with Workbench and directory tools showing only small or even negative amounts of free storage. Some programs even refuse to install because they think there is not enough free space.

Again the same problem. Trying to calculate an amount of storage in bytes with 32 bit numbers easily overflows when dealing with bigger drives. Signed numbers overflow at 2GB (i.e. everything between 2GB and 4GB is treated as negative) and unsigned numbers overflow at 4GB.

There is no easy fix for that. To fix it, every program which has this problem would have to be fixed. You can only work around it.

One workaround is to create only partitions which are smaller than 2GB. This will give a lot of partitions but no problems with applications.

Another workaround is to fill the partition with dummy files until the amount displayed is big enough to install the program. The dummy files can then be deleted when the program was installed successfully.