Applesauce

AppleSauce by John K Morris

The Applesauce project was started in March 2017 by John K Morris of EvolutionInteractive.com. John is a well know programmer in his own right, however this is his most important contribution to the Apple II Community to date.

Applesauce will be a defining project for the Apple II Community which will allow users to simply create exact images of their floppy media. Never before has this be possible. With some effort from the Community as a whole we can all image and save our software heritage from bitrot and being lost to history.

Background

In early 2017 John and Henry S. Courbis from ReActiveMicro were discussing project ideas mostly based around the Raspberry Pi. Henry sent John some hardware and after a few weeks of tinkering and discussing different projects John mentioned an idea he had involving floppy drives and backing up software. John knew Henry had worked on the EDD project and fixed several issue as a consultant for UltimateApple2 and they started to discuss how horrible and unreliable at best the EDD was in its attempt to capture the data which represents "the floppy". John started to talk about what if we attacked the problem of copying protected disks directly from the drive rather than trying to intercept the data stream from the drive under use like the EDD did. This was the start of what would become the Applesauce project.

John discovered it would not be possible to copy disks on-platform as the Apple II is too slow and too dependent on the Disk II Controller card. There would be no way to capture the timing data coming from the drive which occurs every 4 microseconds. In order to fully read a floppy disk as "raw data", also know as the flux patterns, he would need to take the drive off-platform and control it directly. The flux patterns are literally the way the magnetic particles are magnetized on the floppy medium. They are NOT bits of or among themselves. They contain timing information as well as program data.

After some research about how the Apple II encodes floppy disks John found the only one to get things right is Jim Sather in his book Understanding the Apple II. All the other books and resources seem to have major flaws in their interpretation of how things work and plagiarized from one another. It is quite possible that John may be one of the few people besides Wozniak and Sather to actually understand how the whole floppy subsystems actually work.

By March 2017 John had started to look in to how to control the Disk II Drive and its power requirements. After some research he found the drive should be able to be controlled by a microcontroller. However most microcontrollers are not 5v tolerant and would need active and bidirectional level shifting. A proper speed microcontroller would also be needed in order to provide enough resolution to handle all the data being read from the floppy disk and allow enough time to control the drive.

A major issue would also be how to determine where the disk is in relation to the drive. This is called synchronization. IBM PC drives use a sensor to track synchronization which allows for less floppy controller processing requirements and simpler controller design to navigate tracks and sectors. The Disk II drive however does not have any synchronization mechanism which some copy protections relied on and would make copying these protected disks impossible on-platform. In order to over come this issue John designed a sync sensor with the help from Henry and added it to the arbor of the Disk II drive.

The Disk II drive's power requirements also posed some issues as the power supply would need to support universal AC input to allow for international use, and also be tri-output in order to power the Disk II drive and its logic board. John initially used a prebuilt solution (as pictured) in order to reduce prototyping time. Later revisions of the project will use a custom power supply integrated on to the project's PCB.

In April John started in earnest to create the Applesauce User Interface. This will be a GUI based program for the Mac and ported to X86 and Raspberry Pi (ARM). Javier A. Rivera is assisting John with icons and other relates program art work. John has also been looking for programmers to help with different parts of the project so he can better concentrate on the analysis engine which is the heart of Applesauce.

Henry's involvement has been rather limited as John has wanted to get hands on with CAD and circuit design. Henry has advised on several aspects of the project however John has managed to do all the labor and design work in the Proof of Concept and Alpha designs. ReActiveMicro will probably have a hand in a final design review and audit as well as overseeing assembly and distribution.

Applesauce Alpha units have been sent to several test users who have evaluated the hardware and different aspects of the software and file format.

There have been other similar projects such as Kryoflux and Catwheesel, however these projects have fallen short in several areas especially where Apple II is concerned.

What Is Applesauce?

Applesauce comprises two parts - hardware and software.

The hardware component of the project is a "Controller Interface" based on a microcontroller that allows an Apple Disk II 5.25" disk drive to be connected via USB to a modern computer. Only Mac is currently supported, however plans to port the project to X86 and Raspberry Pi (ARM) are in the plans. An expansion port has also been added for upgrades to the hardware for things like 3.5" disk imaging support in the future.

The software component runs on the modern computer and controls the Disk II Drive and captures the raw flux patterns from the inserted disk. Once this data stream is captured, the Applesauce software then process the data for saving as disk image files.

Because Applesauce captures the data on the disk at the flux level, it is able to capture Apple II copy-protection schemes "in place". Creating an image using the flux pattern is an EXACT digital representation of the analog disk. All data on the disk is captured and recorded. In most cases the disk image could be written back to a physical disk by the Applesauce software with protection in place. Or (subject to emulation hardware or software support) it can be used with copy protection left in place on Apple II emulators or solid state Apple II disk drive emulators.

For non-protected (or previously cracked disks), Applesauce is able to capture a .dsk disk image in approximately 11s.

Flux Patterns

Example Of Disk Head Writing To Disk

Magnetic media flux patterns, or just flux for short, are an inherent physical property of all magnetic media. The pattern refers to how each region of the media is magnetized. These patters are used to store track and sector information, as well as program data unrelated to the disk operation.

All magnetic media is coated with a ferrous material. This allows small regions in the magnetic to be used as individual magnets. The smaller the region the more "bit dense" the material is said to be. The media is then able to be used to store data. This is the basis of how all such media is created and used.

Floppy disks use a track and sector coordinate system to store information. Each track holds a set amount of sectors, and these can vary due to OS and drive design. The flux patterns are created when the head of the disk drive writes to the disk. These patterns are permanent unless damaged or overwritten. Damage can occur in several ways but is most likely due to age or scratching of the media.

Individual fluxes are always in a North or South polarity as with all magnets.

Weak Bits, Bubble Bits, Fake Bits, And The MC3470

There has been some odd terms thrown around by different people about individual fluxes and their behavior. One such term used most commonly is "weak bit" which can mean two possible things.

Weak can refer to the state of the individual or group of flux patterns that have physically become damaged and are unable to be reliably read. This can be physically caused by damage to the media such as oxidation, mold, or the magnetic coating flaking off and becoming lodged under the drive head. As debris accumulates under the drive head scratching of the media becomes more likely. Once this kind of damage happens there is no way to repair it. Damage can also happen in the form of magnetic erasure which affects the levels of north or south saturation for a given area making it appear neutral or degaussed. If a disk is mishandled or brought too close to a magnetic source then parts of the disk can be changed randomly. This kind of issue can be repaired by rewriting the affected area and does not change location on the disk.

The other form of weak bits not caused by damage are more correctly called "bubble bits" or "fake bits". These flux patterns appear randomly when read and constantly change location. This can be caused by several factors such as RFI however the majority of random flux patterns are caused by the MC3470 amplifier IC.

The MC3470 takes the raw flux pattern signals from the drive head and amplifies it to the 5v TTL logic standard needed for the Disk II Analog card to process and send to the Apple Disk II Controller card. This amplifier IC however is far from perfect in its task. When a disk is being read the amplifier starts out waiting for a change in flux signals under the drive head. If it does not detect a change for a set period it will automatically turn up the sign gain and the process starts again. This will repeat until the MC3470 has turned the signal gain up so much all that is produced is random noise being sent from the IC. This is inherent to the design of the MC3470 and Motorola had redesigned the IC multiple times the ICs active life to address this noise issue. Some early date codes of the IC are very noisy. Later date codes tend to be less noisy. This is most evident in image file created using the EDD card. The actual noise is also captured during reads and encoded in to the image files. When two EDD files are compared they are never the same, and this is due to the amplifier noise.

This "noise issue" was taken in to account when producing copy protection schemes for the Apple II as the noise would always be dynamic and change locations when being read. A check could be incorporated in to a copy protection scheme to read a "blank", unformatted, or specially formatted track to see if the flux patterns would change. If they did the disk was original. If however these flux bits did not change location and were static this could only happen if the disk was a copy. So now you're starting to wonder "Can I copy a "blank" track?" No, it is impossible on-platform to reproduce a sector or track on a disk which is all north, south, or a degaussed (equal parts north and south) fluxes. This is a limit of the Disk II design and people who wrote copy protections knew this. The manufactures would require special hardware, like Applesauce, in order to create these kind of protected disks. Special drives however are not needed. This is the sole reason some program titles can not be copied using current methods or devices like the EDD. Also lacking in current disk image support is a way to mark a track as random noise and a way for emulators or image to disk programs to reproduce it. This is what the new image Applesauce image format will also address.

There has been rumor over the years that it is impossible to have two zero bits in a row when writing to the Disk II drive. This is correct due to the Disk II Controller board design. It however is not impossible to write to the disk all north or all south fluxes off-platform. The amount of noise however detected by the MC3870 will be extreme, and the Disk II Controller board will have no idea what is going on location wise with the disk medium. For usefulness more than two zero bits has none except for protection schemes.

Considering how noisy the MC3470 can be it's amazing the resilience of the Disk II Controller design and how few issues Apple II users actually experience. We have all experienced the odd issue when writing to a disk, and this is probably related to the slight chance the random noise can play.

Teensy v3.2 Microcontroller

At the heart of the Applesause Controller board is a Teensy v3.2 microcontroller (MCU) module. John has used this visceral and low cost module to act as a Disk II Controller card that can be controlled using a USB connection.

The MCU operates using programmable firmware, or a program telling the IOs on the module what to do and when to do them. It also allows precise control of the stepper motor and the ability to access all possible tracks from the disk including half tracks and quarter tracks, is responsible for reads and writes, and some very complex calculations related to how the different formats of the disk can be constructed.

It does everything the Disk II Controller card and Apple II do but many time faster and under our direct control. Nothing can be hidden from Applesauce. This kind of control is not possible on-platform and why the Disk II Drive was removed from the Apple II.

Extra processing speed allows for a more granular resolution of reading the flux patterns from the floppy disk. This means more precise analysis of the data is possible.

The most important aspect of the extra speed allows for three to five times oversampling of all reads. This along with the Applesauce software will ensure all disks are imaged without errors every time - unlike all other solutions. Should a difference in the three reads be found then an extra 2 reads are taken and used for a tie breaker. The user will also be alerted to this possible issue for final oversight and control. This feature also means Applesauce will be able to help restore or save damaged disks in some cases where the disk is just starting to become unreadable.

Users can painlessly update the firmware on the MUC so that they won't be left behind as new features are added to the Applesauce project. It has been designs with simplicity and longevity in mind.

Sync Sensor

What it is and why we need it.

Notes

I have been doing all kinds of crazy analysis R&D work that has been really paying off. I'm currently working on a disk analysis/editor that is similar to the old nibble editors, but will show you the nibbles after the analysis tools have annotated the data. So, you get color-coded hex views that show you the function of every nibble. Every hidden sync bit. It will also point out where things like the copy protection exists and the techniques they are using to accomplish it. It gives you the ability to go "digital spelunking" through the disk. It will also allow you to repair a disk image. Do you have a flawed sector? Open up a different disk image (A2R, EDD, NIB, etc) of the same software and use it to patch the bad data.

One of the main things that Applesauce is bringing to the table is new disk image formats. The formats are completed and work has started with emulator developers to start integrating support for the new formats. The new image formats have been well received by folks as it addresses all of the shortcomings of current disk image formats like DSK and NIB. Talks have also started with some hardware developers to support the new formats within their virtual floppy drive projects. This will allow still copy protected disk images to run on real hardware.

EDD Is Dead

As quoted by Antwion, and why.