Applesauce: Difference between revisions

Revision as of 09:27, 22 November 2017

The Applesauce project was started in March 2017 by John K Morris of EvolutionInteractive.com. John is a well know programmer in his own right, however this is his most important contribution to the Apple II Community to date.

Applesauce will be a defining project for the Apple II Community which will allow users to simply create exact images of their floppy media. Never before has this be possible. With some effort from the Community as a whole we can all image and save our software heritage from bitrot and being lost to history.

"I have no time to properly disseminate info to people. No time to put up a blog or anything either. 99% of the info about Applesauce exists only here on Facebook which also has a terrible search mechanism." - John K. Morris, 2017-11-18. "Done!" - Henry S. Courbis, ReActiveMicro, 2017-11-21.

This is the defacto Applesauce information site till John can get something official posted. Links will be added as they become available.

Background

In early 2017 John and Henry S. Courbis from ReActiveMicro were discussing project ideas mostly based around the Raspberry Pi. Henry sent John some hardware and after a few weeks of tinkering and discussing different projects John mentioned an idea he had involving floppy drives and backing up software. John knew Henry had worked on the EDD project and fixed several issue as a consultant for UltimateApple2 and they started to discuss how horrible and unreliable at best the EDD was in its attempt to capture the data which represents "the floppy". John started to talk about what if we attacked the problem of copying protected disks directly from the drive rather than trying to intercept the data stream from the drive under use like the EDD did. This was the start of what would become the Applesauce project.

John discovered it would not be possible to copy disks on-platform as the Apple II is too slow and too dependent on the Disk II Controller card. There would be no way to capture the timing data coming from the drive which occurs every 4 microseconds. In order to fully read a floppy disk as "raw data", also know as the flux patterns, he would need to take the drive off-platform and control it directly. The flux patterns are literally the way the magnetic particles are magnetized on the floppy medium. They are NOT bits of or among themselves. They contain timing information as well as program data.

After some research about how the Apple II encodes floppy disks John found the only one to get things right is Jim Sather in his book Understanding the Apple II. All the other books and resources seem to have major flaws in their interpretation of how things work and plagiarized from one another. It is quite possible that John may be one of the few people besides Wozniak and Sather to actually understand how the whole floppy subsystems actually work.

By March 2017 John had started to look in to how to control the Disk II Drive and its power requirements. After some research he found the drive should be able to be controlled by a microcontroller. However most microcontrollers are not 5v tolerant and would need active and bidirectional level shifting. A proper speed microcontroller would also be needed in order to provide enough resolution to handle all the data being read from the floppy disk and allow enough time to control the drive.

A major issue would also be how to determine where the disk is in relation to the drive. This is called synchronization. IBM PC drives use a sensor to track synchronization which allows for less floppy controller processing requirements and simpler controller design to navigate tracks and sectors. The Disk II drive however does not have any synchronization mechanism which some copy protections relied on and would make copying these protected disks impossible on-platform. In order to over come this issue John designed a sync sensor with the help from Henry and added it to the arbor of the Disk II drive.

The Disk II drive's power requirements also posed some issues as the power supply would need to support universal AC input to allow for international use, and also be tri-output in order to power the Disk II drive and its logic board. John initially used a prebuilt solution (as pictured) in order to reduce prototyping time. Later revisions of the project will use a custom power supply integrated on to the project's PCB.

In April John started in earnest to create the Applesauce User Interface. This will be a GUI based program for the Mac and ported to X86 and Raspberry Pi / Linux. Javier A. Rivera is assisting John with icons and other relates program art work. John has also been looking for programmers to help with different parts of the project so he can better concentrate on the analysis engine which is the heart of Applesauce.

Henry's involvement has been rather limited as John has wanted to get hands on with CAD and circuit design. Henry has advised on several aspects of the project however John has managed to do all the labor and design work in the Proof of Concept and Alpha designs. ReActiveMicro will probably have a hand in a final design review and audit as well as overseeing assembly and distribution.

Applesauce Alpha units have been sent to several test users who have evaluated the hardware and different aspects of the software and file format.

There have been other similar projects such as Kryoflux and Catwheesel, however these projects have fallen short in several areas especially where Apple II is concerned.

What Is Applesauce?

Applesauce comprises two parts - hardware and software.

The hardware component of the project is a "Controller Interface" based on a microcontroller that allows an Apple Disk II 5.25" disk drive to be connected via USB to a modern computer. Only Mac is currently supported, however plans to port the project to X86 and Raspberry Pi (ARM) are in the plans. An expansion port has also been added for upgrades to the hardware for things like 3.5" disk imaging support in the future.

The software component runs on the modern computer and controls the Disk II Drive and captures the raw flux patterns from the inserted disk. Once this data stream is captured, the Applesauce software then process the data for saving as disk image files.

Because Applesauce captures the data on the disk at the flux level, it is able to capture Apple II copy-protection schemes "in place". Creating an image using the flux pattern is an EXACT digital representation of the analog disk. All data on the disk is captured and recorded. In most cases the disk image could be written back to a physical disk by the Applesauce software with protection in place. Or (subject to emulation hardware or software support) it can be used with copy protection left in place on Apple II emulators or solid state Apple II disk drive emulators.

For non-protected (or previously cracked disks), Applesauce is able to capture a .dsk disk image in approximately 11s.

Applesauce will work with any Apple II disk or format that was produced. 13 sector, 16 sector, CP/M, ProDOS, custom OSes, etc. It does a physical capture of the fluxes on the media, so it doesn't care about the format whatsoever. If it's on the disk Applesauce can read it. In fact, Applesauce will also image Commodore 64 and Atari (8-bit) disks just fine.

Flux Patterns

Magnetic media flux patterns, or just flux for short, are an inherent physical property of all magnetic media. The pattern refers to how each region of the media is magnetized. These patters are used to store track and sector information, as well as program data unrelated to the disk operation.

All magnetic media is coated with a ferrous material. This allows small regions in the magnetic to be used as individual magnets. The smaller the region the more "bit dense" the material is said to be. The media is then able to be used to store data. This is the basis of how all such media is created and used.

Floppy disks use a track and sector coordinate system to store information. Each track holds a set amount of sectors, and these can vary due to OS and drive design. The flux patterns are created when the head of the disk drive writes to the disk. These patterns are permanent unless damaged or overwritten. Damage can occur in several ways but is most likely due to age or scratching of the media.

Individual fluxes are always in a North or South polarity as with all magnets.

Weak Bits, Bubble Bits, Fake Bits, And The MC3470

The more correct name for these kinds of data bits or phantom flux transitions should actually be "random bits" or "random transitions" and we will explain why.

There has been some odd terms thrown around by different people about individual fluxes and their behavior. One such term used most commonly is "weak bit" which can mean two possible things.

Weak can refer to the state of the individual or group of flux patterns that have physically become damaged and are unable to be reliably read. This can be physically caused by damage to the media such as oxidation, mold, or the magnetic coating flaking off and becoming lodged under the drive head. As debris accumulates under the drive head scratching of the media becomes more likely. Once this kind of damage happens there is no way to repair it. Damage can also happen in the form of magnetic erasure which affects the levels of north or south saturation for a given area making it appear neutral or degaussed. If a disk is mishandled or brought too close to a magnetic source then parts of the disk can be changed randomly. This kind of issue can be repaired by rewriting the affected area and does not change location on the disk.

The other form of weak bits not caused by damage are more correctly called "random bits", but also "bubble bits" or "fake bits". These flux patterns appear randomly when read and constantly change location. This can be caused by several factors such as RFI however the majority of random flux patterns are caused by the MC3470 amplifier IC within the Disk II Drive itself.

The MC3470 takes the raw flux pattern signals from the drive head and amplifies it to the 5v TTL logic standard needed for the Disk II Analog card to process and send to the Apple Disk II Controller card. This amplifier IC however is far from perfect in its task. When a disk is being read the amplifier starts out waiting for a change in flux signals under the drive head. If it does not detect a change for a set period it will automatically turn up the sign gain and the process starts again. This will repeat until the MC3470 has turned the signal gain up so much all that only random noise is being sent from the IC. This is the actual source of the random bits. This is inherent to the design of the MC3470 and Motorola had redesigned the IC multiple times during the ICs active life to address this noise issue and improve reliability. Some early date codes of the IC are very noisy. Later date codes tend to be less noisy. This is most evident in image files created using the EDD card. The actual noise is also captured during reads and encoded in to the image files. When two EDD files are compared they are never the same, and this is due to the amplifier noise added to the image file.

This "noise issue" was taken in to account when producing copy protection schemes for the Apple II as the noise would always be dynamic and change locations when being read. A check could be incorporated in to a copy protection scheme to read a "blank", unformatted, or specially formatted track to see if the flux patterns were if fact random. If they always changes locations during every read the disk was original. If however these flux bits did not change location and were static this could only happen if the disk was a copy. So now you're starting to wonder "Can I copy a "blank" track?" No, it is impossible on-platform to reproduce a sector or track on a disk which is all north, south, or a degaussed (equal parts north and south) fluxes. This is a limit of the Disk II design and people who wrote copy protections knew this. The disk manufactures required special hardware, like Applesauce, in order to create these kind of protected disks. Special drives however are not needed. This is the sole reason some program titles can not be copied using current methods or devices like the EDD. Also lacking in current disk image support is a way to mark a track as random noise and a way for emulators both hardware and software based or image to disk programs to reproduce it. This is what the new image Applesauce image format will also address.

There has been rumor over the years that it is impossible to have two zero bits in a row when writing to the Disk II drive. This is correct due to the Disk II Controller board design. It however is not impossible to write disk tracks of all north or all south fluxes off-platform. The amount of noise however detected by the MC3870 will be extreme, and the Disk II Controller board will have no idea what is going on location wise with the disk medium. For usefulness more than two zero bits has none except for protection schemes.

Considering how noisy the MC3470 can be it's amazing the resilience of the Disk II Controller design and how few issues Apple II users actually experience. We have all experienced the odd issue when writing to a disk, and this is probably related to the slight chance the random noise can play.

As you can now see the more correct name for these kinds of data bits or phantom flux transitions and their behavior should actually be "random bits" or "random transitions".

Applesauce: The Hardware

At the heart of the Applesause Controller board is a Teensy v3.2 microcontroller (MCU) module. John has used this visceral and low cost module to act as a Disk II Controller card that can be controlled using a USB connection.

The MCU operates using programmable firmware, or a program telling the IOs on the module what to do and when to do them. It also allows precise control of the stepper motor and the ability to access all possible tracks from the disk including half tracks and quarter tracks, is responsible for reads and writes, and some very complex calculations related to how the different formats of the disk can be constructed.

It does everything the Disk II Controller card and Apple II do but many time faster and under our direct control. Nothing can be hidden from Applesauce. This kind of control is not possible on-platform and why the Disk II Drive was removed from the Apple II.

Extra processing speed allows for a more granular resolution of reading the flux patterns from the floppy disk. This means more precise analysis of the data is possible.

The most important aspect of the extra speed allows for three to five times oversampling of all reads. This along with the Applesauce software will ensure all disks are imaged without errors every time - unlike all other solutions. Should a difference in the three reads be found then an extra 2 reads are taken and used for a tie breaker. The user will also be alerted to this possible issue for final oversight and control. This feature also means Applesauce will be able to help restore or save damaged disks in some cases where the disk is just starting to become unreadable.

Users can painlessly update the firmware on the MUC so that they won't be left behind as new features are added to the Applesauce project. It has been designs with simplicity and longevity in mind.

Sync Sensor

A major aspect of the Applesauce project would also be how to determine where the disk is in relation to the drive head. This is called drive or media synchronization, synchronization for short, and there is no mechanism built in to the Disk II drive to detect or track it. This was done by design and helped save costs although made Wozniak's design time a lot longer and more complex. It also costs storage space since track and sectors locations must be encoded and stored on the floppy medium. The youtube channel Computerphile has a good video explanation about this (and other great retro videos too).

Apple 2 Floppy Disk Codes - Computerphile

Some copy protections relied on this feature missing from the Disk II Drive. Along with fake bits this lack of synchronization means disks could be made off-platform and there would be zero chance the disk could be copied on-platform. This doesn't mean it couldn't be cracked, however it would be unlikely developers would go to such lengths just to allow simple code edits to bypass security. This had been confirmed by Rebecca Heineman who stated The Bard's Tale III: Thief of Fate although cracked has never had all the built in security checks defeated.

In order to overcome this lack of synchronization issue John designed a sync sensor with the help from Henry and added it to the arbor of the Disk II drive. It connects using a cable to the Applesauce mainboard. Disk locations are tracked by the MCU and the result is Applesauce can very precisely locate a flux transition on the disk medium. This combined with the oversampling means Applesauce is able to guarantee a disk is read correctly every time. This is not possible on-platform or with any other copy related program or hardware like the EDD.

Applesauce: The Software

The User Interface (UI) for Applesauce is GUI based, and is only supported on the Mac (macOS 10.11) at this time. There are plans to have it ported to X86, Raspberry Pi / Linux, and possibly some older macOS versions.

Users will be able to use the UI to painlessly update the firmware on the MUC so that they won't be left behind as new features are added to the Applesauce project. It has been designs with simplicity and longevity in mind.

All of the disk imaging parts of the UI are completed and working great. Imaging unprotected disks takes about 11 seconds. Imaging protected disks takes about 2-3 minutes and will depend on what issues or errors are discovered on the disk. Clean disks will image faster than ones with mould or bad sectors for example. A major feature of the UI is the ability to help salvage damaged disks. Some disks will be able to be saved with the help oversampling. The UI will then take the best of 5 reads and save what data matches consistently. It will then ask the user for direction for the other sectors. The best part is what data is salvaged will not be thrown out if part of a read is bad. All known good data is stored and new good data can be added later. So users with a several bad disks of the same title and version should in theory be able to salvage the disk image by reading all the known good areas of the disks which Applesauce will then compile in to an image until the image is fully built. You'll also be able to open up a different disk image formats (A2R, EDD, NIB, etc) of the same software and use it to patch the bad data.

Another major part of the UI is the disk analysis/editor which is similar in feel and use to the old nibble editors. The UI will process the data read and show you the nibbles after the analysis tools have annotated the data. Users will be presented with color-coded hex views that show you the function of every nibble. Every hidden sync bit. In most cases it will also point out where things like the copy protection exists and the techniques they are using to accomplish it. It gives you the ability to go high tech "digital spelunking" through the disk image in ways that the the early cracker could have only dreamed of.

Applesauce Image File Format

One of the main things that Applesauce is bringing to the table is new disk image formats.

The formats are completed and work has started with emulator developers to start integrating support for the new formats. The new image formats have been well received by folks as it addresses all of the shortcomings of current disk image formats like DSK and NIB. Talks have also started with some hardware developers to support the new formats within their virtual floppy drive projects. This will allow still copy protected disk images to run on real hardware.

The original proof of concept image file extension was ".A2D". However at KFEST 2017 Jason Scott made a request to John that the extension be changed to ".WOZ" as an homage to Wozniak.

The .WOZ file spec has not been fully finalized, published, or released yet. Links will be posted here when it is.

Crack Verses Copy

Two solution exist which would allow a program to be fully copied and useable. One is the difficult "crack" method and the other is the simpler "copy" method.

Cracking a program is basically circumventing any protections built in to it. The protections could be added for multiple reasons however are usually based on enforcing payment for use. Cracking can range from using a copy program, to "digital spelunking" through the entire program, and there were also Copy Boards like the Wildcard which aided users in tracing code and manipulating the CPU.

To correctly crack a program ranges from the very simple to the almost impossible. First the person who is doing the cracking will usually need some moderate to deep understanding of the platform and CPU. This enables them to follow lines of code and hopefully identify all protection schemes and bypass or disable them. The issue with this however is various protection routines can be hidden almost anywhere. There is rarely just one protection check for example. And the protectors have a huge arsenal and bag of tricks at their disposal. Among these are being able to use random flux bits and lack of drive synchronization. Programmers and protection companies would often have layers of protections, checks that checked the checks to see if they were disabled or altered, and even ways to disable the program to make it unuseable, corrupt output, or unwinnable. A cracker may believe that have found all the protections and can start to play a game, however only if you play it against an original copy would you ever notice the difference. To really ensure all protections have been removed would entail going through all lines of code that make up a program and fully following them and understanding what they do and why. And the most obvious issue is a cracked version IS NOT an original. The art of the protection is lost and future wanna-be crackers will never be able to investigate the design.

Copying a program is usually much more simple however along with random bits this lack of synchronization means disks could be made off-platform and there would be zero chance the disk could be copied on-platform. This doesn't mean it couldn't be cracked, however it would be unlikely developers would go to such lengths just to allow simple code edits to bypass security. This had been confirmed by Rebecca Heineman who stated The Bard's Tale III: Thief of Fate although cracked has never had all the built in security checks defeated.

Although copying is the prefer method when archiving compared to cracking, it also has its own pitfalls and can be inconsistent even at the best of time. With copies we are still tied to physical media and drives. Applesauce will finally break this dependence AND allow us to have 100% exact copies with protecting intact.

Applesauce At KFEST 2017

John K. Morris's Presentation At KEFT 2016

EDD Is Dead

Antoine Vignau has announced that imfEDDup will no longer be updated directly due to the superiority and completeness of Applesauce.