ToToTEK.COM Forum Index ToToTEK.COM
Help & Support Forum
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Super UFO PRO 8 SD - USB File transfer alternatives?
Goto page Previous  1, 2, 3  Next
 
Post new topic   Reply to topic    ToToTEK.COM Forum Index -> Copiers and Hardware
View previous topic :: View next topic  
Author Message
dbjh



Joined: 02 Aug 2003
Posts: 167

PostPosted: Tue Jun 12, 2018 4:38 pm    Post subject: Reply with quote

revenant1 wrote:
I noticed that Super Metroid ends up with the value 0x20 at 0x11, which might just be the ROM size in megabits (0x18) rounded up to the next power of two.

Yes, that is why
I wrote:
At least the output of ucon64 --dbuh on actual cartridge dumps matches perfectly with the assumption that offset 0x11 represents the internal size in Mbit.


revenant1 wrote:
Here are my results (for 0x13-15) for most of the games you listed. These were obtained by loading headerless copies of the ROMs from a SD card, then immediately saving them back with whatever default header settings were applied by the firmware.

I do not think that proves much. What it shows is what the Super UFO Pro 8 SD *guesses* the mapping of the original cartridges to be.
The headers it produces when dumping actual cartridges are no accident. There is obviously a reason why the firmware does not produce a more "logical" header for certain cartridges - we just do not know yet why. In the ideal scenario the firmware analyzes the actual mapping of the cartridge. With that said, I reduced your logic in romutils.py for uCON64 to the following lines:
Code:
header.map_control[2] = ufosd_size <= 16 * MBIT ? 0x10 : 0x50;
if (snes_sram_size > 32 * 1024) header.map_control[2] += 0x10;
header.map_control[3] = 0x3f;


revenant1 wrote:
The firmware seems to use 0x2x for byte 0x15 when there's at least 8kb of SRAM, rather than over 32kb. This shouldn't matter for most games, but doing it this way may trigger copy protection in some games that try to detect improper SRAM mapping.

Regarding copy protection, Trenton_net has tested several games that use SRAM based copy protections and none were triggered using the code above.

revenant1 wrote:
dbjh wrote:
Have you found that link again?

Check the bottom of the CH372 page here. CH372EVT.ZIP has the header files and some example code and CH372DRV.EXE is the installers for the drivers and the API DLL.

Thanks for the link.

I have attached a text file with the output of --dbuh for actual cartridge dumps. It is what I used to write the old implementation of --ufosd. Perhaps you find it useful.



ufoheaders.zip
 Description:
Output of "ucon64 --dbuh" for cartridge ROM images produced with a Super UFO Pro 8 SD.

Download
 Filename:  ufoheaders.zip
 Filesize:  8.2 KB
 Downloaded:  416 Time(s)

Back to top
View user's profile Send private message
revenant1



Joined: 25 Mar 2009
Posts: 14

PostPosted: Sun Jun 17, 2018 10:26 am    Post subject: Reply with quote

dbjh wrote:
I do not think that proves much. What it shows is what the Super UFO Pro 8 SD *guesses* the mapping of the original cartridges to be.
The headers it produces when dumping actual cartridges are no accident. There is obviously a reason why the firmware does not produce a more "logical" header for certain cartridges - we just do not know yet why. In the ideal scenario the firmware analyzes the actual mapping of the cartridge.


The fact that the games work with these header values - and that you can determine those values using heuristics similar to the ones that reasonably accurate emulators (including bsnes/higan) use - proves that you don't really need special cases for specific games, I think.

I decided to examine the actual firmware behavior a bit more. It does attempt to figure out ROM/RAM mapping when examining a real cartridge, but because of how some boards work, the UFO may arrive at different values than it would when examining a ROM. For example, the board used by most Link to the Past cartridges leaves banks $a0-bf unmapped rather than mirroring $80-9f, so the firmware treats it as a 24-32Mbit LoROM game for memory mapping purposes, even though it actually isn't one. (It still gets the correct ROM size from just looking at $FFD7.)

You can still treat it like a "generic" LoROM cartridge and it'll work fine, though. Virtually all emulators do the same thing.

(The only time I can think of that it might not work is on the original Japanese 1.0 version of Rockman X, but that's actually due to a bug that causes the copy protection to be triggered even on some real cartridges. Most versions of the game don't have this issue.)

I still haven't tested the USB code you sent me, but I may do it if I have any more free time this weekend. Sorry I've been too busy with work and other stuff to get to that yet.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Trenton_net



Joined: 29 Jun 2006
Posts: 233

PostPosted: Tue Jun 19, 2018 4:03 am    Post subject: Reply with quote

@revenant1: I was able to test USB transfers and it seems to work fine with zidig. However I experienced a strange issue where transfering the game "Super Mario All Stars (U)[!]" would give me a black screen on the UFO when using ucon64, while it would work fine with the official UFO Transfer software. I was able to get Dbjh some dumps from the UFO, which show that some bytes are incorrect when ucon64 sends the ROM, but we're still trying to figure out more. I tried transferring a bunch of games such as:

"Kerby's Dream Course"
"Breath of Fire II (U)"
"Donkey Kong Country 3 - Dixie Kong's Double Trouble (U)"
"Earthbound (U)"
"Front Mission - Gun Hazard (J)"
"Lufia II - Rise of the Sinistrals (U)"
"Mario no Super Picross (J)"
"Super Mario All-Stars (U)"
"Super Metroid (JU)"
"Uniracers (U)"

And they all seem to run and transfer fine, except for "Super Mario All-Stars (U)" which only works when sent using the official UFO software.
Back to top
View user's profile Send private message
dbjh



Joined: 02 Aug 2003
Posts: 167

PostPosted: Tue Jun 19, 2018 2:56 pm    Post subject: Reply with quote

revenant1 wrote:
The fact that the games work with these header values - and that you can determine those values using heuristics similar to the ones that reasonably accurate emulators (including bsnes/higan) use - proves that you don't really need special cases for specific games, I think.

If those specific games work then yes. I just find it strange that people who clearly understand the SNES pretty well, would come up with such a scheme to specify certain mappings. Perhaps it's a case of hardware guys not being very skilled at writing software? I tend to assume that engineers know what they are doing.

revenant1 wrote:
I decided to examine the actual firmware behavior a bit more. It does attempt to figure out ROM/RAM mapping when examining a real cartridge, but because of how some boards work, the UFO may arrive at different values than it would when examining a ROM. For example, the board used by most Link to the Past cartridges leaves banks $a0-bf unmapped rather than mirroring $80-9f, so the firmware treats it as a 24-32Mbit LoROM game for memory mapping purposes, even though it actually isn't one. (It still gets the correct ROM size from just looking at $FFD7.)

It's nice to get a confirmation that the firmware actually does that. Thanks for sharing.

Sorry for bringing it up, but I still don't understand the meaning of the byte at offset 0x15 in the header. In your header document you wrote that the value 0x10 maps entire banks (xx:0000-0xffff, 64 kB) and the value 0x20 maps the lower half (xx:0000-0x7fff, 32 kB). I don't see how it then makes sense that the UFO SD produces a header with 0x15 being 0x10 for games that use 2 kB SRAM and 0x15 being 0x20 for games that use 8 kB SRAM. I mean, theoretically 0x20 already maps more SRAM than those games can access in a cartridge, and could thus trigger SRAM based copy protection code. Besides, 0x10 maps the larger 64 kB, so why does it make sense that it is associated with games that use the smaller amount of 2 kB SRAM, while 0x20 which maps the smaller 32 kB is associated with games that use the larger amount of 8 kB SRAM? Please enlighten me, I would like to understand this :-)

Trenton_net wrote:
which show that some bytes are incorrect when ucon64 sends the ROM

Actually 1 byte. At offset 0x200, so the first byte of the actual ROM image data. We don't know yet whether it is always only this byte that gets corrupted/reset (set to 0).
@revenant1: What happens when you upload Super Mario All-Stars with your upload tool?
Back to top
View user's profile Send private message
revenant1



Joined: 25 Mar 2009
Posts: 14

PostPosted: Tue Jun 19, 2018 5:25 pm    Post subject: Reply with quote

@Trenton_net Thanks for testing it out, I still wasn't able to set it up over the weekend like I was hoping to.

dbjh wrote:
If those specific games work then yes. I just find it strange that people who clearly understand the SNES pretty well, would come up with such a scheme to specify certain mappings. Perhaps it's a case of hardware guys not being very skilled at writing software? I tend to assume that engineers know what they are doing.

Which software are you talking about? If you're just referring to how the firmware generates headers for those games, it's mostly just a matter of the firmware not knowing how to distinguish unmapped/"open bus" memory from an actual large/unmirrored ROM.

dbjh wrote:
Sorry for bringing it up, but I still don't understand the meaning of the byte at offset 0x15 in the header. In your header document you wrote that the value 0x10 maps entire banks (xx:0000-0xffff, 64 kB) and the value 0x20 maps the lower half (xx:0000-0x7fff, 32 kB). I don't see how it then makes sense that the UFO SD produces a header with 0x15 being 0x10 for games that use 2 kB SRAM and 0x15 being 0x20 for games that use 8 kB SRAM.

Games with large amounts of ROM (such as Dezaemon, RPG Maker, Mario Paint, etc) map SRAM only to 0x0000-7fff so that 0x8000-ffff in the same banks can also be mapped to ROM. Dezaemon also has 128kb SRAM but maps it to 0x0000-0x7fff across multiple banks, even though the ROM is only 512kb.

It's probably best to use 0x20 when there's more than 2MB/16Mbit of ROM or 32kb/256kbit of SRAM. For whatever reason I didn't mention ROM size in my header document (and romutils.py doesn't account for ROM or SRAM size there, because I guess I forgot to), but hopefully that clears things up.

dbjh wrote:
@revenant1: What happens when you upload Super Mario All-Stars with your upload tool?

I'll give it a try after work. I'll also update my docs/scripts and make sure that the SRAM settings for large LoROM games are documented/handled properly.

e: Tried it, SMAS uploaded and played without any problems. Also made some small corrections to my docs/scripts.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
dbjh



Joined: 02 Aug 2003
Posts: 167

PostPosted: Wed Jun 20, 2018 4:47 pm    Post subject: Reply with quote

revenant1 wrote:
If you're just referring to how the firmware generates headers for those games, it's mostly just a matter of the firmware not knowing how to distinguish unmapped/"open bus" memory from an actual large/unmirrored ROM.

Is it possible at all to distinguish between open bus and unmirrored ROM?

revenant1 wrote:
hopefully that clears things up.

Of course it does not - you did not address my questions at all :-)
Do not get me wrong. Mentioning that SRAM can be mapped across a range of banks is of course an explanation how 0x20 can work for games using more than 32 kB SRAM. However, I was specifically asking about the difference between 2 kB and 8 kB.

In an earlier post you wrote:
This mostly all agrees with my docs (note that the upper two bits of 0x15 are based on ROM size, not SRAM size).

The firmware seems to use 0x2x for byte 0x15 when there's at least 8kb of SRAM, rather than over 32kb.

In other words, it does *not* agree with your docs. No distinction can be made between 2 kB and 8 kB with just the value of the byte at offset 0x15. I wanted to know why it made sense to you, because with the data presented it does not make sense to me. The logic in a table (using bitmask 0x30 on 0x15 value):
Code:
SRAM size in kB  0x15 value (your logic)  bank size in kB  0x15 value (Super UFO firmware guess)  bank size in kB
2                0x10                     64               0x10                                   64
8                0x10                     64               0x20                                   32
32               0x10                     64               0x20                                   32
64               0x20                     32               0x20                                   32
128              0x20                     32               0x20                                   32

So, you have explained how 64 kB or more can be addressed, but not how that makes sense. Why is 0x15 not always 0x20? It is after all more restrictive than 0x10. Or can the SRAM of LoROM games using 2 or 8 kB SRAM be mapped somewhere in the upper 32 kB? According to that logic apparently the 4 games usable on the Super UFO Pro 8 SD that use more than 32 kB SRAM all map their SRAM in the lower 32 kB.

And then you suddenly bring in ROM size :-) Again, with your knowledge about the SNES it may be obvious, but is a big question mark to me. Yes, you already mentioned that the upper 2 bits are based on ROM size. However, with your latest logic the ROM size determines the lower 6 bits as well. Why is that?

revenant1 wrote:
dbjh wrote:
@revenant1: What happens when you upload Super Mario All-Stars with your upload tool?

I'll give it a try after work. I'll also update my docs/scripts and make sure that the SRAM settings for large LoROM games are documented/handled properly.

e: Tried it, SMAS uploaded and played without any problems. Also made some small corrections to my docs/scripts.

Thanks for testing. Really strange that Trenton_net is experiencing that problem. The only obvious difference is the buffer size (512 bytes in your code vs. 32 kB in uCON64).
Back to top
View user's profile Send private message
revenant1



Joined: 25 Mar 2009
Posts: 14

PostPosted: Wed Jun 20, 2018 9:30 pm    Post subject: Reply with quote

dbjh wrote:
Is it possible at all to distinguish between open bus and unmirrored ROM?

It is possible, but I don't think it was something that the UFO firmware authors actually considered.

dbjh wrote:
Why is 0x15 not always 0x20? It is after all more restrictive than 0x10. Or can the SRAM of LoROM games using 2 or 8 kB SRAM be mapped somewhere in the upper 32 kB?

And then you suddenly bring in ROM size Smile Again, with your knowledge about the SNES it may be obvious, but is a big question mark to me. Yes, you already mentioned that the upper 2 bits are based on ROM size. However, with your latest logic the ROM size determines the lower 6 bits as well. Why is that?


For LoROM boards that support only 2MB of ROM or less, the ROM is mapped to $8000-ffff in banks $00-3f. For boards that support larger ROMs (like Super Metroid), it is also mapped into $40-7d, which overlaps with the banks where SRAM is mapped (normally $70-7d), so these boards only map RAM to $0000-7fff to avoid hardware conflicts. That's why the ROM size matters. (Hopefully this makes sense, sorry for being vague about it / forgetting to mention it previously - it had actually completely slipped my mind at the time).

Games with more than 32k SRAM all use boards which also only map SRAM to the lower 32kb, even when the ROM is not very large (e.g. Dezaemon has iirc 512kb ROM and 128kb SRAM). Using 0x20 for these will also make sure that SRAM for these games works correctly.

Most other LoROM boards map SRAM at the entire $0000-ffff range, and can access it at both the lower or upper 32kb ranges. Using 0x10 for these makes sure that both ranges work correctly.

I think the UFO firmware guesses 0x20 for games with >= 8kb SRAM just because many of them (according to higan's game database) sometimes use boards that only map SRAM to the lower 32kb range. As long as the ROM is also only 2MB or less (like SMAS and Uniracers), then using 0x10 will still also work for these games and will match the behavior of current emulators and other flashcarts.

Anti-piracy stuff, from my experience, only really cares about the amount of SRAM available, rather than where it's mapped - as long as everything's where the game expects it to be then there shouldn't be any issues.

Hopefully that info helps, let me know if anything is still unclear.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
dbjh



Joined: 02 Aug 2003
Posts: 167

PostPosted: Thu Jun 21, 2018 2:32 am    Post subject: Reply with quote

That was a great answer! It all makes sense now. Thanks a lot for your effort.
Back to top
View user's profile Send private message
revenant1



Joined: 25 Mar 2009
Posts: 14

PostPosted: Thu Jun 21, 2018 4:11 pm    Post subject: Reply with quote

Glad to hear Smile My brain's been all over the place trying to manage different projects lately, so sorry if my earlier posts seemed confusing/inconsistent.

I'll try to finally give USB uploading a try with ucon64 sometime soon to see if I'm able to reproduce the issue Trenton_net was having with SMAS (or if I am able to experience it with any other games).
Back to top
View user's profile Send private message Send e-mail Visit poster's website
dbjh



Joined: 02 Aug 2003
Posts: 167

PostPosted: Tue Jun 26, 2018 3:17 pm    Post subject: Reply with quote

Trenton_net performed further tests with a build of uCON64 that uses a transfer buffer of 1024 bytes instead of 32768. However, that didn't change his results -- the first byte of the ROM data is consistently 0 after having been uploaded.
So, I was wondering what backend your tool is using, libusb 0.1, libusb 1.0 or OpenUSB? And if it isn't libusb 0.1, what happens if you force PyUSB to use the libusb 0.1 backend?
Another noteworthy thing is the following:
Trenton_net wrote:
Undocumented secret, but if you hold down the "A" button while the UFO is powering up, you'll enter directly into USB transfer mode (except the GUI colours look different). In addition to the colour change, the UFO will also attempt to do a checksum on the files after the transfer is complete (which doesn't happen normally). I will note that for the roms that I went out of my way to checksum, they all seem to pass just fine under UCON64, but fail when the UFO tries to check them.

I am curious what happens with your tool.
Do you have any ideas how to tackle the issue with the first byte of Super Mario All-Stars being 0 when using uCON64 (libusb-0.1)?
Back to top
View user's profile Send private message
Trenton_net



Joined: 29 Jun 2006
Posts: 233

PostPosted: Fri Jun 29, 2018 12:33 am    Post subject: Reply with quote

@dbjh: Yep, that feature is kind of strange. I don't know why it does a checksum there, but not when you try to send USB data normally. Another strange bit is that I don't think it even waits for your client to start transferring data. It will go into that loading screen, but will only stay there for a few seconds. if you miss your window to start transferring, your out of luck (^_^);
Back to top
View user's profile Send private message
revenant1



Joined: 25 Mar 2009
Posts: 14

PostPosted: Fri Jun 29, 2018 4:58 am    Post subject: Reply with quote

dbjh wrote:
So, I was wondering what backend your tool is using, libusb 0.1, libusb 1.0 or OpenUSB? And if it isn't libusb 0.1, what happens if you force PyUSB to use the libusb 0.1 backend?

I finally got around to testing the build of ucon64 that you sent me, and I was able to reproduce the issue with SMAS. At the moment I'm also using libusb 0.1 / libusb-win32 (via Zadig), so it's strange that they both have different results when transfering the actual ROM.

Try changing the dummy writes at the beginning/end to actually transfer 0 bytes each instead of 1. I wonder if the extra null byte at the end is being written over the intended DRAM contents by the firmware or something.

Trenton_net wrote:
I will note that for the roms that I went out of my way to checksum, they all seem to pass just fine under UCON64, but fail when the UFO tries to check them.


SMAS (and others) passes the checksum when I send it with my tool, but if it fails when sending with ucon64 then it might also be caused by the extra byte getting written at the end.

By the way, there are two other similar undocumented buttons: holding B attempts to load "SFCUFOSD.UFO" from the SD card, and holding Y will dump a cartridge to DRAM (and checksum it).

I'm going to be out of town for a week so I won't be able to test (or build) anything for a while.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Trenton_net



Joined: 29 Jun 2006
Posts: 233

PostPosted: Fri Jun 29, 2018 3:48 pm    Post subject: Reply with quote

@revenant1: Thanks for trying it out. I'm glad I'm not the only person seeing this issue! Your suggestions sound promising as a lead, since the only other idea dbjh had was that maybe we need to throttle our transfers to be a bit slower, so I was going to give that a try over the weekend and see.
Back to top
View user's profile Send private message
revenant1



Joined: 25 Mar 2009
Posts: 14

PostPosted: Sat Jun 30, 2018 4:59 am    Post subject: Reply with quote

I had a bit of free time before I leave, so I modified loadrom.py to send one byte instead of zero bytes at the end and got the same result as with ucon64 (i.e. the extra byte overwrites the first byte of the game in DRAM). Changing that in ucon64 should also fix the issue there as well.

(I assume the UFO firmware sets up the DRAM address mapping before receiving the rest of the actual ROM, so extra data may just be written to a mirror of the previously written game data, depending on the size/mapping of the actual game)
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Trenton_net



Joined: 29 Jun 2006
Posts: 233

PostPosted: Mon Jul 09, 2018 3:50 pm    Post subject: Reply with quote

revenant1 wrote:
I had a bit of free time before I leave, so I modified loadrom.py to send one byte instead of zero bytes at the end and got the same result as with ucon64 (i.e. the extra byte overwrites the first byte of the game in DRAM). Changing that in ucon64 should also fix the issue there as well.

(I assume the UFO firmware sets up the DRAM address mapping before receiving the rest of the actual ROM, so extra data may just be written to a mirror of the previously written game data, depending on the size/mapping of the actual game)


@Revenant1: I was able to test a fixed version of Ucon64 that sends 0 bytes instead of 1. Initially it was set to 1 because usually when you tell a C programmer to send an empty string, they assume it does not include the termination byte. Now that the termination byte is omitted, the game transfers over just fine! Using the UFO's built in CRC checker, I can also confirm that it passes integrity checks.

However while some games now pass the UFO's CRC check (from our sample list), I noted that three of them do not. They are as follows:

Earthbound
Front Mission: Gun Hazard
Breath Of Fire II

That means while they do transfer and seem to boot correctly, they'll likely crash at some point in the future since the actual ROM images fail CRC (while ucon64 gives them a clean bill of health). I took some ROM dumps for Dbjh to look at, but maybe you can try to transfer those games using your tool and see if the UFO can CRC them correctly. I'm not sure why these games fail, while others like Super Mario All Stars now pass. <Shrugs>
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    ToToTEK.COM Forum Index -> Copiers and Hardware All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group