Magic with PencilKit

The SwiftGame is coming along fairly well and I am still working on attack mechanics. My game’s core attack mechanic is going to be influenced by one of my favorite DS games. I have always been a hand held gamer and the Nintendo DS was a day 1 purchase for me. I was immediatly drawn to the gaming device’s ability to track hand drawn gestures, and the early games really flexed creativity. Coincidentally the DS was also the firstonly console I got from Blockbuster.I had to use BB because my favorite video game rental place, DoorStep Video, was shut down. That place was way ahead of it’s time because it offered home delivery, but that’s a post for a different time. This is one about over engineering. It’s a pickle!

Lost Magic is my second favorite Nintendo DS game. First is Pac-Pix, both of which got an average 6.0/10.0 review scores so critical acclaim we begin with. Lost Magic had a really cool attack mechanic where the player could use the DS’ stylus and draw a spell to use as an attack. In Lost Magic you had custom shapes, for instance a triangle with one side missing, and each shape corresponded to a different attack. To bring that same functionality to iOS would only require CoreML and PencilKit.

All of the heavy lifting for drawing is handled by PencilKit, and there is a wonderful representable to make working with PencilKit in SwiftUI easier. Staggering how easy it is to get a drawable surface and all of the drawn points. At one point the only option for something like this would have been tracking touches or trying to handle a custom gesture. As you can tell from my awful mouse drawing skills even PencilKit is great.

The real magic begins pun very much intended when PencilKit is attached to CoreML. Apple provides CoreML models, one of which is the MNIST which can be used to detect handwritten, numerical digits. There is some slight manipilation of the output from PencilKit required to match the model’s training data, hence the black background with white text in the above screenshots, but outside of that only an image reduction is required to get an accurate reading. I also had to play with PencilKit’s canvas tool’s width. It is currently set to 55.