Combined Lab Updates Headed Into Summer 2023
Last Updated: 2023-05-23 08:30:00 -0500
Today we’ll talk about something changing around the lab, dive fairly deep in to the state of our currently-flagship project, PETI, and drill down into some side projects for the back half of the year. We start in with bad news and then it just gets better and better as you go, so tear the proverbial bandage off under the cut.
Ending Twitch Livestreams
I’m going to get the worst-sounding part of the update out of the way first: I’m putting to bed the idea of routine Lab/Development livestreams on my Twitch channel. I originally started the practice for a few reasons, namely that I thought it would keep me honest in terms of encouraging me to work more often on “lab” business, give me a way to share updates without much extra overhead by simply discussing developments as they happen, and that it would put me in a position to funnel the rather limited revenue as an Affiliate into the Lab discretionary fund, which beyond what I can wring out of the budget from my day job more or less entirely funds the lab.
In the end though, this particular experiment was a collossal failure. It turns out livestreaming is its own kind of overhead, requiring me to spend the entire time I’m writing PETI modules split-braining as an entertainer. While it’s been helpful on more than one occasion to have a knowledgeable audience right there with me to help with debugging, it’s also just lead to a fair number of blunders through not letting me devote my full attention to what I’m doing. It also produces a tendency to focus on only what I would consider the most watchable part of the project, and take attention away from more important work.
This doesn’t mean livestreams are going away. If you know me through Twitch rather than from the Lab first, don’t panic. I’m still going to be playing puzzles, building games, and the like on the channel, I’ll just be doing it less often, and moving the schedule to the weekends.
What’s New With PETI
On happier notes, a lot has changed with PETI since the last time we wrote anything down. The PRNG/LCG module has gotten a few adjustments that make it both more performant and more reliably unpredictable, just a few weeks ago I started the process of paring down the memory usage to free up some limited device RAM by “globalizing” some more system state, and then I went ahead and promptly broke the main repository for the project.
Improvements to the PRNG
A short while ago I wrote about our attempts to get some randomness into PETI. I concluded that post with a complaint about three key points in the PRNG’s operation that I didn’t like. The key problem, though, was that of periodicity. The numbers generated by the PRNG were forming a pattern that would, usually, fully repeat itself every 10-15 draws (depending on some other parameters that changed on each boot).
I’ve since largely fixed this issue. The solution was to make a few major changes that addressed all three of the original concerns:
- The periodicity problem in Linear Congruential Generators has a lot to do with their boundary conditions, so I changed those boundary conditions to values yoinked directly from a historical example 16-bit LCG. I can’t remember which historical example but it was likely the one used on the ZX81. This change alone, with no other changes, immediately resolved the periodicity issue to some new value “higher than I can be bothered to count” according to my notes.
- I changed the behaviour of
RNG_getSeedWordand it appears to be doing a bit better at picking a random word out of the MSP430’s factory “random” number to use as the seed for the LCG. This is purely a matter of my own personal cares when it comes to randomness and isn’t actually needed to get the pet to be humanly-unpredictable (which I believe I referred to earlier as “random enough for casual gaming”).
- Replaced all the math in the LCG so that it’s using integers instead of floats almost everywhere. The module still exposes a function to other modules called
RNG_drawFloatwhich still spits out a float from 0-1, but we’ve changed how it works to address my earlier concerns. It now correctly throws out the least-significant byte of the RNG result and then does some division to “normal” the value between 0-1. In my testing this gives a fairly good range for the RNG, but I haven’t tested it as exhaustively as I’d like so we’ll be keeping an eye on it to make sure.
Memory Management Streamlining
Recently, in a valiant effort to make myself do anything other than what I’d actually set out to do, I wasted more or less an entire PETI livestream doing my damndest to squeeze a few bytes back out of memory. I’d had a sudden brain wave that because of the hackiwsh way scenes are boilerplated out in PETI, every single one (at this point, at least 15) was creating three or four bytes of allocated memory they didn’t really need to create.
The issue wasn’t necessarily the scene’s own functions: this was more or less unavoidable. The deeper problem was to be found in things like page and frame counters, cursor positions, pagination variables for things like the menu generator and its specialized derivatives, and exit flags.
By globalizing these values we’ve carved out a grand total of - by my math, about 20 bytes. I know, I know, totally worth the three hours I spent on it. Working on it.
Fixing (Mostly) Our Horrible Input Handling
The single biggest change up to this point was actually going through every single scene in the game a second time and changing the input handler. This was intimidating - it felt like a major refactor - but it was actually doable pretty quickly. But why do it at all?
In PETI, as with many other systems, there’s actually a bit of an interlock between handling human input and the display refresh rate. This makes a degree of sense, right? If you want to scroll down in a menu and press the down button (B), you won’t see the cursor move until the screen refreshes.
There are, however, two major problems with this in the
v 0.1.0 firmware:
- PETI’s display update rate is about 2Hz. While gaming consoles tended to run near 60Hz in NTSC and modern computers can run well over 100Hz, fully refreshing the screen twice a second is trivially perceptable to most humans. An input event at the right time could theoretically trigger an extra screen refresh, but statistically it’s actually most likely to happen while the display is already calculating its next update, doing nothing for you. This is because the display update rate and the main program rate are intrinsically tied together. The display “updates” at the end of a core gameplay loop that updates most of the game’s state machine, right before going to sleep, but unsururpsingly, computing that update actually takes most of the time there.
- PETI can only handle one press of each of its four buttons per display update. This is because a foolish mortal (me) decided to try and cram all of the input events since the last time input was handled into a single byte of memory. I had this whole system where the upper nibble was four bits of toggling flags sort of like latches and the other, lower nibble was whether or not that button had been pressed “that go around”, and this was all legacy code from way back in 2019 or so when I was just trying to get input and the display working together. This would create a “dropped input” feeling where sometimes you’d only scroll once even though you’d felt you’d hit the button a few times.
What I’ve done to “fix” this, albeit temporarily, was to replace the input byte with an 8-byte list that stores up to 8 button presses since the last time any scene’s “handle inputs” function was called (which would bleed the list back to empty). FIFO operation, like a queue.
This means that if I’m on a menu and I somehow manage to press the B button 4 times between screen refreshes, the cursor will now scroll down by 4 positions on the next refresh. This also means it’s not perfect. While this is certainly miles better and worth every minute of the roughly two hours it took to implement it and refactor all the existing scenes that had handle input calls, it’s still perceptually weird. As humans, we don’t really expect to see the cursor just teleport by several positions like that. It makes the menu feel stuttery.
As I see it though, there’s only two real solutions though:
- Implement logic to “scroll” cursor positions one at a time on successive screen inputs, which would slow menu handling back down to a crawl. While this is still probably “better” than the old feeling of dropped inputs, it’s a regression. Can’t have it.
- Speed up the core gameplay loop and screen refreshes. Right now we can’t quite do that. Two Hz is about how fast the display can be refreshed at our current master clock speed, which is about as high as we can set it. The reason for this is a major outstanding problem in PETI: the display refresh calculator isn’t working right.
I’m sure I’ve written about this problem before, but if you’re new, here’s the gist: updating the SHARP mLCD is either computationally expensive or else takes a long time simply on the device side. If you try to update the entire screen at once and the screen is quite full, at the old master clock speed of 8MHz, you could actually watch the scanlines run down the display. Increasing the master clock speed helped with this a bit, but it still takes time to compute all 128 rows of the display, send it over one line at a time, and intersperse the “commit” command word and address bytes and the like.
There’s efficiencies to be found in display handling for certain, along these lines:
- The Frame Comparison Logic is Broken. I know from watching screen capture data come in over the logic analyzer that we’re rewriting rows when we don’t even have to, or at least I strongly suspect that we are. DisplayFrame is meant to be compared to PreviousFrame, a clone of the last DisplayFrame, and only rows that would have changed should get updated. This seems not to be happening. We seem to be sending rows that didn’t change, and this is a huge waste of time.
- I miscalcuated the amount of memory needed to hold the full display frame state as a bitmap. While I still wouldn’t want to manipulate the display this way - it would involve re-inventing how I think about display manipulation then refactoring every single scene in the game to use the new logic - I don’t know that it’s necessarily wrong to do so. I fundamentally misclaculated how much memory this would take and thought it massively exceeded the available memory on the device. In truth, it would take around two KB. Padding for a few extra bytes here and there for a command, it may be possible to send the whole screen update at once. Whether that’s more computationally practical or not is… beyond my skill to work out on paper. It’s worth experimenting with, possibly.
- Even if we can’t do the whole thing at once, it might be faster to do whole rows at once. The display for PETI is textmode, meaning that each DisplayFrame “line” actually controls 8 or more rows of the LCD. We iterate over those lines one row at a time and update the display after each. But the mLCD supports multi-row writes. For marginal overhead it could be possible to at least send the full row’s update at once. If the screen scanning effect is coming from actual “mechanical” delays in the screen, this could reduce the apparent flicker.
Still and all, I strongly suspect we don’t actually have room, computationally, to update the screen more often than we do. I’m quite certain I had settled on the current limiter logic of twice a second (more or less) for a reason. What I haven’t done though is experimentally benchmarked this. It would be interesting to add some sort of debug scene that runs a program main loop and then displays the wall-clock time that took somehwere. If it turns out we’re somehow taking a relatively small sliver of time to do this, then I could very trivially improve the screen speed without any noticable human impact by just… changing the divider on the clock that governs the interrupt that runs the main loop from out of sleep.
Yes, we’d use more power, but the main power savings in operation is actually going to be the low-interaction sleep mode and not the fact that we sleep the main program loop after every display update until it’s time to get started on the next one.
Breaking the Main Development Repo
As a result of all these changes, and the problem I alluded to earlier when talking about only wanting to stream the “watchable” bits of working on PETI, the branch structure of the repo has become a proper mess. I have a named functional change branch (named in a convention that’s totally undocumented because we also have a major documentation gap now) that has a whole whack of flaws, from having a supercommit last week that works several changes in all at once, to somehow being for several new features even though that’s a bad practice and would be verboten if the relevant docs were up to date, and somehow simultaneously being head of and behind master. Also somehow master was created as master instead of as main.
Untangling this will likely take a bit of time, but it’s nothing I haven’t had to deal with before. It’s just worthwhile to note, and why you won’t see most of these changes on
master if you download and flash the firmware today. You want the
f/version0-1-1 branch, which probably won’t be version 0.1.1 when released. Deciding what to do about that version numbering is going to be the hardest part.
What’s to Come For Summer and Autumn?
PETI Development Kit General Availability
It’s my goal over the summer to get the PETI development kit onto Tindie so that folks can get their hands on it, and I’m going to sell it in three forms: boards alone, boards with parts, and the boards and parts as a kit with the one surface mount component already attached. What’s been slowing me down is a few major things:
- There’s not technically a BOM created for the development kit. I mean yes, there nominally is, and with two exceptions everything on both boards are jellybean components. But it’s not compiled as a single document anywhere a person can read it, because;
- The documentation gap also includes the PETI development kit service manual.
Also, I was on the fence about listing them at all because a quick review of the policies from TI mean I can’t just order the launchpad and booster pack modules that go with the dev kit and ship them out alongside. I’ll have to be very clear in the listing that you need to supply your own for both because of this requirement.
But, that being said, getting these kits listed online (and therefore, finishing the service manual) is top priority for the lab at the moment, just behind disentangling the repo itself.
PETI: First Firmware Version with Implemented Minigames Support
Once I get the kits out the door I want to finish what I’m going to benchmark as version
0.2.0 of the PETI firmware. This is comparatively pretty trivial - I just need to get the first proof-of-concept minigame scene finished. Parts of it already technically exist, but basically, the game is going to be to play Rock Paper Scizzors with your pet, first to 3 points wins. Most of the logic already exists in my head and the core pieces are there, so I just have to sit down and code it. DEpending on how much energy I have, I may even release the firmware in paralel with listing the development kits for sale.
Side Project: Experiments in Modular Synthesis
I have a pretty major side project on the go. I originally started it as a distraction, then justified the expense of it by calling it a streaming tool, but the more I think about it the more I realize it’s mostly just a toy for me. I’m building a Eurorack modular synthesizer from kits and a few planned DIY modules. It’s been teaching me a lot about analog electronics and I’ve really enjoyed the process of playing such a synth virtually through the open-source implementation/simulator thing, VCV Rack.
Assembling this thing’s modules - and possibly even designing one from scratch - will occasionally feature on updates around the site and the odd stream.
Review of Tapestry
Tapestry hasn’t had an update in years, and its behaviour is starting to freak me out a bit. It seems to hang at an early, intended-to-be-rapid step and just never progress. I will admit that I’m using it on a larger data set than the usual test corpus, but… not that much larger.
I would like to take some time this summer to review that. Nobody’s opened a bug against it, but I’m not sure that anyone other than me uses it. While I’m not actively developing or improving the damn thing, I want to at least keep it in a usable state, and I appear to have broken it somehow, so I should at least fix it.
If you wanted to show your support financially, for free tools and toys like PETI, your best avenue is via my Github Sponsors account or by making a one-time donation to Arcana Labs via Ko-Fi.com or through other avenues detailed here. Supporters also get access to a special patrons-only section of the Arcana Labs Discord Server as well, and new bonuses are soon to be introduced on the github side!