Lab Update: Q4 2022
Last Updated: 2022-11-23 07:30:00 -0600
Turns out the autumn was mostly a low point for us, but be that as it may, some things actually did get done here in the labs. There was work done on all of our major projects as well as new possible projects and improvements identified, not least of all being an action plan for PETI and a possible “feature” update to Tapestry. Click through to see what’s been going on and what we’re planning over the winter.
PETI Update: Hybernation is Good for Growth
The summer and autumn were sort of a slow period for PETI development, without any of the big leaps and bounds of the front half of the year and the end of last year. That said, a few key milestones were reached, even if they didn’t look especially glorious in their individual pieces.
PETI Has Escaped Its Containment
People who have stopped by the Twitch channel during lab streams have no doubt heard me more than once express my overall displeasure with the linux version of TI’s IDE for the MSP430, Code Composer Studio. In truth, it’s reasonably usable, and for all my whining it was always more usable than the percieved effort of learning to make do without it.
After one especially egregious round of its usual bugs, however, I decided I’d had enough and set out to do something I’d put off for entirely too long - getting out of it. In truth, I can (and fully intend to) write a whole article in and of itself on what was involved in getting it ported out (and what it took to get each of those things working), but as of the most recent commits on github, there’s enough of the infrastructure there to compile, flash, and debug the PETI firmware from any IDE you choose. In summary, this involved:
- Identifying and learning to install and use CLI tooling that would handle the flash and debug operations
- Learning enough about available MSP430 compilers to select one for use and learning how to interact with that, and;
- Learning enough of
makesyntax to glue it all together without having to go so far as to use a dozen nested makefiles to “get there”, which seems to be the largely standard practice.
The end result is that there’s now a single makefile that references one other local config file, and now when working on peti you can just “make flash-target” and have your new firmware rapidly compiled and slapped onto the device. This is letting me use a different code editor with more responsive syntax highlighting, and I have to say so far I’m really enjoying the change.
Even if it did take me two months to get it working.
Correcting the Power Management Issue
As most of you know, there’s an open issue in PETI which concerns its complete lack of power management in the Development Kit hardware, beyond “two AA cells will not exceed the rated voltage of the 3.3V-nominal rail, for any chemistry I’m aware of”. This was… silly, so we fixed it.
I’m currently sitting on a draft set of changes to the schematic which come from recent tests using a TPS60204 “charge pump” DC-DC converter IC from TI to step voltages up from any input voltage all the way down to about 1.2 V up to 3.3 V. This allows the use of nickle chemistry rechargable batteries with the device when it’s in cordless mode, and I’m excited for that because it’s been an open bug for a while.
I kind of can’t believe I didn’t write it up earlier, but at one point over the last few months I implemented the world’s laziest Mersenne Twister formula into PETI, giving the codebase an
entropy.h import under
game that you can include to be able to pull a random float bounded from 0-1 into any other module.
The problem is that the current implementation seems to be extremely periodic. Given the nature of how the RNG is called this is still going to be hard to exploit, but I’d like to make it more reliably unpredictable before I really call this work done.
Looking Forward: A Pet You Can Play With
Sometime in the new year I want to really take the time to finally implement a minigame in PETI. It’s a major component of what was called for in the project spec, but up until now the ability to increase the “fun” stat of your pet by playing games with it is entirely absent from the game. It’s probably the heaviest remaining lift in the entire MVP spec for the toy, so I’m looking forward to getting started on it at long last.
More immediately, though, there’s going to be new boards to mill, and if I’m happy with they come out, I will finally, finally be able to release the Development Kit and its service manual as something available for other people to play with. I’m likely going to have a small number available to purchase as kits owing to the economics of ordeirng such things, and of course the designs will always be available.
Tapestry Update: “Is this going to take much longer?”
A recent unfortunate discovery lead to me reconstructing my Tapestry configuration file locally, and adding several new data sets to it. This dramatically increased the size of the output files and made it clear to me that this script is still not that performant.
As it stands, most of the delay in the process, beyond simply “it takes time to read and write files on a drive”, is to do with an early step in the process known as “build_ops_list”, the phase where it determines which files it wants to back up and gets checksums for all of them.
A weird quirk in Hash Speeds
The most recent version of tapestry,
2.2.1, uses the SHA256 hash algorithm to compute a file contents hash for each file on the backup list. This hash is used both during backup creation and turing
--verify calls to ensure that the file contents were not changed during the handling of the files by Tapestry; a subtly-different type of integrity check than that provided by the overall backup signature validation done during recovery, which is more for detecting tampering.
Thing is, SHA256 is actually not that fast, in part by design. There are well-known faster hashing algorithms that we could use to speed this process up relatively easily (it’d be a one-line change in the application - maybe slightly more, if we want to maintain backward compatibility). There’s just one problem: under my admittedly crude testing just yesterday, I couldn’t get
any of the supported
hashlib algorithims from python’s standard library to out-perform SHA256.
Now, granted, my local python3 install and its attendant libraries are certainly a few months out of date, or even more - I believe I’m using 3.6 and I hear 3.11 is now considered current. It may be that simply upgrading Tapestry to work with newer python would let me also use faster hashes. I just don’t know. Further testing here is needed.
Potential Savings in Parallelism
Short of a faster hashing algorithm, the only meaningful way to improve the performance of this step in tapestry would be to give it a parallel operating capability, similar to all the other steps in the process. This would actually likely be a significant performance improvement now that hashing is part of this step - it was only done serially previously because we didn’t include this kind of hashing in earlier versions of the program.
While that won’t be rocket science, though, it’s a much heavier lift than simply changing the hash. Unfortunately though, if I can’t get the performance increase I need out of changing the hash, I don’t really have a choice.
Two Possible Futures
Ultimately, Tapestry has two potential future states from here: status quo and enhancements.
It may be the case that the investment in time these performance enhancements call for is worthwhile, and there’s precedent in earlier tapestry development that we can draw on to test, in reality, whether or not the proposed changes would actually improve performance. If they do - and the lift doesn’t look miserable - then concievably we’d be looking at a
2.3.X release that includes these performance enhancements, and maybe any other low-hanging fruit from the triage board.
Alternatively, if the testing comes back negatively, it’s possible that tapestry remains in its current, less-than-optimal state, and we stay on the current track of “I’m only fixing major bugs and identified security problems in this software”.
It’s possible that letting Tapestry die is actually the right move. While the project itself had admirable goals, in permitting the average person to use low-cost web hosting or even “toss these files on the shelf in your office” as an offsite backup strategy, it’s also a generally-inefficient way to handle backups. Taking a snapshot of every file every time you run the script leads to a large amount of data needing to be held. Using GPG for file security was an interesting idea, but perhaps it’s not sufficiently performant to justify itself.
It’s possible that the “right” way to handle Tapestry and making it more performant is actually to create its successor - or, depending on my mood, identify its replacement.
Model 2: Nearing the End of the Line
Over the summer, we did exactly no work on the first “completed” Secret Project and taking it over the real finish line. Now, it’s closer than ever to completion.
The Incredible Expense of Needing Help
The biggest delay in Model 2’s completion is down to cost. As I’ve discussed previously, the back plate and “closed” case layer both needed to be redesigned by thickening both by a couple of milimeters. This is… nontrivially costly, given the method of manufacturing I’m bound to without the tooling to create the pards myself. This will also call for ordering new fasteners. I’m happy to say those parts are now on order
Measure Twice, Cut Seventeen Times
With the new parts on order there are only a few small remaining obstacles, all of which have the overall theme of “measure next time, man”:
- The new encoders with the correct body dimensions are in my hands and waiting for me to install them. Such installation is non-trivial because of the floating design of the electrical harness, so I’m putting it off until I feel up to it.
- I still don’t have caps/knobs for the encoders that actually fit. This is a failure of my own ability to do parametric parts search and I have put off purchasing them because I didn’t want to submit a fourth order in the same project; I’ll order them when I order the parts I’ll need for the PETI dev kits.
- Actual assembly of the new back plate, which will be accompanied by once-again touching-up the soldering since I’ll be cleaning up the guts of the board anyway.
And then, of course, the final missing writeup can be released and I can create the project page on the main section of the site, which will require developing some new templates. Funnily enough, that’s probably going to be the most fun part.
Goetic Vitromancy: The Language that Never Was
Last quarter I announced what the new secret project would be: an interpreted language with an extremely sugary syntax called Goetic Vitromancy. This quarter, I’m announcing I will very much not be doing that… just yet.
Nothing is more esoteric than not existing
Goetic Vitromancy was intended to be, essentially, more sugary and eventually-compilable Python. All of its specified features were just things I liked about python. There’s nothing inherently wrong with that. Thing is though, it was pretty much doomed from the start as being used by me, me only, and maybe a crazy person or two from the Lab discord.
Such obscure programming languages are part of a family known as Esoteric Languages. I like the idea of esoteric languages. I like what they show about the mind of the people who develop and use them. The thing is though, while they’re not unheard of, esoteric languages that are just “really common scripting language with a coat of paint on top” tend to be a little… boring.
Simply put, I just don’t feel strongly motivated by novelty or challenge when it comes to Goetic Vitromancy, and I won’t be starting the actual work on that project any time soon.
Someday/Maybe and the Schrodinger’s Project
I think I’ve talked about this on this blog before, but I’m a (heavily variant) practicioner of a somewhat older productivity/project management style called “Getting Things Done”, more easily googleable by its nickname, 43 Folders. One concept in that workflow is the idea of a “someday/maybe” list where activities can go if they’re appealing, but not relevant enough to warrant immediate action or even concrete planned future action.
I’ve moved Goetic Vitromancy into that pile, along with all of my notes and documentation for the project. It wouldn’t be the first time I shelved something and picked it up potentially years down the line, so I’m confident that if I change my mind about the project, I can pick it up with a minimum of state loss.
Social Media Shuffleups
I hear there’s something up with twitter. As always, check the contact page above if you want to know where else to find me. That said, I also hear there’s this flashy new tech called RSS
Twitch Lab Streams To Return?
I hear a rumour that our twitch channel has been getting more use lately because of changes to Twitch’s affiliate policies making it more realistic to actually hit the twitch payout theshold. One of the things I plan to use the channel for more often now that several other problems have been solved is lab nights. PETI development and other side projects is about to accellerate dramatically.
This also isn’t the only video project coming down the pipe. Someone has started this terrible rumour that when the PETI development kits are ready a… curated, edited video promoting it might be released. Hmm.
The New Secret Project!
There’s a new secret project to replace Goetic Vitromancy. As with all secret projects I won’t delve deeply into the details, or blog about it much until it’s done. It is, however, a very large project. Larger than PETI, and likely to take a few years to complete even the minimum viable version of it, in much the same way. This means that once PETI hits its MVP, the new project is likely to become a public project much like PETI. For now though, the usual rules apply: you have to be a Github Sponsor or a Twitch Subscriber and a member of the lab discord to have access to it.
That said, I think it should be really enjoyable: I’m caving to the times and developing a small video game. Or at least, the proof of concept of one.
If you wanted to show your support financially, for free tools and toys like Pyminder, Tapestry, and PETI, your best avenue is via my Github Sponsors account or by making a one-time donation to Arcana Labs via Ko-Fi.com or through other avenues detailed here. Supporters also get access to a special patrons-only section of the Arcana Labs Discord Server, where we talk about the ongoing and upcoming super-secret project.