Pulling a Moore

One of the few things everyone seems to agree on is that code duplication is evil. But. Admit it, you’ve done it too: For whatever contrived reasons you take a piece of source code and copy and paste it into some other project. Problem solved, on to the next one. The proper way of doing it would, of course, be creating a library to avoid having the same code in two separate projects.

I’m not so sure about that anymore.

One of the big problems I see with library design is how to make a library a) general enough to be useful in lots of situations, b) simple to extend if need be, without breaking backward compatibility, and c) simple to use.  The fourth requirement for libraries, not wasting too much space, seems to be a thing of the past by now, so let’s concentrate on the other ones.

If, as an example, I had to access a CANopen network from a PC, I would start looking for a library that allows me to connect to a CAN interface and proceed to upload and download SDOs, send and receive PDOs and whatnot. The problem here is that such a library would have to support the whole CANopen protocol to be considered useful, as you can’t just implement a small part of it — it’s either all or nothing. Adding the extensibility requirement, you can throw simplicity out of the window. Or can anyone out there show me a CANopen library that is easy to use, well documented in a human-readable language, and stable?

If you need a solution to that problem FAST, you can do two things. The default would be to suck it up and use whatever library seems most suited. The other approach would be what I have come to call the Forth path, or “Pulling a Moore”, from Chuck Moore, the inventor of Forth: If the problem is hard, just modify it to make it simpler. Detractors might call it “cheating”. It’s related to the Not Invented Here Syndrome, which describes the perceived need to Do It Yourself even if someone else has already solved the problem. But what if none of the ready-made solutions satisfy you right now?

So if CANopen is too complicated to just use in your programs, look at the problem as it looked like before you started to solve it. We have a PC that will be connected to a CAN bus. No problem so far, we have an interface that allows us to send and receive raw CAN messages. Now we know that a CANopen message is, basically, just a CAN message with a special ID and payload. The ID tells us what kind of message we are looking at, and, most importantly, how to react to it.  If some other node asks you to read a value from your Object Dictionary (OD), which is a kind of database that stores information about your node, you had better send an answer to that request. But as we are a PC that will query other devices, and we KNOW that those other devices won’t start any queries of their own, we can simply omit that part. In fact, we can replace our OD with a few variables. If all we need are heartbeat messages, SDOs and PDOs from the other devices, and we KNOW that heartbeats and PDOs will come in without the need to ask for them, we can skip a big part of CANopen.

What we need to do now is write a few functions that encode and decode SDO upload and download requests, the respective response messages, and PDO and heartbeat messages. Helpfully, our devices go to Operative state without us having to ask them to, so we can skip that part, too. Now all we do is listen for messages on the CAN bus and react to them. If a message is of no use to us, we quietly ignore it.

And lo and behold, problem solved. Weeks of looking for good libraries, experimenting, trying to decipher the few comments that were left in the source, visiting web forums, mailing lists etc., and now we solved the problem by simply… duplicating a piece of work that has been done before. By writing our own stripped-down CANopen layer, that won’t ever be useful to anyone else or in another situation. But boy was that easier and faster.

So there seem to be situations where using, or even writing, a library could make your problem worse. Sometimes, duplicating the parts of a solution that are relevant RIGHT NOW, RIGHT HERE, seems justified. Sometimes you don’t need to pull in 10’000 lines of code just to parse your command line parameters. Just loop over your argv entries and strcmp with “-flag” to set a variable. Done.

That’s the Forth way: Reuse ideas, not implementations. When you write the code, you can adjust it to do exactly what is needed, and nothing more. Simpler code, less configuration hassles. While superficially this looks like a lot of wasted time, I think I’ll use this method a little more often from now on. Like with that CANopen library.

But make sure to encapsulate such shortcuts in your code, just in case you do need a full CANopen stack sometime in the future. You know, with a custom OD containing a few thousand entries that describe some weird communication parameter, and some strange packet type that configures how some other strange packet (that you might receive every four and a half years) should be handled.

Oh, and put that custom CANopen stuff in a library. You might need it somewhere else sometime in the future.

Hello again, Delphi. It’s been a while.

A few weeks ago I had to look into an old program to add a few new features. I had written it in Delphi, a long time ago, before the age of free Versions of Visual Studio hanging around. As all my computers are running some flavor of BSD or Linux now, I fired up a VM, installed the old trusty Windows 2000 license I keep lying around and dusted off the Delphi 7 CD.

A few minutes later, my mind was blown.

It’s amazing what a fine development system Delphi used to be. After all those years of using C# and Java professionally, and playing with Python and Scheme, this old Pascal based language still rocks. Just like Win2k, by the way: It’s fast, stable, responsive, and does just what I need it to do. Right inside a 512 meg VM. So much for living in the future…

Back to Delphi: Say what you want about Object Pascal, I felt productive right from the start. All those old tricks learnt ages ago still work. The changes to that old source code were just too easy, it almost felt as if I had forgotten something. But the tests ran, and the client is happy. And I’m wondering.

What is it that makes good ol’ Delphi 7 still compare favorably to lots of development systems out there today? The language didn’t have Lambdas (newer versions of Delphi do have a stripped-down Lambda-like thing), there is no Garbage Collection, and the runtime and component libraries don’t consist of thousands of classes that do virtually everything for you. And the editing features of the IDE pale in comparison with pretty much everything out there today.

But. The visual designer is great. Even better than the one found in Visual Studio 2013, IMO. The components found in Delphi 7 Pro still make designing GUI applications a snap. Changing screen resolutions? No problem. Layout adjustments? Got it. Database access? Yup. Everything just works. And the language does have reference counting and RAII, something that lots of old school Delphi developers sadly don’t seem to know.

Can’t say good things about Delphi .NET, though. Or Kylix. Or all those ill-fated versions coming after them. Slow, buggy, unstable, bah. The job market crumbled. That was when I abandoned Delphi and went to seek out greener pastures in C# land. Delphi started to suck.

Until, well, now. I tried the XE5 version for a project at work and while I’m not exactly thrilled with the new IDE, it still has some of that old Delphi feeling. Sure, I’m a nostalgic. Memories are written in golden ink. But still, I think trying to emulate the Visual Studio IDE is a step backwards here. The resource hogging, the slow startup, the gruesome stuff they call documentation are Bad Things ™. Delphi used to be quick, snappy, with real documentation coming up at the press of F1. No online search, no links to Youtube videos banned at work. Real, fast offline help. Even PDF files with a complete language reference.

Still, the new XE5 and now XE6 do have a lot going for them. The language has Generics that expose the Java ones as the joke they are. You could call Object Pascal a cleaner version of C++ that allows you to do pretty much the same things, only simpler, and with a way faster compiler, and a fine default GUI library, and database access, and a nice IDE. Unit tests are there, refactorings work. If you do need C++, there’s C++Builder.

What sealed the deal for me was the ability to write your code once, and then compile down to an Android app. Or to iOS. Or old-school Windows, or OS X. Same codebase. Fast compiler. Native code, statically linked. No slow Dalvik shit. No need to try and bend HTML and JavaScript to do what you want. Want to access the native API? Conditionally compile a few classes, and you have hardware access way simpler than with a PhoneGap plugin. Developing an Android app in Delphi blows the ADT bundle right out of the water. And the apps start up fast. Just like Delphi programs start up fast on Windows. Compare that to the second .NET programs take even from an SSD.

So will I switch to Delphi? No. Object Pascal doesn’t stand a chance against C#: GC, real lexical scope, LINQ, and all the dynamic introspection stuff. Nuff said. And Visual Studio has free versions that are good enough for most smaller projects. The cheapest version of Delphi usable for professional development is way too expensive for my taste. But for mobile development at work, or for simpler utility programs that don’t have to be installed on top of the full .NET framework, I’ll use it.

At home, I’ll go for Lazarus instead. It isn’t as stable yet, but it has the good old Delphi feel. And it’s free in both senses of the word. And runs on Linux.

Looks like I’m a fan again.

The perils of having fun at work

As with everything, there are three main reasons why someone would become a software developer:

  • to make money
  • as a means to some other end
  • because it’s fun (l’art pour l’art)

The first one is the simplest: Programmers make good money. Becoming a programmer means being able to get a piece of the cake, too. The problem here is that if you write programs only because of the money, chances are you aren’t that experienced. If you treat it like a 9-to-5 job, and don’t even think of writing code when not at work, you won’t get deeper insights into technologies you aren’t directly confronted with at work. That’s why I’m not a big fan of all those year-of-coding efforts: People may learn pretty quickly how to stack together big pre-made components, but as soon as they need to dive a little deeper, they’re lost. There is a big difference between stacking a few Lego bricks and designing a skyscraper. On the other hand, for lots of jobs out there, that’s just good enough, and it often becomes a stepping stone:

The second kind of programmer got into it because they needed to solve some repetitive problem. Lots of programs originated from the need for a simple, specialized app to address some problem “until a real programmer can take care of it”, or because having a shitty, thrown-together app is still better than doing everything by hand. Good ol’ Visual Basic was what turned lots of people into programmers. From Excel macros to data base backed store management stuff, lots of small and large programs have been written, with horrible source code, but they work. They solve problems. And the people who wrote them had to sit down and think. They learnt something, even if they stayed right within their problem domain.

This is the kind of team I love taking over: People who know their code could be better, but who also know that they solved some problem by themselves. They’re motivated to learn more. And I love teaching. All that stuff you thought was boring? Think about it again. And again. And again. See how others are struggling with that boring stuff? Maybe you could teach it some other, simpler way. Maybe you haven’t really understood it, too. It’s a very interesting, sometimes humiliating experience to revisit stuff you hadn’t thought about anymore, only to find it more complicated than you remembered.

The third kind of programmer is often the next evolutionary step. Programming becomes a hobby, or even more. In the extreme case, programming is a reason for existing:

Thus spake the master programmer:

“After three days without programming, life becomes meaningless.”

From The Tao Of Programming

It doesn’t always have to be this extreme, but you get the point. For some of us, writing programs is simply plain fun. I like making a machine do my bidding. I like the fact that, barring hardware failures, computers do exactly what I tell them to do, which isn’t necessarily what I want them to do. The fun lies in figuring out how to solve a problem. If I were rich enough not to have to work anymore, I’d still write programs.

And that is my biggest problem.

Sometimes, the need for new problems to solve takes over. Banging my head against some hard problem is fun only as long as I haven’t solved it yet. As soon as I see the solution, the problem becomes boring. It’s solved, but only inside my brain: I still have to write the actual code. It has to be tested. Documented. Deployed. And that, to me, is stupid busywork. While I’m sitting at the keyboard hacking away at some source code, my thoughts wander to some other problem that could be interesting to attack next. As soon as a piece of code is good enough, it’s on to the next one. I’m a starter, not a finisher. And I constantly have to keep myself from re-inventing the wheel over and over, just to find out how some piece of technology works. Nothing bad about this by itself, it’s always good to learn something new; but when there’s a deadline looming, the last thing the boss needs is me experimenting with green threads in Forth, just because I found some quirk in the architecture of the .NET runtime and decided to hack away at a stack heavy language.

Which is another reason for me to keep updating lbjScheme from time to time: It’s not finished yet. It’s good enough for me, so it’s in the most dangerous state right now. The code needs some heavy refactoring. Documentation. More tests. Continuations don’t work yet. The compiler/vm combo is slower than the interpreter. And so on. Boy do I hope that continuations will be harder to do than I estimate right now. In theory, it looks so easy that it might be the death of my little side project. And for once, I don’t want to add another good-enough-yet-unfinished code base to my portfolio. This time, I’ll pull through.

As soon as I get those green threads done.

The power of plain text

Another one of these days that makes me appreciate Unix and having read The Pragmatic Programmer. My student was struggling with a log file containing, among others, a hex dump of some weird data needed to talk to a Bluetooth chip that he needed to convert to a C array initializer. He was typing along rhythmically, sighing, having accepted that he would have to press right, right, right, right, comma, right, right… and so on, until somewhere around the evening he would be finished. When I showed him the search/replace function he was jumping with joy: All he needed to do was replace every space with a comma-space combination at certain places in the file. Much easier. Had he been using VIM, he could have recorded a quick macro and his editing would have been over in a minute.

Which begs the question: Why do people not even use a functionaliy as simple as search/replace? He’s a bright guy who works with the Linux shell all day, writes C and C#, but textual editing above cut/paste still eludes him. Well, at least he doesn’t use the mouse to pop up a menu to select the Paste command from. But up until now, he never saw the necessity for looking at his editor a little closer, find out what else it can do. Just like lots of programmers I’ve met, including myself — before I had read The Pragmatic Programmer, and its fervent advocating of learning to use one editor really, really well, no matter which one.

Just having turned my eyes towards Linux, naturally I selected VIM as my tool of choice. Hey, stop hitting me, I do use Emacs when editing Lisp code! But my hands tend to find VIM more comfortable: No more RSI. Nice.

Before learning VIM, I was a happy Gedit user. But still, being a programmer I did work a lot with textual data, but never used anything more advanced than cut/paste. At first, learning how to use VIM was a pain. The learning curve looks more like a vertical mountain slope, and there are way too many bizarre commands you think you’ll never need. After all, I can just move upwards a few lines, press V, go down to the end of the block, and press >> to indent it, right? Why learn a special three-character-sequence for that?

Because it will make you faster. Once those strange commands have found their way into your muscle memory, suddenly your friends and collegues will get all excited when looking over your shoulder while you do some editing. What you do to the source code becomes magic to them, and you become a wizard wielding a powerful tool. Using a system that doesn’t have your editor on it becomes a pain. Which is another reason for my advocating VIM: It’s everywhere! Unless you’re a Windows user, but even then you can install a port.

Back in 1992, I used my trusty old Commodore 64 to do my homework. The Pagefox module (German link) had three modules: A pixel based graphics editor, a layout editor, and — a text editor. You’d write some text, add a few formatting codes1, then you would paint a few boxes in the layout editor, placing bitmaps somewhere on a virtual sheet of paper, and start the formatter, generating a pixel based representation of the text entered before. Using it was pure bliss: While working on the text, you could concentrate on the content, without thinking about how the text would flow around an image placed somewhere on the page. That came later, at layouting time. In contrast, when you write a document in one of the modern WYSIWYG word processors, you are constantly reminded of the “exact” location on the page where a word will have to move into the next line, where a page break will occur and so on. Distracting, isn’t it? If you write lots of textual documents, try an experiment: Write your texts in a plain text editor, then copy and paste into your word processor and add whatever formatting you want. Do you find it easier to concentrate on the actual content when writing in this fashion? What if you had a more powerful editor that could make your editing even faster and more pleasant?

Of course there are lots of systems nowadays that use the same method of formatting textual data2. The simplest of them seems to be Markdown, a very basic but powerful system to turn a plain text document into a formatted one.  Another system worth looking at is LaTeX. Extremely powerful and the de-facto standard in scientific publishing, but not as simple as Markdown. This is perfect for longer, complex documents, like PhD theses. And then, nearly forgotten, there is the *roff family of text processors: nroff, troff, groff and all the others. Quirky little programs that take what can only be described as an obscure programming language as input and produce beautiful papers and reports as output. They were part of the Unix text processing system, and parts of this system are still in use in every Linux and BSD system: Whenever you call up a man page, there is a groff instance running in the background turning the source code of your desired man page into a nice document to view on screen. If you want, you can turn every single man page into a beautifully formatted PostScript document that you can send to your printer or convert to PDF.

What you are currently reading was written in Markdown, using VIM on the text mode console of my Linux box. I like to use text mode to edit longer texts, because there is no browser, email notifications or whatever distracting me from writing. Just VIM and me. Oh, and Git. Wait, what? I thought that was for managing source code? Yes, and no. Not only. As soon as you are dealing with a plain text file, you can use all the goodies programmers use at work: Source code management tools like Git become a safety net for your document. Write a few sentences, then git commit to save the current state of your text. Unlimited undo capabilities! If you’re not sure about how to approach a chapter, create a branch and go wild. If you like the outcome, keep the branch, otherwise just checkout the master branch again. Merging and even cherry picking works too — try that with a WYSIWYG text processor!

And all the other power tools are there, too. Every Un*x box comes with sed and awk, two powerful little programs that let you run automated text processing scripts for everything a simple VIM macro is not enough for. Or write a Perl script to do the heavy lifting for you. Compare different revisions of your document with a basic file differ. The possibilities are endless.

As they were decades ago. We just went on to “simpler” tools, focusing on the looks of our documents while we are writing them, modifying their looks by clicking around with a mouse. Much simpler, sure, and easier to learn. But sometimes you want a power tool, and editing plain text documents can give you just that. I can’t even remember how many times OpenOffice crashed while I was working on my thesis. MS Word kept changing image placements, layout and formatting data. Only when I switched to plain text and LaTeX could I finally concentrate on getting the job done.

Oh, and by the way, nearly everything in a Un*x system is a plain text file. Configuration (always place /etc in a Git repo!), scripts, directory listings, log files… once you have learnt to use sed, awk, perl and a good plain text editor, you have an immensely powerful toolset. So go ahead and learn. You’ll never look back.


  1. Think start bold, end bold and so on. 
  2. Early word processors, e.g. for DOS, used the same approach, where you entered plain text and added special formatting codes to be executed later at print time. 

Moving up and down the technology stack

Programming is a crazy field. One day you fire up a Python interpreter, connect to some PostgreSQL database, retrieve a few records, and print out a Gnuplot script so you can include a nice plot in your LaTeX document. The next day, you connect a scope to that damn 515C controller to find out why the CAN interface timing is botched. Throw in a few Excel macros, a Node.js based web service and some work on the Scheme interpreter written in C#, and you have a typical week at my employer’s.

And that’s exactly how I like it.

Doing the same stuff every day gets boring pretty quickly. Working on something else every other day keeps things interesting. There is so much stuff out there to learn! Lots of high level libraries and tools that let you create fully automated reports typeset in the highest quality, juggle with enormous amounts of data with a near-English “natural” language or detect diodes soldered onto a board with the wrong polarity in an image. It’s incredible how much work has been done for us already. Some days, programming feels like sticking together some Lego pieces.

Then again, sometimes you have to get down to the dirty little details. Be it to find out how stuff works at a lower level, or to make sure things run as fast as possible on tiny systems with an amount of memory that was considered to small to be documented as a “free” range on the Commodore 64.

And then there is learning and teaching. When I walked “my” student employee through writing his first ring buffer in pure C, providing him with a small test suite and Git along the way, there it was again — that sweet, sweet nostalgia. Code::Blocks is a pretty decent IDE, and within half a day the code was “done”. Only it wasn’t, because I’m a bastard and gave him a crippled test suite that omitted a few corner cases, so he could learn to watch out for incomplete or faulty test suites along the way. So far he’s learnt how to use source control, C and C# command line compilers, several IDEs, played with Android programming in Java and C#, and debugged a serial comm line with a digital scope. And I’ve learnt a lot in the process, too: Teaching changes your view about pretty much any technology you touch, making you remember those days when you learnt something yourself, what pitfalls there were, how to get around them — and suddenly you notice corner cases you always just shoved aside, never really thought about, and bingo, another interesting thought to follow.

It’s been a fun week.

Want to go on a nostalgia trip yourself? Implement this:
/** @brief Initializes a ring buffer. 
  * @param buffer The byte array to use for the ring buffer.
  * @param size The size of the array in bytes.
  * @return The number of bytes that can be stored in the buffer, or -1 if
  *         the array is too small to make a ring buffer.
  */
int16_t ring_init(uint8_t *buffer, uint16_t size);
 
/** @brief Gets the next byte from a ring buffer.
  * @param buffer The byte array used for the ring buffer.
  * @return The value read, or -1 on buffer underrun.
  */
int16_t ring_get(uint8_t *buffer);
 
/** @brief Stores a byte in a ring buffer.
  * @param buffer The byte array used for the ring buffer.
  * @param value The value to store.
  * @return 1 if successful, or -1 on buffer overflow.
  */
int16_t ring_put(uint8_t *buffer, uint8_t value);

Set up a git repo, your favorite IDE, a fresh C project, maybe go crazy and set up a Trac instance, and write a few test cases. Implement and test. Then think really hard: Did you handle this “project” with the same care you always use at work? What would you do differently on a “real” project? Why? What could be done better? Could you change something to your usual approach to become better? Interesting thoughts ahead.

Automatically renaming image files

I’ve given up on trying to find a good naming scheme for my digital photos. It’s just too much work. So now I just run a small Python script that parses the Exif info, extracts the time stamp the photo was taken, and renames the image file to yyyy_mm_dd_hhmmss.JPG. Then I just sort the files into a separate folder for each month and I’m done. The probability of two images taken at the same second is reasonably low not to run into naming conflicts.

Not exactly a life-saver, but nevertheless quite a handy script:

#!/usr/bin/python

from PIL import Image
from PIL.ExifTags import TAGS
import os
import sys

def getImageDate(fn):
    info = Image.open(fn)._getexif()
    dt = ""
    for tag, value in info.items():
        decoded = TAGS.get(tag, tag)
        print "%s => %s" % (decoded, value)
        if decoded == "DateTimeOriginal": return value
        if decoded == "DateTimeDigitized": return value
        if decoded == "DateTime": dt = value
    if dt != "": return dt
    return 0

for fn in sys.argv[1:]:
    try:
        path = os.path.dirname(fn)
        dt = getImageDate(fn).replace(" ", "_").replace(":", "_", 2).replace(":", "") + ".JPG"
        dest = os.path.join(path, dt)
        if dt.startswith("0000_00"):
            print "Skipping %s" % (fn)
        else:
            if os.path.exists(dest):
                print "Dest file exists, skipping %s" % (dest)
            else:
                os.rename(fn, dest)
                print "%s -> %s" % (fn, dest)
    except:
        print "Skipping %s" % (fn)