Return on Investment

About a year ago, a friend and I were discussing the evolution of learning new tools while working under time constraints. He claimed that, regardless of the pressure, it is almost always better to sacrifice the upfront time in learning a new tool than to stay in your comfort zone to construct the illusion of progress. By illusion he simply meant that early stage progress is shadowed by the long-term effect of having to maintain and update a homegrown solution.

He used the following illustration to convey his point:

The effect is that if you choose to develop your own tools you satisfy the immediate gratification of results but the final solution is more likely to end up as a house of cards.

I had forgotten about this conversation until recently. The program I am currently working on had me in line to provide relevant results in a shortened amount of time. I am working with the llvm compiler infrastructure and it's unique assembly language. Part of what I needed to do is understand, at a high level, the contents of an llvm assembly file. My natural reaction to this was to implement some minimal parser in Ruby to get to a working set of results quickly. Things went well until I wanted to make a decent tool for broader use within the team: I was starting to spend more time on developing the Ruby code than making headway on my tasks. Not only did I dig myself a grave in that respect but since I was quick to satisyfy the initial demand for results I set an expectation that the work was less complicated than it truly is (and will continue to be). I was building a house of cards.

I ended up scrapping the Ruby implementation and I'm now happily coming up to speed on how to manage llvm code with, well, llvm tools. This is not without effect from the leads of the program, of course. Live and learn, I suppose.

Progress

[EDIT] This code is now available on github: libpbar

cURL ships with one. wget uses it's own. OpenSSH (scp) even comes with it's own version. Indeed, I've written several variations myself over the years. Sooner or later, if you are doing something that processes large amounts of data or takes a long time to complete and you are working in a command-line environment you will write your own, too. A progress bar.

I did a quick search for existing progress bar libraries but came up with little outside of Ruby and Python wrappers around what the packages I mention above are already doing [1]. For mostly selfish reasons, I've decided to write a library that fixes my redundant behavior moving forward. Once I am able to provide a proper build system I will put this code up on github - my unfamiliarity with autotools may delay this a bit.

What this library aims to provide is a consistent interface for programatically displaying a progress bar outside of a GUI environment. It does not try to provide a package similar to Pipe Viewer (pv) which is a complete, standalone program for monitoring processes activity. Instead, to keep things simple, libpbar will initially provide a class with the following public interface:

---

class ProgressBar {
public:
    ProgressBar (::size_t cap);
    void update (::size_t n = 1);
};

---

With three implementations provided with the library: TextProgressBar, ColorProgressBar, GraphicalProgressBar. These provide basic text functionality, color text, and a GTK+-based object respectively. Using any of these facilities is as simple as:

---

int flags = TextProgressBar::SHOW_VALUE | TextProgressBar::SHOW_COUNT;
TextProgressBar pb(100,flags);

---

They each do as you might expect:
TextProgressBar:

ColorProgressBar:

GraphicalProgressBar:

The nice thing about the GUI version is that the hosting program doesn't need to be a GUI application. For this version of the progress bar a separate thread is used to spawn the graphical interface and as much as possible is done to shed the window treatments to expose just the progress bar itself.

I've got to expand on this by allowing some way to encode message text with format strings and allowing user-defined color modifications. The basics are there, however, and I should now be free from writing yet another progress bar implementation.

---

[1] This is a solved problem and certainly my fruitless search does not imply any lack of existing tools

Know Your Tools

I was recently pulled into a project that needed a developer for a short sprint to make a portable version of a set of already existing tools. The functional version of this suite consists mostly of C and Perl code written to one of three platforms: FreeBSD, Linux, or Windows. The Windows support was provided via Cygwin and the version I was to develop intended to remove this and other crutches. It's also relevant to mention that I would only be converting the client side code; the server would remain untouched.

The port itself was relatively straightforward. Moving Perl and C to Python is not terribly taxing in and of itself, it's more of a burden ensuring that you do not break any existing functionality. These tools essentially provide a remote query/response mechanism across multiple platforms which simplified the testing approach. I would just iterate through all supported queries, store the output, and do a diff on each target platform.

I had two weeks to deliver. At the end of the second week the C and Perl files were converted to Python and I was testing the functionality of the system but was getting a 50% fail rate. A random set of commands were only getting a portion of the response - no error codes, just partial responses.

From trace output in each version the behavior looked sane. From tcpdump output I could tell that the handshake and shutdown was happening properly it just looked as if the server had less to send my version of the client. Because it can be easier to elicit salient information through Wireshark's interface I plugged the pcap traces in and looked a them side-by-side. There it was: a single byte was different between the two. The original query:

And the version I was sending:

The difference is a single space (the 0x20 sitting at the end of the first image). It threw me the first time since I was actually looking for non-printable characters in the ASCII-formatted output. Non-printable characters show up as '.' but 0x20 is a space character. Grrr. For whatever reason, the server will only return a full response if the command has a trailing space character. It's intriguing that a partial response is returned for an invalid request without warning. My guess is that there is an off-by-one error somewhere in the server command parsing code.

I pushed a bug up to the server group though I doubt it will see the light of day since I've already covered the ground necessary to make the new tool work. If it ain't broke costing us money, don't fix it.

The moral of the story is that you should know the tools that can help you track down problems. You should also understand the strengths of different versions of those tools (Wireshark vs. tcpdump). Tried and true tools are invaluable to getting good work done and it is your responsibility as a developer to have your set at the ready at all times.