In almost all instances it is a bad idea to place multiple distinct scales on a single plot. Not only does it require more of the reader in terms of deciphering the message, it also lessens the impact of the chart itself. Instead of a reader being able to associate the vertical data with an immediate estimated value in their mind they now have a two step process to determine a value. First, they must choose which scale applies - then they must consult the appropriate scale to derive a value. This is all a bad thing.
I also feel this is a common way to try and hide information in plain sight. Consider the following graph, what points define where the two lines intersect? (Here's a hint - they don't) This layout lends itself to all sorts of slight of hand when talking to or describing data.
Usually, I would claim that any graph that did this could be better provided as either a multiplot or two separate graphs. However, the other day I saw an instance where it made perfect sense: temperature. Plotting temperature with two separate scales is actually a representation of two separate mathematical functions that represent the same property; in effect the graph acts as a lookup table for the user for translating from one function to another. As an example, consider the following:
A glorified lookup table to be sure - but entirely functional.
Moving forward, I've modified my view to something more along the lines of: "Dual scales are useful when they both describe a single property in two distinct ways." Any measurement that can be represented in a variety of ways falls into this category: temperature (Fahrenheit v. Celsius), time (24-hour v. 12-hour), distance (miles v. kilometers), and so on.
Friday, July 15, 2011
Tuesday, July 5, 2011
Progress
[EDIT] This code is now available on github: libpbar
cURL ships with one. wget uses it's own. OpenSSH (scp) even comes with it's own version. Indeed, I've written several variations myself over the years. Sooner or later, if you are doing something that processes large amounts of data or takes a long time to complete and you are working in a command-line environment you will write your own, too. A progress bar.
I did a quick search for existing progress bar libraries but came up with little outside of Ruby and Python wrappers around what the packages I mention above are already doing [1]. For mostly selfish reasons, I've decided to write a library that fixes my redundant behavior moving forward. Once I am able to provide a proper build system I will put this code up on github - my unfamiliarity with autotools may delay this a bit.
What this library aims to provide is a consistent interface for programatically displaying a progress bar outside of a GUI environment. It does not try to provide a package similar to Pipe Viewer (pv) which is a complete, standalone program for monitoring processes activity. Instead, to keep things simple, libpbar will initially provide a class with the following public interface:
With three implementations provided with the library: TextProgressBar, ColorProgressBar, GraphicalProgressBar. These provide basic text functionality, color text, and a GTK+-based object respectively. Using any of these facilities is as simple as:
They each do as you might expect:
TextProgressBar:
ColorProgressBar:
GraphicalProgressBar:
The nice thing about the GUI version is that the hosting program doesn't need to be a GUI application. For this version of the progress bar a separate thread is used to spawn the graphical interface and as much as possible is done to shed the window treatments to expose just the progress bar itself.
I've got to expand on this by allowing some way to encode message text with format strings and allowing user-defined color modifications. The basics are there, however, and I should now be free from writing yet another progress bar implementation.
[1] This is a solved problem and certainly my fruitless search does not imply any lack of existing tools
cURL ships with one. wget uses it's own. OpenSSH (scp) even comes with it's own version. Indeed, I've written several variations myself over the years. Sooner or later, if you are doing something that processes large amounts of data or takes a long time to complete and you are working in a command-line environment you will write your own, too. A progress bar.
I did a quick search for existing progress bar libraries but came up with little outside of Ruby and Python wrappers around what the packages I mention above are already doing [1]. For mostly selfish reasons, I've decided to write a library that fixes my redundant behavior moving forward. Once I am able to provide a proper build system I will put this code up on github - my unfamiliarity with autotools may delay this a bit.
What this library aims to provide is a consistent interface for programatically displaying a progress bar outside of a GUI environment. It does not try to provide a package similar to Pipe Viewer (pv) which is a complete, standalone program for monitoring processes activity. Instead, to keep things simple, libpbar will initially provide a class with the following public interface:
class ProgressBar {
public:
ProgressBar (::size_t cap);
void update (::size_t n = 1);
};
With three implementations provided with the library: TextProgressBar, ColorProgressBar, GraphicalProgressBar. These provide basic text functionality, color text, and a GTK+-based object respectively. Using any of these facilities is as simple as:
int flags = TextProgressBar::SHOW_VALUE | TextProgressBar::SHOW_COUNT;
TextProgressBar pb(100,flags);
They each do as you might expect:
TextProgressBar:
ColorProgressBar:
GraphicalProgressBar:
The nice thing about the GUI version is that the hosting program doesn't need to be a GUI application. For this version of the progress bar a separate thread is used to spawn the graphical interface and as much as possible is done to shed the window treatments to expose just the progress bar itself.
I've got to expand on this by allowing some way to encode message text with format strings and allowing user-defined color modifications. The basics are there, however, and I should now be free from writing yet another progress bar implementation.
[1] This is a solved problem and certainly my fruitless search does not imply any lack of existing tools
Thursday, May 26, 2011
Optical Disillusion
I dislike chartjunk. Not only is there a trend toward the incomprehensible but the movement comes with a ridiculous amount of flair. For all I can tell there exists a competition between infographic creators where the rules are based solely on who can cram more slop on a page.
Besides my obvious distaste for the style, there are sacrifices being made that compromise, or even forgo, the actual message - often without awareness or malice. Take, for example, the evolution of the pie chart.
For the sake of this example lets ignore the fact that a pie chart is a particularly poor way to display data to begin with. As a measure of two variables it provides a rough estimate of dominance but beyond that the human eye can not distinguish relative quantities across the various shapes. In almost all cases a simple bar chart provides a more precise description - even in the two variable case. A visual display of data should be able to stand on its own without the need of labels describing quantities or values. The pie chart fails in this respect, but I digress.
Consider a simple pie chart of two variables.
(As a measure of the strength of a pie chart as a communication tool, can you guess the values of the two areas? Go ahead, take a guess.)
The red portion of the chart is 55% and the blue portion is the remaining 45%. Without labels it is hard to distinguish exactly but serves to at least show the dominance of red over blue. The problem with trendy infographincs is that a simple pie chart is almost never sufficient in the layout. It needs exploding, or gradients, or even a third dimension.
Lets dress it up a bit and make a 3D pie chart with the same values.
So what's my beef about that? Lets consider the new representative areas of the chart. In the first chart, the values were inconspicuous but at least the color representation mapped directly to the underlying data.
Standard Pie Chart (red pixels) : 44295 (55.000%)
Standard Pie Chart (blue pixels): 36188 (44.900%)
In this new 'cooler' version of the chart we have skewed the data representation and thus our understanding of the overall message. In fact, by visible surface area alone we have changed the meaning of the chart entirely!
3D Pie Chart (red pixels) : 44792 (47.300%)
3D Pie Chart (blue pixels): 49740 (52.600%)
What is now required of us in this new chart, along with somehow mapping area to value, is to do accurate mathematical transformations in our heads to convert the 3D surface to an area in 2D. In fact, we need to now be able to deduce that roughly 52% of viewable surface area translates to 45% underlying data. The skew depends on the pitch, yaw, and roll so there is no magical formula here - every view will be a different mapping between surfaces.
I don't think people consider these details when compiling charts. In my estimate they are only trying to provide the most 'eye candy' for the intended consumer. The behavior is facilitated by common built-in chart generators (only 48 out of Excel's 288 pie chart variations are simple 2D charts) but there is no warning about the possible loss of meaning.
I'm certainly not among those pushing the envelope with infographics - this definitely makes my opinion biased. I keep things as simple as possible and for most data hungry crowds my approach is just too boring against current standards. I do believe there is a middle-ground, however; a place where rich graphics convey accurate data with minimal annotation markup. I only wish I knew how to bridge the gap.
A huge thanks to Dana Brown for taking the time to review and provide feedback on the first draft of this post.
Besides my obvious distaste for the style, there are sacrifices being made that compromise, or even forgo, the actual message - often without awareness or malice. Take, for example, the evolution of the pie chart.
For the sake of this example lets ignore the fact that a pie chart is a particularly poor way to display data to begin with. As a measure of two variables it provides a rough estimate of dominance but beyond that the human eye can not distinguish relative quantities across the various shapes. In almost all cases a simple bar chart provides a more precise description - even in the two variable case. A visual display of data should be able to stand on its own without the need of labels describing quantities or values. The pie chart fails in this respect, but I digress.
Consider a simple pie chart of two variables.
(As a measure of the strength of a pie chart as a communication tool, can you guess the values of the two areas? Go ahead, take a guess.)
The red portion of the chart is 55% and the blue portion is the remaining 45%. Without labels it is hard to distinguish exactly but serves to at least show the dominance of red over blue. The problem with trendy infographincs is that a simple pie chart is almost never sufficient in the layout. It needs exploding, or gradients, or even a third dimension.
Lets dress it up a bit and make a 3D pie chart with the same values.
So what's my beef about that? Lets consider the new representative areas of the chart. In the first chart, the values were inconspicuous but at least the color representation mapped directly to the underlying data.
Standard Pie Chart (red pixels) : 44295 (55.000%)
Standard Pie Chart (blue pixels): 36188 (44.900%)
In this new 'cooler' version of the chart we have skewed the data representation and thus our understanding of the overall message. In fact, by visible surface area alone we have changed the meaning of the chart entirely!
3D Pie Chart (red pixels) : 44792 (47.300%)
3D Pie Chart (blue pixels): 49740 (52.600%)
What is now required of us in this new chart, along with somehow mapping area to value, is to do accurate mathematical transformations in our heads to convert the 3D surface to an area in 2D. In fact, we need to now be able to deduce that roughly 52% of viewable surface area translates to 45% underlying data. The skew depends on the pitch, yaw, and roll so there is no magical formula here - every view will be a different mapping between surfaces.
I don't think people consider these details when compiling charts. In my estimate they are only trying to provide the most 'eye candy' for the intended consumer. The behavior is facilitated by common built-in chart generators (only 48 out of Excel's 288 pie chart variations are simple 2D charts) but there is no warning about the possible loss of meaning.
I'm certainly not among those pushing the envelope with infographics - this definitely makes my opinion biased. I keep things as simple as possible and for most data hungry crowds my approach is just too boring against current standards. I do believe there is a middle-ground, however; a place where rich graphics convey accurate data with minimal annotation markup. I only wish I knew how to bridge the gap.
A huge thanks to Dana Brown for taking the time to review and provide feedback on the first draft of this post.
Labels:
plots
,
responsibility
,
style
Thursday, May 19, 2011
Using structs
If you want to group related information in a C program the common way to do so is with a struct. You see this everywhere: various network packets are represented as structs; file system objects (inodes); time values can be stored with separate components in struct members; and so on. In all of these cases it is entirely possible to use basic arrays to maintain this data. The problem with that is that we, as non-machines, find it more natural to think in terms of collections with named members than random offsets into a buffer. In other words, it is easier to design and reason with 'object.member = value' than it is with 'buffer[offset] = value' (or something more obscure). Especially if you deal with members of various sizes.
I feel this is a natural progression - the tendency to want to group related items and operate with them as individual 'things'. I believe this to be a dominating theme with C programmers (and programmers in general). What I dont see too much of, however, is explicitly moving in the opposite direction. That is, given some data in the form of a buffer, C programmers are more likely to develop their own mini-parser to get at that data instead of using a struct to ease implementation.
As an example, I've seen the following many times in a variety of flavors:
And, while that is functional it is also error-prone and cumbersome to write each time you need to do such a conversion. In contrast, I see very little in the form of
Where I think the implementation is simplified
If we remove most of the boilerplate sections of an example and examine just how each of these is available to use we can see what the overall effect is.
and with the struct
The main differences between the two are how data is extracted from the raw buffer and how that implementation affects code design. In both cases the struct provides a cleaner and more understandable solution. In a more generic setting, one where you may not have the exact number of elements in the buffer, the struct approach above doesn't fit exactly.
However, it can be modified:
Where the using the function still requires the buffer offset when calling the function but the method implementation is much cleaner.
This becomes even more useful if you consider cases where an individual member may contain multiple bits of information. For instance, it is common to have a data member of a struct represent a set of flags. The typical implementation involves a set of macros to test or set values in a bit mask.
For example:
With well designed macro names this approach does not imply altogether clumsy code but the macro design is still cumbersome. I think that a more elegant approach can be achieved through the use of structs.
This can even be done without having to change implementation code. Leaving the macro interface while changing the bodies to represent the new implementation allows users of the macro interface to continue uninterrupted.
The struct is no panacea. However, I find that in these types of scenarios the struct provides for much cleaner and manageable code - something I favor whenever I can.
I feel this is a natural progression - the tendency to want to group related items and operate with them as individual 'things'. I believe this to be a dominating theme with C programmers (and programmers in general). What I dont see too much of, however, is explicitly moving in the opposite direction. That is, given some data in the form of a buffer, C programmers are more likely to develop their own mini-parser to get at that data instead of using a struct to ease implementation.
As an example, I've seen the following many times in a variety of flavors:
uint32_t extract(unsigned char *buffer) {
uint32_t value = 0;
int i = 0;
for (i=sizeof(uint32_t)-1; i>=0; i--) {
value = value << 8;
value += buffer[i];
}
return value;
}
And, while that is functional it is also error-prone and cumbersome to write each time you need to do such a conversion. In contrast, I see very little in the form of
struct extracted {
uint32_t v[5];
};
struct extracted * map = buffer;
Where I think the implementation is simplified
If we remove most of the boilerplate sections of an example and examine just how each of these is available to use we can see what the overall effect is.
uint32_t vals[] = {0x0, 0x0f, 0x0f0f, 0x0f0f0f, 0x0f0f0f0f};
unsigned char * cvals = vals;
for (; i < 5; ++i)
printf ("%10lu\n", extract (cvals + i * sizeof(uint32_t)));
and with the struct
uint32_t vals[] = {0x0, 0x0f, 0x0f0f, 0x0f0f0f, 0x0f0f0f0f};
unsigned char * cvals = vals;
struct extracted * map = cvals;
for (; i < 5; ++i)
printf ("%10lu\n", map->v[i]);
The main differences between the two are how data is extracted from the raw buffer and how that implementation affects code design. In both cases the struct provides a cleaner and more understandable solution. In a more generic setting, one where you may not have the exact number of elements in the buffer, the struct approach above doesn't fit exactly.
However, it can be modified:
struct value {
uint32_t data;
};
uint32_t extract(unsigned char *buffer) {
struct value * v = buffer;
return v->data;
}
Where the using the function still requires the buffer offset when calling the function but the method implementation is much cleaner.
This becomes even more useful if you consider cases where an individual member may contain multiple bits of information. For instance, it is common to have a data member of a struct represent a set of flags. The typical implementation involves a set of macros to test or set values in a bit mask.
For example:
#define FOO_FLAG (1<<0)
#define BAR_FLAG (1<<1)
#define BAZ_FLAG (1<<2)
#define ISSETFOO(v) ((v).mask & (FOO_FLAG))
#define SETFOO(v) ((v).mask |= (FOO_FLAG))
#define UNSETFOO(v) ((v).mask &= ~(FOO_FLAG))
/* similarly for BAR_FLAG and BAZ_FLAG */
struct data {
unsigned short mask;
unsigned char stuff[200];
};
int main () {
struct data data;
SETFOO(data);
if (ISSETFOO(data)) {
printf ("FOO_FLAG is set\n");
} else {
printf ("Foo_FLAG is not set\n");
}
/* ... */
With well designed macro names this approach does not imply altogether clumsy code but the macro design is still cumbersome. I think that a more elegant approach can be achieved through the use of structs.
struct flags {
unsigned char foo:1;
unsigned char bar:1;
unsigned char baz:1;
};
struct data {
struct flags mask;
unsigned char stuff[200];
};
int main () {
struct data data;
data.mask.foo = 1;
if (data.mask.foo) {
printf ("FOO_FLAG is set\n");
} else {
printf ("Foo_FLAG is not set\n");
}
/* ... */
This can even be done without having to change implementation code. Leaving the macro interface while changing the bodies to represent the new implementation allows users of the macro interface to continue uninterrupted.
The struct is no panacea. However, I find that in these types of scenarios the struct provides for much cleaner and manageable code - something I favor whenever I can.
Saturday, April 23, 2011
y = mx + b
I had recently compiled a set of plots for a technical report I put together to summarize an evaluation of some of our software. The focus of the tests was performance of multiple file transfer approaches. One of the comments I got back after pushing the report out for review was that I should include the slope as an annotation for a best fit line I had included. I thought this was a curious request at first since the best fit was already intended to give a visual estimate of the behavior. After I thought about it, however, I considered the case when someone would want to compare results to ours without having any of our raw data. A visual estimate would certainly not do in that case - especially if the range of their data was outside of the viewable area of the plot.
This was a simple change. R, the software I was using to generate the plots, has built in functionality for linear regression. Getting the slope and y-intercept is as easy as:
This got me thinking, however. Would I know how to generate the linear regression without software? The answer was 'no', so I learned how. Here are the steps I took to generate the same results as the statistical software I was using.
I was dealing with a set of data that was generated using inputs over the powers of two: [1, 2, 4, 8, ..., 2048, 4096]. I also had multiple samples at each input. The first step was to determine how, in fact, a linear regression is calculated.
Linear regression (used to make a best fit line) can be calculated using the least squares approach. I needed to have the following: mean and standard deviation of both the inputs and outputs and the Pearson product-moment correlation coefficient. Once I had all of those calculations I could generate the slope and y-intercept of the best fit line and thus any point along the line.
In order to calculate the PMCC there is some upfront legwork that I needed to do. The equation requires the standard score and sample standard deviation from the two sets and calculating the y-intercept requires the sample means. The Pearson product-moment correlation coefficient for a given sample is defined here (Wikipedia).
I put together some Ruby code to generate the necessary values:
Running that with my control data set yields the following:
With the results from R above as a sanity check these values look accurate. Now, using these calculations, I could plot a best fit simply by drawing a line from (1,y) to (4096,y) where y is calculated using the equation: y = mx + b.
So, while I'm still an advocate of using the right tool for the job I can now understand the mechanics behind these calculations and talk specifically about why they should (or should not) be used.
N.B. I am no expert on any of this. There are topics at play here I do not fully understand (details of the Pearson product-moment correlation coefficient, for one). This is more a narrative than any official guide to understanding.
This was a simple change. R, the software I was using to generate the plots, has built in functionality for linear regression. Getting the slope and y-intercept is as easy as:
> lm(y ~ x)
Call:
lm(formula = y ~ x)
Coefficients:
(Intercept) x
8.474523 0.007313
This got me thinking, however. Would I know how to generate the linear regression without software? The answer was 'no', so I learned how. Here are the steps I took to generate the same results as the statistical software I was using.
I was dealing with a set of data that was generated using inputs over the powers of two: [1, 2, 4, 8, ..., 2048, 4096]. I also had multiple samples at each input. The first step was to determine how, in fact, a linear regression is calculated.
Linear regression (used to make a best fit line) can be calculated using the least squares approach. I needed to have the following: mean and standard deviation of both the inputs and outputs and the Pearson product-moment correlation coefficient. Once I had all of those calculations I could generate the slope and y-intercept of the best fit line and thus any point along the line.
In order to calculate the PMCC there is some upfront legwork that I needed to do. The equation requires the standard score and sample standard deviation from the two sets and calculating the y-intercept requires the sample means. The Pearson product-moment correlation coefficient for a given sample is defined here (Wikipedia).
I put together some Ruby code to generate the necessary values:
#!/usr/bin/env ruby
class DataSet
attr_reader :mean, :var, :sd, :size
def initialize(ary)
@size = ary.size
@data = ary.dup
@mean = ary.inject(0.0) { |sum,n| sum + n } / @size
@var = ary.inject(0.0) { |sum,n| (n-@mean)**2 + sum }
@sd = Math.sqrt(@var/(@size - 1))
end
def [](i)
@data[i]
end
end
# http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient
def pearson(x,y)
g = []
x.size.times do |i|
g << ((x[i] - x.mean)/x.sd)*((y[i] - y.mean)/y.sd)
end
(1.0/(x.size - 1)) * g.inject(0.0) { |sum,n| sum + n }
end
#http://www.stat.wmich.edu/s216/book/node126.html
def slope(x,y)
pearson(x,y) * (x.sd/y.sd)
end
def intercept(x,y)
x.mean - slope(x, y)*y.mean
end
def load_data(file)
xs, ys = [], []
if file
File.open(file).each do |line|
x,y,_ = line.chomp.split(',')
xs << x.to_f
ys << y.to_f
end
else
while line = gets
x,y,_ = line.chomp.split(',')
xs << x.to_f
ys << y.to_f
end
end
[DataSet.new(xs), DataSet.new(ys)]
end
x, y = load_data(ARGV.shift)
puts "Slope : %0.6f" % [slope y, x]
puts "Y-intercept: %0.6f" % [intercept y, x]
Running that with my control data set yields the following:
Slope : 0.007313
Y-intercept: 8.474523
With the results from R above as a sanity check these values look accurate. Now, using these calculations, I could plot a best fit simply by drawing a line from (1,y) to (4096,y) where y is calculated using the equation: y = mx + b.
So, while I'm still an advocate of using the right tool for the job I can now understand the mechanics behind these calculations and talk specifically about why they should (or should not) be used.
N.B. I am no expert on any of this. There are topics at play here I do not fully understand (details of the Pearson product-moment correlation coefficient, for one). This is more a narrative than any official guide to understanding.
Wednesday, April 13, 2011
Dynamic Code Loading (in C)
Originally, I intended to write this post about function pointers in C. I was inspired (yet again) by activity in a forum where a user asked a question about why you would ever need to use function pointers. As I started thinking about mapping the idea to a useful example I kept considering how the kernel uses function pointers for module writers. Couple that with my new exploration into the Erlang programming language and you get this content instead.
To address the first point directly: function pointers are useful as a means to expose a standard set of functionality. Consider how a character device driver for the linux kernel operates, for instance. When you write a character device driver you are presented with a struct file_operations (fops).
Why? Consider what you do with a character device; read from it, write to it, and so on. But since each device requires it's own implementation there needs to be a way to indicate this within the module code.
One solution to this is to have a set of skeleton functions declared in a header file. Require that all modules include this file and provide definitions for the functions. There are two obvious drawbacks to this approach:
Another solution is to use a structure that contains function pointers and allow developers to use that when writing to the interface. This solves the two issues with the skeleton declaration approach: any function not defined is set to nil in the structure and adding new members to the structure wont break existing code. Ok... if there is some fancy offset calculation being done there is potential for breakage, but that is suspect behavior in the first place.
Obviously removing entries creates compatibility issues in either solution.
Here is how the function pointer approach works in the linux kernel:
Where each of the i2cdev_* functions are defined locally to that file and any operations not defined in the struct are nil (aio_fsynch, for example).
That actually represents dynamic code loading in it's entirety. You can change the behavior of a system by loading modules at runtime. But that is in the kernel and not applicable in userland; what if we wanted to do that for a process we control? That is something that Erlang lets you do and - because I'm bent on reinventing the wheel for experience - I wanted to do it in C as well. Note: I'm sure that Erlang is not the only language that supports this; it just happens to be what I was reading recently.
Lets start with the interface that is exposed:
And two clients writing to that interface:
Client 1
Client 2
This driver is relatively crude. Notification of code change come by way of a signal and a well-known path indicates where the code resides. In addition, the mechanism for copying the structure of function pointers should reside in the driver itself and not in the clients. I think it is sufficient, however, to convey the idea.
And some implementation
With that setup we can now control the behavior of that driver at runtime. The example here runs the driver in it's own terminal and in another terminal commands are executed to load new code. The table below shows the execution in sequence - dynamic code execution in C.
To address the first point directly: function pointers are useful as a means to expose a standard set of functionality. Consider how a character device driver for the linux kernel operates, for instance. When you write a character device driver you are presented with a struct file_operations (fops).
struct file_operations {
struct module *owner;
loff_t (*llseek) (struct file *, loff_t, int);
ssize_t (*read) (struct file *, char *, size_t, loff_t *);
ssize_t (*write) (struct file *, const char *, size_t, loff_t *);
int (*readdir) (struct file *, void *, filldir_t);
/* ... lots of other calls ... */
};
Why? Consider what you do with a character device; read from it, write to it, and so on. But since each device requires it's own implementation there needs to be a way to indicate this within the module code.
One solution to this is to have a set of skeleton functions declared in a header file. Require that all modules include this file and provide definitions for the functions. There are two obvious drawbacks to this approach:
- Module writers are forced to fill in an empty body for all functions that will not be used
- Any changes to the interface require reimplementation of the entire set of modules using the now outdated file
Another solution is to use a structure that contains function pointers and allow developers to use that when writing to the interface. This solves the two issues with the skeleton declaration approach: any function not defined is set to nil in the structure and adding new members to the structure wont break existing code. Ok... if there is some fancy offset calculation being done there is potential for breakage, but that is suspect behavior in the first place.
Obviously removing entries creates compatibility issues in either solution.
Here is how the function pointer approach works in the linux kernel:
/* linux-2.6.37/drivers/i2c/i2c-dev.c:513 */
static const struct file_operations i2cdev_fops = {
.owner = THIS_MODULE,
.llseek = no_llseek,
.read = i2cdev_read,
.write = i2cdev_write,
.unlocked_ioctl = i2cdev_ioctl,
.open = i2cdev_open,
.release = i2cdev_release,
};
Where each of the i2cdev_* functions are defined locally to that file and any operations not defined in the struct are nil (aio_fsynch, for example).
That actually represents dynamic code loading in it's entirety. You can change the behavior of a system by loading modules at runtime. But that is in the kernel and not applicable in userland; what if we wanted to do that for a process we control? That is something that Erlang lets you do and - because I'm bent on reinventing the wheel for experience - I wanted to do it in C as well. Note: I'm sure that Erlang is not the only language that supports this; it just happens to be what I was reading recently.
Lets start with the interface that is exposed:
#ifndef JRB_DYNOPS__H__
#define JRB_DYNOPS__H__
/*
* Operations supported by our dynamic runtime:
* init: Initialize the new code
* speak: process strings in some way
* destroy: Cleanup code
*/
struct dynops {
void (*init)();
void (*speak)(const char *);
void (*destroy)();
};
/*
* Fill in the opeartions for a particular implementation
*/
void get_ops (struct dynops *);
#endif /*JRB_DYNOPS__H__*/
And two clients writing to that interface:
Client 1
#include <stdio.h>
#include <string.h>
#include "dynops.h"
static void lib1_speak (const char * text) {
fprintf (stderr, "[LIB1] %s\n", text);
}
/* only implements speak. Ignores init & destroy */
static struct dynops my_ops = {
.speak = lib1_speak,
};
void get_ops (struct dynops ** ops) {
memcpy(*ops,&my_ops,sizeof(my_ops));
}
Client 2
#include <stdio.h>
#include <string.h>
#include "dynops.h"
static void lib2_init () {
fprintf (stderr, "Initializing lib2\n");
}
static void lib2_destroy () {
fprintf (stderr, "Cleanup lib2\n");
}
static void lib2_speak (const char * text) {
fprintf (stderr, "[LIB2] %s\n", text);
}
static struct dynops my_ops = {
.init = lib2_init,
.speak = lib2_speak,
.destroy = lib2_destroy,
};
void get_ops (struct dynops ** ops) {
memcpy(*ops,&my_ops,sizeof(my_ops));
}
This driver is relatively crude. Notification of code change come by way of a signal and a well-known path indicates where the code resides. In addition, the mechanism for copying the structure of function pointers should reside in the driver itself and not in the clients. I think it is sufficient, however, to convey the idea.
int main () {
unsigned long count = 0;
char iteration[255] = {0};
signal (SIGUSR1, trigger_load);
while (1) {
snprintf (iteration, 255, "Iteration: %lu\n", ++count);
do_ops (iteration);
usleep (500000);
}
return 0;
}
And some implementation
void trigger_load (int sig) {
if (sig != SIGUSR1)
return;
FILE * new_lib = fopen("/tmp/dynlib", "r");
if (! new_lib)
return;
fscanf (new_lib, "%s", lib_name);
fclose (new_lib);
the_lib = lib_name;
reload = 1;
}
/* ... */
void do_ops (const char * str) {
if (reload) {
load_lib (the_lib);
reload = 0;
}
if (ops.speak)
ops.speak(str);
}
With that setup we can now control the behavior of that driver at runtime. The example here runs the driver in it's own terminal and in another terminal commands are executed to load new code. The table below shows the execution in sequence - dynamic code execution in C.
Terminal 1
$ ./a.out 'libops1.so' successfully loaded [LIB1] Iteration: 40 ... [LIB1] Iteration: 52 'libops2.so' successfully loaded Initializing lib2 [LIB2] Iteration: 53 ... [LIB2] Iteration: 62 Cleanup lib2 'libops1.so' successfully loaded [LIB1] Iteration: 63 ...
Terminal 2
$ pgrep a.out 5993 $ echo "libops1.so" > /tmp/dynlib $ kill -10 5993 $ echo "libops2.so" > /tmp/dynlib $ kill -10 5993 $ echo "libops1.so" > /tmp/dynlib $ kill -10 5993
Labels:
C
,
dynamic code execution
,
function pointer
Work Ethic
I do something every day at work, something I've done since my first day; I keep details about all things related to hours worked. Mostly this stems from having to manage working 3 or more programs while still billing appropriately for each accurate to the one-half hour. I'll admit that I've also got a slight tendency toward obsessive compulsiveness for some things.
In either case, I decided to take a look back over the past three years to see how my behavior has changed as I've become more comfortable with my surroundings and settled into a personal groove. I should mention first that I work in a very relaxed environment: jeans and t-shirt, flex hours, 24-hour access to premises and so on. This is reflected in the type of patterns that emerge here. Certainly, if I worked at a typical 9-to-5 this exercise would be moot.
Here is what the overall picture looks like[1]:
The blue points represent the times I arrive at work and the red indicates when I leave. You can see a clear pattern - especially in the start times - that is moving away from that 8:30am time. I split the start times into two groups and added a linear fit[2] to bring this out. It follows from this that there should be some split at the top but as each day is not exactly 8 hours the relationship is obviously not 1:1.
Why do the sets diverge then? It made me chuckle when I saw it because I deal with the problem every day and tire of it the more I do. See, I've got a 1-hour commute in each direction. I'm also forced to travel a particularly busy stretch of highway headed into Philadelphia to get to work. When I first started the job I was eager to impress and dealt with traffic to do so. As I've established myself I've decided to either get in early enough to avoid any traffic or wait until it is all passed before heading out. Again, not really possible in the 9-to-5 setting. I don't see actually following the projection lines; the three years have been enough for me to gauge just when to leave. The pattern has probably plateaued by now.
Since I couldn't decipher it from the above image, I also wanted to known what my average output was for the company (and if it had changed as much as my commuting habits). Here is the result: total hours worked for the same time frame shown above.
Takeaway is that I put in, on average, 9 solid hours a day. More obvious is that there is no telling how I'm going to do that over any given period. My first impression was that the hours would be high on Monday decreasing slowly to Friday where the average would be lowest (I like half-day Fridays). There was no such trend, however. In fact, the only day that had any significant difference from the others was Wednesday. I pick my daughter up from school on Wednesday so regardless of what time I get in I leave between 2:30 and 3:30. This ultimately drives down the number of hours I work on Wednesdays over time.
Nothing particularly groundbreaking. While my opinion of the work I do has fluctuated over time I can see no evidence of that in my behavior (from this data, anyway). The traffic thing was a neat find but I was disappointed to not have had more interesting trends in the distribution of hours over the week.
Still, I got to have some fun playing with R and a bit of data.
[1] In all the data presented here I've removed extreme outliers such as a stint I had working 10:00pm to 4:00am. I intended to explore my general behavior so I took some liberties with the data used in doing so.
[2] I failed Prob/Stat the first time through. Literally. I believe there is some underlying assumption about normal distribution when fitting a line which possibly makes this inaccurate as the distribution of each set of points is slightly skewed.
Here is what the overall picture looks like[1]:
The blue points represent the times I arrive at work and the red indicates when I leave. You can see a clear pattern - especially in the start times - that is moving away from that 8:30am time. I split the start times into two groups and added a linear fit[2] to bring this out. It follows from this that there should be some split at the top but as each day is not exactly 8 hours the relationship is obviously not 1:1.
Why do the sets diverge then? It made me chuckle when I saw it because I deal with the problem every day and tire of it the more I do. See, I've got a 1-hour commute in each direction. I'm also forced to travel a particularly busy stretch of highway headed into Philadelphia to get to work. When I first started the job I was eager to impress and dealt with traffic to do so. As I've established myself I've decided to either get in early enough to avoid any traffic or wait until it is all passed before heading out. Again, not really possible in the 9-to-5 setting. I don't see actually following the projection lines; the three years have been enough for me to gauge just when to leave. The pattern has probably plateaued by now.
Since I couldn't decipher it from the above image, I also wanted to known what my average output was for the company (and if it had changed as much as my commuting habits). Here is the result: total hours worked for the same time frame shown above.
Takeaway is that I put in, on average, 9 solid hours a day. More obvious is that there is no telling how I'm going to do that over any given period. My first impression was that the hours would be high on Monday decreasing slowly to Friday where the average would be lowest (I like half-day Fridays). There was no such trend, however. In fact, the only day that had any significant difference from the others was Wednesday. I pick my daughter up from school on Wednesday so regardless of what time I get in I leave between 2:30 and 3:30. This ultimately drives down the number of hours I work on Wednesdays over time.
Nothing particularly groundbreaking. While my opinion of the work I do has fluctuated over time I can see no evidence of that in my behavior (from this data, anyway). The traffic thing was a neat find but I was disappointed to not have had more interesting trends in the distribution of hours over the week.
Still, I got to have some fun playing with R and a bit of data.
[1] In all the data presented here I've removed extreme outliers such as a stint I had working 10:00pm to 4:00am. I intended to explore my general behavior so I took some liberties with the data used in doing so.
[2] I failed Prob/Stat the first time through. Literally. I believe there is some underlying assumption about normal distribution when fitting a line which possibly makes this inaccurate as the distribution of each set of points is slightly skewed.
Subscribe to:
Posts
(
Atom
)









