From information overflow, to a spotless mind

From Charles-Philippe Larivière...

It seems that I have been suffering an acute case of writer's block for the first few months of year 2012.

And what better way to start the year, than writing a non-technical vaguely metaphysical post. It is fun to see how I got to posting about this. While I write these first few lines, I still haven't chosen a title. Like B. Pascal put it a long time ago, the last thing I'll do is figure what will go first.

So, a piece of news not so recently suggested that the last DRAM chip maker in Japan, called Elpida, would go bankrupt. And then it did. You've never heard of it before, have you ? Well that's how I started to think again about memory.

As self-centred egotistical human beings, we basically build our memories with our own experience. Unfortunately, some things go well beyond our understanding and need taking a few thousands steps back to behold the big picture. That's where History comes into play. You could see it as a purveyor of fabricated memories, or as a conservation mechanism for the collective memory of mankind. Notice how History is both important and out of control at the same time. This is the setting that gave so much strength to the 1984 book in my own self-centered opinion.

The folk memory of fishermen in the Fukushima prefecture says that when a quake hits, a tsunami will follow. It also says that you had better go out to sea before the wave becomes to steep to climb. This is how a few boats were saved.

Our History is full of important lessons that should be remembered, and yet, we are now bathing in useless information, both figuratively and literally speaking. From the electromagnetic waves of wifi, cellphones, satellites, land TV and radio, to the clusterfuck of social media, RSS aggregation and whatever will come next - we are even drowning in information. And although the face value of it depends on how much each piece of information relates to you, chances are that not a billionth of it is worth your attention.

This is like dilution and filtering. The important information is diluted, your interest might not match those of dedicated mainstream portals, and obtaining what you want is frustrating. Meanwhile, the human brain received billions of sensory reports from all over the body, and yet, you could read these words without having to deal with it. Only valuable information made it to conscious level. Why can't our computers do this work of filtering for us, skipping over all that is not useful, skimming the things that we must read ?

Granted, this is hard from a computer software point of view. This is why regular users don't have it. Big companies do, however. That's how your webmail is free of monetary cost. Instead, computer programs are laughing at your information and selling select pieces to each other.

...to Sandow Birk

But eventually, you would expect people governing us to employ such filters, be they humans or computers, to isolate what is important, focus on it, and rule accordingly. So, how come a megaquake more than 1000 years ago went forgotten ? How come a ridiculously low $5 millions cut can be decided on a tsunami advisory program ? When will people stop doing exactly the same wrong thing every once in a long while, that is, long enough to not remember a bit...

NOTE: the two paintings came to my attention through the "google filter" and this page.

Looking back on 2011

Dear readers,

So the year 2011 is coming to an end and I have eventually failed my secret goal of having more articles than last year. The main reason being that I did not  post much in the last three months. And the explanation is just a single word : Tamias. Tamias is a pet name, a kind of chipmunk (although some folks complained that our logo looks like a mutated rat or something, hi Ed!). It could have been the name of a pet-project, but it really is the pet-name of a project we have here, at the office.

I can't really explain it all in one post, so let's say a few very basic things to keep it short and interesting.

1. Tamias is based on Tahoe-LAFS, which makes it awesome from the start
2. Tamias allows users sharing a mutual interest to gather together and create their own cloudish storage system by providing storage space to the distributed storage "cloud". Sorry for the buzz word, you can really replace cloud by network, p2p network, overlay network, whatever.
3. Users from a same network exchange identity information between each others (essentially public keys) and can share files privately and securely. Privately means that someone learning the access information (what is called a capability here, think of it as a self-certifying URL) from someone else can not use it. Securely means that file blocks are stored with encryption and thus not readable by the storage node (this is granted by Tahoe-LAFS).
4. The first beta version of Tamias was released on December 25th here.

This version works, but it is not very user-friendly, so we only expect the most computer savvy users to try it out and report errors, problems, bugs and so on. In the future, we will provide a (much much much) better UI of course, and new features like user self-introduction, user search, sharing delegation, networks federation, etc...

The Tamias website is at https://tamias.iijlab.net and has a few tutorials about compiling the software or setting up a node. As it is based on Tahoe-LAFS, it includes Tahoe-LAFS and builds upon it. However, you can not use your existing Tahoe-LAFS grid without modifications, because we added a couple of features on the storage server side, that were needed for privacy extensions.

Short post for a long story, but here's a scoop for you all : the next release will be much better and is planned for next spring...

Happy year-end holidays to you all !

Jean

Realtime visualization for Xbee+Accelerometer data

Here is the long awaited follow-up to the Xbee experiments that my little brother outsourced to me.

Before I get to the description of the software itself, if you want to try this out, be sure to follow the guidelines in the previous post on the same topic. You are then ready to proceed.

And here is what you get with this software :

• Realtime visualization of all three axis
• Can be adapted to many different accelerometers by tuning some global variables
• Recording of samples to file, with timestamps and X, Y and Z values
• Automatically generated plots for each axis and a combined 3-axis plot
• Exported files named according to experiment date and time (for easier sorting)
• All the above, in pure opensource python goodness
• English and French localization
• Theoretically works from within a LiveCD

Now let me briefly describe what the software does. As you can see in the screenshot on the side, the user interface is quite easy to understand. The three rectangle-shaped black boxes are showing whatever comes from the Xbee chip hooked up to the USB port, in realtime.

There are also three clickable buttons. The leftmost button labelled 'start' triggers the beginning of the recording. You will usually push this just before your experiment. The middle button named 'stop' obviously halts the recording. It also takes care of dumping all the samples into a CSV and generating the four plots : a combined plot with all 3 axis and three separate plots, one per axis. Of course the last button closes the USB port and quits the program. Beware that it does not check if you have a running experiment and will hence lose some data unless you push the stop button first.

Enough talking now, here is the python archive, all you have to do is extract the contents somewhere and run the software by launching the xviz.py script file.

I also uploaded this to the github social coding site, feel free to fork this, submit patches or whatever !

https://github.com/jean-/xviz

Wanted: Volunteers to give a hand on next holiday

Dear readers,

Here is a quick call for volunteering.

An association I am working voluntarily is organizing a trip to Sendai area to deliver home appliances and furniture we have been gathering from the people who left Japan.

As we do not have enough hands for the whole work, we are actively recruiting people who would like to help. Here is the full text : 

Silver Week is coming, the "friends" are hiring
----------------------------------------------
by Les Amis du Tohoku on Monday, September 12, 2011 at 11:20pm

EDIT: the day has changed from 23rd to 22nd in order to avoid congestion on the Tohoku expressway on a national holiday, moreover reduced to a single lane because of (long overdue) construction work.

Dear friends, through our Facebook page you have been able to follow us and our activities, picking up furniture and home appliances around Tokyo. You have been hearing our calls for donation but were not moving and had neither furniture nor home appliances to donate.

Now is your chance to make a difference, to show that you really love Japan, and that you are a friend of the disaster-hit Tohoku area.

Come join us, come help us for the preparations to the next journey, or even the journey itself, on September 22nd 2011.

We are looking for the following kind of volunteers :
- On the day of departure, people to help clean the goods
- On the day of departure, people to help load the truck 
- From the day of departure to following evening, people to drive the truck (standard driving license required*)
- From the day of departure to following evening, people to join us with their own car and come help to unload.

Don't hesitate to contact us if you would like to find out about this trip, to volunteer,or for any other reason. by mail: jean.lorchat@gmail.com

[*]: for any inquiries about driving license, please contact us as well

----------------------------------------------

 And here is the original post. les amis du Tohoku : silver week is coming

Adding Pachube readings to any PHP website

Whew, finally got to clean up almost all the frenchisms from the code...

As promised earlier, here is a PHP code snippet to fetch data from the Pachube web service using their API. If this is the first time you are using a web service, there is something you need to know first : web services don't run for free. So even if you are paying for some service, it does not mean you can abuse it by querying for some information every second.

As any API-backed service, Pachube will require you to sign-up in order to get your API-key that can be used in your scripts. This key is identifying you and must thus not be handed around. For those who do not have an account, obviously go ahead and create one. You might want to start with a 'Basic' plan which is free, until you find that you are reaching the limits of the system and want to switch to something more beefy.

When you're done registering, you need to create an API key for use by the PHP script. One way to go is to use the master API key in your script but this is not such a great idea. Instead, create a "secure sharing key" so that if someone gets to see your code, he won't be able to do much to your existing feeds and so on. You can find these options on the right-hand side menu of the Pachube website, after you have logged-in. It is called "My API keys" and resides withing the "My account" section. Now that you have created some key with at least the 'GET' permission, write it down, we'll use it later.

Now the other thing to take into account is, you only have a limited number of queries available over time. Depending on the web service provider, the type of account (free vs premium for example) and other likely parameters, you can find out this number. In the case of the Pachube Basic account, the limit is 5 requests per minute. I wanted to update some Geiger counter reading every 5 minutes, so this is plenty enough, great ! Here is how it looks on our PTA web site, pretty cool huh ?

How we are going to deal with this in the code is by caching the web server response. The example I am showing here is very crude, but at least it works and respects the web service netiquette by not flooding the server with superfluous requests.

Another final word about web services : JSON. This stands for JavaScript Object Notation. It used to be a very convenient way for service to return data, because you would just get a bit JS code that once evaluated would have populated your namespace with the data you had requested. Obviously, evaluating data returned by a service that can potentially host data created by malicious users is never a good idea. Well, we are doing PHP anyway so we don't care, plus PHP provides json parsing functions.

Now let's have a look at the code :

$json = file_get_contents('json.txt');



$flux = json_decode($json);



$right_now = new DateTime();

$seconds_now = $right_now->format('U');

$json_cache_date = new DateTime($flux->datastreams[0]->at);

$seconds_then = $json_cache_date->format('U');

What we have here is pretty simple. We open the cached json data (once again, crude example, this should probably be stored in a better place like your website database, and named in a more multi-feeds friendly way) and extract the information about when this data was returned. This is specific to Pachube's data format, other web services
might store this information in another place/way. We also generate information about current time, for comparison. Both times are expressed in epoch unit, which is a number of secondes elapsed since a specific event (guess which :P).

if ( (($seconds_now - $seconds_then) > 300) || (property_exists($flux, 'errors')) )

{ /* fetch and cache JSON data from feed every 5 minutes */

$request = curl_init();

curl_setopt($request, CURLOPT_URL, "http://api.pachube.com/v2/feeds/26485.json");

curl_setopt($request, CURLOPT_HTTPHEADER, array("X-PachubeApiKey: your_own_key") );

curl_setopt($request, CURLOPT_RETURNTRANSFER, TRUE);

curl_setopt($request, CURLOPT_BINARYTRANSFER, TRUE);

$json = curl_exec($request);

Now we have some check about whether the data should be fetched again or not. The interval I chose is 5 minutes because it does not really make sense to get the radiation reading in realtime except for some specific events from March. If the data is stale, we build the web service request using the curl library. Note the line that adds a header with your API key. This is where you should put the information you wrote down earlier. Don't forget the space between the double-colon and the start of your key. When the request is built, we throw it at the Pachube API server and get ready to handle the result.

if (($json !== FALSE) && (curl_getinfo($request, CURLINFO_HTTP_CODE) < 400)) 

{   

$flux = json_decode($json);

if ($flux !== FALSE)     

  file_put_contents('json.txt', $json); 

}

}

Several things can happen here. The server can return an error, it sometimes happen at night for some reasons I could not find out because I was sleeping... Or the fetching can fail if you have network connectivity problems, etc... So we check that we got an answer from the API server and that this answer was a real json object. In which case we cache it and decode it.

if ($flux !== FALSE)

{

$date = new DateTime($flux->datastreams[0]->at);

$date->setTimeZone(new DateTimeZone("Asia/Tokyo"));

echo 'On : ' . $date->format('Y-m-d') .' at '. $date->format('H:i:s');

echo '<br>';

echo 'value was: ' . $flux->datastreams[0]->current_value;

echo ' ' . $flux->datastreams[0]->unit->label;

}

Whether we got the information from the cache or from the API web server, there are cases where the data has errors, in which case you do not want to show it. But if we got the requested information, all that's left to do is parse it and push it to the user in a human readable way. Et voila !