Jean's tech blog

Looking back on 2011

2011-12-27T10:17:00.000+09:00

Dear readers,

So the year 2011 is coming to an end and I have eventually failed my secret goal of having more articles than last year. The main reason being that I did not post much in the last three months. And the explanation is just a single word : Tamias. Tamias is a pet name, a kind of chipmunk (although some folks complained that our logo looks like a mutated rat or something, hi Ed!). It could have been the name of a pet-project, but it really is the pet-name of a project we have here, at the office.

I can't really explain it all in one post, so let's say a few very basic things to keep it short and interesting.

Tamias is based on Tahoe-LAFS, which makes it awesome from the start
Tamias allows users sharing a mutual interest to gather together and create their own cloudish storage system by providing storage space to the distributed storage "cloud". Sorry for the buzz word, you can really replace cloud by network, p2p network, overlay network, whatever.
Users from a same network exchange identity information between each others (essentially public keys) and can share files privately and securely. Privately means that someone learning the access information (what is called a capability here, think of it as a self-certifying URL) from someone else can not use it. Securely means that file blocks are stored with encryption and thus not readable by the storage node (this is granted by Tahoe-LAFS).
The first beta version of Tamias was released on December 25th here.

This version works, but it is not very user-friendly, so we only expect the most computer savvy users to try it out and report errors, problems, bugs and so on. In the future, we will provide a (much much much) better UI of course, and new features like user self-introduction, user search, sharing delegation, networks federation, etc...

The Tamias website is at https://tamias.iijlab.net and has a few tutorials about compiling the software or setting up a node. As it is based on Tahoe-LAFS, it includes Tahoe-LAFS and builds upon it. However, you can not use your existing Tahoe-LAFS grid without modifications, because we added a couple of features on the storage server side, that were needed for privacy extensions.

Short post for a long story, but here's a scoop for you all : the next release will be much better and is planned for next spring...

Happy year-end holidays to you all !

Jean

Realtime visualization for Xbee+Accelerometer data

2011-09-21T10:22:00.000+09:00

Here is the long awaited follow-up to the Xbee experiments that my little brother outsourced to me.

Before I get to the description of the software itself, if you want to try this out, be sure to follow the guidelines in the previous post on the same topic. You are then ready to proceed.

And here is what you get with this software :

Realtime visualization of all three axis
Can be adapted to many different accelerometers by tuning some global variables
Recording of samples to file, with timestamps and X, Y and Z values
Automatically generated plots for each axis and a combined 3-axis plot
Exported files named according to experiment date and time (for easier sorting)
All the above, in pure opensource python goodness
English and French localization
Theoretically works from within a LiveCD

Now let me briefly describe what the software does. As you can see in the screenshot on the side, the user interface is quite easy to understand. The three rectangle-shaped black boxes are showing whatever comes from the Xbee chip hooked up to the USB port, in realtime.

There are also three clickable buttons. The leftmost button labelled 'start' triggers the beginning of the recording. You will usually push this just before your experiment. The middle button named 'stop' obviously halts the recording. It also takes care of dumping all the samples into a CSV and generating the four plots : a combined plot with all 3 axis and three separate plots, one per axis. Of course the last button closes the USB port and quits the program. Beware that it does not check if you have a running experiment and will hence lose some data unless you push the stop button first.

Enough talking now, here is the python archive, all you have to do is extract the contents somewhere and run the software by launching the xviz.py script file.

I also uploaded this to the github social coding site, feel free to fork this, submit patches or whatever !

https://github.com/jean-/xviz

Wanted: Volunteers to give a hand on next holiday

2011-09-15T14:18:00.000+09:00

Dear readers,

Here is a quick call for volunteering.

An association I am working voluntarily is organizing a trip to Sendai area to deliver home appliances and furniture we have been gathering from the people who left Japan.

As we do not have enough hands for the whole work, we are actively recruiting people who would like to help. Here is the full text :

Silver Week is coming, the "friends" are hiring
----------------------------------------------
by Les Amis du Tohoku on Monday, September 12, 2011 at 11:20pm

EDIT: the day has changed from 23rd to 22nd in order to avoid congestion on the Tohoku expressway on a national holiday, moreover reduced to a single lane because of (long overdue) construction work.

Dear friends, through our Facebook page you have beenable to follow us and our activities, picking up furniture and home appliances around Tokyo.You have been hearing our calls for donation but were not moving and had neither furniture nor home appliances to donate.

Now is your chance to make a difference, to show that youreally love Japan, and that you are a friend of the disaster-hit Tohoku area.

Come join us, come help us for the preparations to the next journey, or even the journey itself, on September 22nd 2011.

We are looking for the following kind of volunteers :
- On the day of departure, people to help clean the goods
- On the day of departure, people to help load the truck
- From the day of departure to following evening, people to drive the truck (standard driving license required*)
- From the day of departure to following evening, people to join us with their own car and come help to unload.

Don't hesitate to contact us if you would like to find out about this trip, to volunteer,or for any other reason.by mail: jean.lorchat@gmail.com

[*]: for any inquiries about driving license, please contact us as well

----------------------------------------------

And here is the original post. les amis du Tohoku : silver week is coming

Adding Pachube readings to any PHP website

2011-08-12T11:01:00.013+09:00

Whew, finally got to clean up almost all the frenchisms from the code...

As promised earlier, here is a PHP code snippet to fetch data from the Pachube web service using their API. If this is the first time you are using a web service, there is something you need to know first : web services don't run for free. So even if you are paying for some service, it does not mean you can abuse it by querying for some information every second.

As any API-backed service, Pachube will require you to sign-up in order to get your API-key that can be used in your scripts. This key is identifying you and must thus not be handed around. For those who do not have an account, obviously go ahead and create one. You might want to start with a 'Basic' plan which is free, until you find that you are reaching the limits of the system and want to switch to something more beefy.

When you're done registering, you need to create an API key for use by the PHP script. One way to go is to use the master API key in your script but this is not such a great idea. Instead, create a "secure sharing key" so that if someone gets to see your code, he won't be able to do much to your existing feeds and so on. You can find these options on the right-hand side menu of the Pachube website, after you have logged-in. It is called "My API keys" and resides withing the "My account" section. Now that you have created some key with at least the 'GET' permission, write it down, we'll use it later.

Now the other thing to take into account is, you only have a limited number of queries available over time. Depending on the web service provider, the type of account (free vs premium for example) and other likely parameters, you can find out this number. In the case of the Pachube Basic account, the limit is 5 requests per minute. I wanted to update some Geiger counter reading every 5 minutes, so this is plenty enough, great ! Here is how it looks on our PTA web site, pretty cool huh ?

How we are going to deal with this in the code is by caching the web server response. The example I am showing here is very crude, but at least it works and respects the web service netiquette by not flooding the server with superfluous requests.

Another final word about web services : JSON. This stands for JavaScript Object Notation. It used to be a very convenient way for service to return data, because you would just get a bit JS code that once evaluated would have populated your namespace with the data you had requested. Obviously, evaluating data returned by a service that can potentially host data created by malicious users is never a good idea. Well, we are doing PHP anyway so we don't care, plus PHP provides json parsing functions.

Now let's have a look at the code :

$json = file_get_contents('json.txt');



$flux = json_decode($json);



$right_now = new DateTime();

$seconds_now = $right_now->format('U');

$json_cache_date = new DateTime($flux->datastreams[0]->at);

$seconds_then = $json_cache_date->format('U');

What we have here is pretty simple. We open the cached json data (once again, crude example, this should probably be stored in a better place like your website database, and named in a more multi-feeds friendly way) and extract the information about when this data was returned. This is specific to Pachube's data format, other web services
might store this information in another place/way. We also generate information about current time, for comparison. Both times are expressed in epoch unit, which is a number of secondes elapsed since a specific event (guess which :P).

if ( (($seconds_now - $seconds_then) > 300) || (property_exists($flux, 'errors')) )

{ /* fetch and cache JSON data from feed every 5 minutes */

$request = curl_init();

curl_setopt($request, CURLOPT_URL, "http://api.pachube.com/v2/feeds/26485.json");

curl_setopt($request, CURLOPT_HTTPHEADER, array("X-PachubeApiKey: your_own_key") );

curl_setopt($request, CURLOPT_RETURNTRANSFER, TRUE);

curl_setopt($request, CURLOPT_BINARYTRANSFER, TRUE);

$json = curl_exec($request);

Now we have some check about whether the data should be fetched again or not. The interval I chose is 5 minutes because it does not really make sense to get the radiation reading in realtime except for some specific events from March. If the data is stale, we build the web service request using the curl library. Note the line that adds a header with your API key. This is where you should put the information you wrote down earlier. Don't forget the space between the double-colon and the start of your key. When the request is built, we throw it at the Pachube API server and get ready to handle the result.

if (($json !== FALSE) && (curl_getinfo($request, CURLINFO_HTTP_CODE) < 400)) 

{   

$flux = json_decode($json);

if ($flux !== FALSE)     

  file_put_contents('json.txt', $json); 

}

}

Several things can happen here. The server can return an error, it sometimes happen at night for some reasons I could not find out because I was sleeping... Or the fetching can fail if you have network connectivity problems, etc... So we check that we got an answer from the API server and that this answer was a real json object. In which case we cache it and decode it.

if ($flux !== FALSE)

{

$date = new DateTime($flux->datastreams[0]->at);

$date->setTimeZone(new DateTimeZone("Asia/Tokyo"));

echo 'On : ' . $date->format('Y-m-d') .' at '. $date->format('H:i:s');

echo '<br>';

echo 'value was: ' . $flux->datastreams[0]->current_value;

echo ' ' . $flux->datastreams[0]->unit->label;

}

Whether we got the information from the cache or from the API web server, there are cases where the data has errors, in which case you do not want to show it. But if we got the requested information, all that's left to do is parse it and push it to the user in a human readable way. Et voila !

Accelerometer + Xbee = geekish fun

2011-08-12T09:18:00.007+09:00

Hi there, I hope you are all having a nice holiday season.

As for me, I am keeping quite busy despite the loneliness and am desperately trying to do my job, some volunteer help for the Tohoku area, turning the website for our PTA upside-down, transforming a two storeys (stories?) open room type house in a 3+LDK, and helping out my lil brother on his science assignment. And there's today's story.

So introduce my brother, who has been playing with water rockets since he was in high school. Now for his first year at university he managed to get himself a science assignment dealing with those

very things. I won't explain the principle at hand here because I already have lots of things to say, so look up wikipedia (I suggest reading the discussion page there too, it is abysmal in epic proportions) or check out this NASA picture.

Now you can imagine that you can do lots of fun stuff with your own rocket, especially when you learn that the record height achieved is around 600 meters... Well, a 2L soda bottle won't go that high, but it is still a very interesting way to conduct extreme experiments around embedded/autonomous systems.

When I was around the parents' home back in France, he tells me about his latest experiments

and how they thought they wanted to get telemetry readings for the system. That sounded cool, and I was thinking about my co-worker here who does all those neat things with arduinos, xbees, sensors and so on...

Actually, my brother and his friends' main skills are neither geekygadgetry nor computer science, so the final picture is not as complex as I imagined, but still beyond their reach. Based on a survey around his friends and professors, he gathered that Xbees are cool and can work in an autonomous way without any microcontroller, which is good when you don't have a clue. They are small chips (see pic on the right) that do 802.15.4, present a serial emulation to the user, feature 5 analog inputs and an associated ADC, and have a decent (depending on your own definition of decent) sampling frequency.

Well, to be honest he actually told me about this all on the phone before I came back, so that I could do a small shopping trip around akiba and bring an accelerometer. There it is : KXM52-1050, straight from Akizukidenki, behind the now defunct Yamagiwa/Livina (just saw they were doing some construction work there, something happening soon I guess). So this little chip eats 3.3V which is all great because most of the boards that can host the Xbee provide a 3.3v regulator to feed the sensors.

Now here's his plan : one Xbee on the rocket, with some form of power supply, hosted on a regulated board, where to hook the accelerometer sensor as well. On the receiving side, another Xbee (using just one doesn't make sense, does it ?) on a USB board plugged directly into a computer, and voila. It sounded so EASY.

Now that I look back on the whole process, it was. But for some reasons, the Xbee manufacturers went through great pain to write a very detailed yet completely useless documentation. All the information needed is inside, but after reading the whole 68 pages, I felt like I did not know what to do. Also there has been several versions of the chip, things changed, and there is no clear history of what went and what didn't. One important thing I learned at my expense is that the ADC used to do the conversion on the fixed 0~1.2V range only. But, in the latest document there is no mention about the voltage range. There is a two-line section about ADC that states that "Xbee supports ADC and here is how to enable it". Period. After lots of digging I found out that the "Pin signals" table stated that Pin14 is for "Voltage reference for A/D input". If the 68 pages were a haystack, this line would be the needle.

And then, the plan worked ! That is, until I saw the "software" that some friend gave to him in order to do the Xbee monitoring. If you can call a VB 20-liner a software, that is. Which is way I made another one for him that does .csv export and much more. More on it in another article but meanwhile, enjoy the output. Guess which line's the Z ?

PS: I know I am getting behind schedule with the follow-up to the pachube article. It is still in the pipeline...

The internet of things is good for your health

2011-06-07T07:45:00.006+09:00

At some point, the "Internet of Things" (IoT) was a quite popular topic. I guess they lost the buzz war against the clouds but yeah, it's a fight you'll never win... However, the whispering shadows of sensor mania and IoT can still be heard if you listen closely. And, IoT can actually be good for your health !

As a concept, IoT is as loosely defined as "cloud", but what did you expect from a buzz war contender anyway ? For this article we will focus on an instance of a very specific IoT concept and platform, called pachube. In just one sentence, Pachube is a web service that provides centralized hosting for sensor data. As most web services it is rate limited, and offers different limits based on the premium you pay. There is also a free service.

On the other hand, I guess that by now most of the people reading this have learned about the twin disaster that hit Japan, followed by both an unlikely nuclear meltdown and unfortunate radioactive leaks in the air, soil and water. In this context, people suddenly realized about Geiger counters and they quickly sold out. A few brave people put theirs on the web, through various means (digitizer software, webcams, etc...).

Now this is the kind of Thing that can go into the Internet of Things. So a few of the people that had digital probes or probes with digitizer software set up a pachube account to host measurement data. Then someone stepped forward to put together a user interface for all those values and plot near-realtime measurements on a map. Here it is : http://japan.failedrobot.com

Next I will show you how to get some values from pachube to put on display in your own webpage, so stay tuned...

User interfaces gone fuzzy, a case against new interaction devices

2011-04-18T10:19:00.015+09:00

So this is a rant about all the trendy new interaction devices. In fact, not so much against the devices themselves, but the way people keep trying to use them to emulate old things.

Our knowledge, our science, our tools have been built in an incremental way. Of course, who would invent a spanner when nuts and bolts don't exist ? And even though sometimes, someone has a great idea that is very far-fetched, it also often meets with doubts and perplexity. This is how we just avoided getting computers 100 years early, which might have bumped human technology on a very different path.

And so this is how after getting computers and monitors, people feeling the urge to interact with those beasts soon made out the keyboard, and then the mouse. For a very long time keyboard and mouse have reigned supreme over the world of user interfaces.

Hypertext Editing System.
Original photo by Greg Lloyd
source: wikipedia

Of course there had been attempts at alternative ways, but these interfaces never took off and were doomed to stay a niche market. Probably a few schoolboys and girls from the 80s in France remember the Thomson light pen ? And there have been graphical tablets for decades now, and people using 3DVR stations have those gloves, and videogames have their own things filled with buttons.

But we all know the latest trend that seems to take off for real, is touch screens. This is not the dawn of touch interfaces, it has been pioneered for long by the graphics industry. Despite a few attempts at PDAs and smartphones (P900 anyone ?) that were expensive toys, the widespread use of touch devices has more likely begun with the nintendoDS and then the iPhone.

So what's wrong with touch devices ? I would say feedback. When you learnt to use a keyboard for so long, feedback is what makes you know you mistyped even before seeing it. This does not happen with touchscreen emulated keyboards. Now worse, the software thinks it is smart enough to guess what you intended to type, even in some situations where you did *NOT* make a mistake.

This is all because we want to use touchscreens the way we used keyboards. But touchscreens will never be keyboards, they are screens ! Kids like to look at pictures and slide them to see the next one. This is intuitive and cool. Probably the kids born around now will never have to use a real keyboard and won't miss it. But for me, as much as the mouse supplements the keyboard but can't substitute to it, so much can be said about the touchscreen. How many people use mouse emulation with the numpad ?

Don't type technical words !

This all-touch trend, I call it "fuzzy interface". And by the way, I find it much more straining to type on the touchscreen because you have to check continuously that no error has been made, either by you OR the software. Here is an anecdote to close this case, redacted from a Google Summer of Code mailing list for mentors :

. Random person says ''company XYZ can help assess candidate students abilities blablabla....''
. Person in charge from Google answers ''actually I am hot interested in other companies reviewing our students..."
... 8hrs later, same Google person : ''Not, not hot. Touch screen keyboard :-)"

Happy touching !

Best wishes for 2011

2011-01-06T15:29:00.004+09:00

Hi dear readers, and welcome to the year 2011 (of the cute bunny) that should hopefully be more peaceful than year 2010 (of the tiger).

As some people might know, my hobbies include (a lot of) cooking so it is with a recipe that I am going to kick off the new year. All ingredients can be bought here in Japan, so you can probably cook this one almost everywhere in the world.

First, the ingredients :

. 2 pie sheets
. 100g sugar
. 100g almond powder
. 50g butter
. 2 eggs
. Rum or any other flavor (e.g. vanilla essence)

Preparation :

. Place butter, sugar, almond powder and eggs in a bowl (in that order)
. Mix slowly (especially if using an electric appliance)
. When the dry ingredients start getting wet, add flavor and mix faster (to obtain an emulsion)
. This emulsion is the almond cream

. Cut two round shapes from the pie sheets, one being bigger than the other
. The small diameter disk is the bottom of the pie, the large one will be the lid
. Wet the outer part of the small disk (so that the lid will stick when you lay it on top)
. Spread the almond cream in the center
. Insert a bean or anything hard that can withstand oven temperature
. Lay the larger disk on top and weld it on the small disk around the cream so that it doesn't leak (you can use a round cake mold for example to do that)
. "Flute" the edges, by pushing with your finger and chipping with the point of a knife, repeat this all around the disk
. Drill a small hole a the center so that vapor can leave the pie without making the whole thing burst
. Beat another egg and spread it on the pie, then do some fancy drawings with the point of your knife
. Let it dry then spread the beaten egg again
. Bake at 200C for 10 mins, then at 170C for 20 mins

You're done, enjoy.

Small cultural note : this pie is called "La galette des rois" and is connected to the christian celebration of "Epiphany", somewhat related to the persons who visited baby Jesus to bring him presents.

In our family, this pie is just linked to January 6th and a good way to stuff our stomach !

read.air : How to sort a "Vacances" worth of pictures on-the-go ?

2010-08-31T15:59:00.001+09:00

This is the story of a pet project I called read.air.

So the holidays are nearing the end, and I am so happy that I am back to Tokyo for the hot days. When I say I am going to take 3 weeks off, people here dub it "vacances" (read: bakansu) because apparently French people are customary to long holidays, so much so that the name has been borrowed.

More alarmingly, my wife has been gone for 2 months with a borrowed SLR digital camera (thanks to the generous sponsor by the way). So this means that instead of having thousand of pictures piling up in the iphone camera roll, they are now piling up on SD cards. She took around 4 or 5 of these 4Gb cards and since she is experimenting with the camera a lot, there are many many blurry, over/under-exposed, or just failed pictures.

As convenient as the back panel LCD might be, it is not as efficient as handling them from a much more powerful touch device, however I hate it. Also, the designers of the iphone did a perfect job at crippling it as far as interoperation is concerned. So how do you read a SD card with an iphone ? You don't.
When I surveyed the electronic products market for such a thing, I found out that there were one or two products that could read an SD card and plug on the iphone connector on the bottom, but they relied on proprietary apps and also there was no review or preview video to be found. I am not going to buy without a try, and it was nowhere to be found in shops, online retail only, send money first. No way.

So another product I found out was the AirStash and I have to admit I had been thinking about something like this for months but never got "unlazy" enough to actually build it. In fact, I got the idea nagging at the back of my mind everyday when I was looking at the unused FONera router on my desk at the office. The Airstash price tag at 100$ was the carrot I needed because I knew that I did not want to pay this price, and also that my wife would make me buy one if "read.air" was not ready before she left...

Now one thing I didn't need in this AirStash was the harddisk. I just wanted a SD card-reader over WiFi that would be accessed by Web interface. This means no proprietary app, just plain old Safari.

So in the end, if you have a FON router (it is now an old model, also they used to be almost free at some point) all you need to do is buy a USB card reader, 4 regular batteries (I'm using eneloops from sanyo) and some wires and plugs. Overall much below 100$. Mission accomplished, yay !

Next time I will tell you about some custom software I wrote for read.air, which is based on OpenWRT.

NOTE: I will add a pic of the thingie whenever I find one or she comes back, whichever happens first.

Floating point (of no return)

2010-07-02T15:28:00.003+09:00

As the song goes "No return, from the other side of Heaven". And indeed, welcome to Floating Point Hell !

I recently got involved in GSoC (Google Summer of Code, ask Google web search for the homepage) helping a student make Wideband audio codec become a reality in Sip Communicator. Which is a Java application.

And as expected from such a mathematical object, one needs to do all kind of clever calculations to compress a high rate audio source to a low rate bitstream. Of course, nowadays we receive a lot of support from the FPU making this all a piece of cake, with respect to calculation speed.

So, what's the big deal ? Let's see, as usual, it all begins with a standard. And a good one at that, the IEEE 754 standard. The problem with this standard is how it defines several bit format for floating point numbers, and how people decided, or not, to implement them.

And of course, as for most of the hardware things, there is a software counterpart that can start ruining things all the same. Well, the opposite is true and you can ruin good software with horrible hardware, but we are getting side tracked.

So, the wideband codec I am thinking about is CELT and I must say that it is pretty interesting, and much more geeky than the streamlined SILK codec (the one behind skype, also available to developers, kind of).

And a part of CELT implements mdct for some reason, and includes a test program. You don't see test-units for C code so often, the last I can think of where in NS (the network simulator) and I can't say it left me with any good keepsake.

So I try to run them, and guess what ? Results can differ. A lot. But not from one run to another, rather from one 'configuration' to another. For example, I get different results depending on the operating system. Or the compiler flags. An imac gives results that differ from a dell server (although they have the same FPU), which can themselves differ depending on the compiler flags I am using.

In the end, the reason I realized this is because this student tries to adapt libcelt (the CELT codec in a library form) to Sip Communicator, and gets very bad results when running the test. Not that bad, but much worse than the C counterpart. On the same hardware+OS obviously, or it wouldn't be fun.

The final word ? Floating point calculation seems to have been historically complicated (see references below) in the Java VM and it resulted in the current situation with FP-Strict and so on.

Technical explanation : actually, on x86 machines, the FPU supports the extended double-precision format that specifies at least 80 bits numbers. This being available, all calculations are done using the highest precision by the hardware, then rounded before being handled back to the software.

The thing is, this setting can be disabled, and probably has been, in some of the configurations that produce different results. Including within the JVM, although it sounds like it could be configurable according to the Java Language Specification (i.e. how Java should be, not always how it is practically).

Well, I had already had a glimpse of the FP woes that plague Java when doing the ilbc codec implementation, where I realized that C and Java implementations gave different results for the same sample data. I solved the problem by deciding that 'to my ear' the difference is inaudible.

This is probably the way to go, geez.

Further reading material, if you want to have some floating fun :

Ye olde How Java Floating Point Hurts Everyone Everywhere by William Kahan, that might not make the test-of-time award, but sounds so nostalgic. Remember the Sun vs Microsoft Java war...
The floating point chapter of the 'Introduction to GCC' book
A good technical article on Floating Point computations, aimed at what can (will?) go wrong
This webpage on the Sun/Java website with the simple title of "What Every Computer Scientist Should Know About Floating-Point Arithmetic" and apparently triggered by a W. Kahan lecture

Debian, IPv6 and ... Java ?

2010-05-06T16:07:00.001+09:00

Today is the first time for me to talk about Debian. It has been my favorite Linux flavor for quite some time now. I would say at least ten years, when compiling stuff became less fun and more painful, or my spare time ran out. I don't know exactly.

But whatever the cause, I ditched compiling everything in favor of the flexibility (and frustrating nature at times) of a packaging system. At that time, RPM did not prove better than Slackware's plain .tar.gz packaging and although dselect could be stubborn at times, it soon got better with apt-get and friends.

Let's add IPv6 to this landscape.

I discovered the details of this tentative IPv4 replacement protocol at around the same time I got acquainted with Debian. I will not try to cover the details of why IPv6 is better (or not) and why it takes so long to deploy (or not) but just make sure that you know the following important thing : IPv6 is not designed to be a drop-in replacement for IPv4, but is made so that it closely resembles.

And this is probably a very important point in today's mess. Being close enough, it might make people think that it takes a one-liner to update everything. Or just a new board/firmware for the router.

Let's focus on applications on Linux.

IPv6 API uses sockets, much like IPv4, and even uses the same calls for the basic socket creation, binding, listening and so on. Of course, the upper protocols are not modified, which means that UDP, TCP and friends still work.

When binding a socket with the AF_INET6 family, it is specified that the socket can be bound to both IPv4 and IPv6 address spaces. Really ? Yes, probably as a means to make it easier for people to port applications, the system will start listening on both families and you only have to create one socket.

This is made possible by adding a specific format for the IPv4 address which is called IPv6-mapped-IPv4 address. Basically it means that the 32 bits of the IPv4 address are mingled with a special meaning IPv6 prefix. Then you just check on those bits and figure if the incoming connection is IPv4 or IPv6, if you ever care. Which you should.

So how do we enable (or disable) this behavior ? That is the very reason of this post : the default behavior is undetermined. In fact, the default behavior depends on a sysctl value. So imagine you are a software developer, you made a software assuming that dual-stacked sockets were the default. Now imagine that the Linux distribution you (or your user) runs has a different setting. Or even better, changes the setting at some point. This is what happened for Debian.

This is exactly what happened. The variable is called net.ipv6.bindv6only and used to be 0, i.e. a AF_INET6 socket is bound to both v4 and v6. When it changed, it took people a little bit of unpleasant time to find out why the Java VM provided by Sun decided to stop working and report a network connection error. The thing is, the vm listens on some port on localhost and while using IPv4 as a default mean of connection, the socket was probably created with AF_INET6 family to help support both IPv4 and IPv6 (just guessing).

To sum it all up, man is your friend in the coding world : man ipv6 should provide you with all the necessary information. Now if you want to stick to the old style, you have two solutions.

The first solution is to update the sysctl config file in /etc/sysctl.d folder. You will find the file under the name bindv6only.conf

But this is an ugly workaround. As we all learned when writing our first C code program dealing with int arrays, there should be no hard-coded value for anything (at that time, I believe the teacher was referring to the define directive that decided how large the array was).

To fix it properly, we need to stop expecting other people's system to work like ours. Since the default behavior is undefined because of a sysctl value, we can force the behavior we want with a simple setsockopt call. Indeed !

This brings us to the morale of this story :

  int arg = 1;

  setsockopt(fd6, IPPROTO_IPV6, IPV6_V6ONLY, &arg, sizeof(arg));

which makes sure that the socket listens only on IPv6. Which prevents you from receiving a bind error in case another program listens on the same port in the IPv4 address space, and at the same time allows to receive IPv4 and IPv6 traffic on two separate sockets if you so wish.

Howto get a list of network interfaces in Linux, part 3

2010-03-15T16:05:00.003+09:00

Here we are, this is the ultimate post from these series. Thanks both previous posts, we are now able to get a list of interfaces using any language from shell script to whatever language which can speak ''sockets''.

However, as we could realize earlier, the ioctl-based method does not work exactly the way we want, because it does only report IP enabled devices. So if you want to deal with network interfaces before they start running, you are stuck with the /proc method. There must be another way. Remember that man netdevice we did last week. They mentioned netlink... What are those ?

Netlink sockets are sockets from the AF_NETLINK family. Being sockets, they are used to communicate between two addresses. The fun part is, netlink communications are local. And they usually (in the most simple scenario) link a user process with the kernel, especially the networking stack of the kernel.

Oh great, so we could query the kernel directly, to find out the contents of the global list of interfaces ? Indeed, we will. And since this tutorial is the most tricky of all three, let's have a detailed look at the source code. You can always grab the full source by following the link at the bottom of this post.

First we will open a socket :

int fd; 

(...)

fd = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);

We already discussed about the socket family, the type should be set to SOCK_RAW (the only one that makes sense here anyway) and the protocol is NETLINK_ROUTE. Why _ROUTE ? There are a few netlink protocols, each dealing with a specific part of the networking stack. NETLINK_XFRM deals with the transform layer used by security and mobility stacks, for example. And the protocol we chose deals with both routes and links.

Now we have to bind that socket. Netlink addresses are based on process ID, so let's do that :

struct sockaddr_nl local; 

(...) 

memset(&local, 0, sizeof(local));
local.nl_family = AF_NETLINK;

local.nl_pid = pid; /* can be obtained using getpid*/
local.nl_groups = 0;

if (bind(fd, (struct sockaddr *) &local, sizeof(local)) < 0)

... error handling ...

If an error occurs it might mean that you are not running the program as super-user or with sufficient rights, or that netlink support is missing in the kernel. And if the bind system call is successful, we can now start sending requests. The request we will be sending, is the one that makes the kernel answer by sending a list of all the interfaces, whatever their state.

typedef struct nl_req_s nl_req_t; 

struct nl_req_s {
  struct nlmsghdr hdr;
  struct rtgenmsg gen;
};

(...) 

struct sockaddr_nl kernel;

struct msghdr rtnl_msg;

struct iovec io;

nl_req_t req;

(...)

memset(&rtnl_msg, 0, sizeof(rtnl_msg));

memset(&kernel, 0, sizeof(kernel));
memset(&req, 0, sizeof(req));
(...) 

kernel.nl_family = AF_NETLINK; /* fill-in kernel address (destination) */

req.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg));
req.hdr.nlmsg_type = RTM_GETLINK;
req.hdr.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP; 
req.hdr.nlmsg_seq = 1;
req.hdr.nlmsg_pid = pid;
req.gen.rtgen_family = AF_PACKET; /*  no preferred AF, we will get *all* interfaces */

io.iov_base = &req;
io.iov_len = req.hdr.nlmsg_len;
rtnl_msg.msg_iov = &io;
rtnl_msg.msg_iovlen = 1;
rtnl_msg.msg_name = &kernel;
rtnl_msg.msg_namelen = sizeof(kernel);

sendmsg(fd, (struct msghdr *) &rtnl_msg, 0);

Okay, so this is the longest piece of code we will be quoting here. First we need to define a structure that can hold our specific message, since netlink can encapsulate so many things, we need to give some structure for this encapsulation. This the the nl_req_t type we just defined.

Then we prepare the destination address for our message. As expected, it is the ''kernel'' address which is specified by all zeroes except the family that is set to AF_NETLINK.

Now we prepare our request. It has a header, and a single message. The header follows the struct nlmsghdr format. In this structure, we will fill information about :

the message type : RTM_GETLINK, obviously used to get information about links.
the message length. For this, we use the NLMSG_LENGTH macro that makes some alignment calculation and also accounts for the header length. We thus have to pass the size of the payload only, and the result will be the aligned size of the whole netlink message. Our payload is a struct rtgenmsg structure, that is used to specify which address family we are interested in.
the message flags : NLM_F_REQUEST because we are sending a request (i.e. requiring an answer) and NLM_F_DUMP.
the message sequence number : not very important here because we will only send one message. Note that this number must be monotonically increasing.
the message pid : this is our the pid of the sending process, which as we said earlier, is related to the netlink address.

We also fill the rtgen_family field of our payload structure to make sure that we receive information about ALL interfaces, not only IP-related ones. Then we prepare the regular sendmsg stuff using struct msghdr and struct iovec friends.

The message is then complete, and is sent through a sendmsg call. And now, let's get ready for the answer.

int end = 0;

char reply[IFLIST_REPLY_BUFFER]; /* a large buffer */

(...)

  while (!end)
    {
      int len;
      struct nlmsghdr *msg_ptr;    /* pointer to current part */

      struct msghdr rtnl_reply;    /* generic msghdr structure */
      struct iovec io_reply;

      memset(&io_reply, 0, sizeof(io_reply));
      memset(&rtnl_reply, 0, sizeof(rtnl_reply));

      io.iov_base = reply;
      io.iov_len = IFLIST_REPLY_BUFFER;
      rtnl_reply.msg_iov = &io;
      rtnl_reply.msg_iovlen = 1;
      rtnl_reply.msg_name = &kernel;
      rtnl_reply.msg_namelen = sizeof(kernel);

      len = recvmsg(fd, &rtnl_reply, 0); /* read lots of data */

This is the first part of the loop that contains most of the initialization logic. It is relying on a end flag that will be updated upon receiving the message that defines an end of answer. Then recvmsg structures are prepared to read a very large message. This netlink message itself can have several message in it's payload, this is why we are going to need a nested loop :

  if (len)
    {
      for (msg_ptr = (struct nlmsghdr *) reply; NLMSG_OK(msg_ptr, len); msg_ptr = NLMSG_NEXT(msg_ptr, len))
        {
          switch(msg_ptr->nlmsg_type)
          {
        case 3:        /* this is the NLMSG_DONE end msg */
          end++;
          break;
        case 16:    /* this is the RTM_NEWLINK msg */
          rtnl_print_link(msg_ptr);
          break;
        default:    /* for education only,
                   should not happen here */
          printf("message type %d, length %d\n", msg_ptr->nlmsg_type, msg_ptr->nlmsg_len);
          break;
          }
        }
    }

Here we use a few of those NLMSG macros, to walk the big netlink message. Basically, NLMSG_OK checks that the message specified by the parameter pointer is indeed a valid netlink message, while NLMSG_NEXT steps over a single iteration, moving from the parameter pointer current message to the next message in the netlink payload, while the global length is specified to avoid buffer overrun. Upon receiving NLMSG_DONE message, we just update our end of loop condition. And we use a separated function to print out new link information. Here it is :

void
rtnl_print_link(struct nlmsghdr *h)
{
struct ifinfomsg *iface;
struct rtattr *attribute;
int len;

iface = NLMSG_DATA(h);
len = h->nlmsg_len - NLMSG_LENGTH(sizeof(*iface));

for (attribute = IFLA_RTA(iface); RTA_OK(attribute, len); attribute = RTA_NEXT(attribute, len))
    {
      switch(attribute->rta_type)
      {
      case IFLA_IFNAME:
      printf("Interface %d : %s\n", iface->ifi_index, (char *) RTA_DATA(attribute));
      break;
      default:
      break;
      }
    }
}

Very simple function made of mostly a big loop. Here we can find some new macros that look a lot like the NLMSG ones. They have an equivalent behavior but are handling different data. NLMSG_ macros are handling the netlink message and allows to parse all parts of a multi-part query or request. Then, each part itself is made of several attributes (think of route information here, routes have lots of attributes, and so do links). So RTA_ stands for route attribute, and the effect of the RTA macros is identical to NLMSG ones. This allows us to loop over ALL attributes of a link.

Also, in order to catch the first of those RTAs, we need to know where they start. This depends on the kind of message we received, but as far as this function is concerned, the message type is known to be RTM_NEWLINK. So IFLA_RTA allows us to get a pointer to the first attribute in the RTM_NEWLINK message.

For the sake of this exemple, we will only consider the interface name, but there is much more information (MTU, hardware address, link type, flags, state, etc...). You can check out for more information, like the list of possible attributes.

And now, let's run our example :

Interface 1 : lo
Interface 2 : eth1
Interface 3 : wlan1
Interface 4 : pan0

Where we can see that they are all returned, including the down ones. Victory ! By the way, the number we are printing here is the interface index, that is very useful for subsequent actions, you'd better store it next to the interface name.

I hope this was an interesting (long) read. Again, feel free to use this code provided you give proper credits.

PS: for the lazy people who want to try it right away, here is a link to the full source.

Howto get a list of network interfaces in Linux, part 2

2010-02-19T14:38:00.002+09:00

Welcome back, networkers !

Now, we are able to loop over all network interfaces within a shell script. How about we do it in C ? Let's make a first attempt using syscalls, shall we ? The alert reader already guessed that like all major things, there will be three parts (talking about major things, Google suggests me that there are three little pigs, stooges, musketeers. But Rs beat them all with 3 billion hits, what an ignorant fool I was, to not even know about RRR).

So, whenever it comes to system and kernel stuff, system calls are your friends, and thus we naturally turn to ioctl and friends, in order to find out more about the network interfaces. A quick "man ioctl" reveals that there is a manual page that knows them all ioctls. So, let's summon it right away by doing "man ioctl_list". Now that you have it in front of your eyes in it's full glory, you will just trust me and use SIOCGIFCONF to know about network interfaces. Because that manual page was rather obscure, wasn't it ?

So basically that's how it works : you prepare an ifconf structure to issue the ioctl itself, and add within another pointer to a few ifreq structures that will hold the answer to your request. It's that simple. Got it ? Let's proceed to the source code then :

#include <stdio.h>#include <stdlib.h>#include <string.h>#include <unistd.h>#include <sys/socket.h>#include <sys/types.h>#include <sys/ioctl.h>#include <linux/if.h>intmain(int argc, char **argv){  int devices, n, i;  struct ifconf config;  struct ifreq ifreq[128];  int ret;  /* Initialize the structure to 0s */  memset(&config, 0, sizeof(struct ifconf));  /* Open a socket to issue the ioctl */  devices = socket(AF_INET, SOCK_STREAM, 0);  if (devices < 0)    {      perror("cannot open socket");      return 1;    }  /* fill-in the config request structure, including a buffer to hold the answer */  config.ifc_buf = (char *) ifreq;  config.ifc_len = 128 * sizeof(struct ifreq);  /* issue the system call */  ret = ioctl(devices, SIOCGIFCONF, (char *) &config);  if (ret < 0)    {      perror("ioctl failed\n");      close(devices);      return 2;    }  /* and parse the results, the length of the answer hints at the number of interfaces */  n = config.ifc_len / (sizeof(struct ifreq));  for (i = 0; i < n; i++)     {      printf("Interface : %s\n", ifreq[i].ifr_name);    }    close(devices);}

And now let's try it :

$ ./iflist Interface : loInterface : eth1

Assuming I ran it on the same hardware, there seems to be some inconsistency. Because there are missing interfaces, and these are the ones that had no traffic in the stats. Which leads us to believe that SIOCGIFCONF does not return information about non running interfaces.

By digging a little bit further, we will realise, doing "man netdevice", that SIOCGIFCONF belongs to IP layer and thus only returns information about up interfaces that have an IP address. So, let's discuss next week about another solution using netlink sockets. To be continued...

PS: of course you can use that code if you need, although it is not that great. Well it depends on what you need ;-)

Howto get a list of network interfaces in Linux, part 1

2010-02-19T12:15:00.005+09:00

Hi folks,

Today we are going to become a little bit more technical. Some of you might wonder how to get a list of the network interfaces. For our first part, we are going to consider the shell script case.

Little disclaimer first, I am not an expert at shell-scripts at all, my abilities are limited to understanding the random scripts I encounter during my daily use of Linux on my computers, and writing the occasional script that saves some time when having to automate some boring tasks (converting a few gigs of videos is boring).

So, where shall we begin ? This is all quite easy : the kernel has a list of devices that is updated whenever a new device is added (through hotplugging, module loading, virtual device creation, and so on) or removed. As for many important kernel hosted pieces of information, this list can be accessed through the /proc filesystem. If you are curious and want to know more about all this, you can check net/core/dev.c from your current kernel, it should be there. Look for dev_base_head and dev_seq_printf_stats.

Let's have a look at that proc entry :

Inter-|   Receive                                                |  Transmit face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed    lo:    1384      20    0    0    0     0          0         0     1384      20    0    0    0     0       0          0  eth1:14664583   22189    0    0    0     0          0       660  3069813   15105    0    0    0     0       0          0 wlan1:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0  pan0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0

Great, It's all there in a neat and easy to parse format. Let's trash the first line, keep only the leftmost part of all the other ones, and trim the leading whitespace.

As I told you earlier, I am not claiming to be a shell-script expert, but that script does work ! Here it is, using bash.

#!/bin/bashfor i in `cat /proc/net/dev | grep ':' | cut -d ':' -f 1`do  ifname=`echo $i | tr -d ' '`  echo "Interface : $i"done

And of course, we get the expected results, at least on my debian box.

Results for a random debian GNU/Linux box :

bash$ ./iflist.sh Interface : loInterface : eth1Interface : wlan1Interface : pan

That code snippet is given to you for free, just leave a comment if you love or hate it. Next time we will look at a little longer piece of C code to obtain the same kind of information, without resorting to opening the /proc file, of course.

Standards : How do you like them Apple ? (continued)

2010-01-19T14:13:00.002+09:00

So, we are back to our iphone standards woes. If, like my wife, you shoot between 100 and 500 pictures a months, it will still take 20 to 100 months to fill up the camera roll. But that's still a lot to scroll through, so I am making a few Album folders within iTunes, and grouping the pictures by month. Which means that I remove the old pictures from the camera roll. Which also means that when the "10000 glitch" appeared, the camera roll was looking empty for some time until the update. Had to plug the phone to the comp to be sure everything was still there.

So here's the big question : how could we reach 10'000 pictures in the camera roll so quickly ? The answer is, broken software. As I am suspecting more and more after seeing this "10000 glitch", the software quality is much below the anticipated quality from a company as notorious as Apple. I have never owned a Mac, out of hatred. But I have always heard good feedback and was looking forward to that phone from a software point of view.

And I am so disappointed. This phone reminds me those cheap scripted games that are pretty much like a movie. You can not access the contents of the phone. There are no folders, no documents. It is thus impossible to upload a file. Even more ridiculous, for a very long time it was impossible to cut and paste text. Even my Toshiba (!) phone could do it... By the way, I know most of this is possible with add-on software, but not always free, and never well integrated. I even read some enthusiast saying that : "If the iphone doesn't support it, it means we don't need it". The same person probably acclaimed the release that featured the never seen before cut and paste capability. Which reminds me how safari lacks a "search" function...

So, back to software quality. Even though there are no folders, Safari at least gives you the ability to save images to the camera roll. And so does the iphone OS by allowing you to make screenshots, pressing both the square button and the power button simultaneously. Which has a high probability to screw up your camera roll ! Yes, this is the culprit. I found out that every now and then, when saving pictures from the web or making a screenshot, the software smartly creates a new 1XXAPPLE folder inside the DCIM root folder. So instead of 10'000 pictures, all we needed to get to the infamous 110APPLE folder was a dozen of screenshots or saved files...

Now the story wouldn't be complete without some more ranting from my side. So here it goes : when I realized that this picture stuff was happening i filed a bug in the iphone developer section of apple developer connection (the Apple developer resource center, like MSDN for the windozers, if I'm not mistaken). Of course I never had any reply, but that update I mentioned was fixing part of the problem : Apple decided to work around the issue and allow the camera roll to display files with out-of-specifications names (i.e. 4 letters 5 digits).

Unfortunately, it looks like something in the 3.x update completely screwed up the battery life for some phones in some circumstances. Not my case, but it still resulted in a last update, the latest and greatest 3.1.2 iPhone OS. And what a mess it is : it features a very nice bug where the phone is not detected as a storage device anymore (neither by PCs nor Macs). Remeber the thin obfuscation layer I was mentioning in the previous post ? It looks like it is broken, oh my ! After a few days posting on the apple forums, I met someone who found the cause for this issue : it only happens if there are PNG files in the camera roll. Well, not any PNGs, it needs to be very specific ones, but at least I was not the only person with the issue.

And to my knowledge, the problem is still not fixed today. So what do you do when you want to extract the pictures from your glitched phone ? It depends on budget and policy :

strict policy : you wait for Apple to sort it out
loose policy, no budget : you use a free application to send all the pictures (e.g. Pixelpipe, or any of those wifi sharing apps)
loose policy, low budget : you buy a <30$ software to extract the pictures from the 1XXAPPLE folders (e.g. Touchcopy)
high budget : you buy a real digital camera and repeat after me : I phone with my phone, iphone will just phone.

Okay enough joking. If you can manage to copy those PNG files to your computer and erase them from the iphone using a third-party app, the phone will appear again as a storage device, after you unplug (from the computer) and reboot it.

Standards : How do you like them Apple ?

2010-01-13T14:58:00.002+09:00

Okay I have been trying hard to find some funny way to introduce this post, don't worry if you don't think it is (funny).

Actually, there are two reasons why I wanted that funny start, the first one being because it has been a long time since I posted anything -or since I posted at all- and the second one because, I am still not sure if I should post it. I have been wrestling with myself over this, but I believe that accumulated frustration finally triggered the whole thing, so let's go. I will try to explain here how I am getting more and more disappointed by an overrated phone...

It all began when I bought an iPhone for my wife. Very good idea by the way, because being very busy all day along, she could not use her computer anymore and this was a kind of pocket substitute. She had tried a few other tiny gadget notebooks like the Zaurus, various VAIOs, but apparently she was much more pleased with this one. She mainly uses it for : browsing the web, reading RSS flows, writing emails and taking pictures.

And these last two were iPhone killers. At the time we got it (I believe 2.0 upgrade, the one with Google Streetvie) there was still no cut and paste (ha ha !) and it was not possible to send more than 1 picture by e-mail, with butchered resolution. This led me to design a small web-based mailing application allowing a bit of formatting, and image inlining which is much better than attaching everything at the bottom. If anyone is interested, I can talk about that another day. So e-mail problem was solved.

What about pictures then ? There is a camera on the iPhone so it should be okay. Should be. This is where we will discover the meaning of the title. But first, let's digress a bit and discuss Standards.

I never loved anything as much as standards when it comes to computers. Either hardware or software, standards are good for us. If a manufacturer, software developer fails, you don't have to worry if they were operating within a standard : data can be losslessly converted, parts can be replaced, etc...

Now back to the camera. Under a little obfuscation layer that we will discuss again later, the iPhone has folders to organize the pictures it takes. And of course, people over there at Apple didn't want to re-invent the wheel, so they just went for the famous ABCD1234.JPG naming scheme. If you have ever looked at the data produced by digital cameras on their memory cards, you know what I mean. And guess what ? This naming has been standardized. Yes ! By the Japan Electronic Industry Development Association, and it can be found here. Read it if you want, the important part is, the name should be made of 8 chars including 4 free chars and 4 digits.

Okay, so just plug your iPhone into your computer and check out the storage device that is created. You can see the pictures there, sorted neatly in a few 100APPLE and such folders. Let's see how Apple people decided to name the files. The folders look like 100APPLE, 101APPLE, 102APPLE and so on. In each folder, there are files called IMG_numb.JPG, where numb is unique and incremented after each shot. And now is the interesting part : this unique number was also decided, by the designers of the iPhone and out of the specification scope, to be unique among directories. The quick and very dirty way is to decide that 100APPLE has pictures from 0001 to 0999, then 101APPLE has pictures from 1000 to 1999, and this sounds so much like "640Ko should be enough for all" that I am surprised they could do the exact same mistake !

What happens when someone has more than 9999 pictures ? I will tell you, a new folder is created, called 110APPLE and it contains files named IMG_10001,JPG and so on. I would laugh if it didn't deprive me from sleeping for a few days. So here we have a clear violation of the DCF standard, right ? For people like Apple this is no big deal because they make the picture taking, picture viewing, picture synchronizing (iTunes, iPhoto, whatever) applications, so they just have to be aware of this standard "enhancement". But they are not. Which is why during a few weeks (before an update that silently fixed the camera roll without telling) it was impossible for me to see any pictures taken after the 9999th in the Camera Roll application within the iPhone. Good job guys...

Hmmm this is getting longer than expected so i will cut here and resume the story in the next post soon... Because let me reassure you, we never shot ten thousand pictures with the iphone, yet we reached the 10'000 bar...

Welcome / ようこそ / Bienvenue !

2008-07-08T07:40:00.000+09:00

Welcome to the 'Hayai Jean' blog. On this site, I will try to present all kind of technical stuff that I find interesting.

「ハヤイ　ジャン」へようこそ！このウェブサイトで色々な面白い情報を見せたいです。　

Bienvenue sur le blog Hayai Jean. J'essaierai d'y présenter des informations techniques que je trouverai intéressantes.

早い　ジャン　d^.^V

EDIT: This blog was renamed "Jean's tech blog" after a major lack of content