Atmel SAMD21 and WINC1500 WiFi

I integrated an Atmel SAM D21 Xplained dev kit with the Internet of Things data service, and wrote about it.


How to Write Good Ops Documentation

In my travels I’ve worn the hat of a DevOps guy a few times, and have written documentation, manuals, and guides countless times wearing an entire rainbow of hats. I’ve also had the good fortune to read documentation left by previous engineers at the larger organizations I’ve worked with. Both of these experiences have given me some useful insights into techniques for documentation, good and bad, and what can result from both.

Monkey Proofing

A long time ago, in a startup far, far away, I was building the CyberJocks LAN game center brand into what we hoped would become a national franchise. We only ended up with two locations, in New York and Texas, but that’s a story for another time (and requires beer). Building any brand requires consistency. Consistency in public image, trade dress, operations, and product, to start. Without this consistency, the customer will end up confused, and the brand identity is diluted and far less effective. We figured that comprehensive documentation on The Right Way To Do Things™ would yield consistency.

It did not.

We wrote mighty tomes of operational procedures, guidelines, and standards. We had every last step spelled out, to ensure that the most green member of our team would be able to pick up the three-ring binder from under the front counter, turn to page 314, and follow a step-by-step guide leading her to consistency nirvana. We called it monkey-proofing, the idea to write things so explicitly and clearly, that even a monkey could follow the directions. And we were very good at it. Our directions divined every detail, clarified every concern, and indicated every idiom required to run the business. Whether it was making a pepperoni pizza panini (16 slices of pepperoni), or creating a new user account (legal guardian signature, please), we had it all laid out.

But nobody read the damn thing!

Hundreds of pages of policy and procedure. Step by mind-numbing step. All laid out. Nobody had time to read that much documentation, and nobody was inclined to read it. Of course, any national franchise is going to have The Book just like this, but it is a poor solution.

Later in life, we learned that the best way to set things up is to carefully design a whole system to encourage the right outcomes. Make the easiest thing to do the correct thing, and leverage the natural human tendency of laziness instead of fight it. Engage the team, infuse them with the core beliefs, the driving principles, the why we do it, not the how or the what. Armed with the why, let people think for themselves, empower them to decide the course of action. Amazing things happen this way.

Technical Docs

So what does a franchise manual have to do with technical documentation? Quite a bit, it turned out. As I made my way through the docs of some larger companies, and worked with their operations and development teams, I started to discover a few patterns in the writing, and patterns in the behaviours of those consuming it.

The writing itself fell into several categories:

  • Non-existent
  • Hastily sketched and incomplete
  • Stale, obsolete, or inapplicable
  • Cookbook-style

These all fell short. The first three are obvious in their faults, and serve to illustrate that curation of a document repository is a complex and sizable task, and important to the organization. It’s best to have someone who does this and nothing else, looking after the docs.

The last one ended up being very common. It would tend to specify a problem-solution type of pattern, the same thing you seen in all those cookbook books for tech stuff. Run into a problem with Foo? Follow these steps! The steps would then specify specific commands, options, files, servers, etc., and if followed claimed that the problem with Foo would go away.

But there’s an insidious problem with cookbooks! I’m sure you could guess that they are at high risk of going stale, and quickly. They do precisely that. Beyond the information rotting into irrelevance, the deeper trouble is in the way the consuming engineer responds to a cookbook.

It discourages independent thought and understanding. By providing exact steps, the author believes they are helping out the next guy as much as possible. Simple! Paste this code in your terminal and you’re done! But by doing this, they are short-circuiting the engineer’s mind. They lose the chance to learn, to deepen their understanding of the system.

Understanding The Whole Trumps Specified Recipes

Understanding lets you cover the things a cookbook doesn’t. When situations change, when the docs go stale, when the system throws a fit, and the unexpected becomes the reality, the engineer needs to be able to work it out. They need the resources to think through things. They need a map that can point them in the right direction, clue them in to non-obvious aspects, and offer wisdom hard-earned from previous challenges that were solved.

So how to write better docs? Focus on communicating the fundamental concepts. The goal for better docs is to imbue understanding of the system as a whole. What are the high-level components? How does data flow through? Where are things located, and what bits interact with each other? If something odd happened in the past, document the situation and the solution, and include hard data. Armed with this foundational material, an engineer has a fighting chance when the system goes haywire. It’s ok for some procedures to be outlined, but they should be annotated to account for expected, and where possible, unexpected change. Where cookbooks may leave you wanting, true understanding is a recipe for success.


How to Transfer your OTR Private Key Between Hosts

OTR is a good protocol for encrypted chat. I use it often, but have always wondered how to transfer my OTR private keys between computers and devices. Up until now, every machine would have a separately-generated key, and I’d have to re-verify keys with my friends before chatting. Always verify your keys! Today I finally looked around for where the keys are stored, on a Mac using Adium, and on a Debian Linux box using Pidgin.

The Pidgin OTR keys were easy to find. Pidgin is run under the hood by libpurple, and there was simply a .purple directory in my home dir. In here we find the OTR files:


Inside the otr.private_key file we see the keys in a parenthesis-delimited data structure (how (lispy!)), which are set per-account (this is not my actual key, of course):

(name your_username_here)
(protocol prpl-aim)
  (p #9AD61CB50561A45116DC9735ED1DAABA372308628ABDCA1E92B7283189B10945DAB0D20594E4DE5E92B2334635208D78D17371D6012426A347C831B89D5EA7D2CED4CAD0D5DADA46DCCC13E6CB436324E226D68DBB7165BE69BDCCE667B59AD47423C586A8700D47BB0821D1BB8086E73073DBA847AE358B0231D3A9BC112A96358#)
  (q #ECAD3933491A19B6A4170EE921D480B0AB736E244C1B0#)
  (g #9AD61CB50561A45116DC9735ED1DAABA372308628ABDCA1E92B7283189B10945DAB0D20594E4DE5E92B2334635208D78D17371D6012426A347C831B89D5EA7D2CED4CAD0D5DADA46DCCC13E6CB436324E226D68DBB7165BE69BDCCE667B59AD47423C586A8700D47BB0821D1BB8086E73073DBA847AE358B0231D3A9BC112A96358#)
  (y #9AD61CB50561A45116DC9735ED1DAABA372308628ABDCA1E92B7283189B10945DAB0D20594E4DE5E92B2334635208D78D17371D6012426A347C831B89D5EA7D2CED4CAD0D5DADA46DCCC13E6CB436324E226D68DBB7165BE69BDCCE667B59AD47423C586A8700D47BB0821D1BB8086E73073DBA847AE358B0231D3A9BC112A96358#)
  (x #ECAD3933491A19B6A4170EE921D480B0AB736E244C1B0#)

The bit inside of the private-key section is the bit you want to transfer.

On the Mac, using Adium, you can find your OTR key files in the directory ~/Users/username/Library/Application Support/Adium 2.0/Users/Default but the contents are slightly different. Adium names the protocls differently. This is why it’s best to transfer the private-key section of the file only, and make sure you copy from and to the intended protocols.

You can also see the otr.fingerprints file, which has the list of verified (or simply seen) fingerprints for your friends. Copy that around also, and you won’t have to re-verify keys! You could also do this on your mobile devices in theory, but that’s going to vary widely, so good luck and have fun!

Make sure you don’t mess up the file access permissions during this process – you wouldn’t want someone to steal your keys! chmod 600 is probably what you want, but security is complex, so don’t take my word for your setup.


Novena Laptop Linux Samsung 840 EVO Hard Drive Temperature Sensor Fix

The Novena laptop’s Linux build doesn’t support the Samsung 840 EVO hard drive that it ships with. While this isn’t a problem specific to the Novena by any means, it will affect the users. It’s a simple fix, just edit the hddtemp.db file and add the proper ID string and temp code to it. You can just echo-append it, or if you are fussy open it in an editor and put it neatly in the Samsung section!

echo '"Samsung SSD 840 EVO 250G B" 190 C "Samsung SSD 840 EVO 250GB"' >> /etc/hddtemp.db
                               ^---that space needs to be there; see below

If you want to find how to add this to hddtemp.db yourself, or on a different model drive that’s not in the db file, use smartctl and hddtemp --debug. The -A flag on smartctl lists the drive’s attributes. Look for the ID# for the temperature sensor. It also may indicate Celsius or Fahrenheit, but more likely it will be Celcius. Here we can see it’s attribute number 190, and based on the name, it’s in Celsius.

[email protected]:~# smartctl -A /dev/sda
smartctl 6.4 2014-10-07 r4002 [armv7l-linux-3.19.0-00268-g04e9d08] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke,

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       169
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       12
177 Wear_Leveling_Count     0x0013   100   100   000    Pre-fail  Always       -       0
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   067   064   000    Old_age   Always       -       33
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       8
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       39490266

You can get the drive’s name with hddtemp --debug and also make an educated guess as to the sensor ID#. Notice the space in “250G B” below – you’ll want to use this exact model string in the db file. Not sure precisely why that space should be there, but from the hddtemp source it seems that’s how the drive identifies itself via SATA command. Perhaps other utils like smartctl get the name in other ways, or fix it up. I don’t have time to dig further on that…

[email protected]:~# hddtemp --debug /dev/sda

================= hddtemp 0.3-beta15 ==================
Model: Samsung SSD 840 EVO 250G B              �@

field(5)	 = 0
field(9)	 = 170
field(12)	 = 12
field(177)	 = 0
field(179)	 = 0
field(181)	 = 0
field(182)	 = 0
field(183)	 = 0
field(187)	 = 0
field(190)	 = 34
field(195)	 = 0
field(199)	 = 0
field(235)	 = 8
field(241)	 = 50

If one of the field value seems to match the temperature, be sure to read
the hddtemp man page before sending a report (section REPORT). Thanks.

Novena Laptop Thermal Imaging

I’ve been wondering how much heat the CPU on the Novena puts out, and if the tiny heatsink (no fan) is sufficient to cool it. After letting the machine sit idle for an hour or so, running X with a browser but nothing else with high load, I shot a few images with a Flir E-series camera. Allowed the Flir to stabilize its self-calibration for a few minutes, and then got these images:

Whole laptop innards:

Novena Laptop Thermal Image

Novena Laptop Thermal Image

Close-up image of the PVT2-A board, charger, display adapter, and GPIO breakout board:

Novena PVT2-A Thermal Image

Novena PVT2-A Thermal Image

And the same, with data overlay:

Novena PVT2-A Thermal Image with Data Overlay

Novena PVT2-A Thermal Image with Data Overlay

The CPU heatsink temp was calculated using an emissivity of ε0.83 for red anodized aluminum, and I used ε0.95 for the display board chip’s black plastic. Remember kids, thermal emissivity is important to keep in mind!

The i.MX6 datasheet lists the operating temperature range as 90C typical, with 105C maximum. Even still, it seems to run a bit hot at idle in my opinion, and I need to test it under extended load. Also will test with the lid closed (not sure it was designed to be operated that way) and in different orientations.

I’ve also found a possible power supply issue when running on battery under high CPU load, but I still need to investigate.

Update 3 May 2015: I’ve been compiling KiCad for over an hour, and it’s sitting at near 100% CPU use on all four cores, and running at max clock speed. Figured I should take another thermal pic while it’s hot. Not terribly bad for the hotspots, within the datasheet tolerances. Hotter than I’d prefer, and I don’t think I’d run it this hard for long with the lid closed, but no troubles.

Novena PVT2-A Thermal Image with Data Overlay at 100% CPU

Novena PVT2-A Thermal Image with Data Overlay at 100% CPU

Also, the power supply issue under high load that I suspected, I now think is probably related to a timing issue in the eDP chip as reported here by xobs:


Novena Laptop Assembly Timelapse Video

Recently assembled my shiny new Novena laptop, and decided to make a time-lapse video of the assembly process. Shot in 4k on RED – because one awesome tech deserves another!

In the video, you’ll see that I use threadlocker everywhere. I’m a big fan of the stuff, and in my opinion so very many of the so-called maker projects and kits out there should be recommending it. Especially the ones full of stepper motors and other vibrating bits, that are made of wood or acrylic, which prevents you from pre-tensioning the bolts correctly because you’ll crush or crack the chassis.

I also spill superglue all over my hands, when a tube decides to goosh out everywhere. (was securing one of the peek array threaded inserts, that pulled out while trying to install a speaker) That was fun, trying to not touch anything for a few minutes until it dried, and then going at it with acetone… I do not recommend gooshing superglue all over your hands. Avoid.

Very excited to dive into this phenomenal hardware hacking platform. Thanks, bunnie and xobs!