I integrated an Atmel SAM D21 Xplained dev kit with the bip.io Internet of Things data service, and wrote about it.
In my travels I’ve worn the hat of a DevOps guy a few times, and have written documentation, manuals, and guides countless times wearing an entire rainbow of hats. I’ve also had the good fortune to read documentation left by previous engineers at the larger organizations I’ve worked with. Both of these experiences have given me some useful insights into techniques for documentation, good and bad, and what can result from both.
A long time ago, in a startup far, far away, I was building the CyberJocks LAN game center brand into what we hoped would become a national franchise. We only ended up with two locations, in New York and Texas, but that’s a story for another time (and requires beer). Building any brand requires consistency. Consistency in public image, trade dress, operations, and product, to start. Without this consistency, the customer will end up confused, and the brand identity is diluted and far less effective. We figured that comprehensive documentation on The Right Way To Do Things™ would yield consistency.
It did not.
We wrote mighty tomes of operational procedures, guidelines, and standards. We had every last step spelled out, to ensure that the most green member of our team would be able to pick up the three-ring binder from under the front counter, turn to page 314, and follow a step-by-step guide leading her to consistency nirvana. We called it monkey-proofing, the idea to write things so explicitly and clearly, that even a monkey could follow the directions. And we were very good at it. Our directions divined every detail, clarified every concern, and indicated every idiom required to run the business. Whether it was making a pepperoni pizza panini (16 slices of pepperoni), or creating a new user account (legal guardian signature, please), we had it all laid out.
But nobody read the damn thing!
Hundreds of pages of policy and procedure. Step by mind-numbing step. All laid out. Nobody had time to read that much documentation, and nobody was inclined to read it. Of course, any national franchise is going to have The Book just like this, but it is a poor solution.
Later in life, we learned that the best way to set things up is to carefully design a whole system to encourage the right outcomes. Make the easiest thing to do the correct thing, and leverage the natural human tendency of laziness instead of fight it. Engage the team, infuse them with the core beliefs, the driving principles, the why we do it, not the how or the what. Armed with the why, let people think for themselves, empower them to decide the course of action. Amazing things happen this way.
So what does a franchise manual have to do with technical documentation? Quite a bit, it turned out. As I made my way through the docs of some larger companies, and worked with their operations and development teams, I started to discover a few patterns in the writing, and patterns in the behaviours of those consuming it.
The writing itself fell into several categories:
- Hastily sketched and incomplete
- Stale, obsolete, or inapplicable
These all fell short. The first three are obvious in their faults, and serve to illustrate that curation of a document repository is a complex and sizable task, and important to the organization. It’s best to have someone who does this and nothing else, looking after the docs.
The last one ended up being very common. It would tend to specify a problem-solution type of pattern, the same thing you seen in all those cookbook books for tech stuff. Run into a problem with Foo? Follow these steps! The steps would then specify specific commands, options, files, servers, etc., and if followed claimed that the problem with Foo would go away.
But there’s an insidious problem with cookbooks! I’m sure you could guess that they are at high risk of going stale, and quickly. They do precisely that. Beyond the information rotting into irrelevance, the deeper trouble is in the way the consuming engineer responds to a cookbook.
It discourages independent thought and understanding. By providing exact steps, the author believes they are helping out the next guy as much as possible. Simple! Paste this code in your terminal and you’re done! But by doing this, they are short-circuiting the engineer’s mind. They lose the chance to learn, to deepen their understanding of the system.
Understanding The Whole Trumps Specified Recipes
Understanding lets you cover the things a cookbook doesn’t. When situations change, when the docs go stale, when the system throws a fit, and the unexpected becomes the reality, the engineer needs to be able to work it out. They need the resources to think through things. They need a map that can point them in the right direction, clue them in to non-obvious aspects, and offer wisdom hard-earned from previous challenges that were solved.
So how to write better docs? Focus on communicating the fundamental concepts. The goal for better docs is to imbue understanding of the system as a whole. What are the high-level components? How does data flow through? Where are things located, and what bits interact with each other? If something odd happened in the past, document the situation and the solution, and include hard data. Armed with this foundational material, an engineer has a fighting chance when the system goes haywire. It’s ok for some procedures to be outlined, but they should be annotated to account for expected, and where possible, unexpected change. Where cookbooks may leave you wanting, true understanding is a recipe for success.
OTR is a good protocol for encrypted chat. I use it often, but have always wondered how to transfer my OTR private keys between computers and devices. Up until now, every machine would have a separately-generated key, and I’d have to re-verify keys with my friends before chatting. Always verify your keys! Today I finally looked around for where the keys are stored, on a Mac using Adium, and on a Debian Linux box using Pidgin.
The Pidgin OTR keys were easy to find. Pidgin is run under the hood by
libpurple, and there was simply a
.purple directory in my home dir. In here we find the OTR files:
accounts.xml blist.xml certificates/ icons/ logs/ otr.fingerprints otr.instance_tags otr.private_key plugins/ prefs.xml smileys/ status.xml
otr.private_key file we see the keys in a parenthesis-delimited data structure (how (lispy!)), which are set per-account (this is not my actual key, of course):
(privkeys (account (name your_username_here) (protocol prpl-aim) (private-key (dsa (p #9AD61CB50561A45116DC9735ED1DAABA372308628ABDCA1E92B7283189B10945DAB0D20594E4DE5E92B2334635208D78D17371D6012426A347C831B89D5EA7D2CED4CAD0D5DADA46DCCC13E6CB436324E226D68DBB7165BE69BDCCE667B59AD47423C586A8700D47BB0821D1BB8086E73073DBA847AE358B0231D3A9BC112A96358#) (q #ECAD3933491A19B6A4170EE921D480B0AB736E244C1B0#) (g #9AD61CB50561A45116DC9735ED1DAABA372308628ABDCA1E92B7283189B10945DAB0D20594E4DE5E92B2334635208D78D17371D6012426A347C831B89D5EA7D2CED4CAD0D5DADA46DCCC13E6CB436324E226D68DBB7165BE69BDCCE667B59AD47423C586A8700D47BB0821D1BB8086E73073DBA847AE358B0231D3A9BC112A96358#) (y #9AD61CB50561A45116DC9735ED1DAABA372308628ABDCA1E92B7283189B10945DAB0D20594E4DE5E92B2334635208D78D17371D6012426A347C831B89D5EA7D2CED4CAD0D5DADA46DCCC13E6CB436324E226D68DBB7165BE69BDCCE667B59AD47423C586A8700D47BB0821D1BB8086E73073DBA847AE358B0231D3A9BC112A96358#) (x #ECAD3933491A19B6A4170EE921D480B0AB736E244C1B0#) ) ) ) )
The bit inside of the
private-key section is the bit you want to transfer.
On the Mac, using Adium, you can find your OTR key files in the directory
~/Users/username/Library/Application Support/Adium 2.0/Users/Default but the contents are slightly different. Adium names the protocls differently. This is why it’s best to transfer the private-key section of the file only, and make sure you copy from and to the intended protocols.
You can also see the
otr.fingerprints file, which has the list of verified (or simply seen) fingerprints for your friends. Copy that around also, and you won’t have to re-verify keys! You could also do this on your mobile devices in theory, but that’s going to vary widely, so good luck and have fun!
Make sure you don’t mess up the file access permissions during this process – you wouldn’t want someone to steal your keys!
chmod 600 is probably what you want, but security is complex, so don’t take my word for your setup.
The Novena laptop’s Linux build doesn’t support the Samsung 840 EVO hard drive that it ships with. While this isn’t a problem specific to the Novena by any means, it will affect the users. It’s a simple fix, just edit the
hddtemp.db file and add the proper ID string and temp code to it. You can just echo-append it, or if you are fussy open it in an editor and put it neatly in the Samsung section!
echo '"Samsung SSD 840 EVO 250G B" 190 C "Samsung SSD 840 EVO 250GB"' >> /etc/hddtemp.db ^---that space needs to be there; see below
If you want to find how to add this to
hddtemp.db yourself, or on a different model drive that’s not in the db file, use
hddtemp --debug. The -A flag on
smartctl lists the drive’s attributes. Look for the ID# for the temperature sensor. It also may indicate Celsius or Fahrenheit, but more likely it will be Celcius. Here we can see it’s attribute number 190, and based on the name, it’s in Celsius.
[email protected]:~# smartctl -A /dev/sda smartctl 6.4 2014-10-07 r4002 [armv7l-linux-3.19.0-00268-g04e9d08] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 1 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 169 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 12 177 Wear_Leveling_Count 0x0013 100 100 000 Pre-fail Always - 0 179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0 181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0 182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0 183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0 187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0032 067 064 000 Old_age Always - 33 195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always - 0 199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 8 241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 39490266
You can get the drive’s name with
hddtemp --debug and also make an educated guess as to the sensor ID#. Notice the space in “250G B” below – you’ll want to use this exact model string in the db file. Not sure precisely why that space should be there, but from the hddtemp source it seems that’s how the drive identifies itself via SATA command. Perhaps other utils like smartctl get the name in other ways, or fix it up. I don’t have time to dig further on that…
[email protected]:~# hddtemp --debug /dev/sda ================= hddtemp 0.3-beta15 ================== Model: Samsung SSD 840 EVO 250G B �@ field(5) = 0 field(9) = 170 field(12) = 12 field(177) = 0 field(179) = 0 field(181) = 0 field(182) = 0 field(183) = 0 field(187) = 0 field(190) = 34 field(195) = 0 field(199) = 0 field(235) = 8 field(241) = 50 If one of the field value seems to match the temperature, be sure to read the hddtemp man page before sending a report (section REPORT). Thanks.
I’ve been wondering how much heat the CPU on the Novena puts out, and if the tiny heatsink (no fan) is sufficient to cool it. After letting the machine sit idle for an hour or so, running X with a browser but nothing else with high load, I shot a few images with a Flir E-series camera. Allowed the Flir to stabilize its self-calibration for a few minutes, and then got these images:
Whole laptop innards:
Close-up image of the PVT2-A board, charger, display adapter, and GPIO breakout board:
And the same, with data overlay:
The CPU heatsink temp was calculated using an emissivity of ε0.83 for red anodized aluminum, and I used ε0.95 for the display board chip’s black plastic. Remember kids, thermal emissivity is important to keep in mind!
The i.MX6 datasheet lists the operating temperature range as 90C typical, with 105C maximum. Even still, it seems to run a bit hot at idle in my opinion, and I need to test it under extended load. Also will test with the lid closed (not sure it was designed to be operated that way) and in different orientations.
I’ve also found a possible power supply issue when running on battery under high CPU load, but I still need to investigate.
Update 3 May 2015: I’ve been compiling KiCad for over an hour, and it’s sitting at near 100% CPU use on all four cores, and running at max clock speed. Figured I should take another thermal pic while it’s hot. Not terribly bad for the hotspots, within the datasheet tolerances. Hotter than I’d prefer, and I don’t think I’d run it this hard for long with the lid closed, but no troubles.
Also, the power supply issue under high load that I suspected, I now think is probably related to a timing issue in the eDP chip as reported here by xobs: http://www.kosagi.com/forums/viewtopic.php?pid=1385#p1385
Recently assembled my shiny new Novena laptop, and decided to make a time-lapse video of the assembly process. Shot in 4k on RED – because one awesome tech deserves another!
In the video, you’ll see that I use threadlocker everywhere. I’m a big fan of the stuff, and in my opinion so very many of the so-called maker projects and kits out there should be recommending it. Especially the ones full of stepper motors and other vibrating bits, that are made of wood or acrylic, which prevents you from pre-tensioning the bolts correctly because you’ll crush or crack the chassis.
I also spill superglue all over my hands, when a tube decides to goosh out everywhere. (was securing one of the peek array threaded inserts, that pulled out while trying to install a speaker) That was fun, trying to not touch anything for a few minutes until it dried, and then going at it with acetone… I do not recommend gooshing superglue all over your hands. Avoid.
Very excited to dive into this phenomenal hardware hacking platform. Thanks, bunnie and xobs!