[push]
# Much saner than the old behavior, and new default.
default = simple
[user]
# Duh.
email = thahn@tcbtech.com
name = Ted Hahn
# Corresponsed to my signing key.
signingkey = 1CA0948A
[pull]
# When pulling, rebase my feature branches on top of what they’ve just pulled.
rebase = true
[commit]
# Sign all commits
gpgsign = true

Here’s some things you should start most bash scripts with:

#!/bin/bash

set -e
set -x
set -o pipefail
set -u


TMPDIR=$(mktemp -d)
trap 'rm -rf $TMPDIR' EXIT

Explanations of the lines:

#!/bin/bash

The shebang line is a unix convention that allows scripts to specify their interpreter. Since this is a bash script, we tell it to run this file with bash.

set -e

Exit immediately if any command fails. Makes it easy to spot when a script did not complete, and prevents things further down the line from doing the wrong thing because they were only partially setup.

set -x

Print each command as it’s run. It’s fantastically useful debug output, though some production scripts should have this disabled.

set -o pipefail

Exit with failure if any substage of a pipeline fails. This is about commands chained together with a pipe; e.g. If your grep command fails, the execution will fail, rather than simply outputting nothing to the next stage of the pipeline.

set -u

Makes referencing unset variables an error.

Further explaination of the above three can be found in the Bash Reference Manual entry on Set.

TMPDIR=$(mktemp -d)
trap 'rm -rf $TMPDIR' EXIT

Create a scratch dir, automatically delete it when you’re done. It’s often useful to comment out the trap line during debugging.

See also Pixelbeat’s blog on Common shell script mistakes

I’ve got two amusing anecdotes related to symlinks. By amusing anecdotes, I of course mean incredibly frustrating weird behaviors that took hours to debug. One java, one chef.

Chef

Chef handles environments very well… except when it comes to databags. From my perspective, this is a critical flaw, since the things I want to keep out of the main chef repo (API keys and passwords) are also the things most likely to be affected by the environment. So,  when building, we specify the path to the chef databags, separating out the prod, canary, and dev environments.

For the parts that are common between the databags, I figured I’d use symlinks. Our databags are stored in a git repo, and git interprets symlinks correctly. The full set of databags were copied everywhere, so I could simply include a relative symlink to ../../prod/foo/bar.json for each databag I wanted consistent.  I got the following error:

syntax error, unexpected end-of-input

pointing to a character in the middle of the first line in the file. This made no sense.

It took me several tries with different files to figure out what was going on. The character that was being pointed out, x, was the same as the number of characters in the symlink path. A symlink is sorta just a text file with a pathname and a special flag on it. If you stat the symlink file, you’ll get the length of that pathname, not the size of the file it points to. What Chef seems to be doing is stat-ing that file, then taking that length as gospel – It doesn’t process it as a stream, but as a block of the stat’d size.

I should probably get around to testing that with the latest version and writing a bug.

Java

Java has a really simple package deployment mechanism: JARs. You can put a bunch of classes into a jar, and deploy them as one. If you have a project with a bunch of dependencies, you can ‘shade’ your jar and wrap all your classes into a single mono-jar.

However, for some use cases it’s not that simple. Java up to 1.7 simply won’t accept more than INT_16_MAX class files in a jar (and remember that anonymous classes are a separate file). Further, signatures can’t be retained; A jar has a signing key attached, and all files must be signed using that same signing key, so a ‘shaded’ jar can’t include the original signatures of dependencies.

So, since monolithic jars don’t work in some cases, what do you do instead? You ship several jars. It’s well documented but not well understood that when you specify a jar with java -jar that your classpath is ignored. How do you load multiple jars, then?

Inside the jar is a META-INF folder containing a MANIFEST.MF file. This manifest file contains a bunch of key-value pairs, and one of those keys can be Class-Path. This class-path key can specify additional jars or directories, and it usually will. However, because of deployment concerns, it will generally list them as relative paths or just as filenames. How does java find those files?

In about the worst way possible. Java will dereference any symlinks in the jar it is loading, then search the base directory of the final file it reads for the class-path includes. So, if you have a bunch of projects with common includes, you cannot simply symlink in all your dependency jars; You need hard copies of every jar you include. This also means you can’t simply update a dependency jar in one place, you have to hard-link it in to the working directory of every app you want to deploy.

I guess an option is to simply have a big folder full of all the jars for all the apps you want to run, but that folder can get very cluttered, and it becomes unclear what’s there why – is one of your dependencies shared? Do you have a garbage-collection mechanism for older jars in that folder?

I’ve seen some talk about whether to keep everything in one codebase, vs having per-project repositories. The answer is very clear to me: Monolithic repos are a must, but Git submodules are functionally equivalent (as I’ll describe later); You should start with one repo, and then subdivide when you have clear submodules.

The importance of monolithic repos vs per project is not about performance, or even directly about organization of your code. Both are fairly clear. It’s about organization of your build system. Good builds are fully deterministic and idempotent, and that is very hard to achieve with a set of per-project repositories.

The unix standard has a very good layout for where code goes. Shared libraries go in /lib, /usr/lib, /usr/local/lib; Headers go in /usr/include or /usr/local/include. But this isn’t a structure for how to organize your code when writing it; It’s a build environment structure. It makes more sense when your entire organization is sharing just one unix machine and environment, but we’re well beyond that.

Because you’re aiming for determinism, you need to be sure that your build environment is the same each time. The traditional unix file structure is not good for this purpose, at least, not directly. That structure is not rebuilt every time; You copy files on top of it, you add and remove, but you don’t reset. You can accomplish a reset – Reimaging your build machine, or using a docker container. And, in fact, the latter is what many people have switched to. But that docker container is another layer of abstraction, another piece you need to manage for full determinism. You need to kill and rebuild it each build, and most solutions don’t. The run new builds in the same container repeatedly, creating uncertainty.

This is why I like Bazel so much. It removes much of the uncertainty, by rebuilding from scratch each time. It has a separate, well defined environment, that it manages and assures is in the correct state. It’s not magic; You can change things, break things, and fool it. But if you don’t touch it, it does *the right thing* and keeps your structure clean, without taking any risks of breaking your whole machine.

Bazel operates on a concept called a ‘workspace’. There’s not a whole lot to it; You pick an arbitrary root directory, and a flag file defines the workspace root. Everything underneath is considered one logical unit. If you’ve got a monolithic codebase, this is a no-brainer.

Git submodules complicate builds a little, but not much. Instead of saying “this build was built at commit A”, you need to know that it was built at commits (A, B, C). But you probably don’t really care to always know the full set of A, B, C; They may move somewhat independently, but your build infrastructure can and should simply serialize them; Keeping a mapping of simple, linear commit numbers to a tuple of commit hashes for each submodule. There’s one disadvantage – It makes ‘who broke the build’ a race condition, if two modules change at the same time. That is solved by a simple answer: Suck it up, both commiters should debug 🙂

I was trying to install some software from source today. After dicking around for a full day last week trying to port it’s build chain to Bazel, I decided to just install the build toolchain it expects.

I install Cmake, and try to build my program. No dice! This source file requires cmake version 2.8.12, and CentOS 7 comes with 2.8.11. Ergh. So, I go back to source, and grab and make the latest cmake. A quick make, make install, and I go back to the original directory and… get the same error message?

/cmake .
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
 CMake 2.8.12 or higher is required. You are running version 2.8.11

What can this be? Did it not install? I make clean, make install again, and this time I search for where it’s installing. /usr/local/bin/cmake ; It seems to work. It’s in path. I run ‘which cmake’ and it tells me ” /usr/local/bin/cmake”, as I expect. I run directly from that path, and check the version; 3.2.2, as I expected. What gives? Why does simply running it not give me the expected results? I throw strace at it; The top line of strace shows the correct path:

execve("/usr/local/bin/cmake", ["cmake"], [/* 26 vars */]) = 0

At this point, I’m fairly stumped. Clearly, the old executable still exists, so I search my path for it; The old version of cmake is in /usr/bin. I then wonder at what could be causing the redirection, so I launch a new terminal, type cmake –version… and I get the correct result! Launching bash as a subshell also gives me what I want. So, what’s wrong with my initial shell?

At this point, I was fairly stumped, so I asked my dad, an old unix hand, and he pointed me to the ‘hash’ builtin. Bash keeps a dictionary of commands to their full pathnames; Presumably for speed, as doing several directory listings for each command would be wasteful, even with all of the PATH directories in ram.

The solution is simply to call ‘hash cname’ or ‘hash -r’ to cause bash to redo the dictionary entry for cname, or for everything, respectively.

I’ve been using tmux for a while, because it’s got a couple cool advantages over screen, mostly in the areas of “readable code” and “not locking up”. What it fails at, however, is having an interface like screen; What I really liked about screen is that I could have a dozen terminal windows open and use screen as a tiling window manager; I could swap between windows at will with keypresses, having some windows in multiple locations, others nowhere to be seen, and re-arrange how I had with button-presses.

The functionality can be replicated with a little known feature called “grouping”. When you create a new session, you can specify the -t command to “group” it with an existing one. This session-group will share all windows, but unlike having just one attached session, switching which window is active does not switch it in all.

I also wrote a quick script to automatically look for an existing session, and, if none are detached, create a new one.


tmux ls |grep -v attached |grep -v daemon
if [ $? -ne 0 ]; then
tmux new-session -t master ;
else
tmux attach;
fi


You know what’s ironic? Society’s use of the word “Irony”. Most people seem to think it means “when something happens when least expected or at the worst possible time”. However, as we know, readers, Irony actually stems from the theatrical practice where a character makes a statement of belief that the audience knows is contrary to fact. That makes the very usage of Irony in a day to day basis ironic to those who know it’s true meaning.

Dramatic Irony is based on withheld information. So much of daily drama is, too. Drama, Strife, and Conflict in relationships is very rarely caused by too much common knowledge – In my experience, it is caused by the exact opposite, secrecy and refusal to acknowledge the minor and natural conflicts, instead choosing to subsume them under a friendly veneer. This causes them to fester and boil, to be communicated (whether directly or indirectly) to subsets and individuals, and for people to have to feign ignorance while dancing around contentious subjects. This leads only to more irony, more drama.

As Thumper says; “If you can’t say anything nice, don’t say anything at all”. This advice is terrible, and deprives the world. What good are positive review, without negative ones to set the scale? Further, what criticism is leveled and acknowledged is often done with bitter invective and biting satire than with any constructive advice – Just look at the contrast of popularity of Yahtzee’s  “Zero” and “Extra” punctuation.

So, yes; I have little intention of seeing these people again, so I’ll air my grievances. I’ll be at PAX, but as an attendee, and I’ll be pretending these people, as well as the areas they represent, do not exist.

If there’s any criticism I have to level at the Enforcers as a whole, it’s of the general malaise afflicting the group. People are almost autistic in their ability to get things done at and for the convention, but unable to apply those same skills to bettering their outside lives.

James Fiske – I do not doubt your technical merits, nor your ability to get people to do things. I doubt your ability to do such without burning through people’s patience, and I know mine is not the first to have been destroyed.

Khahil White – You have terrible taste in choosing who to trust and respect, and screw over those that aren’t on that list.

Kenneth Kuan – I used to think you were cool; Then I thought you were an ass. Partially because it seemed that everything I wanted flowed into your hands. Then, I realized not that long ago, I was simply coveting things because you had them, and I realized how pathetic and fleeting your grasp on those treasures really was. What really got me tonight was how aptly you simply didn’t get it – Rather than understanding the why or how of Appengine architecture, you flat out dismissed it. I’m ashamed of how long I mistook your idiocy for insight.

Matt Buchanan – You could be an excellent friend, a solid planner, and an excellent organizer and bringer-together of people. Instead, you choose to be an ignorant, abrasive jerk, drunk on what little power you wield over people. I appreciate you throwing parties, but you’re very bad at being a gracious host.

Rachel Chateaubriand – I never liked you, and I could never put my finger on it. Finally, it was when I overhead you advocating theft from your employer that I realized it; You’re incredibly selfish and low class. Why in the nine heavens anyone bothers to continue to speak to you, I am uncertain. My best guess is that you are simply good enough at camouflaging yourself as whatever group you’re talking with that you’re able to hide it. For all I know, even the conversation I overheard was simply camouflage among the equally low class people it was with; Somehow, I doubt it. You’re a snake; And I will continue to crash whatever parties I damn well please. If you’re unable to deal with my completely unveiled disgust with you,

To Smalllady (whose real name I haven’t even the slightest guess at), who seemed upset that I would not be attending any more events; These are the reasons why. I am tired of playing second fiddle to halfwits; but it’s their sandboxes that I’m playing in.

(Name Redacted) posts to facebook:

“I know you guys trying to be friendly and nice and all but dont post on my friends posts when I comment them and you can see it, unless your friends with them. Thanks :S”

My response, archived here because said person is no longer in my friends list.

Privacy options are in the top corner, under Account settings. What your friend, Party C, is looking for is “Permission to comment on your posts”. If it has been set to “Friends Only”, then even if you should comment, and I should see it, I cannot comment on Party C’s post.

But don’t set that. It’s stupid. It’s immature. You’re posting to Facebook, you’re having a conversation in a public room of a party that everyone’s invited to. Should other partygoers feel that they are threatening if they should happen to speak up and try to join the conversation?

You’re living in a ridiculous “filter bubble”. Certainly, people add stupid comments to posts; You can remove them if you wish, or you can simply ignore them like a rational person. By setting this setting (or making passive aggressive postings in which you fail to realize that the setting exists), you’re perpetuating a culture that’s unable to comprehend humans as a fungible resource; You’re treating the world as your own personal monkeysphere and a group of “others” who are somehow malicious.

Everyone in their right mind is playing Portal 2, As was I. But when my co-op partner (my little brother) had to go to bed, I decided to check out what else there was. And, much to my surprise, the extras section of Portal 2 had something I wasn’t expecting – In addition to the trailers and videos that had previously been released was “Super 8 Interactive Trailer”.
You start off in a train (Big surprise, if you’re familiar with Valve games). The tooltip pops up, encouraging you to use the middle button to zoom. There’s boxes of military looking supplies, and a radio in the center of the car that’s on, and indistinguishable speech playing. You can walk towards the front of the car, and that’s when things start to happen. A clear shout comes over the radio; “There’s something on the track!”. The view warps, as the boxcar collapses around you.

Seconds later from the player’s perspective, you wake up surrounded by the debris of the train. Fires light your way as you wander towards the front. A barrel catches fire and explodes as you approach, teetering telephone lines suddenly give. You continue walking forward, through the wreckage, til you come to a relatively intact boxcar, and it’s clear why it’s survived better: It’s armored. Suddenly, the area around you explodes, and you’re thrown back, to get a better view of the armored hatch. The hatch starts to thud, then dents appear, and finally, it flies up –

Fade to black, and launch webbrowser. A flash movie plays, and the credits at the bottom says “JJ Abrams” and “Steven Spielberg”. A curious marketing trick (I’d not heard of Super 8 before), but it seems in the vein of both Cloverfield and ET; A young boy with his Super 8 camera is helping his brother film a movie, and they seem the only ones in the town who believe in the alien entity, despite the fact that the military has shown up en-masse.

What does this actually mean, though? Is this just a quick marketing stunt? Is Valve making a Super-8 game? Will there be more interactive components to this story? What did I miss in the train car that I should’ve zoomed in on? If anything is clear from Abrams past works, there will be much secrecy, and many more small snippets of information to build hype. After the almost-letdown that was the Portal 2 ARG, is this what Valve should be doing?