sparkfun is down

Sparkfun's IT administrator laments about MySQL while site is down. Copied here because it'll be gone tomorrow.

tl;dr: SparkFun's down for a while this evening, starting at 18:30pm MDT, while we work on some database stuff. We should get better at uptime, but right now we're mostly trying to get better at databases. Related web nerd musings below the llama, for those interested.

So tonight, sparkfun.com's going to be down for a while. We're hoping for no more than an hour or two, although it could very well run longer.

This is where I concede that we could really get better at keeping the site up all the time than we are right now. We're working on that bit, but right now what I want to ramble about instead is databases.

Since approximately the dawn of time (which is to say since some time last decade, before I worked here), SparkFun has run the core of its business on MySQL (and recently MariaDB). When I got hired in the fall of 2007, a lot of the important code was still split between a hacked-uposCommerce installation and a pile of MS Access clients. That stuff has been gone for years now, but the database itself persists. In places it's practically a fossil record of offhand design decisions made by other people writing a shopping cart some time in 2003.

This probably happens everywhere. Application code may change drastically, but in a working shop, the data store has a way of becoming the scaffolding around which (for better or worse) everything is built and must be maintained.

As it happens, when we first realized that we needed a better toolset than not-very-object-oriented old-school PHP, we made some decisions with long-term consequences:

• We decided to keep using PHP instead of rewriting everything in Ruby.
• We decided to write our own lightweight web application framework instead of using existing PHP projects.
• We decided to write our own ORM layer instead of using existing libraries.

A lot of things informed those decisions: We didn't want to have to replace all the working software at once just to use better tooling for new code. Everyone on our team already knew PHP. The frameworks we looked at seemed lacking in comparison to the stuff we'd used in languages like Ruby and Perl. The available PHP ORM libraries were either heavier than we wanted or built on assumptions that didn't fit our existing (frequently terrible) schema. Finally, rolling our own sounded kind of interesting, and we had no idea what we were getting ourselves into.

(Ok, so in the interest of full disclosure, what happened is that I talked louder than people who knew what we were getting ourselves into.)

Fast-forward a couple of years during which we hire more programmers and SparkFun gets another half million orders. We're sitting around in the mountains drinking beer and talking about what we really should get better at and the subject of PostgreSQL comes up.

See, MySQL and its forks, for all of the really huge and amazing stuff built on them in the last decade, have got some drawbacks. On this point, I risk triggering a lot of strong feelings from the sort of people who have strong feelings about databases, so for the time being, let's just say that it started to seem like we'd have a better time writing business software on a database that embodied a more rigorous approach to formal types and relationships. And because of years of our nerd friends telling us we should probably just use PostgreSQL, it kind of seemed like PostgreSQL might be that database, especially since we already knew we were going to use something with open code.

So we had some more beers and we talked about it and we had some more beers and we talked about it some more, and maybe six months later we finally decided to go for it.

So that's what we're doing.

Sort of.

It turns out that, when you write your own ad hoc ORM layer against a piece of software like MySQL without the faintest idea of how far you're about to be in over your head, it's pretty easy to make a bunch of assumptions without even noticing them. Rumsfeldian unknown-unknowns, if you will, creep in at every turn. Like fish and the concept of water, people who work with a given data store for a long time begin to forget that certain abstractions are even objects of possible consideration. These bits of data are represented as strings in PHP? Well, that's natural, right? What else would they be? String comparisons in your database of choice are all case-insensitive? Well, heck, that's almost a feature, right?

So then you decide you're going to point things at a different database and you get a crash course in all of the stuff you were wrong about, and all of the things you made impossible to test, and all of the ways your implicit assumptions about the two or three or five different type systems you rely on across languages and representations of your model are actually pretty fractured and subtly wrong.

So what we're doing tonight is rearranging a bunch of MySQL schema and deploying a terrifyingly long list of code changes that get us about three quarters of the way to being able to actually migrate to PostgreSQL without all of our software exploding continuously. We sincerely hope. (As regards exploding software, I also have a lot of recently developed thoughts about how I'll never again write a large project in a dynamic language without just tons and tons of unit tests.)

I should really write something coherent about all of this, but I just realized we actually need to deploy the code and such now.

Anyway, thanks for bearing with us while the site's down. The colored wires and blinkenlights and such will be for sale again soon.

☮,

— Brennen / SparkFun IT

postscript: This is the best sloth, right?

The State of Mobile VOIP: Redphone

A couple years ago, Whisper Systems released a product called Redphone that is quite intriguing. Redphone is both an app for Android and iPhone, as well as a service (like Skype). The easiest way to describe Redphone is that it's SIP with a fixed configuration, ZRTP and TURN. That is:

• all the parameters are set by the system making configuration trouble-free
• all the calls are encrypted with ZRTP
• while SIP is usually peer-to-peer, Redphone calls go through Redphone's servers.

You register with Redphone using your phone's number. (It also works with Google Voice.) Redphone verifies it with a text message that it automatically responds to.

When you call, you specify another phone number. You can use the stock dialler and, if the other person is registered, it will use that instead. Redphone "rings" this device using either an SMS message, or a Google or Apple push notification, whichever the receiver prefers.

NOTE: In my experience, Google push notifications don't appear to work (or at least not be reliable) over a NAT'd local network (most wifi). If you aren't receiving calls, try switching to "always use SMS".

If the call is accepted, the two phones begin an elaborate protocol to both encrypt the conversation and make sure that no one has intercepted the call in between the two phones.

After a call has begun, Redphone will display two words on the phone's screens. You read one word; the other party reads the other word. If the pair matches what the phones show on the screens, then you can be reasonably sure that no one is in between you two, relaying the call's data.

ZRTP specifies that, once you have called someone, you shouldn't have to go through the "compare words" protocol again, however rumor has it that Redphone doesn't store this information. I can't say I mind, since it seems (to me) like the best place to attack the protocol, and the word pair protocol is quite unobtrusive.

Redphone is fairly hassle-free. ZRTP doesn't require previously trading public keys with anyone. The security relies on you recognizing the voice at the other end, and the word pair. Calling someone is as simple as selecting them out of your contact list. If you or your communicado's are tech-newbs, but you want security, this is the app to use.

Redphone was written by Moxie Marlinspike, a name well known in the security business, and Stuart Anderson. Redphone was acquired in 2011 by Twitter, who seems to be trying to make a name for itself among Middle Eastern protesters. However, the service can and has gone away before. Have a back-up.

The State of Mobile VOIP: Republic Wireless

A company called Republic Wireless made the headlines a while back. They advertise that they will send a call to either wifi or the cell network, whichever is cheaper and available. This is basically the Holy Grail of mobile communication. As promising as this is, there are gotchas.

For one, Republic Wireless basically licenses coverage from Sprint not unlike Metro PCS and others. This means that your back-up coverage is fairly spotty.

Second, they want $20 a month. This is cheaper than a cell phone account, but only half what Metro PCS wants including data. I have a suspicion that you're paying for a "data-only" account.

Finally, you have to use their phone (a Motorola Defy XT). Even though the software would work fine on any phone, and even though there are other Sprint phones, they want you to use a specific (rapidly aging) model.

If you put this all together, Republic Wireless is not exactly the "liberator" that it claims to be. In fact, they are fairly rigid. It would be nice if one could use their own phone on their own provider, even if it's more expensive. Given all this, it may be just as cheap and convenient to track down a data-only provider, and just use SIP all the time.

The State of Mobile VOIP: Skype

As mentioned in an early post, Skype is another audio/video conferencing tool available on Android devices. Skype has been pretty well supported for a few years. It started out requiring cellphone connections, however for a while you've been able to make wifi calls for free.

Again, Skype supports both audio and video. Skype has a proprietary encryption protocol which should keep out your neighbors from overhearing over wifi, however the government has been hacking at it for a while. (The NSA was actively hiring to crack this one protocol a few years ago.)

Recently, Skype was purchased by Microsoft. At least one blogger has proposed that Microsoft gave the government a back-door into the service. Then of course, the whole PRISM thing blew up. Therefore, I would not assume that is safe for planning your next coup.

Skype has a friendly social environment, where you can look up your friends and see if they are online and available for chat. Multi-party conferences are possible.

Finally, Skype has excellent telephone integration. You can get a Skype-in number such that people can call you and have it ring over wifi. Also, you can buy credits such that you can dial-out. Obviously, when talking with a legacy network, there is zero encryption (however the wifi is still relatively safe.) You can also activate (for free) voicemail for your account. Call forwarding is available but I believe that it incurs standard charges.

I would go so far as to say that Skype is a better version of Google Voice. That is, it is what everyone wishes Google Voice would be. When you're off-line, incoming calls go to voicemail, and you are notified via email about them. (It doesn't do transcription (yet).) You can receive calls over wifi for free - both cost free and hassle free. You can also make cheap 2c/minute calls, again, over wifi if necessary. Skype to Skype calls can include video and are theoretically encrypted, unlike Google Talk. [ Is this true ? Talk doesn't use SSL ? ]

You would think that Google Voice would make a nice companion to Skype. That is, you would have Google Voice ring both your cell phone and your Skype account so that, if your cell phone is out of the reception area, you could just pick up the call over wifi.

Unfortunately, it isn't this simple. When Google Voice rings both your cell phone and Skype account, your phone literally leaps out of your pocket ringing both numbers. Its a toss up which call will come in second (which trounces the first call) but it is usually Skype.

Additionally, if you have voicemail for these accounts, they will all pick up, conspiring to bury your Google Voice voicemail. I disable voicemail on Skype for this reason, but if I got rid of Google Voice, I would make Skype my primary voicemail.

What you need is a service that rings *either/or*. That is, if you're signed in with Skype, use that. Otherwise, use the cell phone. Once again, Skype trumps Google Voice here. Skype will do this, but you incur their cheap rates during the call. Also, Skype wants $5-6/month for the DID number.

hmm, if you pointed Skype at your Google Voice number, then all your phones would ring when you weren't signed in...

My latest annual

I believe that capitalist markets do work, and that they promote competition among the best contenders. However, I believe they only work when people have the information to choose between those contenders, and are free to choose between them. If every candidate is a blind date, how are you supposed to pick the best ? How do you let others evaluate which is best ?

An airplane annual inspection doesn't fit into the perfect market model very well. You have to pay the entire cost up front, before you know anything about whether the plane will pass inspection or how much work it will require to make it airworthy again. Therefore, you have no idea what a mechanic will say before paying the money and getting the inspection. A mechanic can come back with a huge list of work that has to be done before he will sign off your airplane. You have no idea if a mechanic is picky or not, except through word of mouth.

Therefore, I feel it is helpful to the market that I share my experience with my last annual, with Able Air Corporation at Whiteman Airport in Pacoima. I will not say anything subjective about this experience - nothing good, nothing bad - only the facts so that other airplane owners can compare the experience to their's and make up their own minds, prior to laying down cash.

I own a 1975 Piper Warrior, basically a Cherokee 150.

I contacted Able because my previous mechanic left several items inoperative or degraded after the last annual. I fixed all of these issues except for one: The plane had the classic Cherokee "hard starting" problem. I suspected everything from the starter to the fuel system, but the owner of the shop had a look at the plane and suspected the magnetos. He then took a look in the logs and, sure enough, the magnetos hadn't been touched in over 1000 hours.

I let him inspect and repair the mags. When I got it back, the plane started right up. The starter still had trouble getting over the compression hump, but the "bump trick" still worked. However, instead of having to crank the engine for dozens of revolutions until the battery died, it started after a couple revolutions. My battery recharged from this effort much faster too.

From this experience, I felt comfortable letting Able Air do my annual inspection. They asked $1275 (15 x $85) for the inspection. I dropped off the airplane on May 7th.

I didn't get a call back until May 17th. When they came back with the results, they said that a couple ADs had not been complied with and that they'd have to be re-done. I have taken excellent care of my airplane, I've had every AD that was mentioned to me dealt with and I've had half a dozen mechanics look over my logs. None of them ever expressed an issue with the logs.

One of the ADs was to inspect and replace the attachment hardware of the flap lever. The AD had been addressed but the shop owner didn't like the entry in the AD compliance sheet. It said, "replaced bolt" when the AD asked for the bolt and bushing to be replaced. Note that the FAA's own suggested AD compliance sheet format leaves less space for "description" than the AD number itself, but this didn't sway him from demanding that the description be longer.

Complying with the AD was only estimated to cost 15 minutes and a bushing so I let them do it.

The other AD was the Lycoming crankshaft ID inspection AD. The plane's engine was overhauled in 2006. I suspect that, since it was being upgraded to 160HP, that mechanic now felt that the AD applied and dealt with it.

However, again the shop owner didn't like the evidence of compliance. All he could find was an entry about SB505(inspecting the ID), not SB530 in which you apply a simple coating to the inside diameter of the crankshaft. We called the other mechanic and he said there was no way they would have done SB505 without SB530.

After much haranging with the shop owner, I could not convince him that that mechanic simply forgot to add the additional entry. I reluctantly agreed that the next step was to take off the propeller and see if there was the word "PID" stamped into the prop plate which indicates "painted inner diameter." I had no idea if this would satisfy them but if they were going to force an inspection on me anyways, they would need to do this.

They took off the prop and sure enough, PID was stamped into the side of the plate. They accepted this as proof of compliance with SB530.

The number of issues that they found on the airplane during the inspection were, what I would call, small. They had to address the new control cable AD, they wanted to add a bushing to the airbox, tighten a rod in an aileron, re-charge the nose strut and replace the pilot yoke universal joint.

They had given me a price on the major parts, except strangely for the universal joint. I told them to go ahead with the repairs.

On the 29th, I got a call about the status of the plane, and to get confirmation that they could buy some parts. When they took out the airbox, they decided that the valve within it had to be replaced. Replacement airboxes are quoted at $1600. (Getting a valve alone is cheaper.) I asked what was wrong with it. He gave me his reasoning but fundamentally his concern was that warmer air would leak in and cause detonation. The readers may ask themselves, if warm air isn't supposed to cause detonation when carb heat is on, why would it do it when it's off, even with a leak ? In any case, I couldn't change his mind.

Then he informed me of the price of a new universal joint. The cheapest he could find was $265. I then asked why it needed to be replaced. He gave me his reasoning but the fundamental issue was that it was loose, that it had too much play. I protested that that wasn't enough to make it not airworthy. Again, the previous mechanics, including the one who addressed the control yoke AD, thought it was fine. I said I'd be out to look at the airbox, but I also wanted to check this loose play too.

On May 31st, I went out to the plane to check the play in the pilot yoke. There was a little play but it was, in fact, less play than what was in a nearly new 172 that they had in their hangar. I again tried to convince the mechanic that it was fine, but couldn't.

At this point, I just wanted the annual over so I "bought" the parts and allowed the mechanic to install them. I provided them the airbox valve on June 8th.

On June 13th, I called to check the status of the plane. The mechanic said it wasn't done. I informed them that I would be leaving on a trip, and that, if they would like the bill paid before July, it would have to be done by Saturday. The mechanic said not to worry about paying in July, and the bill certainly wouldn't be ready by Saturday.

On July 3rd, I called to check the status of the bill. The secretary said it wasn't done, but I could have the keys, and that the log had been signed off. I went and got the keys.

By July 16, no one had called me so I called and asked if the bill was ready. They said it was. I went, paid it and picked up the logs.

The bill was one third higher than the highest annual I've ever paid, including last years' during which they replaced all my brake lines. 7 out of 16 items on the bill are complying with ADs. One item I specifically asked them not to do (main strut servicing). Another item I have no idea what it is (main cup bearing ?); the mechanic never mentioned it. The hours are not itemized; there is only a total - 40.5.

The new, supposedly correctly written AD compliance sheet is no better than the previous ones. Many entries still say "C/W" ("complied with"), which the shop owner said was insufficient. And despite the lecture, during which the shop owner said that showing compliance had to include the measurements taken (for instance, if a bolt is supposed to be a certain diameter, the measurement of the diameter should be included) this shop owner didn't write down the compression values of the cylinders (for AD 2009-26-12) which my previous mechanic had always done. So for all that trouble, it appears that the ADs are in worse shape.

I hope this information helps someone else make a decision about their next airplane mechanic.

The State of Mobile VOIP, part 5: SIP clients revisitted

Sorry no post in a while. Been on vacation, without broadband. :(

I'd like to correct an earlier statement, and report some encouraging findings.

- The Android SIP client -

I said that the Android built-in SIP client wasn't very good, and listed a few features that it lacked to provide proof of that statement. I've learned that its feature set isn't as small as I thought. (Instead, I simply couldn't get something to work, and assumed that since CSipSimple worked, that the built-in stack was insufficient. Alas, that is one of the sore spots of SIP: When something doesn't work, you don't know who to blame.)

The feature(s) that I mentioned didn't work concerned dialing regular phone numbers from the stock dialler over SIP. Namely, I said that the only thing that the built-in stack could do was dial someone@somewhere, i.e. SIP-to-SIP calls. Furthermore, that the only way to actually dial those numbers was to enter the address into the Contact list and select it. These 2 statements are untrue.

Assuming that you have an account where you can dial regular phone numbers (dial out capability), (and that your account has some money in its balance), you can dial any number - either from your address book or dialled manually - using this SIP account, straight from the stock dialler. I'll explain how after some soapboxing.

I like the idea of using the stock dialler because it doesn't require using extra software. CSipSimple isn't a small download. Furthermore, it doesn't always play nice with the stock dialler and Google Voice, if installed (not that making a SIP call from the stock dialler is trouble free either; see below.)

Don't get me wrong; CSipSimple is an awesome SIP client. It has CODECs galore, it has re-write rules, it can register multiple accounts, it has encryption and it has video (and these are just the features I can think of off the bat.) It really is a huge pile of awesome.

But if all you want to do is make SIP calls, and cheap international calls, it appears to work with some tweaking.

End of soapboxing; here's how to get it to work. I have an account with Mondotalk, so I'll use it as an example. Go into the stock dialler. Click Menu | Settings | Accounts. (Note that Menu | Accounts goes to an entirely different place; don't go there.) Click Add Account. Enter your username, password and, in my example, sip99.mondotalk.com. Click on Optional settings | Outbound proxy address. In my example, enter sip99.mondotalk.com again. Click Save.

You can play with the Receive incoming calls setting if you like, but it's not relevant to making calls.

Back out one level and click on Use Internet calling | Ask for each call or For all calls when data network is available.

Here's the important part: If you have Google Voice installed, set it to Ask for each call. If you have Skype installed, also set it to Ask for each, if that's possible.

When you want to make a SIP call, dial the number normally (from the Contact list or enter manually or click on a phone number in the webbrowser). If it then asks about Google Voice, say 'no'. If it asks about Phone vs Skype, pick Phone. It will then ask Mobile vs Internet call. Select the latter of course.

Update:

I've been using the stock dialler for a while now. I have to say I'm quite pleased with it.

One thing you should be aware of, if you're going to use it: Provide the complete country code while dialling. Mondotalk is an international carrier and assumes that the first digit(s) is part of a country code. Unfortunately, the stock dialler isn't as smart as CSipSimple and won't prefix a number for you. Therefore, you have to do it yourself.

For example, put a '1' in front of all your numbers that reside within the US. If you don't, your call likely won't connect, but if it does, you could be charged a fair amount of money for that first minute. Guess where country code 95 is ?

The State of Mobile VOIP, part 4: SIP providers

There's a wide variety of SIP providers. Some are aimed at businesses trying to squeeze more phone lines in, some are aimed at immigrants trying to call home cheap, etc. Take a look around and try different ones (they're cheap, after all) until you find one you like. I will share my experience with 2 of the major providers that I've tried.

Most providers tend to be pretty careful about their customers. For instance, OnSip and Mondotalk require that you call them to confirm a purchase of minutes. I suppose this is to make sure that your set-up is working. This is a good thing because your client may not work with the provider.

The problems that you will see with clients are various. Depending on the provider, I can get certain combinations to register, but then it won't make a call. Or there won't be sound. In one direction. Or the other. Or both.

This may be because the NAT support isn't working, or that there isn't a compatible codec (even though they all support uLaw/ALaw.) Keep in mind, I'm not doing anything exotic. For many providers, I can't even call their own main number, much less a subscriber.

One provider that I've had some success with is CallCentric. They have cheap, simple plans and you can still get a DID number from them. In other words, there's nothing special about them. They're just who I was able to get a SIP stack to work with.

One issue with them is that they feel they need to collect a monthly fee to cover their 911 expenses. This will use up your balance steadily (but not quickly.) I don't make many calls and $20 would last me a long time, except for this fee.

The provider that I'm currently using is Mondotalk. They provide all the usual features. In call testing, I hear a slight delay but it's comparable to Google Voice, or Skype. I haven't tried any exotic codecs. My default works fine (and sounds better than my cell phone.)

Despite having a good client and provider, I wouldn't rely solely on SIP on my mobile phone. The networks you will run into are too varied and SIP's NAT hacks aren't reliable. Running SIP over your cell data may work if you a large or infinite bandwidth/data limit.

You may be asking, "But I see all these providers on the net ? Who's buying ?" Small businesses are using SIP to lower their phone bills. It works for them because they don't have to deal with NAT, and they have a static configuration with a provider that they are familiar with.

Next post: What *does* work reliably ?