When SCALE approved my talk proposal, Fun with Linux and Devices, I had a challenge: I needed some good, visual Arduino demos that would work in front of an audience.

In particular, I wanted something that moved. A little toy truck? A moving penguin? A rotating sunflower? I fiddled with this and that, not fully satisfied with anything. And then suddenly I realized what I needed. Something cool. Something BIG. Something I'd been wanting an excuse to buy anyway.

An Air Swimmers Shark.

I'd seen these things on video, but never in person. They're available all over, even on Amazon, so I put in an order there and got a shark in a few days.

These things are rediculous and cool. It's huge, about 5 feet long, and filled with helium. It takes maybe half an hour to assemble. It has a small motor to beat the tail, an infrared transmitter, and a weighted receiver that moves back and forth on a track to tilt the fish up or down as it swims.

Once it's assembled, you can get it filled with helium at a party store (which costs $3 to $6 depending on where you go). Once the shark is filled, you add clay as ballast until the shark is neutrally buoyant, neither rising nor sinking. It's quite sensitive: you'll find yourself needing to add or remove pea-sized chunks of clay as the temperature in the room changes, but being a little over- or under-ballasted doesn't hurt it much. With its tail beating, the shark really does look like it's swimming through the air.

My shark is named Bruce, after the mechanical shark used for the movie "Jaws". My Bruce, I'm happy to say, has been much more tractable than his famously intemperate namesake.

Okay, now how do we turn this ridiculous-but-cool thing into an Arduino project?

Hacking the transmitter

There were two possible approaches. First, mount an Arduino directly on the shark, and let it be totally self-directed. Second, patch the Arduino into the shark's transmitter and control it from Linux. I chose the second option, for several reasons. First, I was fairly sure it would be easier, and less invasive (the shark would still be usable with manual control). I also liked the idea of keeping the transmitter as a manual override, in case my control program didn't work right. Finally, I liked the idea of keeping a Linux machine in the loop -- the shark would actually be controlled by Linux, not just by the Arduino.

So the first thing I did was take the transmitter apart (4 Philips screws). Inside are 4 pushbuttons, for right, left, up, and down, and the circuit board is nice and simple. Whew -- this might be doable!

Four more screws and I had access to the back of the board, which was equally simple. Now I could get my voltmeter on the contacts while I pushed buttons.

It turned out the contacts (indicated with arrows on the photo) on the downstream side of each switch were normally high (4.5 volts -- the transmitter uses 3 AAA batteries). When I pushed the button, the contact went to ground. Okay, so what I needed was some way for the Arduino to ground those contacts at will.

First I needed to solder some wires to the contacts. (How do you tell which side of the pushbutton is the one you need to solder? Well, one side changes voltage when you press the button, and the other side stays constant. The one that changes is the one you need to connect to the Arduino, so the Arduino can change it too.)

I figured I needed 6 wires: ground, power, and one for each switch. (It turned out I didn't need the power wire, but I figured it didn't hurt to include it just in case.) I wanted to have a nice small connector on the side of the transmitter, but I couldn't find any 6-pin connectors that didn't look big and bulky, so I gave up and decided I'd just let my ribbon cable dangle from the transmitter. If I got a nice multi-colored one, maybe it would look festive.

I couldn't find any 6-conductor ribbon cable, so I got a wider one and separated 6 wires from the rest. Then I soldered the six wires to the appropriate places (marked by arrows in the photo). On the other end, I tinned the six wires with solder so I could plug the stranded wires into my breadboard.

Simulating button presses

I've done enough reading to know of three ways to simulate a button press. You can put a relay between the two contacts of the switch; you can do the same thing, but with an optocoupler (opto-isolator) instead of a relay; or you can do some magic with a transistor. I was fuzzy on the transistor magic bit, so a relay sounded easiest.

I played around with a relay and a spare switch and convinced myself I knew how to wire them up. Then it was off to my local parts store to buy four matched relays small enough to fit on my little mini breadboard.

There followed a several-day fiasco wherein I bought lots of relays that turned out not to be suitable, and got increasingly frustrated at how large and clunky all the available relays were. There are smaller ones, but I couldn't get them to work. And I learned that relays mostly come without documentation on which pin does which, so there's a lot of experimenting with each new type.

Frustrated, I tried some optocouplers I'd bought on a whim last year. No dice ... couldn't get them to work either.

Desperate, I turned to IRC, #arduino on Freenode. The folks there are mostly electronics wizards, and I'm sure my questions must have seemed very dumb, but they were patient with me, and pointed me toward a very simple circuit, LED4dummies, that was just what I needed. (They also suggested Wikipedia's Open collector article, but I found that less clear.)

It took me some experimenting with a transistor, an LED and a couple of resistors (I blew out a couple of transistors before I realized I had the R2 resistor in the wrong place) but eventually I got it working, and felt confident enough to try it with the real shark transmitter. The key was to simplify the circuit so it had no extra parts, then once it was working, add more parts to build it up to what I needed.

At left, the circuit I ended up with. For each button, I have one transistor and one resistor (I don't need the second resistor from the LED4dummies circuit, since that was just to keep the LED from burning out).

At right is the circuit assembled on a mini-breadboard on top of the proto-shield. Note that the ends of the ribbon cable are plugged in to a spare header I had lying around; a header makes a passable connector, so I can plug it in fairly easily right before a talk. The green and blue wires in the back are going to Arduino digital output pins 3 through 6 (leaving 0 and 1 for serial I/O). The red wires go from the transistors back to the ribbon cable wires that go to the shark's transmitter buttons.

The software side

Now I could make the shark think I'd pressed a button on its transmitter. How do I control that from Linux?

On the Arduino side, I wrote a simple program that reads and parses commands coming over the USB cable. So from the computer, I might send a line like L 300, and the Arduino would "press" the Left button for 300 milliseconds. I had already written something like this for a couple of other Arduino programs. That program is shark.pde.

On the Linux side, first I needed something that established a serial connection and sent commands to the Arduino. I wrote a Python class for that, shark.py. That let me send commands from the Python console to test the shark.

Then I needed something more visual, something I could show during a talk. In particular, the shark doesn't swim unless someone's pressing left, right, left, right buttons over and over. Of course I wanted the computer to handle that part.

So I wrote a little Python-GTK application that keeps the shark swimming, and lets me drag a mouse around to adjust its left/right up/down direction: sharkwindow.

Purely by coincidence, the week before SCALE, Scott Adams introduced a roboshark character: Dilbert, Jan 11 2012. Nice timing for the debut of my own roboshark!

Sadly, I don't have any photos or video of the shark in action. But if you're a LWN subscriber, there's an article on my talk with a couple of great pictures: Robots rampage (in a friendly way) at SCALE 10X. And you can see my slides and notes at Arduino notes.

I've long wanted a way of converting my HTML presentation slides to PDF. Mostly because conference organizers tend to freak out at slides in any format other than Open/Libre Office, Powerpoint or PDF. Too many times, I've submitted a tarball of my slides and discovered they weren't even listed on the conference site. (I ask you, in what way is a tarball more difficult to deal with than an .odp file?) Slide-sharing websites also have a limited set of formats they'll accept.

A year or so ago, I added screenshot capability to my webkit-based presentation program, Preso, do "screenshots", but I really needed PDF, not images.

Now, creating PDF from HTML shouldn't be that hard. Every browser has a print function that can print to a PDF file. So why is it so hard to create PDF from HTML in any kind of scriptable way?

After much searching and experimenting, I finally found a Python code snippet that worked: XHTML to PDF using PyGTK4 Webkit from Alex Dong. It uses Python-Qt, not GTK, so I can't integrate it into my Preso app, but that's okay -- a separate tool is just as good.

(I struggled to write an equivalent in PyGTK, but gave up due to the complete lack of documentation of Python-Webkit-GTK, and not much more for gtk.PrintOperation(). QWebView's documentation may not be as complete as I'd like, but at least there is some.)

Printing from QtWebView to QPrinter

Here are the important things I learned about QWebView from fiddling around with Alex's code to adapt it to what I needed, which is printing a list of pages to sequentially numbered files:

• To print, you need to wait until the page has finished loading, so connect a function to SIGNAL("loadFinished(bool)"), then load(QUrl(url)).
• That loadFinished function remains registered, so as you load new pages, it will be called each time. So you can load() the next URL as the last step in your loadFinished callback.
• If you get confused about callbacks and connect more than one of them, bad things happen, and only the last page gets printed, or QApplication.exit() doesn't exit at all.

Things I learned about QPrinter():

• All the examples I found online set the page size with lines like QPrinter.setPageSize(QPrinter.A4) or setPaperSize(QPrinter.A4) (setPageSize is apparently deprecated in favor of setPaperSize); but
• If you want to set a specific size, you can do that with a line like QPrinter.setPaperSize(QSizeF(1024, 768), QPrinter.DevicePixel) The second argument (DevicePixel) is a unit, from this list.
• That line gives you the right aspect ratio. But if you think "DevicePixels" means the size will correspond to pixels in your browser window (just because the documentation says so), you're sadly mistaken.
• If you want a PDF page that actually corresponds to the size of your browser window, you can get it by calling QWebView.setZoomFactor(z) You'll have to experiment to find the right value of z; I found I needed about 1.24 if I wanted to capture my full 1366x768 slides, or exactly 2.0 if I wanted to restrict the saved PDF to only the 1024x758 part that shows up in the projector.

Anyway, it's a little hacky with that empirical zoom factor ... but it works! The program is here: qhtmlprint: convert HTML to PDF using Qt Webkit.

And it does produce reasonable PDF, with the text properly vectorized, not just raster screenshots of each page.

Printing the slides in the right order

Terrific -- now I can feed a list of slides to qhtmlprint and get a bunch of PDF files back. How do I print the right slides?

My slides are listed in order in an array inside a Javascript file, one per line. If I grep .html navigate.js, I get a list like this:

    "arduino.html",
    "img.html?pix/arduinos/arduino-clones.jpg",
    "getting_started.html",
    "img.html?pix/projects/led.jpg",
    //"blink.html",
    "arduino-ide.html",

To pass that to qhtmlprint, I only need to remove the commented-out lines (the ones with //) and strip off the quotes and commas. I can do that all in one command with a grep and sed pipeline:

qhtmlprint ` fgrep .html navigate.js  | grep -v // | sed -e 's/",/"/' -e 's/"//g' `

And voiaà! I have a bunch of fileNNN.pdf files.

Creating a multi-page slide deck

Okay, great! Now how do I stick those files all together into one slide deck I can submit to conference organizers?

That part's easy -- Ghostscript can do it.

gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=slidedeck.pdf -dBATCH file*.pdf

And now slidedeck.pdf contains my whole presentation, ready to go.

When I give talks that need slides, I've been using my Slide Presentations in HTML and JavaScript for many years. I uploaded it in 2007 -- then left it there, without many updates.

But meanwhile, I've been giving lots of presentations, tweaking the code, tweaking the CSS to make it display better. And every now and then I get reminded that a few other people besides me are using this stuff.

For instance, around a year ago, I gave a talk where nearly all the slides were just images. Silly to have to make a separate HTML file to go with each image. Why not just have one file, img.html, that can show different images? So I wrote some code that lets you go to a URL like img.html?pix/whizzyphoto.jpg, and it will display it properly, and the Next and Previous slide links will still work.

Of course, I tweak this software mainly when I have a talk coming up. I've been working lately on my SCALE talk, coming up on January 22: Fun with Linux and Devices (be ready for some fun Arduino demos!) Sometimes when I overload on talk preparation, I procrastinate by hacking the software instead of the content of the actual talk. So I've added some nice changes just in the past few weeks.

For instance, the speaker notes that remind me of where I am in the talk and what's coming next. I didn't have any way to add notes on image slides. But I need them on those slides, too -- so I added that.

Then I decided it was silly not to have some sort of automatic reminder of what the next slide was. Why should I have to put it in the speaker notes by hand? So that went in too.

And now I've done the less fun part -- collecting it all together and documenting the new additions. So if you're using my HTML/JS slide kit -- or if you think you might be interested in something like that as an alternative to Powerpoint or Libre Office Presenter -- check out the presentation I have explaining the package, including the new features.

You can find it here: Slide Presentations in HTML and JavaScript

I've been having (mis)adventures learning about Python's various options for parsing HTML.

Up until now, I've avoided doing any HTMl parsing in my RSS reader FeedMe. I use regular expressions to find the places where content starts and ends, and to screen out content like advertising, and to rewrite links. Using regexps on HTML is generally considered to be a no-no, but it didn't seem worth parsing the whole document just for those modest goals.

But I've long wanted to add support for downloading images, so you could view the downloaded pages with their embedded images if you so chose. That means not only identifying img tags and extracting their src attributes, but also rewriting the img tag afterward to point to the locally stored image. It was time to learn how to parse HTML.

Since I'm forever seeing people flamed on the #python IRC channel for using regexps on HTML, I figured real HTML parsing must be straightforward. A quick web search led me to Python's built-in HTMLParser class. It comes with a nice example for how to use it: define a class that inherits from HTMLParser, then define some functions it can call for things like handle_starttag and handle_endtag; then call self.feed(). Something like this:

from HTMLParser import HTMLParser

class MyFancyHTMLParser(HTMLParser):
  def fetch_url(self, url) :
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    link = response.geturl()
    html = response.read()
    response.close()
    self.feed(html)   # feed() starts the HTMLParser parsing

  def handle_starttag(self, tag, attrs):
    if tag == 'img' :
      # attrs is a list of tuples, (attribute, value)
      srcindex = self.has_attr('src', attrs)
      if srcindex < 0 :
        return   # img with no src tag? skip it
      src = attrs[srcindex][1]
      # Make relative URLs absolute
      src = self.make_absolute(src)
      attrs[srcindex] = (attrs[srcindex][0], src)

    print '<' + tag
    for attr in attrs :
      print ' ' + attr[0]
      if len(attr) > 1 and type(attr[1]) == 'str' :
        # make sure attr[1] doesn't have any embedded double-quotes
        val = attr[1].replace('"', '\"')
        print '="' + val + '"')
    print '>'

  def handle_endtag(self, tag):
    self.outfile.write('</' + tag.encode(self.encoding) + '>\n')

Easy, right? Of course there are a lot more details, but the basics are simple.

I coded it up and it didn't take long to get it downloading images and changing img tags to point to them. Woohoo! Whee!

The bad news about HTMLParser

Except ... after using it a few days, I was hitting some weird errors. In particular, this one:
HTMLParser.HTMLParseError: bad end tag: ''

It comes from sites that have illegal content. For instance, stories on Slate.com include Javascript lines like this one inside <script></script> tags:
document.write("<script type='text/javascript' src='whatever'></scr" + "ipt>");

This is technically illegal html -- but lots of sites do it, so protesting that it's technically illegal doesn't help if you're trying to read a real-world site.

Some discussions said setting self.CDATA_CONTENT_ELEMENTS = () would help, but it didn't.

HTMLParser's code is in Python, not C. So I took a look at where the errors are generated, thinking maybe I could override them. It was easy enough to redefine parse_endtag() to make it not throw an error (I had to duplicate some internal strings too). But then I hit another error, so I redefined unknown_decl() and _scan_name(). And then I hit another error. I'm sure you see where this was going. Pretty soon I had over 100 lines of duplicated code, and I was still getting errors and needed to redefine even more functions. This clearly wasn't the way to go.

Using lxml.html

I'd been trying to avoid adding dependencies to additional Python packages, but if you want to parse real-world HTML, you have to. There are two main options: Beautiful Soup and lxml.html. Beautiful Soup is popular for large projects, but the consensus seems to be that lxml.html is more error-tolerant and lighter weight.

Indeed, lxml.html is much more forgiving. You can't handle start and end tags as they pass through, like you can with HTMLParser. Instead you parse the HTML into an in-memory tree, like this:

  tree = lxml.html.fromstring(html)

How do you iterate over the tree? lxml.html is a good parser, but it has rather poor documentation, so it took some struggling to figure out what was inside the tree and how to iterate over it.

You can visit every element in the tree with

for e in tree.iter() :
  print e.tag

But that's not terribly useful if you need to know which tags are inside which other tags. Instead, define a function that iterates over the top level elements and calls itself recursively on each child.

The top of the tree itself is an element -- typically the <html></html> -- and each element has .tag and .attrib. If it contains text inside it (like a <p> tag), it also has .text. So to make something that works similarly to HTMLParser:

def crawl_tree(tree) :
  handle_starttag(tree.tag, tree.attrib)
  if tree.text :
    handle_data(tree.text)
  for node in tree :
    crawl_tree(node)
  handle_endtag(tree.tag)

But wait -- we're not quite all there. You need to handle two undocumented cases.

First, comment tags are special: their tag attribute, instead of being a string, is <built-in function Comment> so you have to handle that specially and not assume that tag is text that you can print or test against.

Second, what about cases like <p>Here is some <i>italicised</i> text.</p> ? in this case, you have the p tag, and its text is "Here is some ". Then the p has a child, the i tag, with text of "italicised". But what about the rest of the string, " text."?

That's called a tail -- and it's the tail of the adjacent i tag it follows, not the parent p tag that contains it. Confusing!

So our function becomes:

def crawl_tree(tree) :
  if type(tree.tag) is str :
    handle_starttag(tree.tag, tree.attrib)
    if tree.text :
      handle_data(tree.text)
    for node in tree :
      crawl_tree(node)
    handle_endtag(tree.tag)
  if tree.tail :
    handle_data(tree.tail)

See how it works? If it's a comment (tree.tag isn't a string), we'll skip everything -- except the tail. Even a comment might have a tail:
<p>Here is some <!-- this is a comment --> text we want to show.</p>
so even if we're skipping comment we need its tail.

I'm sure I'll find other gotchas I've missed, so I'm not releasing this version of feedme until it's had a lot more testing. But it looks like lxml.html is a reliable way to parse real-world pages. It even has a lot of convenience functions like link rewriting that you can use without iterating the tree at all. Definitely worth a look!

Like most Linux users, I use virtual desktops. Normally my browser window is on a desktop of its own.

Naturally, it often happens that I encounter a link I'd like to visit while I'm on a desktop where the browser isn't visible. From some apps, I can click on the link and have it show up. But sometimes, the link is just text, and I have to select it, change to the browser desktop, paste the link into firefox, then change desktops again to do something else while the link loads.

So I set up a way to load whatever's in the X selection in firefox no matter what desktop I'm on.

In most browsers, including firefox, you can tell your existing browser window to open a new link from the command line: firefox http://example.com/ opens that link in your existing browser window if you already have one up, rather than starting another browser. So the trick is to get the text you've selected.

At first, I used a program called xclip. You can run this command: firefox `xclip -o` to open the selection. That worked okay at first -- until I hit my first URL in weechat that was so long that it was wrapped to the next line. It turns out xclip does odd things with multi-line output; depending on whether it thinks the output is a terminal or not, it may replace the newline with a space, or delete whatever follows the newline. In any case, I couldn't find a way to make it work reliably when pasted into firefox.

After futzing with xclip for a little too long, trying to reverse-engineer its undocumented newline behavior, I decided it would be easier just to write my own X clipboard app in Python. I already knew how to do that, and it's super easy once you know the trick:

mport gtk
primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)
if primary.wait_is_text_available() :
    print primary.wait_for_text()

That just prints it directly, including any newlines or spaces. But as long as I was writing my own app, why not handle that too?

It's not entirely necessary on Firefox: on Linux, Firefox has some special code to deal with pasting multi-line URLs, so you can copy a URL that spans multiple lines, middleclick in the content area and things will work. On other platforms, that's disabled, and some Linux distros disable it as well; you can enable it by going to about:config and searching for single, then setting the preference editor.singlelinepaste.pasteNewlines to 2.

However, it was easy enough to make my Python clipboard app do the right thing so it would work in any browser. I used Python's re (regular expressions) module:

#!/usr/bin/env python

import gtk
import re

primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)

if not primary.wait_is_text_available() :
    sys.exit(0)
s = primary.wait_for_text()

# eliminate newlines, and any spaces immediately following a newline:
print re.sub(r'[\r\n]+ *', '', s)

That seemed to work fine, even on long URLs pasted from weechat with newlines and spaces, like that looked like

http://example.com/long-
    url.html

All that was left was binding it so I could access it from anywhere. Of course, that varies depending on your desktop/window manager. In Openbox, I added two items to my desktop menu in menu.xml:

  <item label="open selection in Firefox">
    <action name="Execute"><execute>sh -c 'firefox `xclip -o`'</execute></action>
  </item>
  <item label="open selection in new tab">
    <action name="Execute"><execute>sh -c 'firefox -new-tab `xclip -o`'</execute></action>
  </item>

I also added some code in rc.xml inside <context name="Desktop">, so I can middle-click or control-middle-click on the desktop to open a link in the browser:

      <mousebind button="Middle" action="Press">
        <action name="Execute">
          <execute>sh -c 'firefox `pyclip`'</execute>
        </action>
      </mousebind>
      <mousebind button="C-Middle" action="Press">
        <action name="Execute">
          <execute>sh -c -new-tab 'firefox `pyclip`'</execute>
        </action>
      </mousebind>

I set this up maybe two hours ago and I've probably used it ten or fifteen times already. This is something I should have done long ago!

My SJAA planet-observing column for January is about the Analemma and the Equation of Time.

The analemma is that funny figure-eight you see on world globes in the middle of the Pacific Ocean. Its shape is the shape traced out by the sun in the sky, if you mark its position at precisely the same time of day over the course of an entire year.

The analemma has two components: the vertical component represents the sun's declination, how far north or south it is in our sky. The horizontal component represents the equation of time.

The equation of time describes how the sun moves relatively faster or slower at different times of year. It, too, has two components: it's the sum of two sine waves, one representing how the earth speeds up and slows down as it moves in its elliptical orbit, the other a function the tilt (or "obliquity") of the earth's axis compared to its orbital plane, the ecliptic.

The Wikipedia page for Equation of time includes a link to a lovely piece of R code by Thomas Steiner showing how the two components relate. It's labeled in German, but since the source is included, I was able to add English labels and use it for my article.

But if you look at photos of real analemmas in the sky, they're always tilted. Shouldn't they be vertical? Why are they tilted, and how does the tilt vary with location? To find out, I wanted a program to calculate the analemma.

Calculating analemmas in PyEphem

The very useful astronomy Python package PyEphem makes it easy to calculate the position of any astronomical object for a specific location. Install it with: easy_install pyephem for Python 2, or easy_install ephem for Python 3.

import ephem
observer = ephem.city('San Francisco')
sun = ephem.Sun()
sun.compute(observer)
print sun.alt, sun.az

The alt and az are the altitude and azimuth of the sun right now. They're printed as strings: 25:23:16.6 203:49:35.6 but they're actually type 'ephem.Angle', so float(sun.alt) will give you a number in radians that you can use for calculations.

Of course, you can specify any location, not just major cities. PyEphem doesn't know San Jose, so here's the approximate location of Houge Park where the San Jose Astronomical Association meets:

observer = ephem.Observer()
observer.name = "San Jose"
observer.lon = '-121:56.8'
observer.lat = '37:15.55'

You can also specify elevation, barometric pressure and other parameters.

So here's a simple analemma, calculating the sun's position at noon on the 15th of each month of 2011:

    for m in range(1, 13) :
        observer.date('2011/%d/15 12:00' % (m))
        sun.compute(observer)

I used a simple PyGTK window to plot sun.az and sun.alt, so once it was initialized, I drew the points like this:

    # Y scale is 45 degrees (PI/2), horizon to halfway to zenith:
    y = int(self.height - float(self.sun.alt) * self.height / math.pi)
    # So make X scale 45 degrees too, centered around due south.
    # Want az = PI to come out at x = width/2.
    x = int(float(self.sun.az) * self.width / math.pi / 2)
    # print self.sun.az, float(self.sun.az), float(self.sun.alt), x, y
    self.drawing_area.window.draw_arc(self.xgc, True, x, y, 4, 4, 0, 23040)

So now you just need to calculate the sun's position at the same time of day but different dates spread throughout the year.

And my 12-noon analemma came out almost vertical! Maybe the tilt I saw in analemma photos was just a function of taking the photo early in the morning or late in the afternoon? To find out, I calculated the analemma for 7:30am and 4:30pm, and sure enough, those were tilted.

But wait -- notice my noon analemma was almost vertical -- but it wasn't exactly vertical. Why was it skewed at all?

Time is always a problem

As always with astronomy programs, time zones turned out to be the hardest part of the project. I tried to add other locations to my program and immediately ran into a problem.

The ephem.Date class always uses UTC, and has no concept of converting to the observer's timezone. You can convert to the timezone of the person running the program with localtime, but that's not useful when you're trying to plot an analemma at local noon.

At first, I was only calculating analemmas for my own location. So I set time to '20:00', that being the UTC for my local noon. And I got the image at right. It's an analemma, all right, and it's almost vertical. Almost ... but not quite. What was up?

Well, I was calculating for 12 noon clock time -- but clock time isn't the same as mean solar time unless you're right in the middle of your time zone.

You can calculate what your real localtime is (regardless of what politicians say your time zone should be) by using your longitude rather than your official time zone:

    date = '2011/%d/12 12:00' % (m)
    adjtime = ephem.date(ephem.date(date) \
                    - float(self.observer.lon) * 12 / math.pi * ephem.hour)
    observer.date = adjtime

Maybe that needs a little explaining. I take the initial time string, like '2011/12/15 12:00', and convert it to an ephem.date. The number of hours I want to adjust is my longitude (in radians) times 12 divided by pi -- that's because if you go pi (180) degrees to the other side of the earth, you'll be 12 hours off. Finally, I have to multiply that by ephem.hour because ... um, because that's the way to add hours in PyEphem and they don't really document the internals of ephem.Date.

Set the observer date to this adjusted time before calculating your analemma, and you get the much more vertical figure you see here. This also explains why the morning and evening analemmas weren't symmetrical in the previous run.

This code is location independent, so now I can run my analemma program on a city name, or specify longitude and latitude.

PyEphem turned out to be a great tool for exploring analemmas. But to really understand analemma shapes, I had more exploring to do. I'll write about that, and post my complete analemma program, in the next article.

Today is the winter solstice -- the official beginning of winter.

The solstice is determined by the Earth's tilt on its axis, not anything to do with the shape of its orbit: the solstice is the point when the poles come closest to pointing toward or away from the sun. To us, standing on Earth, that means the winter solstice is the day when the sun's highest point in the sky is lowest.

You can calculate the exact time of the equinox using the handy Python package PyEphem. Install it with: easy_install pyephem for Python 2, or easy_install ephem for Python 3. Then ask it for the date of the next or previous equinox. You have to give it a starting date, so I'll pick a date in late summer that's nowhere near the solstice:

>>> ephem.next_solstice('2011/8/1')
2011/12/22 05:29:52

That agrees with my RASC Observer's Handbook: Dec 22, 5:30 UTC. (Whew!)

PyEphem gives all times in UTC, so, since I'm in California, I subtract 8 hours to find out that the solstice was actually last night at 9:30. If I'm lazy, I can get PyEphem to do the subtraction for me:

ephem.date(ephem.next_solstice('2011/8/1') - 8./24)
2011/12/21 21:29:52

I used 8./24 because PyEphem's dates are in decimal days, so in order to subtract 8 hours I have to convert that into a fraction of a 24-hour day. The decimal point after the 8 is to get Python to do the division in floating point, otherwise it'll do an integer division and subtract int(8/24) = 0.

The shortest day

The winter solstice also pretty much marks the shortest day of the year. But was the shortest day yesterday, or today? To check that, set up an "observer" at a specific place on Earth, since sunrise and sunset times vary depending on where you are. PyEphem doesn't know about San Jose, so I'll use San Francisco:

>>> import ephem
>>> observer = ephem.city("San Francisco")
>>> sun = ephem.Sun()
>>> for i in range(20,25) :
...   d = '2011/12/%i 20:00' % i
...   print d, (observer.next_setting(sun, d) - observer.previous_rising(sun, d)) * 24
2011/12/20 20:00 9.56007901422
2011/12/21 20:00 9.55920379754
2011/12/22 20:00 9.55932991847
2011/12/23 20:00 9.56045709446
2011/12/24 20:00 9.56258416496

I'm multiplying by 24 to get hours rather than decimal days.

So the shortest day, at least here in the bay area, was actually yesterday, 2011/12/21. Not too surprising, since the solstice wasn't that long after sunset yesterday.

If you look at the actual sunrise and sunset times, you'll find that the latest sunrise and earliest sunset don't correspond to the solstice or the shortest day. But that's all tied up with the equation of time and the analemma ... and I'll cover that in a separate article.

A friend had a fun problem: she had some XML files she needed to import into GNUcash, but the program that produced them left names in all-caps and she wanted them more readable. So she'd have a file like this:

<STMTTRN>
   <TRNTYPE>DEBIT
   <DTPOSTED>20111125000000[-5:EST]
   <TRNAMT>-22.71
   <FITID>****

   <NAME>SOME    COMPANY
   <MEMO>SOME COMPANY    ANY TOWN   CA 11-25-11 330346
</STMTTRN>

and wanted to change the NAME and MEMO lines to read Some Company and Any Town. However, the tags, like <NAME>, all had to remain upper case, and presumably so did strings like DEBIT. How do you change just the NAME and MEMO lines from upper case to title case?

The obvious candidate to do string substitutes is sed. But there are several components to the problem.

Addresses

First, how do you ensure the replacement only happens on lines with NAME and MEMO?

sed lets you specify address ranges for just that purpose. If you say sed 's/xxx/yyy/' sed will change all xxx's to yyy; but if you say sed '/NAME/s/xxx/yyy/' then sed will only do that substitution on lines containing NAME.

But we need this to happen on lines that contain either NAME or MEMO. How do you do that? With \|, like this: sed '/\(NAME\|MEMO\)/s/xxx/yyy/'

Converting to title case

Next, how do you convert upper case to lower case? There's a sed command for that: \L. Run sed 's/.*/\L&/' and type some upper and lower case characters, and they'll all be converted to lower-case.

But here we want title case -- we want most of each word converted to lowercase, but the first letter should stay uppercase. That means we need to detect a word and figure out which is the first letter.

In the strings we're considering, a word is a set of letters A through Z with one of the following characteristics:

1. It's preceded by a space
2. It's preceded by a close-angle-bracket, >

So the pattern /[ >][A-Z]*/ will match anything we consider a word that might need conversion.

But we need to separate the first letter and the rest of the word, so we can treat them separately. sed's \( \) operators will let us do that. The pattern \([ >][A-Z]\) finds the first letter of a word (including the space or > preceding it), and saves that as its first matched pattern, \1. Then \([A-Z]*\) right after it will save the rest of the word as \2.

So, taking our \L case converter, we can convert to title case like this: sed 's/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g

Starting to look long and scary, right? But it's not so bad if you build it up gradually from components. I added a g on the end to tell sed this is a global replace: do the operation on every word it finds in the line, otherwise it will only make the substitution once, on the first word it sees, then quit.

Putting it together

So we know how to seek out specific lines, and how to convert to title case. Put the two together, and you get the final command:

sed '/\(NAME\|MEMO\)/s/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g'

I ran it on the test input, and it worked just fine.

For more information on sed, a good place to start is the sed regular expressions manual.

One of the distros I'm trying on my Dell Latitude 2120 laptop is Arch Linux. I like a lot of things about Arch; I had to stop using it on my previous laptop because something broke in the wi-fi drivers, but I always regretted giving it up. And it seems to work quite well on the Dell, with one exception: the system beep (which other distros don't support at all) was ear-shatteringly loud, far too loud to consider being able to use this laptop in a public space. Clearly that needed to be fixed.

The usual approach to system beeps, unloading or blacklisting the pcspkr module, had no effect. xset b off turned the beep off in X; I could also set its pitch and duration to change it to a nice quiet click, though I wasn't able to change the volume that way. I actually do like having a system beep, as long as it's fairly quiet and won't disturb people nearby. Unfortunately, xset b only affects the bell while in X; it didn't have any effect on the deafening sound Arch gave upon shutdown or reboot.

It turns out that on some laptops, including this Dell, the system beep goes not through the old-style pcspkr driver, but through the normal sound card. And the sound card has a separate channel for the system beep, so even if you have your volume turned down, the beep may still be at 100%. All I needed to do was run alsamixer and find out what the channel was called: "Beep".

Given that, I could use the amixer program to ensure the beep volume will be sane when I log in. I added the following to .zlogin (for zsh; obviously, adjust for your own shell):

amixer -q -c 0 set Beep 5

That gave me a nice quiet beep. If I need to turn it off completely, amixer can do that too: amixer -q -c 0 set Beep mute (curiously, amixer -q -c 0 set Beep 0 doesn't actually set the volume to zero, just sets it very low).

That volume setting applies to the shutdown beep, too, fortunately. Though what I'd really like is to have quiet beeps while I'm running, but no shutdown beep at all. I don't understand the purpose of the shutdown beep; obviously I know when I've told my machine to shut down or reboot, so why do I need an audible reminder? But I've been unable to find anything explaining what's causing this beep. I tried adding a amixer -q -c 0 set Beep mute to /etc/rc.local.shutdown, but it didn't help; apparently the shutdown beep is called before that file is run. Which strongly suggests it is being run by Arch Linux, not by something in the BIOS. But nobody I've asked had any suggestions as to its source, or how to change it. Another enduring Linux mystery ...

Update: I mentioned xset b as a way to adjust beeps inside X -- in addition to turning beeps totally off, you can also set pitch, duration, and sometimes volume though the volume part didn't work for me. But outside X, you can make similar adjustments with setterm, e.g. setterm -bfreq 400 -blength 50. Thanks to Mikachu for the tip!

Need your Ubuntu clock to stay in sync with a dual-boot Windows install? It seems to have changed over the years, and google wasn't finding any pages for me offering suggestions that still worked.

Turns out, in Ubuntu Oneiric Ocelot, it's controlled by the file /etc/default/rcS: set UTC=no.

Apparently it's also possible to get Windows to understand a UTC system clock using a registry tweak.

Ironically, that page, which I found by searching for windows system clock utc, also has the answer for setting Ubuntu to local time. So if I'd searched for Windows in the first place, I wouldn't have had to puzzle out the Ubuntu solution myself. Go figure!