Col

Small data hack: bin day calendar

Wed, 28 Dec 2016 15:34:51 GMT

I'm very lazy. Rather than having to keep track of Cambridge bin collection days manually, especially around holidays, I wrote a thing to convert it into an iCalendar file for me so that I could import it into Google Calendar. Here it is in case it's useful to anyone else:

#! /usr/bin/python3

from argparse import ArgumentParser
from datetime import datetime
import os.path
import re

from bs4 import BeautifulSoup
import dateutil.parser
from icalendar import (
    Calendar,
    Event,
    )
import requests


parser = ArgumentParser(
    description="Generate iCalendar file for Cambridge bin collection days.")
parser.add_argument(
    "id",
    help=(
        "Unique identifier for this calendar, normally your host name.  Make "
        "sure that this does not collide with any other calendars."))
parser.add_argument("address", help="Your street address in Cambridge.")
parser.add_argument("postcode", help="Your postcode.")
args = parser.parse_args()

now = datetime.now()
req = requests.get(
    "http://bins.cambridge.gov.uk/bins.php",
    params={"address": args.address, "postcode": args.postcode})
soup = BeautifulSoup(req.text)
cal = Calendar()
cal.add("prodid", "-//riva.pelham.vpn.ucam.org//bin-days//EN")
cal.add("version", "2.0")
cal.add("calscale", "GREGORIAN")
cal.add("x-wr-calname", "Bin days")
cal.add("x-wr-timezone", "Europe/London")
for div in soup.find_all("div", style=re.compile(r"^text-align:center")):
    desc = div.contents[0]
    when = dateutil.parser.parse(div.b.get_text(" ").rstrip("*"))
    while when < now:
        when = when.replace(year=when.year + 1)
    event = Event()
    event.add("uid", "bin-days/{:%Y%m%d}@{}".format(when, args.id))
    event.add("dtstart", when.date())
    event.add("summary", desc.capitalize())
    event.add("transp", "TRANSPARENT")
    cal.add_component(event)
with open(os.path.expanduser("~/public_html/bin-days.ics"), "wb") as out:
    out.write(cal.to_ical())

On Debian, this requires the python3-bs4, python3-dateutil, python3-icalendar, and python3-requests packages. You'll probably want to change the output path to somewhere that your calendar software can see (so if it's a web service such as Google Calendar then it needs to be something that corresponds to an accessible URL). The web-scraping is pretty gross, but it's the best I can do given the council's published data. Ideally this would itself be a web service that could generate calendars on demand for a given address and postcode, but like I say I'm lazy.

comments

December days: Programming language improvements

Fri, 26 Dec 2014 16:21:56 GMT

As a follow-up to my post about my personal history of programming languages, cartesiandaemon asked me to expand on "which language(s) do you use most now (C and Python?) and what improvements would you like to see in them?"

Mostly I do indeed use C and Python these days. I'm still comfortable enough in Perl and often reach for it particularly for shorter text-processing kinds of jobs (some of which grow a bit more than I'd intended), but I really haven't kept up with work on the core language much since about Perl 5.10, so I'm not desperately qualified to talk about things I'd like to see improved. The trends I've seen there seem to be towards making it easier and more natural for people to use modern and safer techniques rather than perl4isms, things like Modern::Perl and Moose, and these seem like good things but I don't have a lot of experience to share there.

C

C is in many ways an awful language for the sorts of things it's often used for: manual memory management, no bounds checking, a standard library with a host of peculiar warts, support for closures that could be described as poor if you were feeling generous, and so on. (C++ fixes some of these at the cost of truly insane programmer-facing complexity.) Huge numbers of security vulnerabilities can be ascribed to defects in the design of C. On the other hand, it underpins so many other things, especially on Unix, that it isn't really possible or reasonable to avoid entirely, and it's pretty much the greatest common denominator for library interfaces that want to be usable from more than one language environment. (You can sometimes make C++ libraries usable from C, but you have to be pretty careful.)

GLib (and its object system add-on, GObject) is pretty pervasive these days, including in various things that otherwise have nothing to do with GNOME, and it's not at all a bad general supplemental library: better-designed and more comprehensive than much of the C standard library, and, sure, it may be a megabyte or so but you almost certainly had it installed anyway. I think for a new C project these days I would probably turn to it, unless it were particularly small or needed to function in particularly minimal environments. Its main flaw is verbosity, particularly where callbacks are involved. For this I think it's possible to do an excellent job with domain-specific languages that provide a thin wrapper over C. I converted performance-critical parts of a project at work recently from Python to Vala recently, and was very impressed: it made it actively enjoyable to program using GObject, which wasn't something I could really say before, and it was still possible to inspect and roughly understand the generated C code. GObject also provides very close to automatic binding generation for various other languages by way of gobject-introspection, which is extremely powerful for being able to use the right tool for individual jobs without having to commit to it for your whole project. I think that this kind of thing is probably the right path for many C projects.

For things that need to stick with plain C, it's certainly possible to incrementally evolve the facilities they're using to make things safer and easier. valgrind made quite a few waves when it was introduced a little over ten years ago: in C you often find that your memory management mistakes are only reported as a crash in some entirely different part of your program much later, and this is excruciatingly difficult to debug directly, so valgrind keeps track of absolutely everything you do with memory and tells you if it looks invalid. Compiler developers have been doing all kinds of interesting things recently such as AddressSanitizer and Undefined Behaviour Sanitizer which promise to make it easier to spot problems early, and there are all sorts of proactive hardening techniques one can use to stop bugs escaping into the rest of your system as exploits.

Library-wise, there's always more to be done. I made my own small contribution to this with libpipeline, which I'd like to see used by more C projects that invoke other programs, since it's very easy to get this kind of thing wrong.

Python

The Python world spent a long time collecting all the things that were hard to improve in the core language without breaking compatibility in some way, and batched up a lot of these into Python 3. Unfortunately the transition to Python 3 has been an extremely painful and protracted one. Despite considerable work on migration strategies such as 2to3, the changes were substantial enough that it's taken quite some time for many projects to be ported, and you can only start using Python 3 once all your dependencies have been ported and are available anywhere you might want your code to run. (In particular, the Unicode string changes are I think a significant net improvement - they're not without problems, but Python 2 was even worse for anything that might care about internationalisation - but porting to them requires going through your program and for each string-like variable in it determining whether its essential nature is to contain binary data or text. Easy enough to get right from scratch, but painful to retrofit.) The right answer for many projects is to write "bilingual" code for a while that works in both Python 2 and Python 3, which is largely possible with libraries like six to help, but this is a bit of extra cognitive load on programmers and of course not everyone cares. So I think if I got one wish here it would be for all the remaining stragglers to be magically ported to Python 3 so that we could stop caring about the old stuff and simplify things, but that's not likely to happen any time soon.

I guess the main general improvement I can think of for Python, short of "please be faster" and various standard library warts, would be some form of partial static typing so that I don't have to rely quite so completely on test suites to defend me against my own foolishness. I hear that something like this may be underway for Python 3.5, which will be interesting.

This post is part of my December days series. Please prompt me!

comments

December days: Teaching children programming

Sun, 21 Dec 2014 16:49:27 GMT

As a follow-up to my post about programming languages, liv asked me: "can you talk about whether you're planning to teach your children programming, and if so how?"

I'd very much like to teach my children programming, yes, for a variety of reasons. One is that it's a thing I'm personally enthusiastic about that I want to share with them. Another is that it's an increasingly useful secondary skill in all kinds of other academic disciplines, whether that's for data analysis or driving complex machinery or whatever, and I'd like them to have that be accessible to them if at all possible. And of course I think it's a worthwhile skill in its own right, as computers become more and more a part of everyday life.

On the other hand, I don't want to teach them just single bespoke skills, such as just one programming language: what I really want to impart is the mental discipline of ordering your thoughts in order to instruct a computer accurately in how to do them, which I think is an aptitude that transfers itself well to all kinds of other things, even though doing that clearly involves learning the nuts and bolts of programming (preferably in more than one languages) and especially for children it needs to involve having fun along the way. There's no point trying to teach programming to children if they find it boring, or if it's too early in their development. (I tried to teach B how to program some years ago, but honestly I hadn't prepared well enough, it fell rather flat, and by the time we revisited it he wasn't really interested, so I definitely want to prepare better this time round.)

ghoti has been planning to start with a plan she'd previously started on while TAing at primary level, namely to start with Scratch (hmm, appropriately the top featured project there is currently a dreidel game) and move on to Rhodri James's Python course. I've generally been of the opinion that it will work better if we wait until J's reading is a fair bit more fluent, and so to be honest I hadn't yet thought much about the details yet; Scratch is more visual than a lot of languages but it still has a very significant textual component.

I think this is still an area where I very much don't think I have the answers and am listening for suggestions. My criteria are that I want them to be able to progress quickly to doing things that will interest them, I don't want them to get bogged down in syntactic vinegar, but I also want them to be using (if not necessarily as the very first step) a language that isn't a toy and that they can write real non-trivial programs in, and preferably one that won't get them stuck in particularly bad habits. Python seems like a pretty good thing to aim for with the support of some decent code libraries and teaching materials, so ghoti's plan generally seems sound here, but I sort of feel the need to work through it ourselves first to make sure we aren't caught by surprise along the way.

Does anyone else reading this have experience with teaching children (other than themselves!) to program? I'd be interested in hearing about what you did.

This post is part of my December days series. Please prompt me!

comments

December days: Programming languages

Sat, 13 Dec 2014 16:12:53 GMT

ewx prompted me to write about programming languages, and said "be as specific or general as feels appropriate".

Answering prompts with a personal-history kind of framing seems to be working quite well this month, so I'll stick with that for this one. I started programming when I was seven; that was on a ZX Spectrum+ which was a lovely piece of hardware for the time, so the language available was Sinclair BASIC. This wasn't a bad BASIC dialect, and I remember its PRINT command having some useful gadgets. The limited memory and the very odd encoding (numbers were always five bytes; reserved tokens were always one byte) meant that it was sometimes worthwhile to go through some ridiculous circumlocutions which have stuck in my head: INT PI was two bytes in memory against five bytes for 3 (ditto SGN PI for 1 and NOT PI for 0). At any rate this kept me going until I was 11, at which point we got an 8086-based PC.

There was a dark stage for about a year here where I didn't have any programming language worth mentioning, because MS-DOS 3.3 (I think it was) didn't ship with one. Batch files were sort of barely functional for a few things, and if you were really determined it was possible to do things like write a minimal calculator in them, but it wasn't exactly pleasant. Out of a mix of desperation and curiosity I resorted to teaching myself assembly language via DEBUG.COM; I haven't used that experience much since, but it was probably worth having done at least for a while.

When I was about 12 we upgraded to MS-DOS 5.0, which shipped with QBasic, so I had a more or less familiar language available again with slightly different quirks. Not having to number lines manually was a bit of a revelation, and I seem to remember subroutines being noticeably better. Some things were more annoying, and I seem to remember putting in some effort to add some of the facilities that the Spectrum's PRINT command had. I stuck with this until I was about 16 or so, including going to the All-Ireland Schools' Programming Competition (which I think is now the All-Ireland Programming Olympiad); I placed respectably enough but not particularly close to the medals, and reckoned that some of that was due to trying to use BASIC, which was just a bit too clunky for the job.

That summer I went on a residential course in Dublin where I learned Pascal, and that made a big difference, though part of that was probably just due to the mental flexibility involved in having learned a second language. The next year I went back to the same competition and felt I'd done much better. I sat through the awards ceremony, of course presented in reverse order, figured I'd just missed out on the medals and might try again next year, and almost fell over when they announced I'd come first. This of course did a lot for my confidence and meant I got to go to the international competition, which was mainly useful in that it was the first time I'd really hung out with other serious programmers.

I learned C at some point before going to university. If memory serves it was initially a dialect called Pacific C, although I haven't retained any of its idiosyncrasies. I think I then used Visual C++ for a bit, until I switched to Linux-based systems in 1998ish and taught myself GNU C, which even at the time was much closer to the standard. (It wasn't until a year or two later that I got hold of a copy of K&R II, at which point things started making a lot more sense than whatever I'd picked up through osmosis.)

At university they taught us ML (on the reasonable basis that virtually nobody knew it already so it levelled the playing field) and Java. ML was excellent at broadening the mind but I never kept it up. I used Java for a while, but eventually I decided that it just felt too clunky for me and I was running into its limits a lot in terms of what systems programming was reasonable at the time, and I've avoided it since.

Once I was using Linux, the scripting facilities were a lot more powerful and it often made sense to use them for things where I'd previously have used full-fledged programming languages. I used a lot of bash, sed, awk, and such around this point. In 1999 or so I taught myself Perl, and that was another big step up in terms of how quickly I could get things done. Perl is often criticised for being a write-only language, and I admit that I've occasionally contributed to this, but I think my normal style is rather more readable and it's mostly fine if you refrain from playing silly buggers too much. It remained my language of choice for some years, including lots of work on the Debian bug tracking system which is written in Perl.

When I started working for Canonical, Python was the preferred language there, so I taught myself that. That was just as well since I soon found myself being handed stuff like Germinate to maintain (nowadays it looks more like this, which is a bit more tractable). Nowadays I'm very comfortable in Python, either 2.x or 3.x, and my new job will involve even more of it. It's a reasonably nice general-purpose language with generally excellent library facilities; it's maybe not the fastest thing in the world with the standard implementation but it's not terrible, and I feel pretty comfortable with its quirks.

Apart from the time spent learning ML at university, all of these fall into the general category of procedural/imperative languages, and I have to admit I really haven't done much outside that sphere. A number of people I respect are big fans of functional languages, and I suspect that at some point I will end up teaching myself Haskell. I've done some work related to that before in a rather strange way, but I never really taught myself more than a minimal amount of the language as a result. (On the other hand, it did cause some local Haskell developers to buy me drinks, so that was a win.) Once I have more energy for free-time experimentation again then this might be reasonably high up my list.

Virtually all of this has been self-taught. I did have some formal programming education (A-level computing, which in this area at least mostly covered ground I already knew, and university), but it's mostly been driven by what I needed at any given point rather than learning new languages for the sake of it. That said, if you're learning to program I strongly recommend learning more than one language once you're over the initial hump, just so that you have a better idea of what's intrinsic to the field in general and what's just a quirk of the language you learned first.

This post is part of my December days series. Please prompt me!

comments