Skip navigation

A long time ago at a Human Factors lab on an Air Force base in Texas, a group of human factors space scientists and Air Force pilots were sitting in the O Club and got to talking about cats and zero gravity. How would a cat orient in micro gravity? Visually? They always land on their feet. But what if they couldn’t feel which way was down?

A few drinks later we realized that one of the pilots wasn’t having a drink because he had to do a proficiency flight later that afternoon. And we already had a camera rigged in the cockpit of a T Bird, and if a couple of us certified this as a human factors experiment it wouldn’t cost the government anything it wasn’t going to spend on the proficiency flight, and it would be an interesting experiment, and — Well, it seemed like a great idea at the time, and the captain who’d be flying thought it would be a good idea.

You can find the rest here. Just search that page for “Here Kitty”.

So I have a Motorola Xoom, and I mostly love the thing. It can handle the majority of my computing needs that don’t involve SSH’ing to a server and using a console–and on a good day, with a bluetooth keyboard, it can handle most of that, too.

It doesn’t have everything, of course. So here’s my quick wishlist for apps which should exist on Android, but don’t.

X11

It needs an X server. Yes, there’s VNC. Yes, there are NX clients. Those aren’t really enough. VNC can’t hold a candle to X11’s integration and performance in most settings. NX would be nice, but I’ve had no luck, so far, getting FreeNX to work. If I ever do, you can expect to see a blog post about it.

For me, I’d prefer both X11-over-TCP and X11-over-SSH, but X11-over-SSH is the most widespread usage I’m aware of.

SSH

Yes, there’s ConnectBot. Yes, there’s IRSSI ConnectBot, which is ConnectBot plus a few modifications for irssi+screen users…but that seems to be about it.

They’re reasonably nice applications…but when your SSH connection drops every time your device’s DHCP lease comes up for renewal (even if it gets the same IP it just had), you can go mad quite quickly. I had DHCP leases at 900s on my network to improve DDNS behavior…I finally bit the bullet and upped it to 86400, because I was tired of what felt like inscessent connection droppages.

They also have a strange problem where your key encodings can get completely fubar’d. I notice this especially when I use a bluetooth keyboard, where I can type quickly enough that I sometimes typo and chord keys. I don’t think it’s the keyboard’s problem; this keyboard works great with the PS3 and with Linux desktops. On Android, under the ConnectBot suite of applications, things like Ctrl-A and Ctrl-D work fine initially, but the Ctrl key eventually stops working chorded. Around the same time, I usually lose things like the entire upper form of the numeric row…all of ~!@#$%^&*( and ) began coming back as ~. Imagine being stuck in a Python REPL, unable to hit Ctrl-D or type exit() to get out. Ouch.

Generic MediaWiki

MediaWiki has a powerful, straightforward and well-documented API, both through index.php and api.php. It seems silly that there hasn’t been a MediaWiki-specific browser, editor and monitor.

Others

There are other apps which don’t yet exist, but should, but that’s a discussion for another time.

I mentioned I wanted to set this up at Casey DuBois’ place, modeled after my own setup, and Jeff Bosch asked me to pass on the instructions, so he could set it up over at The Geek Group. While Casey’s garage, my network and TGG are very different networks, a basic setup is pretty simple, and should work for a broad range of configurations. So here it is, blogged, for the entire GRLUG to play with.

First, I’m assuming you’re running Debian. I don’t think there’s any notable distinction between Debian’s packaging of Squid and, say, RHEL5, and the configuration directives are almost entirely compatible (AFAIK) between squid2 and squid3, so the only necessary significant difference between my setup and one for those packages is likely to be how to retrieve and install the package, and where the configuration files are kept. (Drop a note in the comments if you’ve got an answer for those.)

The first step, obviously, is to install the package itself. There are two Squid packages, squid and squid3. Squid 3 is a ground-up rewrite of Squid in C++. The language it’s written in isn’t really important, though. What’s important is that Squid 3 is where future development and improvements are going, and there’s at least one practical consequence of that: Between the two, the squid3 package is the only one that has support for IPv6. That’s the only practical difference between the two that I’m aware of.

In Debian, if you want to install the newer Squid 3, you run:

apt-get install squid3

If you want the pre-rewrite edition, you run:

apt-get install squid

Next, we need to configure Squid. If you’re using Squid 3 on Debian, the Squid configuration will be found in:

/etc/squid3/squid.conf

If you’re using the previous version of Squid, the configuration goes in:

/etc/squid/squid.conf

Probably the hardest part about setting up Squid is getting the Access Control Lists, or ACLs, right. You need to define your ACLs in the file before you try to say who is allowed access to what. (You don’t, for example, want to allow someone on the public internet to use your Squid installation to connect to arbitrary hosts on your internal network!)

Here’s the first ACL in my squid.conf file:

acl manager proto cache_object

Later, this will allow us to say who can and cannot explicitly manipulate cached objects.

My next ACL defines ‘localhost’ as a source–if it comes from these IP addresses, it’s from the machine Squid is running on. Here, I show both IPv4-only and IPv6-supporting versions. Use only one of them.

acl localhost src 127.0.0.1/32 ::1 # Use this if you’re using squid3.
acl localhost src 127.0.0.1/32 # Use this only if you’re using squid2, not squid3.

Next, we define ‘localhost’ as a destination. Again, two versions are shown.

acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1 # Use this if you’re using squid3.
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 # Use this only if you’re using squid2, not squid3.

Now we come to some details that are particular to your network. I got a little fancy and distinguished between IPv4 and two scopes of IPv6 networks. You’ll want to change ‘localnet4’ and ‘localnet8gl’ to reflect your network’s numbering.

acl localnet4 src 192.168.22.0/24 # our local IPv4 network.
acl localnet8ll src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl localnet8gl src 2001:470:c5b9::/48 # our local IPv6 network.

If you’re running Squid 2, you’ll want to skip the localnet8ll and localnet8gl lines. If you’re running squid3, but you don’t have a public IPv6 prefix, then you’ll want to skip localnet8gl. (BTW, getting public IPv6 space is pretty easy and free.)

Now, we use the ACLs we defined to control who has access to what. These directives are checked in order, and the first match wins.

http_access allow manager localhost
http_access deny manager

First, we allow the local system access to manage Squid, and then we close the ‘manager’ feature from anyone further.

http_access deny to_localhost

Next, we deny any request claiming to access ‘localhost’. This way, nobody should be able to access any services on your machine by asking your Squid service “hey, go grab http://localhost/some_sensitive/web/service”.

http_access allow localnet4
http_access allow localhost
http_access allow localnet8ll # Don’t use unless you’re at least running squid3.
http_access allow localnet8gl # Don’t use unless you’re at least running squid3, *and* you have a piece of the global IPv6 address space.

Next, we allow anyone on our local networks to use the squid proxy server to connect out. Note the comments.

http_access deny all

Wrapping up the access controls, we deny anyone left access to any service we haven’t already covered.

Here’s a quick dump of what’s left in my configuration, and some basic explanation of it:

http_port 8123

Configure Squid to listen on port 8123.

hierarchy_stoplist cgi-bin ?

Recommended Squid default; it’s probably in your existing squid.conf if you didn’t obliterate the whole file.

cache_mem 1024 MB

Configure Squid to use up 1024MB of RAM for caching. This is in addition to any additional overhead it may have. I have Squid running on a router with 2GB of RAM. Excessive, probably, but it works nicely for me.

cache_dir aufs /var/spool/squid3 81920 16 256

Configure Squid to use 80GB of disk for caching, and tell it where to put those cached objects.

maximum_object_size 5120 MB

I’d like Squid to at least be able to cache a full Linux liveDVD ISO image, so I configure it to cache objects up to around 5GB.

cache_store_log /var/log/squid3/store.log

Where Squid puts its log files. (This is a default value for squid3 on Debian. You probably don’t need to change it.)

coredump_dir /var/spool/squid3

If Squid crashes, this is where it will put the dump. (Another default for squid3 on Debian. You get the idea.)

refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320

These control how long and likely Squid will consider a retrieved object ‘fresh’, meaning that it won’t go out and try to grab another copy of the file. These were default values on my installation.

quick_abort_min -1 QB

If I’ve aborted a program performing a download, I’d honestly like the proxy server to continue downloading the file. For me, it usually means I screwed up some parameter, and will be trying again shortly. Having the proxy server continue the download will mean much of the work will be already done by the time I try again. It may also mean that my system might continue downloading a DVD ISO image I’ve decided I don’t need, however, and cause my network to run slower. Your needs may vary from mine, so you may end up wanting to change this parameter.

read_ahead_gap 1 MB

How much data Squid should try to buffer on the client’s behalf. Tune this value if “buffer bloat” is relevant to you.

positive_dns_ttl 30s
negative_dns_ttl 1 second

Values controlling how long Squid should consider a DNS request fresh; this can significantly improve performance, especially if DNS is unreliable. (Though if it is, why aren’t you already running a local recursing DNS server as a cache?)

minimum_expiry_time 600 seconds
chunked_request_body_max_size 4096 KB

Some miscellaneous values tuning behavior of the HTTP protocol. You can probably leave these at their defaults.

With that configuration in place (and after you’ve restarted squid), your proxy server should be up and ready to run. Next up, you need to configure your client machines to use it. On Windows, follow KB135982, and tell non-IE applications to “Use system proxy settings.” On Linux, add these two lines to /etc/environment:

http_proxy=”http://hostname_or_ip_address_to_your_proxy_server:8123/”
ftp_proxy=”http://hostname_or_ip_address_to_your_proxy_server:8123/”

Replacing the obvious. If you used something other than 8123 (and you may have; Squid’s default is 3128, I’m just weird), then you should change the port specification here as well. After making those changes, restart your user session (or even your whole machine). There’s also WPAD, but I’ve never used that; you’re on your own there.

If you’re running Gentoo, you probably have “-march=native” in your CFLAGS, since that flag gives gcc permission to examine the running processor and decide for itself what CPU features are available.

If you have any interest in distcc or cross-compiling, though, you most definitely do not want to use that flag; gcc will be making assumptions based on the processor your programs are compiling on, not on the processor your programs will be executing on.

There’s a fairly neat solution to this. Run this line on the machine you’re compiling for, and it will emit the gcc arguments that -march=native would translate to:

gcc -march=native -E -v – </dev/null 2>&1 | sed -n ‘s/.* -v – //p’

You can take the output of that command, add any additional flags you like (in my case, -O2 and -ggdb), and add it to your CFLAGS on that system. Should you choose to use distcc in the future, the machines in your compile pool will use the appropriate set of architecture-specific optimizations.

Incidentally, variations on this line have been floating around the gentoo-users mailing list since this summer; this is the cleanest form I’ve seen of it, which I pulled from a comment in Stefan G. Weichinger’s make.conf file.

I got my hands on an awesome system recently; it has two E5345 Xeon processors in it, which means it has eight physical Xeon cores. Here’s what my CFLAGS looks like for it:

CFLAGS=”-O2 -pipe -D_FORTIFY_SOURCE=2 -march=core2 -mcx16 -msahf –param l1-cache-size=32 –param l1-cache-line-size=64 –param l2-cache-size=4096 -mtune=core2 -ggdb”

Now, if you’re using Gentoo, you’re probably utilizing parallel make via the -j parameter in MAKEOPTS in /etc/make.conf. To find out what value to pass -j, I go by N*2 or N*2+1, where N is the number of CPU cores. That works well for my Phenom 9650, and it happens that it works well for the Intel E5345 in my setup.

-j only gives you a benefit when the Makefile structure of a program can be parallelized. Most aren’t structured perfectly for this; there points where Make must get one part finished before it can do anything else. At that point, it doesn’t matter how many CPU cores you have; everything’s waiting on one instance of your compiler to finish executing before things may continue.

As it happens, the emerge  command also takes a -j parameter, and it behaves very much like Make’s, except it parallelizes the building of multiple packages, rather than of multiple components within the same package. Also, as it happens, package dependencies result in the same pattern of limitations as you see with -j in Make. Worse, if you have Make dispatching 16 jobs at once, and emerge dispatching 16 jobs at once, you’ll have up to 256 jobs flying at the same time. If you’ve only got eight cores, you’ll lose so much time to CPU context switches bouncing between different processes, you’ll see your builds take longer than if you’d told it to only dispatch one job at a time…

It turns out that Make has another useful option: -l

-l tells Make not to spawn another process if the system load average is above a certain value. Roughly speaking, your system load is calculated as the number of processes executing or waiting to be executed by the system scheduler.

My first “emerge -e world”, with -j16 in MAKEOPTS, took 103 minutes.

With:

MAKEOPTS=”-j16 -l10″

“emerge -e -j8 @world” took 89 minutes.

That’s a respectable improvement!

*thump* *thump*

Is this thing on?

And, uh, serif font? Really? Or is that just the editing interface?

I love my LUG

17:41 < shortcircuit> Today is a real Monday.
17:41 < shortcircuit> In every negative sense I know of.
17:52 < tyrok> Well, at least if it’s an even number of negatives, it still comes out okay.  It’s those days with odd numbers of negatives that are trouble.
17:54 < shortcircuit> It’s more complex than that. 🙂
18:01 < brousch> you should make it ‘metallica monday’ like i did. it makes the day livable
18:02 < tyrok> shortcircuit: Oh, so now you’re taking square roots of your day outcomes?  Really, I’d have thought you could ignore the imaginary part for our purposes.  🙂
18:02 < shortcircuit> Hey, I need to get to the root of my problems, don’t I?
18:03 < tyrok> Not in any real sense.  🙂
18:03 < brousch> the groans per minute in here is getting out of hand

17:52 < shortcircuit> Oh boy, I think I’m in for it.
17:52 < shortcircuit> My sense of humor is just awful today.
17:52 < slestak> and this is different *how*?
17:53 < slestak> 🙂
17:53 < shortcircuit> Well, day got pissed this time.
17:53 < slestak> oops
17:53 < shortcircuit> “awful today” -> “awful to day”
17:54 < slestak> pure offal
17:54 < shortcircuit> As in, being mean to an entity named ‘day’.
17:54 < shortcircuit> It’s that kind of weird 2-3 step mutation that my humor’s settled on.
17:54 < shortcircuit> Hopefully something else comes along and provides evidence of a prior land claim.
17:55 < slestak> could be a centuries long holy war
17:55 < shortcircuit> My humor needs to convert or die.
17:57 < slestak> the Spanglish Inquisition
17:57 < shortcircuit> …I didn’t expect that.

20:30 < brousch> i am a fan of ubuntu server
20:33 < shortcircuit> brousch keeps it cool.
20:34 < tyrok_laptop> Aargh.
20:36 < brousch> groan
20:36 < brousch> it took me 3 minutes to get that one