It was a rough holiday week around here. The Payphone Project and all my other web sites got knocked offline, some for a couple of days others for a couple of weeks. The important things were back in order quickly enough but a full restore turned out to be a lot of unexpected heavy lifting for me. Slinging 40- and 50-gigabyte web sites around takes time and patience under most circumstances, but as an individual with no humans to back me up and only foggy memories of how I got many technical things done before the outage I had a lot to stare at and to try and remember. I wandered through thousands of lines of code I either wrote from scratch or took from others and manhandled so it would behave as I wanted — and not “As Specified.” (There’s a programming term to hate.)
I am not a virtuoso programmer or sysadmin, but I do well enough that I’ve maintained my web sites on dedicated servers almost entirely on my own for about 15 of the 21+ years I’ve been doing this. I had one other blowout like this several years ago. I traced that one back to China, and ultimately blame it on the folks at the data center where the server was hosted. They had, unknown to me, originally set up the server with an ancient version of phpMyAdmin, a standard MySQL admin tool with a notorious history of security vulnerabilities. I was ignorant that that software was just sitting there on the open web, waiting to be discovered. It was over a year later that I figured out, mostly by chance, what had happened. That server outage cost me dearly. Not a day went by for years afterwards that I did not find myself looking for some piece of code, some file, some piece of e-mail lost to that incident.
This time, not so bad. Even if it took a few extra days of rummaging around I appear to have lost nothing except some holiday cheer.
I like to think I am more of a content and creative sort. But the longer I do this web stuff the more I must concede that the technical guts lingering beneath the content demands constant attention, and in my world these demands routinely gets in the way of creating.
An upshot of this server blowout is that I recognize my need to simplify things, and cut or repurpose some content running on obsolete software. I bid adieu to the beloved Menalto Gallery2 software, among my favorite web packages ever. As soon as I can get it converted I will show my collection of “Etude Magazine” covers either using the actively maintained Piwigo, or I might just stuff it all into good old flat HTML pages and free myself from future obsolescence.
I finally figured it out but for days after this blowout I could not fathom why I was seeing a “SECURITY VIOLATION” when I tried to access Gallery2 content at its usual URLs. As is often the case, problems like this which I confront are uniquely blamed on some crazy fiddling I did with the software. No amount of “Googling on” will find a solution until there is a way to Google my freakin’ brain. I understand they are working on that, by the way.
In this case it came back to how I integrated Gallery2 with the Moveable Type content management system. I don’t remember seeing anyone else even attempt such an integration. While my success in it was not comprehensive (it didn’t need to be) I think it worked as well as I had in mind. Yet the intertwining of Gallery2 and Movable Type came with all kinds of little bitty command line tweaks and symlinks that I forgot about in the 3 or 4 years since last working with it.
Another favorite website package I must abandon is the great Swish-e package of search scripts, which indexed and made searchable local filesystem and flat HTML web pages in a way I found amazingly powerful and (even more importantly) flexible. I integrated Swish-e, too, into Moveable Type, in ways I can only reminisce about now. The swish-indexes which I had backed up are not portable across servers, and even if they were I cannot find a working RPM or other distribution that will let me install Swish-e on top of 64-bit Centos7. The web site for Swish-e has gone dark, which would seem to indicate that the product is finally dead. Swish++, is a no-go for now, too, as I can’t get that ancient script so install either. Other options exist but I may or may not sort through them later. If the mantra is to simplify then building and maintaining custom search indexes doesn’t fit the bill. And as far as I can tell none of the options compare to Swish-e for ease of use. I had Solr up and running in no time but gave up on parsing the JSON and XML outputs.I just want my Swish-e back! J For now I have resorted to the poor man’s Big G Site Search, which utterly sucks on coverage and configurability compared to Swish-e, but it’s better than nothing. If/when I can get Swish-e to come alive I would consider going back to that in a hot second, except that it is abandoned ware and thus probably best left to my fond memories.
A major revelation in all this was the forced move from DirectAdmin to cPanel. These are two of the web-based control panels that let someone like me easily do things like add web sites, e-mail accounts, subdomains, etc. Manually editing config files and system resources is something I can but why spend the time when, again, simplicity is the goal. And holy crap, how was I futzing with the prickly and puzzling DirectAdmin all these years? cPanel is head and shoulders better than DirectAdmin. I might not know this except that DA does not, apparently, work on Centos7, at least not at my particular data center. So I am not complaining about being “forced” over to cPanel.
My only gripe so far is that through cPanel one cannot easily edit virtualhosts within the web interface, as you can through DirectAdmin. The cPanel documentation explains, seemingly clearly, that Apache container files must be edited on the web server itself. From there the cPanel documentation gives the wrong location for those include files, so I got to enjoy spending some holiday cheer figuring out where they really were. Just a small annoyance but it was just those kind of inaccurate instructions in one bit of documentation after another that made the holiday week recovery so tedious.
And on and on… So many lines of code, so much minutæ buried in one configuration file after another, so much rapid-fire industry prestidigitationally spilled into a computer keyboard never to be seen by any but these two eyes, and that’s if I’m even paying attention. No one has ever seen my crontab, my Alpine e-mail configurations, my ugly-to-look-at-but-truly-beautiful mod_rewrite rules, my intentionally obfuscated offsite FTP backup regimen, my .htaccess files (one is 1720 lines long but I expect to trim it significantly)… how much code or how much of this kind of work remains unseen to all but its executor?
That’s a rhetorical question. Don’t let it get to you.
For the cleanup I’ve done things are still kind of a mess around here though it should not be as obvious now as a few days ago, when traffic was at a lull anyway. One of the big web projects I had to let go was payphonepictures.com. All its content and more is to be found at the new Payphone Pictures right here at the original WWW.Payphone-Project.COM. I only kept payphonepictures.com around because I had plugged in so many hard-coded links to images located directly on that server. I did that partly out of laziness but also thinking it created an efficiency of disk space. That last point doesn’t really matter now, with storage so cheap and plentiful.
Another bit of cleanup I’d been meaning to do for a long time was to move the news.payphone-project.com stuff back home to www.payphone-project.com. I really hated that news.pay… URL. I set it up in the first place to get some mileage out of the sometimes painful hours spent setting up a WordPress multisite installation. That process both relied on and was stymied by an astoundingly incoherent body of documentation I can only describe as chaotic. Dots had to be connected among comment boards, official documentation, lone blogger-coders who found solutions no one else seemed to have, etc. All this not to mention finding countless solutions on my own that seemed not to exist out there in the Intertubes.
In the end the effort put into setting up the multisite proved to be worthwhile but keeping the Payphone there made no sense.
If this site ever made money I would love to hire a webmaster, and I would love even more to pay a designer what they deserve to make this site look like something more in the spirit of what it is. And it would be cool to hire contributing reporters, on an ad hoc basis even. But I would not expect anyone to do that sort of stuff for free, even though some have done so in the past.
As I sorted through some of the unexpected carnage of this web server blowout I found myself re-reading some stories I’ve posted here. I rarely re-read anything I write, except maybe years later, when and only if it somehow feels safe to do so. In cleaning house after this outage I also re-read some of my semi-private blog which I started in late 2002, and to which I posted almost daily ever since. In most of those ramblings I remember writing the words, and I remember what my station in life was at the time I wrote them. But in a surprising number of them I do not remember or recognize anything about the accounts, the stories, or even the time spent writing them down. It’s kind of eerie.
The same could be said of the stories posted here. My reactions vary from “That was kinda cool” to “OK, sometimes I know what I’m talking about” to “That was pretty stupid.” Here and there my reaction is “Who the hell wrote this?” I just do not recognize my own voice sometimes.
That semi-private blog remains semi-private mostly because of its confessional nature, which is intentionally reminiscent of the quiet, early days of the WWW, when I and others felt safe pouring our little hearts out to a captive worldwide audience of potentially billions!
Today such an outlet is intended solely for the people who actually care about or are interested enough in me to do the work of seeking it out. Everything else remains “public”. I put the word in quotes to cite none other than myself. Who else is going to do it? In the summer of 2004, a playwright asked me for one word that I thought would describe the coming century. I said “Public. This will be the Public Century.”
I stand by that prediction which I think, if I am allowed to say so myself, is coming true.
As anyone who knows me in person will confirm, my attitudes toward the subject matter of payphones and public telephony has vacillated wildly over the years. Today I am more than fine with it. It is an enduringly robust interest of mine which represents a harmless if sometimes amusing preoccupation. There have been times when the subject seems positively moribund, but then I am lifted from such thoughts by contacts from interesting private investigators, journalists, and everyday individuals with thought-provoking discussions about public telephones.
But there have been and will likely continue to be periods where the very word “payphone” echoes in my mind like the sound of a dog that will not stop barking.
On that note: Happy New Year from The Payphone Project! Enjoy my newly formatted and ongoing series of pictures: The Payphone Project: Where People Still Use Payphones.
