Written by Joshua Paine, known on the #helma irc channel as jsp and midnightmonster. Known most other places as Joshua Paine. Not so big on aliases.
One way to make Cross-Site Request Forgery pretty impossible and significantly cut down on form spam is to generate a one-time use token when a form is retrieved by GET and then check for it on POST and don’t do the action unless the token is valid. Last night I spent a little time (half an hour? an hour?) writing some app-wide functionality to do both those parts automatically.
In probably 11 working hours over the last two days I’ve gotten to where 84 of the sermons are in the database and all the data structures connect and load and save correctly. The big exceptions are:
Sunday I know I didn’t work at all. Saturday, very little. Thursday and Friday? Eeek. I don’t know. I actually haven’t a clue what I did on those days. I’ve looked briefly at the SVN logs, and I know I checked in some things, but it’s a total blur. The eleven hours for the last two days is very much a guess, and it brings the total up to 23 hours, but obviously the accuracy of this number is way, way down. Between times of great distraction and times of great concentration, I haven’t kept up with this log.
I’ve been busy on other things and haven’t spent much concentrated time on the site code. Unfortunately I have done bits here and there, but it’s hard to say how much time that amounted to. It certainly wasn’t time spent with full concentration. I’d ballpark a couple hours. CD ripping has also occupied somewhere from five to seven hours. I’ve put all the information I have about each into the MP3s’ metadata, with some made-up conventions of my own, so entering the CDs into the database won’t actually involve any typing. I’ll still need to create an editing form, though, for future use when I just want to update the recording of an existing sermon or add a few at a time. I think the form will need to allow me to either specify metadata explicitly or leave it off and pull the information from the MP3. That way the forms work quickly for me but they still work as expect if anyone else is uploading.
The one puzzle I still haven’t figured is how exactly to determine and store the schedule of sermons. I need some way to store date calculations for an arbitrary number of holidays. I think perhaps the way to do it is set the schedule for a year at once. Just randomize the order of the semon list and start adding them to the schedule. Let the ones with specified dates bump any others as they are placed.
Nope, that doesn’t quite meet the requirements. Fortunately I was able to explain the problem out loud to my wife, and in so doing figured out the solution. I don’t want to spend the time right now to write it here.
One thing I didn’t specify in my introductory message is that I was sort-of bracketing the time spent on ripping these discs. It wasn’t meant to count toward the twenty if it took more than two.
I think I’ve spent another hour today pondering, so let’s say it was 1.5 hours over the last couple days and an hour today for a total of twelve project hours so far.
Oops. Turns out there’s a whole set of user management functions attached to the Application object, only you can’t find them unless you know to look there, because the information on User only links to the relevant Session methods. So I have to redo my whole user setup. I’m going to see if I can do it in 30 minutes.
Afterwards. It actually only took me twenty minutes to retrofit my user system, and then I spent another ten adding nicities like messages announcing that your user has been created or your changes saved. Helma lets you set a response.message property which survives an HTTP redirect. I really have no idea how they do it behind the scenes, but it makes such this-just-happened messages extremely convenient to do.
Total project time is now 9.5 hours.
On reddit there’s been kerfluffle lately about how you should never warn when you mean undo. E.g., no big scary “Do you REALLY want to delete this?” messages when the better behavior would be just to let the user undo the action if they change their mind. It’s good advice, and I’ve read some of it before, but not always convenient to implement. Well, I’ve been thinking about a way to solve undeleteing generically for Helma objects in a way that would also be extensible for objects that require special behavior. And of course I would have to implement the extensions for the built-in extended HopObjects, particularly Users. I think I will have to resist implementing this for now when it’s not really essential for this project and I’m already way behind.
It took another hour and fifteen minutes—much of which was finding that session.login() doesn’t work as advertised, trying to make it work, and finally just working around it—but I can login as my newly created user and access to the user management is controlled. The access control, admittedly simple, turns out to take just one line, which is pretty cool. I can’t delete or edit users, but edit should be trivial and delete not difficult.
Project time is up to 9 hours. Like I said, we’re not going to make this in under 20, but I’m still glad I’m going the Helma route.
I spent a while today waiting around for a new working Rhino build (and trying older ones). I could have spent the time more productively, probably, by working on another project, but when there’s still so much I’m learning it’s hard to switch gears. Then I spent a while in IRC explaining E4X, which I certainly enjoyed but which didn’t get any useful code written. Finally, I spent probably close to three hours today trying to get user creation working. I’m still in the process of developing my own conventions and best practices for common operations, so even simple things take me a while, so I don’t think it’s too early to conclude that convential project management wisdom wins: the trials and mental training involved in switching to a new, more productive platform definitely slow you down initially far more than the better platform speeds you up.
Next to figure out is logging-in users and limiting access to the user list.
Project time so far is 7.75 hours. It looks extremely improbable that I’ll make my estimate, but so far I don’t plan on charging the client for the overage, at least not all of it—we’ll see how the total ends up. It’s a great learning experience so far, and I don’t regret the choice to do this project in Helma: I’m even more convinced than before that it’s going to be my platform of choice going forward.
Chris Zumbrunn was kind enough to build an updated copy of rhino.jar for me that fixes the E4X bug. All is sweetness and light again, and I need to get back to work on users.
Helma uses the latest Rhino from CVS. For the last month and a half, Rhino has had a serious bug with E4X that was not fixed until about 30 hours ago. The Helma user community is small. Of that community, hardly anyone uses E4X yet (though I’ve been evangelizing). No one noticed the problem before, but it has completely stopped my work. I had perhaps an hour to an hour and a half of productive work, then about the same trying to understand and workaround this bug before finding the Rhino bugzilla entry for it and posting to the Helma mailing list assking for help. Now I just have to site back and wait. Project time is at 4.75 hours; time lost to issues with the new framework is about 6 hours. Looks like I got involved in a land war in Asia.
I’ve spent more than an hour on the Helma IRC channel talking about documentation (still weak in places but getting better) and gushing about E4X (still glorious). I think I have a convert, but that doesn’t pay any bills.
For the user form and others later, I’m making a function to output my preferred form field markup style without the bother of actually writing all of it.
Helma provides a templating system. I don’t like it. Someone wrote an extended version that would be an incremental improvement (for the things I care about) over PHPTAL which I was using with PHP. That’s not good enough. I’m tired of typing so daggum much. I’m tired of forcing all my data through the little hole between my logic and my template. The appealing thing about templates usually construed is that the presentation will be not only separate from but loosely coupled with the business logic. That turns out to be fantasy. You end up writing volumes of special-purpose code in the ‘business’ part just to setup variables that the presentation part can use without embedding too much logic in that layer, but then you still have to write tons of template code to use all the variables correctly.
I’m taking a new (to me, anyway) approach. The various ways an object may get rendered to HTML are now methods of that object. Who was I kidding? There is no separate Designer who works on the templates while I do the backend code. When the intial HTML is handled by someone else, I still have to rework it myself to mash it into my style and the template system. And in many places even the template logic gets so complicated that no non-coder could find their way anyhow. And if I ever were working with a Designer on an ongoing basis, they’re going to know their stuff well enough that if I give them a semantically marked-up page with plenty of hooks to hang CSS on, they can make it look however they want. In Helma’s defense, HTML rendering as a method of the object rendered is the view that Helma fundamentally takes, but Helma seems to think that the best way to do that is to have a separate template file for each method. This may have been a good idea when Helma got started. Certainly the technology I’m using didn’t exist then.
So I started to write my own set of JavaScript functions to build up pages loosely based on HyperScript and similar works. It was gonna be pretty sophisticated. I even had a pretty clever and novel (AFAIK, which is not very) solution to the whole namespace issue that afflicts all similar DOM builders. I sent about 2.5 hours down that hole. Then I remembered: Rhino (and hence, Helma) supports E4X, which makes XML a first-class language construct in JavaScript. As soon as I remembered, I knew that would be the faster-running and natural way to build HTML documents. But I’d never actually used it or paid much attention before, because it’s not available in non-Mozilla browsers. So I started using it. Oh. My. E4X is so easy, so natural and so powerful, if once you use it you will ever after regard any other method of dealing with XML, including just writing XML, as [almost] unbearably awkward, ugly, counter-intuitive, and verbose. E4X is a competitive advantage. JavaScript is the most popular programming language in the world because of the accident that it happened to be built into every web browser. It has long been a surprisingly powerful language, but now if you work with and generate XML or XML-like documents, you will find JavaScript not merely a ubiquitous, but in fact a superior language. (Although you actually have to trade in browser ubiquity to get this kind of superiority. Come join us on the server-side. Or in Flash. ActionScript has E4X, too.)
I’m gushing. The bottom line is I wasted 2.5 hours. They definitely count against me in my battle of wits with conventional wisdom, but I couldn’t reasonably charge the client for them. I also spent 1.5 hours figuring out and arranging things I’m actually using, which brings project time up to 3.5 hours.
Helma has a built-in User prototype. It’s not Brilliant, but it should do for the very minimal purpose here, which is just to allow a few administrators to upload sermons and a few other things. So I’ll build a form to create a user, create myself, protect the form so that only admins can use it, and then create the other users.
I ran into a problem. In Helma, urls map to objects. So / maps to the Root object, and /faq might be the faq ‘action’ of the Root object. These url-mapped objects can also have a collection of children all the same type, so in an addressbook application /joe_smith might map to the Person object which is part of Root’s children collection and named ‘joe_smith’. But what if there are several features of the addressbook application and we want to access people by /people/joe_smith instead? Assume there’s a People object to which you would like to delegate this responsibility. It turns out to be really easy, but I didn’t find it in the documentation. It’s “people = mountpoint(People)” in Root’s type.properties file. But I’ve now spent another hour just getting to where the Root and the users object can be found and say hello.
Turned out to be the inappropriate Java runtime that came with my server’s Fedora Core 6. The documentation tells you that java-gcj-compat works as a drop-in replacement for the Sun JRE. What they don’t tell you is you’ll have time to write your own JRE before execution finishes. Problem completely solved by trashing java-gcj-compat and installing Sun’s latest JDK (which includes the JRE). On my production server the anagrams script now runs in 1.2 seconds.
Just for interest I thought I’d try out my anagrams script on my server. On my workstation it now runs in as little as 1.5 sec, thanks to various helps from the Helma users list, which makes it only about 60% slower than PHP for the same job (down from more than 200% slower). On my server, it takes 59 seconds (!!!), except when apache times out waiting for Helma, which seems to happen at 60 seconds. Obviously I’m not building a production web application to solve anagrams, but 40 times slower than my dev machine is unacceptable for anything. I need to figure out why there’s this huge speed issue. I’m not counting these experiments against the time for this project, but it’s definitely real, potentially productive time I’m losing.
Basic Setup: 1 hour It took me about an hour to get Apache and Helma configured correctly on both the server and my dev machine. There wasn’t anything hard about it that I hadn’t already figured out, I just had to make a bunch of decisions about how I wanted to lay things out since this is the first time I’ve really tried to run a Helma app on the production server or done more than mess around on my development server. And there are still further configuration changes I’ll need to make later to get static files (stylesheets, images, etc.) working, but I won’t need those for a bit since I’m building all the functionality before I worry about the appearance at all. Design tends to take me as much time as I give it, so by doing it last-ish, I’ll be giving the designing a hard deadline. So it’s disappointing that it took so long, but it’s time to move on. And if you want you can see Hello World.
We’ve had some discussion recently about project time estimating and controlling client change requests and how long things take. This page is a contribution to that discussion and hopefully also a help to me as a record for later and something to keep me on pace for now. I’m going to follow my own progress on a project I’m beginning from scratch today and record what I’m working on and how long it takes for the duration. Hopefully I’ll get in some notes on setbacks and discoveries as well.
I’m building a website to archive the recorded sermons of a Baptist preacher and offer them freely to the web. I’m not sure how many there are—more than 100. The sermons should be browseable by scripture text and topic. The site will need to allow an administrator to assign particular dates to particular sermons. The site should automatically arrange the sermons into a weekly podcast based on the specified dates and fill in the blanks with a randomly selected sermon where necessary. Visitors need to be able to send a link to a sermon to a friend from the site. There will also be a few static informational pages, a contact form, and some kind of moderated guestbook. All the interactive features need some anti-spam measures, especially the “send to a friend” feature which otherwis could be abused to send junk mail to anyone. Obviously we’ll need some administrative tool to upload and set the properties of the sermons.
One unusual requirement is the time scale. The client plans to endow a fund to keep it running even after his death. So the core functions should keep working with no human intervention indefinitely.
I’ve estimated 20 hours to build this site. I’ve worked with this client before, and I think he’s likely to trust me to make the best decisions about the site functions, user experience and overall design, but he’ll probably have more particular concerns about the details of the static pages. I’m leaving out three hours to handle building the static pages and communication around them. I was originally thinking only two, but with the necessary communication that’s probably unrealistic. So I’ve slipped an hour already, and I haven’t even started yet. That only leaves me one hour of fudge space, because I set 10% over budget as the point where we would stop and have another conversation about revised estimates and/or cutting or changing features. For all the advice I give other people, you might be surprised how rare it is for me to setup the terms of a job this clearly, so I feel like I’m starting on the right foot.
I’m building this site on Helma, a server-side JavaScript framework. I spent much of the last week learning about the details of Helma, but all I’ve built in it so far is an addressbook application tutorial and a simple script that finds anagrams. I’m betting that the benefits of the framework will outweigh the slow down from the learning curve, but that bet is actually “one of the classic blunders” of software development. After twenty hours we’ll find out who is right—and who is dead.