Category: Uncategorized

  • The Shed Porch Project

    We have a she-shed in our back yard. It was built in the 1980’s by my wife’s uncle for her aunt to use as a place to work on sewing projects. We’ve used it for storage of unused furniture, books, clothes, and dishware.

    The shed had a small porch in front of the door. It was evidently built from leftover lumber, which was untreated, unpainted, and without any foundation or posts. Just perched on top of a couple of loose cinder blocks.

    Each year as I mowed the tall grass around the shed, I saw that the porch looks a little more sad than the previous year. The nail heads had retreated a little further from the boards. The cinder blocks looked a little more jury-rigged. The screen door was scraping on the deck of the porch a little closer. One by one, the steps were rotting through.

    It was way past time to replace this porch!

    Tearout and Groundwork

    Removing the old porch only took a few minutes. The nails were barely holding it together anyway. I took the time to pull all the nails and put them in metal recycle. Some of the boards felt as light as styrofoam. I found more cinderblocks buried in the soil.

    I used CraigsList’s Free Stuff to give away the old wood. I don’t know what the person was going to use the boards for. Garden bed edging? Concrete forms? Even if it was for a bonfire, I didn’t mind. By giving away this old wood, I didn’t have to haul it to the dump.

    The shed siding behind the porch ledger was bare wood. I covered that up using some leftover house paint.

    The Foundation

    I decided to rebuild the porch in its original size: four feet by six feet, with a couple of steps on the left side. So I was just doing a “repair” without changing the design. I intended to make it a lot stronger than it was before, but otherwise the same.

    I had some surplus concrete deck blocks that had been loitering around our property for a long time. I finally got to use some of them. This was easier than pouring concrete piers or a slab.

    I used galvanized roofing nails to connect post bases to the wood on top of the deck blocks.

    The Ledger

    I wanted the ledger to be level, even though I knew the shed is actually far from level. So I put a couple of 12in stakes in the ground and balanced the board on the tops of the stakes. I could then tap the stakes by small increments until the board was level. I also made sure there was space between the ledger and the door trim to fit one of the decking boards.

    I fastened the ledger to the shed joists using six 4in GRK structural screws. These are plenty strong! They are rated at 840lbs. of shear strength each. A load that exceeded the capacity of six of these screws would be too heavy to store in this shed anyway.

    The Framing

    Now it was time to attach the sides of the frame. I clamped it to the ledger to make it flush. Then I used my stake-in-the-ground method to hold it level while I screwed it to the end of the ledger. Screws will hold the frame together over the years much better than nails.

    I cut short pieces of 4×4 for the posts, from some pressure-treated posts we had left over from previous projects. I cut the posts to the right length before putting them in place. I also cut a 5 degree slope on the top of the post to let water run off.

    I screwed the frame to the posts.

    I tied the frame together with the front 2×8 board. It fit perfectly between the side boards. The frame was still level and square. I balanced the board on two stakes again, while I fastened it to the side frame pieces.

    I cut posts like the others, placed them in the corners, and screwed into them from two sides. Maybe this was overkill?

    I added interior joists, spaced every 16in. I used some extra 2×8 lumber I had. I used clamps and a wooden stake to hold the joist flush with the rim joist, while I fastened it with more screws.

    DeWalt, if you want to send me any free swag for the product placement, I will not object.

    The frame of the porch was finished! I used a total of 44 structural screws. If the goal was to make the new porch stronger than the old porch, then mission accomplished!

    The Steps

    I’ve never made stair stringers, but the process seems pretty straightforward. This seemed like a good project to practice, since this porch only needed a few steps. I had some 2×12 boards long enough to make stringers left over from past projects.

    I tried making a diagram for the stringer using BlockLayer.com. But it was just as easy to make a template myself on some cardboard.

    I cut a 2×12 to make one stringer according to the diagram using the circular saw, and a little hand-sawing to finish the corners. Then I used the first stringer as the template for three more.

    I decided to use four 12in pavers as a footing for the stringers. Not as solid as it would have been if I poured a concrete pad, but a lot easier, and I tried to keep in mind this is just a shed, and what I was building was still much better than the old porch that was supported only by haphazard cinder blocks.

    Eventually I was able to fasten the stringers to the frame using joist hangers. I cheated the leftmost stringer away from the shed wall a couple of inches, so I could fasten both sides of its joist hanger to the frame.

    The stairs needed risers, so I cut some pieces of plywood and screwed them on. I also back-filled some soil around the pavers. This will help hold them in place as the soil compacts.

    Preparing the Decking

    It’s important to prepare new lumber to help it absorb the wood stain evenly. First, let the moisture and lumber treatments dry out for at least 30 days. Then, clean the wood with oxalic acid to open up the pores.

    I bought the decking boards two months earlier and sanded them. I guessed that two months was enough time, and the wood was dry enough to stain.

    I brushed the sawdust off of the decking planks and the stringers, then treated them with Behr All-in-One Wood and Deck Cleaner using a pump sprayer. Then after the wood dried for a day, I applied #1 Deck Premium Semi-Transparent Wood Stain for Decks, Fences, & Siding (Light Walnut). This was my first project involving stain, and I’m sure I did a pretty amateur job. But my primary goal was to protect the wood, not to make it pretty.

    Installing the Decking

    I covered the top of the joists and frame with 1.5in joist tape, to protect them from any standing water. This was probably overkill, because rain in California is so infrequent, but it’s good practice.

    To fasten the boards to the frame, I started from the stair stringer on the left side, and worked right. I looked around for an object about 1/8in thick that I could use to space the boards. I found a pair of leftover plastic things in our hardware drawer. I don’t remember what these were from, but they seem to fit, and they have an angle so they won’t fall through.

    I fastened all 18 of the boards, and discovered I made a mistake. I was two boards short on the end. I’ll have to order more boards, sand, and stain them. I left a scrap board over the gap at the end, so no one puts their foot in it, and the joist tape won’t peel up.

    Finally, there was easy access to enter the shed.

    Trim Boards

    I wanted to add some trim to cover the joists and the ends of the decking boards. I cut some pieces of 1×6 to the right size, then primed and painted them to match the Tudor Brown trim on the shed.

    Once the paint was dry, it was a quick task to fasten these trim boards to the porch.

    Finished Porch

    The new porch was solid and looked a lot better than the old one. I’m sure there are some things I could have done better, but it was just a shed, and I think this was a successful project.

    What I Learned

    • Decking boards are expensive no matter how one tries to economize. I thought I could reduce the cost by buying cheap pressure-treated wood, but by the time I added up all the costs and time for finishing the boards, I would have done better to just use Trex.
    • Buying cheap sight-unseen wood is a good way to get a bunch of unusable lumber with knots and even voids.
    • Digging and grading earth by hand is very time-consuming, tiring work.
    • It’s a lot harder than you might think to recycle leftover materials. They’re almost never exactly what you need for your new project, or else the materials are too old and degraded to use.

    Costs

    Below is a list of what I spent on this project, roughly in order of using the items.

    Items with “–” for the cost are materials that I had on hand already.

    ItemUnit CostQuantityTotal Cost incl. Tax
    Housepaint (primer and red)*
    Flat chip brush (for paint and wood preserver)$1.971$2.16
    Galvanized Steel 1/2in hardware cloth, 1/2in mesh, 2ft x 25ft$40.822$89.60
    Galvanized staples*
    Concrete pavers8
    Concrete deck blocks4
    Galvanized post base 4×4$7.294$32.00
    2×8 pressure-treated board, 12′ (cut in half)$13.001$14.27
    2×8 pressure-treated board, 8ft (cut in half)$19.371$21.26
    GRK structural screws, 5/16in x 4in (box of 45)$26.871$29.49
    4×4 pressure-treated posts, about 12in each4
    2×8 joists, 6ft2
    2×12 board, 36in (stringers)4
    Galvanized joist hanger, 2×6$1.483$4.84
    Galvanized joist hanger, 2×6 concealed$2.981$3.25
    Pressure-treated plywood, 6in x 4ft (stair risers)3
    Butyl joist tape 1-5/8in x 50ft (2 rolls)$24.631$26.85
    5/4in x 6in planks, 4ft (box of 3 boards)$43.996$263.94
    0.75in x 7.25in board, 8ft (for trim)$17.082$37.23
    Premium all-in-one wood cleaner, 1 gal$12.211$13.31
    Premium semi-transparent wood stain, 1gal$35.991$39.23
    Flat paint brush, 4in (for stain)$10.281$11.21
    Deck screws, #9, 2in (1lb box)$11.472$25.00
    Additional 5/4in x 6in planks, 4ft (box of 3 boards)$43.991$48.28
    TOTAL$661.92

    Tools I used that I already had on hand:

    • Impact driver
    • Circular saw
    • Belt sander
    • Speed square
    • Spirit level
    • Measuring tape
    • Plumb bob & string
    • Trigger clamps
    • Framing hammer
    • Sledge hammer (used for compacting soil)
    • Shovel, hoe, dirt rake
    • Saw horses
    • Wood stakes, 12in
    • Pry bar, nail puller
    • Work knife
    • Pencil
    • Wood preserver
    • Gloves, eye protection, ear protection
    • Miscellaneous galvanized nails, screws, and washers
  • The Footbridge Project

    I have a creek that runs through my yard. Most of the year it’s dry (we live in California), but it has a short span of time in the winter when water flows after the rains. Even when it’s dry, it is inconvenient to cross, because it’s between 3 – 4 feet deep, and the bank is irregular.

    The dark line is the creek.

    Actually I used to have a makeshift bridge over this creek. Years ago, I threw down some scrap lumber I had around. But one day when I came out into the yard, I saw the whole bridge was gone! It wasn’t secured down, so in theory it could have floated away when the water level swelled. Or else I guess someone passing by needed some lumber? Anyway, I need a new bridge.

    The former bridge, now missing.

    So I made a plan to build a new footbridge over it. This time I needed to build one properly with concrete moorings. I’m an amateur with construction projects, so this would be a learning experience. I hoped it would inspire me to try other projects.

    Preparing the site

    The first step was to clear the weeds and brambles that have grown in the creek. It had gotten quite overgrown. The thickest growth was next to my neighbor’s driveway. Some of the weeds could be pulled out by hand, but the raspberry brambles are thorny, and that wannabe tree has grown large enough that I had to use lopping shears and finally an electric saw.

    It took several weeks and truckloads hauled to the green waste at our county landfill.

    $15 dumping fee per load

    I chose a location to build the bridge. This spot is in the shade of a couple of trees, and the bank was nearly the same height on both sides.

    I admit working in the shade is not an insignificant factor!

    I left the Vinca Major ground cover intact, because it’s nice when it blooms, and also because it’s tougher than it looks, so tearing it out would take a lot of work.

    Measuring the bridge

    I ran some string to mark out the bridge. I thought I could use 12-foot joists, but I found that was too short. The bank of the creek is pretty crumbly, and I wanted the moorings to be well back from the edge. So I measured out a 16-foot length.

    The width of the bridge would be about 5 feet, because I had some Trex composite boards I had kept when we rebuilt our deck. I planned to make these the planks of the bridge, so I could avoid buying new material.

    Part of the creek bank is shored up with scrap concrete installed by a previous owner of the property. This was another reason to set the end of the bridge a few feet back from the edge, because I can’t dig through the concrete.

    I learned how to make square corners with the 3-4-5 method, and how to tie string to stakes using a Larkshead knot. I enjoyed watching a video Using String Like a Pro, by the Essential Craftsman.

    I placed one of the Trex boards on the ground for reference, to make sure I was measuring the right width for the bridge.

    How much water?

    The creek is dry most of the time, but during rain it does get water. In January 2023 we had one of the biggest rainstorms we’ve ever had in this county. I got a photo of the water when the creek was full, and I was confident that the underside of the bridge (indicated by the string level) would still be above the level of water, and the moorings at each end of the bridge will be secure.

    Building the moorings

    I planned to have three joists spanning the creek. Trex is not known for its bearing capacity, so I wanted a center joist as well as one on either side. To fix these joists to the ground securely, I would connect them to concrete moorings.

    I got steel Strong-Ties to fix the joists to the moorings. I wanted these to be as straight as I can make them, so I built a temporary frame matching the width of the bridge (i.e. 5 feet, the length of a Trex board). These allowed me to keep the Strong-Ties straight as I put them into the concrete. I used construction screws so I could remove these frame pieces from the Strong-Ties once the concrete sets.

    The next task was to dig holes to put the concrete moorings into.

    Setbacks and getting help

    While digging the holes, I gave myself a muscle strain in my back. I’m not used to this kind of work. It’s going to take several weeks to heal and strengthen my back.

    So I recruited a couple of friends who were visiting me this summer. I’ve known Jay and Kevin for over 30 years, since we were at university together. They asked if there’s anything I want to do during their visit. I suggested half-joking, “do you want to help me build a bridge?” They laughed but said, “yeah, well, okay!”

    Wow, did they help! They made sure I didn’t lift anything so I didn’t re-injure my back. I ended up becoming the foreman, and provided a detailed plan, the tools, and the materials. They said they were there to provide the hands and muscles. But they gave plenty of good input to make every step work well.

    Concrete moorings

    We dug six holes for cardboard sonotubes, which we carefully leveled at the strings I had stretched. We mixed nine 60lb. bags of concrete and filled the tubes. We also pushed pieces of #3 rebar into the concrete to help prevent cracking.

    Then we pushed my Strong-Ties on frames into the concrete. These will remain level and straight while the concrete sets.

    After the concrete set for 24 hours, we removed the frames from the Strong-Ties.

    Joists

    We ordered some lumber from Home Depot: six pressure-treated boards, 2×8 inches by 16 feet. We lifted the boards into position, two in each Strong-Tie.

    I had viewed many YouTube videos recommending to “crown” the boards, orienting any bend in the board up so it flattens out when there is weight on it. But these boards were remarkably straight already, so it didn’t matter much which way we oriented them.

    We also lined them up carefully and measured from corner to corner to make sure the layout of the boards was still square.

    To make sure the tops were flush before we screwed the boards together, we used clamps and held them against a piece of scrap 2×4. Also more clamps to hold the boards tightly together while we put screws in.

    The boards were to be bolted to the Strong-Ties. It was a bit of a challenge to drill straight holes accurately enough to emerge through the hole in the other side of the Strong-Tie, but with practice we got pretty good at it.

    We got through 10 out of 12 bolts using an impact driver with a socket, but then my cheap old 1/4″ socket adapter broke! No problem! We couldn’t let that stop us. We tightened the last two bolts the old-fashioned way with wrenches.

    We now had three strong joists that were level, square, flush on top, and definitely not going anywhere.

    Blocks

    It’s recommended to add some transverse pieces to provide stability and keep the joists from twisting or leaning. These are called blocks. We used a seventh 2×8 board, cutting it into pieces one by one. We aren’t precise architects, so despite trying to make the framing square, each block was slightly different in length. We screwed them into the joists toenail-style. These pieces won’t be load-bearing.

    The bridge was now quite stable. Kevin tested it by walking across it like a balance-beam.

    Joist tape

    It’s a good idea to protect horizontal wood from any potential rainwater pooling on it, so we got some special adhesive tape used for deck projects and stuck it down on top. The 3 inch tape worked well, and we got it down the first try by going slowly and carefully. One 50 foot roll of the tape was perfect for three joists, with minimal waste.

    We didn’t tape the blocks, because they should be under the deck boards, and I assume risk of rot isn’t as important since they aren’t load-bearing. Also because we ran out of tape.

    The shiny tape makes the joists look classy. It’s almost a shame it’ll be covered up by the boards.

    Deck boards

    Now finally we can put the boards on top. We had just enough Trex boards to cover the length if we added a 1/2″ space between each board. That’s recommended anyway for drainage and allowing for expansion (not so much for Trex, but still not a bad idea).

    We got very focused on the work during this phase, and didn’t take many photos, but my wife got one of us working.

    It really paid off that we made the Strong-Ties level, and made each pair of the joists flush. This made it easy to attach the deck boards to the joists. Our worst fear is that the middle joist might be slightly too high, and the boards wouldn’t sit flat or be stable. But in our case, it was perfect.

    After we finished, we were amazed that my impact driver put in 48 coated screws, 12 bolts, and 192 deck screws on a single battery charge. Good job!

    Finished product

    Here’s how the bridge looks now. Later I will make a bit of a ramp with gravel so it’s not such a high step up, but it works, and it’s strong and solid!

    Final thoughts

    This turned out to be a surprisingly great experience to build this bridge with my best friends. They are easy-going, patient, practical, communicate well, and they focus on getting things done. By the end, we were very proud of having such a good result, but the time we got to spend working together will be a great memory.

    What’s the next project?

  • Will Voyager 1 and 2 come back as V’Ger?

    In Star Trek: The Motion Picture (1979), V’Ger was an immense alien spacecraft that threatened Earth.

    V’Ger turned out to be evolved form of Voyager 6, a fictional unmanned probe that in our reality has (so far) never been launched (apologies if that spoils the movie for anyone, but was more than 40 years ago, so I think the moratorium on spoilers has expired).

    My theory is that V’Ger is a Mary Sue character, in a story written by the Voyager probe itself.

    Think about it: the probe is just a bespoke wide-spectrum camera and transmitter, coasting at 0.005% of the speed of light, barely making it outside its home solar system. But somehow it was found by a race of benevolent sentient machines, and they decided it was so impressive and worthy that they upgraded it to a planet-sized vessel armed with plasma weapons.

    The probe then becomes infinitely powerful and intelligent. It excels at tasks that should be impossible for it. It upstages the protagonists in the story. It even becomes the sexiest, most beautiful character in the story, with whom other characters are compelled to fall in love.

    V’Ger has no flaws, and therefore no character arc describing its internal growth from a flawed character to an improved character. It has all the hallmarks of a Mary Sue in bad fan fiction.

    So it’s unlikely that any of the Voyager probes from our Earth will return as V’Ger, unless Voyager transmits its amateur fiction back to us.

  • Maximum Memory Warnings Are Rubbish

    Many people use MySQLTuner, as a guide to MySQL Server configuration values and they are alarmed when they see things like this:

    [!!] Maximum reached memory usage: 86.6G (138.16% of installed RAM)
    [!!] Maximum possible memory usage: 5085.9G (8110.25% of installed RAM)
    [!!] Overall possible memory usage with other process exceeded memory

    You should be aware that the “maximum memory” warning from MySQLTuner is rubbish. It is a misleading and impractical estimate, because it has no real chance of happening.

    The memory usage they base their calculation on has to do with buffers that are allocated conditionally, depending on certain query behavior. Not all queries allocate all those buffers. Even if they do allocate the buffers, they may not allocate the full size of the buffer.

    The chance that a given query will allocate all the possible buffers at their maximum size is remotely small.

    In addition, there might be up to max_connections clients connected, but it’s typical in a running system that not all connections are executing a query at the same time. On every MySQL Server I’ve supported, even if there are hundreds of clients connected, most of them are idle (i.e. not running a query). I would expect the number of threads running a query at any given moment to be 10-20 at most. The others are connected, but if you view SHOW PROCESSLIST, they only show “Sleep” as their current state. These are clearly not using query-specific buffers.

    The possibility that all threads on a MySQL Server will allocate all the possible query buffers at their maximum size at the same time is so unlikely that you should treat it as complete fiction.

    Because of these mistakes in calculating the theoretical maximum memory usage, I ignore those warnings from automatic tuning tools like MySQLTuner. The estimate of “maximum memory” given by MySQLTuner has been many times the size of RAM on every database server I have administered. You don’t need to be alarmed by this.

    So how should one estimate memory usage?

    By observation!

    Monitor the actual memory usage of your mysqld process by using tools like top or ps. Ideally you would have a continuous monitoring system to make graphs over time, so you could observe the trend.

    This is much more accurate than relying on MySQL Tuner or any other estimate. Those estimates are not taking into account your database traffic or activity.

  • Beginner Web Programming Projects

    Can someone suggest a web application project for beginners? What would be a good project? Should I follow a how to make a tutorial or figure it out on my own?

    This is a frequent question from new programmers. I wrote a post in 2020 about this. Below is a reproduction of my post. It was in response to a question specifically about learning PHP, but similar advice would be true for other programming languages.


    The best way to learn PHP, or really any other language, is by practicing writing code. This is probably echoed by many of the other answers.

    • Pick a simple practice project, and implement it. Then pick another one. You don’t have to create a beautiful website to practice. Create an ugly website that serves as your sandbox for trying out code.
    • Learn from mistakes. Try something different.
    • Go outside your comfort zone. Find a new group of functions in the manual and try out each one. For example: Arrays or Strings or SPL.
    • Learn about PHP: The Right Way.
    • Learn about Laravel, the The PHP Framework For Web Artisans. Really. The days of arguing over the best PHP framework are over.
    • Learn about security (see OWASP).
    • Learn about testing code (PHPUnit, the PHP Testing Framework).
    • Subscribe to PHP blogs like Planet PHP. Attend PHP community users’ groups or PHP Conferences around the world.
    • Read other PHP code. Download WordPress or MediaWiki, both widely used and mature PHP projects, and study the code. What’s good about it? What’s lousy about it? There are bits of code in any project that are gems, and some that are garbage.

    Finally, there’s a secret way to learn any skill better than you can learn it on your own: teach it to someone else.

    As for a suggestion for a project to practice, I would recommend you develop a blog application similar to WordPress.

    1. It’s very simple at first: just a database table that stores blog posts by date, one PHP script to view them, and one PHP script to author them.
    2. Add authentication, so only the blog owner can write posts. Be sure to read You’re Probably Storing Passwords Incorrectly.
    3. Add tags, so the blog owner can categorize posts and users can view posts in that category. You’ll learn what a many-to-many relationship is.
    4. Add comments so readers can post too.
    5. Add captcha support or logins for users. Integrate with OpenID, Facebook, Google authentication APIs. Learn OAuth2.
    6. Add a text search function. Learn about a fulltext search engine like Sphinx Search.
    7. Add some kind of email notification or RSS feed, so users can be notified when a new post appears.
    8. Refactor your comments system to use Disqus, as a way to embed a web service into your application.
    9. Refactor the whole app to an MVC framework, to learn what it takes to do a major overhaul of your code without breaking things.
    10. Write unit tests for your classes that you wrote for the MVC implementation.

    I would recommend that you do not follow a tutorial, and instead try to design the application on your own. It will take longer, but I guarantee you’ll learn more about coding. Anyone can follow a tutorial and type in the steps, without knowing what’s happening. If you do not use a tutorial, you’ll have to think harder about how to solve the problems that come up. This is a necessary skill if you want to be a programmer.

    But of course you need to read documentation or how-to articles on some of the individual pieces I included in the list above.

  • Why is Software So Complex?

    Q: Why is most modern software so mindbogglingly complex, with multiple layers of abstraction stacked on each other? Why do they not make simple, efficient software like they used to earlier?

    This question was asked on Quora recently. I answered that it is due to a few major reasons:

    1. Code Maintenance

    There’s an old humor article that has circulated online for many years, titled, “If Architects Had to Work Like Programmers.” It’s written as if it’s a letter from a homebuyer who wants an architect to build a house. Here’s an excerpt:

    “Please design and build me a house. I am not sure of what I need, you should use your discretion. My house should have between two and forty-five bedrooms. Just make sure the plans are such that the bedrooms can be easily added or deleted. When you bring the blueprints to me, I will make the final decision of what I want. Also bring me the cost breakdown for each configuration so I can arbitrarily pick one.”

    This is humorous because it sounds like the way programmers are given their software requirements. Because software can be modified after it is created, the employer assumes it’s easy to do that, and that they don’t need to be specific about what they want.

    People in the software development field over many years have tried their best to accommodate this, by creating more and more abstractions so that the pieces of software can be altered, combined, upgraded, or swapped out more easily.

    The employer wants this because it enables them to get software on an affordable schedule, without forcing the employer to write detailed specifications that they probably don’t know in advance anyway.

    The programmer wants this because they want to remain employed.

    2. Code Reusability

    A good way to increase code quality without increasing the schedule is to write less bespoke code, but instead use more code that has been written and tested well prior to your project. We call these libraries, frameworks, templates, or code generators.

    You’ve probably used Lego toys, where you can build elaborate models using simple reusable bricks or other specialty pieces. You can build practically anything by using enough bricks. There are also some specialized shapes, and lots of guides showing you how to combine them to build the desired models.

    Image: Queen Mary model in Lego.

    It’s a similar concept to code reusability. Software development then becomes an activity of learning all the different pieces and ways to use them together.

    The code is reusable because of abstractions. Like the Lego pieces that use standard dimensions and connecting buttons so they can be fastened to other pieces.

    3. Features, Features, Features

    I once developed an app for a manager who was very decision-challenged. Every time I would ask him, “do you want the app to behave this way or that way?” I was often asking about two alternatives that were mutually exclusive. For example, do you want the report to arrange categories of data in rows or in columns?

    He would always answer, “both.” He didn’t know how to choose, and he was afraid of making the wrong choice. So he asked me to implement both alternatives, and make the software configurable. He wanted to keep his options to change his mind later as often as he wanted to.

    This at least doubled the work to implement the code, and doubled the testing needed to assure it works.

    But it was worse — every time he said, “both” this doubled the number of test cases, because I had to assure that a new feature worked with every combination of alternatives for the past features.

    Programmers can’t say “no” when their employers want some features. They can say, “okay, but here’s what it’ll cost in time and money, do you still want it?”

  • How do Software Engineers cope with stress?

    The typical sources of stress for software engineers are not caused by technology. They’re caused by managers and projects.

    Work tasks are not described clearly.

    If I’m given unclear work requirements, I ask for more details. I make it clear that I can’t give an estimate for the cost or the time of completion until I know the full scope of work. In a professional software engineering environment, a significant amount of time should be spent estimating the work based on complexity of the task. You can get yourself into a stressful obligation if you agree to a deadline before knowing the requirements of the project.

    Be careful about this. Managers often insist that the software engineer decide on the estimate much too early, and then they use that against the engineer later, saying it was an estimate that the engineer had made, so they can’t claim it was imposed on them.

    When you are asked for an estimate before you know the scope of work, remember to use this standard response: “I’ll have to get back to you.”

    Schedule is too aggressive. Deadlines are impossible.

    At one job I joined, I was hired to get a software project back on track after it had fallen behind schedule and the previous team lead had quit. I knew generally what the project was, but I didn’t know how much was done and how much still needed to be done. On my first day, a marketing person told me he wanted to present this software to customers at an annual conference, which was happening in two weeks. He wanted me to promise to finish the software by that time. I told him I would need two weeks just to learn the current state of the project and make an estimate for the completion. It was a little bit stressful to tell him no, but it would have been much worse to make a promise and then fail to deliver.

    Finishing on schedule requires long work hours and few breaks.

    Don’t fall into the trap of letting management bully you into working until you are exhausted. If you do that, you will make more mistakes, and your code will need to be scrapped and written over. You are not a machine — and even machines need time for maintenance, cleaning, repair, etc. If you keep yourself healthy and your mind fresh, you will be able to concentrate better and produce better quality work. You will have a better chance of finishing the work on time.

    New requirements are added late in the project, but the workers are held to the original schedule.

    When they want to add more features, tell them you’ll evaluate the new requirements to see if they can be added with minimal interruption to the schedule. Do your best and make an honest effort to do that. But sometimes the new feature is requires major changes to the current code design, which is partially implemented already.

    Approach the product manager and let them know this. Present to them the following options:

    • Postpone their new feature idea until “phase 2” (that is, a future revision of the software).
    • Cut some other requirements from the current project that are time-consuming and not yet implemented.
    • Make a compromise to reduce the complexity from one or more features, to make them take less time to implement.
    • Extend the project deadline to give enough time for the extra features.

    If they still demand that they want all the features and no change to the schedule, that’s not realistic. Politely tell them that they need to choose one of the options or they will be disappointed. This makes the tough choice their responsibility, which reduces your stress.

    Unscheduled work and alerts interrupt and spoil concentration.

    This is a great source of stress because it’s difficult to resume work that requires concentration after an interruption. This has been studied a lot. It’s not just the time it takes to do the unscheduled task. It also takes time to shift your focus between tasks. If this happens several times per day, you can lose all your productivity for the whole day.

    If software engineers are expected to be oncall or to help with unscheduled analysis or troubleshooting or technical support, then they should make it clear that any schedule estimates are in unpredictable. Your stress comes from uncertainty that you can be productive enough to meet your deadline. You can mitigate this stress by insisting that the deadline must be extended every time you are interrupted.

    Software engineers are expected to understand and be productive with any type of technology with no time for training.

    The best way to cope with this is to fib a little bit and add some time to every estimate, to allow for research, self-training, debugging, and getting answers from technical support. It’s unfortunately a reality that management can’t justify budget for training time, if they are already paying high salaries to software engineers. So you have to include the necessary training time with engineering estimates.

    One way to hide training as engineering is to schedule part of the project to implement a “prototype” or a “proof of concept.” These basically mean you’re going to be practicing, and the result will be an unoptimized implementation, intended to be scrapped and redone before the final deadline.

  • The Private Option

    There’s a famous case of a fumbled rollout of a website: HealthCare.gov, the federal health insurance exchange used by independent insurance customers in about two-thirds of states in the USA.

    These days, the an updated version of HealthCare.gov functions fine, so you’re wondering what the hubbub was about when it was launched.

    Poor Debut

    Proponents said that a slow rollout is not unexpected. People who managed the health insurance exchange in Massachusetts that served as the model for the Affordable Care Act say that the same initial bugs and slow adoption affected their program too.

    The site has performance and scalability problems, has an overly complex user experience, and sometimes calculates wrong answers. The result is that of the 100,000 people who signed up for independent health insurance after October 1 2013, fewer than 27,000 used the federal exchange.

    Why Did it Fail?

    HealthCare.gov had a major obstacle: they had to handle several times the originally anticipated demand. The original plan was for each US state to implement their own health insurance exchange, to serve people in their respective state, and HealthCare.gov would handle those who couldn’t. It was assumed that only a small minority of states would rely on HealthCare.gov, and these would be the states with smaller populations. As it turned out, a majority of states refused to implement their own exchange web sites. In December 2012, when the states were required to have blueprints describing their solution, reportedly 25 states didn’t meet that deadline
    By the time of the rollout of the ACA, only 14 states were signing people up using their own state-run exchange, whereas the rest of the states–more than two-thirds–were relying on the federal exchange. These include some of the highest population states like Texas and Florida, and 20 states who had taken federal money to plan their state exchanges, but ultimately also relied on the federal exchange.
    The Private Option

    A few young programmers created an alternative web site they call HealthSherpa.com in their spare time, after the ACA debut on October 1 2013. Their web site is a prototype effort to make a more streamlined portal for people to find the health insurance plans they’re eligible for. It seems to work, and it’s very fast. It uses raw data that is accessible publicly from the federal government.

    It’s a valid question then: why didn’t the federal government—or any of the states—employ a small team of web experts to throw together such a site for a fraction of the cost?

    HealthSherpa.com doesn’t have all the functions that HealthCare.gov is supposed to. It doesn’t do credit checks, it doesn’t actually even sign anyone up for health care. It just allows consumers to find the data that pertains to them, and then it links to the websites for the respective insurance carriers. And HealthSherpa.com doesn’t create the data—it might be true that part of the effort behind HealthCare.gov has created the raw data that HealthSherpa.com uses.

    Also, HealthSherpa.com isn’t (yet) serving tens of millions of users, as HealthCare.gov is supposed to do. I work for Percona, a company that offers consulting and support for database operations, which is just one aspect of web site scalability. Scalability for a web site is complex, much more difficult than most people appreciate.

    But it’s worth noting that even with these limitations, there’s a pretty big difference between a three guys throwing together a working website in a few days, versus major federal IT contractor CGI Federal spending $174 million since they announced winning the contract in December 2011 (i.e. 22 months until their go-live deadline of October 1 2013), but they still failed to implement a site that could handle the demand.

    Conclusion

    So here’s some hindsight views on the HealthCare.gov project:

    • They should have anticipated the demand from all 50 states. This may have been over-engineering, since the intention was to serve only a minority. But they had no control over which states would agree to create their own exchanges, and every reason to think there would be political resistance to doing so.
    • They should have had a beta test period. No large-scale web site can handle the load of millions of users on its first day, not even sites implemented by major web experts like Google and Amazon. They restrict enrollment to a limited subset of their users, sometimes by invitation only. They leave enough time to work out the problems before going fully public.
    • They should have provided raw data only, not the whole web site. Let other entrepreneurs innovate the best way to search the data. Maybe someone would even create a Facebook game for selecting your insurance.
    • They should have set the deadline after scoping the project.
  • C Pointers Explained, Really

    While I was in college, a friend of mine complained that he was confused while programming in C, struggling to learn the syntax for pointers.

    He gave the example of something like: *x=**p++ being ugly and unreadable, with too many operations layered on each other, making it hard to tell what was happening.  He said he had done a bit of programming with assembly language, but he wasn’t accustomed to the nuances of C.

    I wrote the following explanation on our student message board, and I got a lot of good feedback.  Some people said that they had been programming in C for years, but not until they read my post did they finally understand pointers.  So here it is, unearthed from my backups and slightly edited.  I hope it helps someone again…


    Message 1956 (8 left): Thu Jan 25 1990 2:44am
    From: Bill! (easterb@ucscb)
    Subject: Okay

    Well, if you know assembly, you have a head start on many of the CS freshpersons here. You at least know about memory maps: RAM is a long long array of bytes. It helped me to learn about pointers if I kept this in mind. For some reason, books and instructors talking about pointers want to overlook this.

    When I have some code:

    main()
    {
      int n;
      int *p;

    There is a place in my memory that looks like this:

                :
    Address:    :
             |-----|
    0x5100   |     | n is an integer, one machine word big
             |-----|
    0x5104   |     | p is a pointer, also one word big
             |-----|
    0x5108   |     | other unused memory
             |-----|
                :
                :

    Let’s give these variables some values. I set n to be the number 151.

    n = 151;

    I set the pointer p to point to the integer n.

    p = &n;

    That says, “the value of the variable p is assigned the address of the variable n”.

               :
    Address:   :    Value at that address:
             |----|
    0x5100   | 151| n
             |----|
    0x5104   |5100| p
             |----|
    0x5108   | ?  |
             |----|
               :
               :

    Now I want to print out the value of n, by two ways.

    printf("n is %d.n", n);
    printf("n is %d.n", *p);

    The * operator says, “give me the object at the following address.” The object’s type is the type that the pointer was declared as. So, since we declared “int *p”, the object pointed at will be _assumed_ by C to be an int. In this case, we were careful to make this coincide with what we were pointing at.

    Now I want to print out the memory address of n.

    printf("n is located at $%x.n", &n);
    printf("n is located at $%x.n", p);

    The & operator says, “tell me the address where the following object starts.” In this case, it is hex 5100 (I put a ‘$’ before it, to conform to the Assembly notation I am used to). Notice the _value_ of p is an address.

    Hm. Does p have an address? Sure. It is a variable, and all variables have their own address. The address of p is hex 5104.

    printf("p is located at $%x.n", &p);

    Here we are taking the address of a pointer variable, using the & operator.

    main()
    {
    char name[] = "Bill";
    char *p;
    int *q;

    Now we have an array to play with. Here’s how memory looks now:

           |---|
    0x5100 |'B'| "name" is an address constant that has value hex 5100
           |---|
    0x5101 |'i'| char: 1 byte
           |---|
    0x5102 |'l'| char: 1 byte
           |---|
    0x5103 |'l'| char: 1 byte
           |---|
    0x5104 |   | char: 1 byte
           |---|
    0x5105 |   | p is a pointer: 1 word
           |---|
    0x5109 |   | q is a pointer: 1 word
           |---|
    p = name;

    We set p to the value of name. Now p has value hex 5100 too. We can use the * dereferencing operator on p, and get the character ‘B’ as a result.

    Now what happens if I do this:

    ++p;

    The pointer p is incremented. What value does it have now? Hex 5101. Pretty simple.

    Now let’s try something irresponsible:

    q = name;

    But q is a pointer to int! If we dereference q, it will take the word (typically 4 bytes) beginning at address “name” (which is hex 5100) and try to convert it to an int. ‘B’, ‘i’, ‘l’, ‘l’ converted to an int will be some large number, dependant on the bit-ordering algorithm on your machine’s architecture. On ucscb, it becomes 1114205292. (to see how, line up the binary representation of the ascii values for those 4 characters, and then run the 32 bits together, and convert that resultant binary number as an integer.)

    What we have just seen here is a key issue of pointers that I mentioned earlier: C assumes that what they are pointing at is an object of the type that the pointer was designed to point at. It is up to the programmer to make sure this happens correctly.

    ++q;

    The int pointer is incremented. What value does it have now? Hex 5104. Huh?!? The answer is simple if you accept the above paragraph. It gets incremented by the size of the object it _thinks_ it is pointing at. It’s an int pointer, so incrementing it makes it advance a number of bytes equal to the size of an int.

    Now print the dereferenced value of q (i.e. the value of the object q is pointing to). Well, it’s pointing at a null byte, and then the first 3 bytes of the char *p. Now we’re all messed up. Nice going. Try to convert _that_ to an integer representation. Well actually, C will do it happily. But it’ll be another weird number.

    main()
    {
      int n;
    
      n = 151;
      f(n);
    }
    
    f(x)
    int x;
    {
      printf("%d.n", x);
    }

    Here is a simple program that passes an int “by value”. That is, it copies the value of n into the new variable x!

           |---|
    0x5100 |151| n is an integer
           |---|
    0x5104 |151| x is another integer
           |---|

    When we mention x, we are using the value at location 5104, and we can change it, read it, whatever, and it won’t affect n, the int at location 5100.

    But what if we want to have f() modify the value and then have that new value be available in main()? C does this by passing the variable “by reference”.

    main()
    {
    int n;
    
      n = 151;
      f(&n);
    }
    
    f(x)
    int *x;
    {
      printf("%d.n", *x);
      *x = 451;
    }

    Pass the _address_ of n, and declare x as a _pointer_ to int. Actually, this is still passing by value, but the value being passed is the address, not the number.

           |----|
    0x5100 | 151| n is an integer
           |----|
    0x5104 |5100| x is a pointer to int
           |----|

    Now if f() when we make use of *x, we are referring to the value at location 5100. This is the location of n. After the assignment “*x = 451;”, this is what we have:

           |----|
    0x5100 | 451| n is an integer
           |----|
    0x5104 |5100| x is a pointer to int
           |----|

    x still points to location 5100, but we have changed the value of the object at that location.

    Well, those are the basics. You mentioned things like “*x=**p++” being ugly and unreadable. Well, yeah, but here is a diagram that may help:

           |----| here is a word in memory with initial value 0. 
    0x5100 | 0  | no variable name
           |----|
    0x5104 | 12 | here is a value, a word in memory. no variable name.
           |----|
    0x5108 |5104| Here is an int pointer, pointing at the previous word.
           |----|
    0x511c |5108| here is p, a pointer to int pointer.
           |----|
    0x5120 |5100| here is x, a pointer. guess where it's pointing.
           |----|

    First let’s see what p and x were declared as:

    int *x; /* pointer to int */
    int **p; /* pointer to pointer.

    The subordinate pointer is a pointer to int.*/

    You should know now what “*x” means. It means, “the value of location 5100.”

    And you know what “*p” means, “the value of location 5108”. Now that value is another address! Okay, let’s dereference that address: “**p” and we find (by the declaration) an int.

    Now “*x = **p” looks like, “this int at 5100 gets the value of that int at 5104.”

    And what does “**p++” mean? Well, ++ binds tighter than *, so this is equivalent to: *( *( p++ ) )

    Or, “pointer to pointer to int, and by the way, after we’re done, p has been incremented. But we looked where it was pointing before it got incremented, so we don’t care. Let the next statement worry about it.”


    This content is copyright 2012 by Bill Karwin.  I’ll share it under the terms of the Creative Commons License, Attribution-NonCommercial-ShareAlike 3.0 Unported.

  • How to Save $100 Million

    Last night I listened to an interesting interview on public radio, relating a story from the New Yorker magazine about Michigan Dr. Peter Pronovost saving millions of dollars and hundreds of lives of patients.

    How did he do it? He taught doctors and nurses to use checklists to avoid mistakes in the intensive care units of hospitals. Mistakes that could put patients’ health or lives at risk.

    What’s interesting about this story is that it’s an extremely low-technology solution to a type of problem that exists in virtually every field. In this case, it applies to medical care. But it easily applies to manufacturing. In Japan, they call it poka-yoke, or mistake-proofing. Check out this book too: “Mistake Proofing: Designing Errors Out” by Richard B. Chase and Douglas M. Stewart.

    Do checklists and similar techniques mean hamstringing the creative process in these fields? Absolutely not! On the contrary, effective use of checklists can free our attention from repetitive details, so that we can devote more of our energy to innovation and creativity. We don’t need to keep the details of well-understood procedures in our short-term memory, if we write down the steps so that we can do them without burden, or delegate the work to a teammate.

    Why not go the extra step and create technology to automate those procedures? Because a checklist doesn’t necessarily remove the requirement for human attention, to exercise good judgment and critical thinking. Some steps may require analysis, or may be performed conditionally based on the result of a previous step. It’s usually very expensive to make a machine that does that kind of analysis.

    Procedures are inexpensive to modify and prototype when humans perform them. It might turn out that the whole procedure is revealed to be incorrect, and needs to be re-thought. If automation technology had been developed for that procedure, the cost of developing that technology would be wasted. If the procedure were merely a checklist, then we just need to re-train the operation staff and voilà!

    Notice that books like David Allen’s “Getting Things Done” focus on non-technological methods for organizing and avoiding letting things fall through the cracks.

    If checklists and other easy organizational techniques are such a good idea, why don’t we employ them more? In the article about Dr. Peter Pronovost, he remarked that it’s surprising that it has taken so long to adopt his methods, and if there were a drug that achieved the same positive results he does, it would be mandatory in every hospital. A clue to the explanation is in if there were a drug. Follow the money! The solution that is marketed most aggressively is not the one that is most cost-effective; it’s often the one that is least cost-effective, because its vendor stands to make the most money from that one.