Free download:

Software Wars, the Movie

Soundtrack for the book:


If you enjoyed the free download, a donation of the cost of a newspaper would be appreciated.



The best book explaining free market economics:

The best explanation of how we can build a space elevator in 10 years:


Book explaining a political solution to our economic problems:


Shipping the LibreOffice HiDPI Patches, or How I Learned to Love Heartbleed

I wrote my story about getting some HiDPI patches into LibreOffice, but it was an unfinished one because while the code had gotten accepted into the main Master branch, there was a lot remaining. It hadn’t shipped, I’d only tested it on Gnome and KDE, it hadn’t been tried on Windows, and didn’t work on the Mac. Worst of all, because it missed the December 20 cutoff date for 4.2.0, it was on the default next release train for 4.3.0 in late July.

LibreOffice has a two-part release process. You are encouraged to submit code into the main tree where it can sit for up to 6 months. Twice per year, it is branched and shipped. In between these major releases, every month or so, a minor release is made containing high priority fixes. Meanwhile most of the development team moves ahead adding features and cleanup for the upcoming release.

The code was too late for 4.2.0, and so it wasn’t getting much feedback. Very few people run the daily builds, with all the releases and release-candidates to test. A friendly chap named Darcy from Australia showed up on the QA alias, built LibreOffice on his Fedora 20 laptop, and verified that it worked, but that was it. The only way to get this code tested was to get it out there.

I had also decided to stop working after the second batch. I had improved the most noticeable parts of the product, but I was also steadily making more potential problems for myself. Software isn’t just about code, it is about standing behind your work. I could prove most of my changes were fine, but I was changing places where I didn’t always understand what was going on around. I could justify my fixes, but not the code around it.

Even though LibreOffice is built by a community where other people can fix your bugs, other people can also find your bugs. If your code is causing problems, and no one has time to look into it, it can be reverted. The work is interesting, but I was just a motivated user with some free time during the Christmas holiday. I wanted to reserve time to deal with the inevitable complaints.

In spite of the lack of feedback, since it makes a visual and usability difference, why wait? More computers with these beautiful screens are coming out every day. A very high-resolution screen is the best reason to buy a new laptop. There should be a free software experience that we can enjoy looking at. LibreOffice has plenty of ways to improve, but it can look good in the meanwhile.

Sidebar

The Sidebar was one of the areas I had kept putting off and almost didn’t work on. It is about 70,000 lines of new code providing an alternate UI, and I wanted to focus on the existing one first. I have been using this codebase for almost 10 years and had never missed it. The code was also written by someone from IBM, and so I couldn’t trust that anyone in LibreOffice would be able to help me.

I also hoped that maybe the Apache team would notice bug reports and fix the Sidebar themselves. LibreOffice grabs fixes from Apache on a daily basis. Only about half the changes are useful because in many cases LibreOffice has already done the work, but any pieces of value are ported over, and just a small part of LibreOffice’s churn of the their thousands of improvements per release.

I considered sending an email to the Apache OpenOffice alias asking if they were aware of the problem, and whether they were planning to work on it. It is more efficient to coordinate efforts and not have multiple people re-learning each other’s work. It took me hours to fix problems that could have been fixed over lunch by the person who wrote the code. However, while LibreOffice is taking patches from Apache OpenOffice, the groups are generally not actively planning work, and so asking for HiDPI support for the Sidebar would have been a breach of protocol.

In addition, the codebases are diverging so fixes might not be usable. Apache OpenOffice doesn’t have the OutputDevice::DPIScaleFactor API, so the patches wouldn’t have been directly usable. I didn’t try Apache OpenOffice on Linux, but I did try it on Windows 8.1 and sidebar bitmaps looked doubled. However, Windows 8.1 might have been artificially scaling the entire Apache OpenOffice UI because the text was a blurry mess compared to LibreOffice:
OOLO2
So I could imagine that fixing the Sidebar for LibreOffice would not be a high priority for Apache.

While I personally don’t care about the Sidebar, it is now turned on by default for Impress. I knew that the more places I fixed, the stronger argument I would have to convince people to get the improvements out there. I realized I just needed to motivate myself to learn the code. So while I was in rural northern Michigan with my family over the holiday, there were some quiet nights, and I dug in and learned the Sidebar well enough to fix the major issues. It was the same process and techniques I had used for the other parts of the code. As per usual, finding the correct place to put in a fix was the hardest part.

I’m glad I worked on it because not only did it make the Sidebar fit in visually with the other improvements, it fixed a crashing bug. LibreOffice’s Sidebar is better than OpenOffice’s in that it has dynamic layout when docked. However, the bigger buttons created something wider than the maximum width allowed. With the sidebar starting in an invalid state, the product would still work, but if you tried to resize it with the mouse, you could sometimes get LibreOffice to hang. If you push software beyond its limits, bad things can happen.

I don’t even remember clearly how I found the place to fix, but by just reading enough code, I found the routine SidebarController::RestrictWidth.

@@ -1109,7 +1112,8 @@ void SidebarController::RestrictWidth (sal_Int32 nWidth)
const sal_uInt16 nSetId (pSplitWindow->GetSet(nId));
	pSplitWindow->SetItemSizeRange(nSetId,
- 	Range(TabBar::GetDefaultWidth() + nWidth, gnMaximumSidebarWidth));
+ 	Range(TabBar::GetDefaultWidth() * mpTabBar->GetDPIScaleFactor() + nWidth,
+ 	gnMaximumSidebarWidth * mpTabBar->GetDPIScaleFactor()));
	}
}

Getting on the 4.2.3 release train

Since it was a batch of new code to a stable branch, the LibreOffice Engineering Steering Committee had a discussion about it in one of their weekly meetings:

* HiDPI patches for 4.2.x? (Kendy)
    + LibreOffice does not look good on HiDPI screens at all - close to unusable
    + proposal to merge the HiDPI work to 4.2.x as a late feature
        + I would split the work into 'safe' and 'need real review' parts & push to a branch
        + can I get ESC approval / 3 independent reviews for that ?
    + is it a feature or a bug-fix ? (Michael)
        + most of the pieces going in enclosed in an if (hidpi) ...
            + checking that everything in the right if.
        + less safe part is checking / setting that flag; a small VCL piece.
    + Michael / Caolán signed up to review as/when there is a branch.

Once they gave their support, Kendy prepared for the patches for review.

Working with free software can be fun, but it can also be upsetting because problems can show up at any time. I was happy to see Kendy’s mail to the alias asking for review of the Gerrit patches, but not for very long because Norbert Thiebaud objected to them for the Mac. So after a flurry of emails over 3 days, we eventually got that resolved with another patch. So within a few days after that, the patches got reviewed and into the build for 4.2.3-rc1.

Windows

I was happy to see my efforts finally get into RC build, but I wasn’t happy for very long because this screenshot with clipped toolbar buttons showed up in my inbox:

This was upsetting for several reasons. The first is that it could have been found on master months before. Instead, it showed up on a day when I had my own things to work on. But instead of being able to focus on my tasks, I kept thinking about the toolbars, and what was I going to do about it?

I had been wondering for months what would happen when this code was tried on Windows. LibreOffice is created primarily by Linux developers, but Windows is the most popular OS for their users. My machine shipped with Windows but I had wiped it within a few hours and so couldn’t try it. I had used Windows for 15 years, but the last time was 9 years ago and I had no plans to go back.

I had envisioned various possibilities for what might happen on Windows, but cut-off toolbar buttons was not one of them. I also didn’t understand the toolbar layout code. I had written a toolbar manager before and didn’t want to volunteer any of my life on that boring problem.

However, I felt bad and that I should at least put some effort into trying to fix the problem. So that evening, I decided to just read through the toolbar code from beginning to end and see if I found anything obviously wrong. You don’t have to be very clever to notice a bloody knife at a murder scene.

And so I read through toolbox.cxx, including the toolbar layout code, and didn’t notice anything that stood out. So I next went to toolbox2.cxx. About halfway down, I found something suspicious:

/*static*/ Size
ToolBox::GetDefaultImageSize(bool bLarge)
{
    const long TB_SMALLIMAGESIZE = 16;
    if (!bLarge) {
        return Size(TB_SMALLIMAGESIZE, TB_SMALLIMAGESIZE);
    }

    OUString iconTheme = Application::GetSettings().GetStyleSettings().DetermineIconTheme();
    return vcl::IconThemeInfo::SizeByThemeName(iconTheme);
}

That 16 was a sign of a place that needed doubling. I didn’t yet know who used that function and whether it would make a difference, but opengrok helped me and found that it was called by the main toolbar layout routine.

So I made a change:

1738     // set defaults if image or text is needed but empty
1739     nDefWidth       = GetDefaultImageSize().Width() * GetDPIScaleFactor();
1740     nDefHeight      = GetDefaultImageSize().Height() * GetDPIScaleFactor();

However, I couldn’t tell whether it would fix the problem. The comment above the code, while happily in English, explained it was relevant only for empty toolbars. This was not a case I care about.

So I changed the code, and hoped the comment was wrong, but didn’t understand what the implication was. It was a change I would try if I had a Windows box. It requires much less mental effort to test a change than to manually prove what happens in a big function. As long as you can test something, you can postpone the necessity of full understanding.

Here is the main toolbar layout routine. I’ve highlighted in red all the references to the variable I changed.

bool ToolBox::ImplCalcItem()
{

    // recalc required ?
    if ( !mbCalc )
        return false;

    ImplDisableFlatButtons();

    long            nDefWidth;
    long            nDefHeight;
    long            nMaxWidth = 0;
    long            nMaxHeight = 0;
    long            nMinWidth   = 6;
    long            nMinHeight  = 6;
    long            nDropDownArrowWidth = TB_DROPDOWNARROWWIDTH;

    // set defaults if image or text is needed but empty
    nDefWidth       = GetDefaultImageSize().Width() * GetDPIScaleFactor();
    nDefHeight      = GetDefaultImageSize().Height() * GetDPIScaleFactor();

    mnWinHeight = 0;
    // determine minimum size necessary in NWF
    {
        Rectangle aRect( Point( 0, 0 ), Size( nMinWidth, nMinHeight ) );
        Rectangle aReg( aRect );
        ImplControlValue aVal;
        Rectangle aNativeBounds, aNativeContent;
        if( IsNativeControlSupported( CTRL_TOOLBAR, PART_BUTTON ) )
        {
            if( GetNativeControlRegion( CTRL_TOOLBAR, PART_BUTTON,
                                        aReg,
                                        CTRL_STATE_ENABLED | CTRL_STATE_ROLLOVER,
                                        aVal, OUString(),
                                        aNativeBounds, aNativeContent ) )
            {
                aRect = aNativeBounds;
                if( aRect.GetWidth() > nMinWidth )
                    nMinWidth = aRect.GetWidth();
                if( aRect.GetHeight() > nMinHeight )
                    nMinHeight = aRect.GetHeight();
                if( nDropDownArrowWidth < nMinWidth )
                    nDropDownArrowWidth = nMinWidth;
                if( nMinWidth > mpData->mnMenuButtonWidth )
                    mpData->mnMenuButtonWidth = nMinWidth;
                else if( nMinWidth < TB_MENUBUTTON_SIZE )
                    mpData->mnMenuButtonWidth = TB_MENUBUTTON_SIZE;
            }
        }

        // also calculate the area for comboboxes, drop down list boxes and spinfields
        // as these are often inserted into toolboxes; set mnWinHeight to the
        // greater of those values to prevent toolbar flickering (#i103385#)
        aRect = Rectangle( Point( 0, 0 ), Size( nMinWidth, nMinHeight ) );
        aReg = aRect;
        if( GetNativeControlRegion( CTRL_COMBOBOX, PART_ENTIRE_CONTROL,
                                    aReg,
                                    CTRL_STATE_ENABLED | CTRL_STATE_ROLLOVER,
                                    aVal, OUString(),
                                    aNativeBounds, aNativeContent ) )
        {
            aRect = aNativeBounds;
            if( aRect.GetHeight() > mnWinHeight )
                mnWinHeight = aRect.GetHeight();
        }
        aRect = Rectangle( Point( 0, 0 ), Size( nMinWidth, nMinHeight ) );
        aReg = aRect;
        if( GetNativeControlRegion( CTRL_LISTBOX, PART_ENTIRE_CONTROL,
                                    aReg,
                                    CTRL_STATE_ENABLED | CTRL_STATE_ROLLOVER,
                                    aVal, OUString(),
                                    aNativeBounds, aNativeContent ) )
        {
            aRect = aNativeBounds;
            if( aRect.GetHeight() > mnWinHeight )
                mnWinHeight = aRect.GetHeight();
        }
        aRect = Rectangle( Point( 0, 0 ), Size( nMinWidth, nMinHeight ) );
        aReg = aRect;
        if( GetNativeControlRegion( CTRL_SPINBOX, PART_ENTIRE_CONTROL,
                                    aReg,
                                    CTRL_STATE_ENABLED | CTRL_STATE_ROLLOVER,
                                    aVal, OUString(),
                                    aNativeBounds, aNativeContent ) )
        {
            aRect = aNativeBounds;
            if( aRect.GetHeight() > mnWinHeight )
                mnWinHeight = aRect.GetHeight();
        }
    }

    if ( ! mpData->m_aItems.empty() )
    {
        std::vector< ImplToolItem >::iterator it = mpData->m_aItems.begin();
        while ( it != mpData->m_aItems.end() )
        {
            bool bImage;
            bool bText;

	     // indicates if text will definitely be drawn, influences dropdown pos
            it->mbVisibleText = false;

            if ( it->meType == TOOLBOXITEM_BUTTON )
            {
                // check if image and/or text exists
                if ( !(it->maImage) )
                    bImage = false;
                else
                    bImage = true;
                if ( it->maText.isEmpty() )
                    bText = false;
                else
                    bText = true;
		 // default to toolbox setting
                ButtonType tmpButtonType = determineButtonType( &(*it), meButtonType ); 
                if ( bImage || bText )
                {

                    it->mbEmptyBtn = false;

                    if ( tmpButtonType == BUTTON_SYMBOL )
                    {
                        // we're drawing images only
                        if ( bImage || !bText )
                        {
                            it->maItemSize = it->maImage.GetSizePixel();
                        }
                        else
                        {
                            it->maItemSize = Size( GetCtrlTextWidth( it->maText )+TB_TEXTOFFSET,
                                                   GetTextHeight() );
                            it->mbVisibleText = true;
                        }
                    }
                    else if ( tmpButtonType == BUTTON_TEXT )
                    {
                        // we're drawing text only
                        if ( bText || !bImage )
                        {
                            it->maItemSize = Size( GetCtrlTextWidth( it->maText )+TB_TEXTOFFSET,
                                                   GetTextHeight() );
                            it->mbVisibleText = true;
                        }
                        else
                        {
                            it->maItemSize = it->maImage.GetSizePixel();
                        }
                    }
                    else
                    {
                        // we're drawing images and text
                        it->maItemSize.Width() = bText ? GetCtrlTextWidth(it->maText)+TB_TEXTOFFSET:0;
                        it->maItemSize.Height() = bText ? GetTextHeight() : 0;

                        // leave space between image and text
                        if( bText )
                            it->maItemSize.Width() += TB_IMAGETEXTOFFSET;

                        // image and text side by side
                        it->maItemSize.Width() += it->maImage.GetSizePixel().Width();
                        if ( it->maImage.GetSizePixel().Height() > it->maItemSize.Height() )
                            it->maItemSize.Height() = it->maImage.GetSizePixel().Height();

                        it->mbVisibleText = bText;
                    }
                }
                else
                {   // no image and no text
                    it->maItemSize = Size( nDefWidth, nDefHeight );
                    it->mbEmptyBtn = true;
                }

                // save the content size
                it->maContentSize = it->maItemSize;

                // if required, take window height into consideration
                if ( it->mpWindow )
                {
                    long nHeight = it->mpWindow->GetSizePixel().Height();
                    if ( nHeight > mnWinHeight )
                        mnWinHeight = nHeight;
                }

                // add in drop down arrow
                if( it->mnBits & TIB_DROPDOWN )
                {
                    it->maItemSize.Width() += nDropDownArrowWidth;
                    it->mnDropDownArrowWidth = nDropDownArrowWidth;
                }

                // text items will be rotated in vertical mode
                // -> swap width and height
                if( it->mbVisibleText && !mbHorz )
                {
                    long tmp = it->maItemSize.Width();
                    it->maItemSize.Width() = it->maItemSize.Height();
                    it->maItemSize.Height() = tmp;

                    tmp = it->maContentSize.Width();
                    it->maContentSize.Width() = it->maContentSize.Height();
                    it->maContentSize.Height() = tmp;
                }
            }
            else if ( it->meType == TOOLBOXITEM_SPACE )
            {
                it->maItemSize = Size( nDefWidth, nDefHeight );
                it->maContentSize = it->maItemSize;
            }

            if ( it->meType == TOOLBOXITEM_BUTTON || it->meType == TOOLBOXITEM_SPACE )
            {
                // add borders
                ImplAddButtonBorder( it->maItemSize.Width(), it->maItemSize.Height(),
					mpData->mbNativeButtons );

                if( it->meType == TOOLBOXITEM_BUTTON )
                {
                    long nMinW = std::max(nMinWidth, it->maMinimalItemSize.Width());
                    long nMinH = std::max(nMinHeight, it->maMinimalItemSize.Height());

                    long nGrowContentWidth = 0;
                    long nGrowContentHeight = 0;

                    if( it->maItemSize.Width() < nMinW )
                    {
                        nGrowContentWidth = nMinW - it->maItemSize.Width();
                        it->maItemSize.Width() = nMinW;
                    }
                    if( it->maItemSize.Height() < nMinH )
                    {
                        nGrowContentHeight = nMinH - it->maItemSize.Height();
                        it->maItemSize.Height() = nMinH;
                    }

                    // grow the content size by the additional available space
                    it->maContentSize.Width() += nGrowContentWidth;
                    it->maContentSize.Height() += nGrowContentHeight;
                }

                // keep track of max item size
                if ( it->maItemSize.Width() > nMaxWidth )
                    nMaxWidth = it->maItemSize.Width();
                if ( it->maItemSize.Height() > nMaxHeight )
                    nMaxHeight = it->maItemSize.Height();
            }

            ++it;
        }
    }
    else
    {
        nMaxWidth  = nDefWidth;
        nMaxHeight = nDefHeight;

        ImplAddButtonBorder( nMaxWidth, nMaxHeight, mpData->mbNativeButtons );
    }

    if( !ImplIsFloatingMode() && GetToolboxButtonSize() != TOOLBOX_BUTTONSIZE_DONTCARE )
    {
        // make sure all vertical toolbars have the same width and horizontal have the same height
        // this depends on the used button sizes
        // as this is used for alignement of multiple toolbars
        // it is only required for docked toolbars

        long nFixedWidth = nDefWidth+nDropDownArrowWidth;
        long nFixedHeight = nDefHeight;
        ImplAddButtonBorder( nFixedWidth, nFixedHeight, mpData->mbNativeButtons );

        if( mbHorz )
            nMaxHeight = nFixedHeight;
        else
            nMaxWidth = nFixedWidth;
    }

    mbCalc = false;
    mbFormat = true;

    // do we have to recalc the sizes ?
    if ( (nMaxWidth != mnMaxItemWidth) || (nMaxHeight != mnMaxItemHeight) )
    {
        mnMaxItemWidth  = nMaxWidth;
        mnMaxItemHeight = nMaxHeight;

        return true;
    }
    else
        return false;
}

So I looked through the code, but didn’t understand it and didn’t want to. However, it was easy to see the change was reasonable and helpful in some cases. So I submitted it to Gerrit to see if it could get it reviewed and approved. If I can sneak it into a daily build, then maybe I could convince someone to try it out. So I submitted and waited for input. 3 days later, Caolán McNamara of Red Hat reviewed and accepted the patch. In fact, Caolán appears to be an email alias that 3 developers are attached to. I think Arch is a better distro for me than Fedora, but I am very grateful for the useful investments that Red Hat is making.

Once my patch got into the daily builds, I emailed the tester asking he could try one out, but he replied that he didn’t have time. So I was stuck and frustrated again. I couldn’t know if the change fixed the bug, which made me uncomfortable to ask for a patch to be triple-reviewed and back-ported. Blind-fixes to stable branches are generally not a good way to work.

I also didn’t know how to submit changes to anything other than the master branch. The LibreOffice wiki is great for new developers, but didn’t cover that specific topic. So I decided to ask Caolán if he could submit them to Gerrit to get them in before the 4.2.3 release. Caolán did that and even sent me the Git incantations so I can do it in the future.

With some reviews from Norbert and Miklos Vajna, the patch got into the 4.2.3 branch. So I was very happy, but not for very long. Because I soon noticed that the final 4.2.3 RC build had already been made. You can still push changes to the Git branch, but it doesn’t matter, the digital equivalent of air guitar. It is possible to mark bugs as release critical and delay the release, but this didn’t meet the bar. I should just have been happy that the issue was likely now fixed, but instead I was upset that after all my stress it had missed the train by a few hours.

However, the Heartbleed bug and a few other important ones showed up, and so another RC was made. I’m probably the only person on the Internet, other than the NSA, who was happy for Heartbleed.

KDE Regression

Just as LibreOffice 4.2.3 was shipping, another bug showed up from a KDE Ubuntu user:

Someone with a 1920×1080 15.6” monitor was seeing the HiDPI mode kick in. This is a bad bug because it is a regression. The goal of the feature is to improve the experience for HiDPI users, not break it for everyone else. Degrading a product for other people is the fastest way to get your code reverted.

This would have been stressful for me, but several weeks earlier I had studied the Linux DPI detection code and multi-monitor support. Since I knew exactly what 4.2.3 was doing here, I didn’t worry about being able to quickly solve the problem. I just needed to figure out what data LibreOffice was getting from the OS. You can stare at code as much as you want, but if you depend on hardware-specific information, you can’t prove it correct until you test on other computers.

I had taken the time to learn the code because I submitted a patch to LibreOffice for 4.3 that simplifies it to only fetch from xrdb and never bother to fetch from X Windows. I found the information unreliable for my laptop. On my machine, X tells me it is 96 DPI on a 33” by 18” monitor. It is pretty impressive to squeeze all that into a 13.3” screen. The patch to ignore X isn’t in 4.2.x, but that wasn’t a problem because the hard part is understanding the code. The tester was very helpful in quickly giving me the information I needed.

The problem is simple to describe. The monitor was 141 DPI, but X said it was 139×144. Of course it is crazy with bad data of different DPI values in the X and Y direction, but that was irrelevant here. The issue was that LibreOffice’s doubling kicked in at 144 DPI in the Y direction:

mnDPIScaleFactor = std::max((sal_Int32)1, (mpWindowImpl->mpFrameData->mnDPIY + 48) / 96);

144 + 48 == 192 / 96 == 2

I could think of several fixes, but I wasn’t sure what was best, so I decided to ask Kendy who had written that line of code. Within a few days, he submitted an improvement that will not cause this new mode to kick in until at least 168 DPI. That fixes the problem for this machine, and hopefully others. So with these fixes, things are in decent shape. The next issue is Unity, which is currently broken.

Unity

While LibreOffice can look good with Gnome, KDE, and Xfce, I hadn’t tried Unity. It is difficult to get it running on Arch because it has patches to a lot of key components they’ve not convinced upstreams to accept. And so I’d have to replace a lot of system packages, and I didn’t want to deal that risk for my personal laptop.

Unity isn’t even the only environment Ubuntu supports, but it is their premier one and so I had been curious. The Unity team had done a bunch of HiDPI work for version 8, and so I hoped it would work. Ubuntu 14.04 is shipping 4.2.3 with the LibreOffice icons prominently placed on the dock.

After hearing nothing for months, I tried out a live USB image of the final Ubuntu 14.04. Unfortunately, I discovered that even though they based some of their work on the Gnome 3.10 design, they didn’t fix the xrdb values. I could also find no way to force apps to be 192 DPI, as the other environments enable. Apparently there are new Unity APIs.

I wish Unity would fix their xrdb values. I don’t know where the Unity documentation is or how to write Ubuntu-specific code on LibreOffice. I recommended to Ubuntu’s developer, Bjoern Michaelson, to requisition a new machine. For now, it doesn’t work on Unity and the fix is unknown.

The end, for now

This story ends, but there is plenty more that can be done. It would be great to have higher-resolution toolbar bitmaps, but it isn’t high priority because only a few are pixelated. The Mac is still broken. It appears that the OS doesn’t work in pixels on retina displays. Good luck fixing that.

The splash screen is a little embarrassing, but is is only visible for a moment. I found the relevant code in splashx.c, but didn’t find an easy fix because this early code apparently can’t use the BitmapEx class with its scaling routines. In fact, it often draws bitmaps pixel by pixel. Maybe someone can just make a bigger splash screen and fix the progress control.

The status bar still needs to be improved. The bitmaps look fine but because the layout is done in pixels and stored in an XML file, there is no way currently to make it be different widths for normal and HiDPI screens.

The hyperlink dialog box has a few problems:

HyperlinkLO423

The Insert-special characters dialog needs to be taller by default. The letters are all much too small. I had spent some time looking through the code but never managed to find the place to make a fix. Fortunately, the dialog box is resizable so this isn’t a big problem.

It also needs testing on Windows 7, although from my casual re-reading of the MSDN documentation, it should work.

The underline wave character property isn’t scaling yet. I didn’t bother with that because it is very little used compared to mis-spellings, and is a more complicated codepath.

The line styles toolbar dropdown preview draws lines too thin. I looked around the code, but it was very voluminous. Kohei Yoshida spent a couple of weeks re-working it recently so I’m hoping when he gets a new laptop, he’ll quickly notice and be able to fix it.

It doesn’t appear that multiple monitors of different resolutions is handled properly in LibreOffice. Fixing that could be tricky. So the work continues!

LibreOffice HiDPI Patches

I bought a HiDPI laptop in October to replace my 5-year old Thinkpad. Between the 5.7 million pixels, and the bright LED backlight replacing my dying and dim CFL bulb, it makes the daily computing experience much easier on the eyes. I’d put up with a lot for this screen. It turns out I have to compared to my old Lenovo, as there is an incompatible and inferior keyboard layout, the Synaptics mouse drivers are flakey, it is difficult to replace the battery or hard drive, etc.

I run Arch with Gnome 3.10 in Classic Mode. Gnome is the only DE that recognizes the high-res screen out of the box and picks a good font size. It has specific bugs I described in my review of the laptop, but it generally looks fine albeit bland and crippled. Firefox needs the No-Squint plugin, but then the web looks acceptable. The next most noticeable problem for me was LibreOffice. The text and dialogs looked beautiful:

On Retina Macs, LibreOffice and other apps by default run in a compatibility auto-doubling mode until an application flag is set to tell the OS to unleash the true resolution. On Linux (and Windows 8.1), the text looked like a magazine, but the bitmaps were far too small, and there were other problems.

LibreOffice offers 16×16 or 24×24 toolbar icons today, but the small ones looked like dead bugs on the screen, and even the large ones required concentration to recognize. One day, rich 48×48 toolbar bitmaps can be created, but in the meanwhile, you can double them to look reasonable and be easy to click on. Given that LibreOffice has hundreds of bitmaps, this temporary solution could be useful for a while.

In addition to the toolbars, the status bar controls, the navigator, the sidebar, and various dialog boxes had tiny bitmaps. One of the most annoying problems was that the spelling underline was so thin you didn’t notice it while casually looking at the screen. And so you had to stare carefully to see the lines, which imprinted them causing remnants when you shut your eyes. It was maddening!

And so I thought: perhaps some volunteer developer in LibreOffice could be given new hardware by The Document Foundation to work on this problem. The Gnome 3.10 HiDPI work happened because of a computer donation and perhaps it could happen here. And so I wrote an email to their discuss alias asking if there was interest but I received no public response.

Apparently, everyone is so busy delivering a new product, fostering a young community, paying down technical debt, making it run on Android, improving import and export, rewriting the Calc engine, removing Java, etc., that no one has time to make it look good on these beautiful screens. There is a lot happening without any rich benefactor anymore, and a split community. If you think LibreOffice is amazing, just imagine what it would be if IBM gave them $10M / year, and the trademark, and didn’t seduce away naïve volunteers and donations. (I believe if IBM were to ask Watson whether it should end the fork, the AI would recommend it. Watson is only being applied to customer problems instead of their own. One could spend a lot of time correcting the inaccurate FUD written on the AOO dev alias. Imagine we lived in a society that celebrated divorce instead of marriage.)

I considered setting up a bounty on a website, but I decided to next see if I could find someone in the community with expertise who could be persuaded to work on this issue. It seemed that fixing the toolbars would be a great first step and I felt I’d be happy with that one improvement.

So I next emailed Michael Meeks and asked if he knew anyone who might have a few extra hours. I was put in contact with Andrzej Hunt, who eventually wrote a patch to double the toolbar bitmaps. But I felt that the spelling underline was the next highest priority issue. I decided to go through the codebase, and see if I could find the place that should be fixed. If I thought I could do it, I would buy an external SSD to build and do some experiments.

My first stop was opengrok.libreoffice.org. It is a simple and fast way to search through the codebase. This is one of best tools LibreOffice has setup since they started. Even when you have the code on your local machine, it is better to use their smarter system. The Opengrok source is color-coded and everything is clickable to take you to the definition and references of classes and functions.

I don’t remember exactly how many searches it took, but eventually I found the code to draw “wave lines”. If I had known that term in advance it would have been faster, but I always thought of them as squiggly underlines and that term brought up nothing. Eventually I found interesting routines such as DrawText, ImplDrawTextLine, ImplDrawWaveLine, DrawWavePixel, DrawWaveLine, etc. One usually reads code for understanding, but at first, I was just trying to figure out if the code I was looking at was relevant.

LibreOffice draws wavy underlines as a font property and as part of spelling / grammar errors, and for other reasons, so I had to read through the code, understand who called who, etc. Eventually, I discovered that DrawWaveLine was the function I wanted. It was called from the end of the main routine DrawText, which called lcl_DrawLineForWrongListData, which went through the grammar and spelling error list and called DrawWaveLine for every problem area. And in that routine, I found a strong clue:

5302 if ( nStyle == WAVE_NORMAL )
5303 {
5304     nWaveHeight = 3;

I knew the spelling underline was 3 pixels tall, so at that point I knew my search had ended. I decided to spend money on an external SSD so I could download the code, build it, and make changes.

The wiki is superb and the build process is ridiculously easy. Here are the commands I ran after installing the build dependencies:

$ git clone git://anongit.freedesktop.org/libreoffice/core libreoffice
$ cd libreoffice
$ ./autogen.sh --enable-dbgutil --without-java --without-help --without-myspell-dicts
$ make (wait two hours)
$ instdir/program/soffice --writer

With those few steps, I could then run LibreOffice and try things. After a few builds, I found what looked best. Here is the first diff I sent to the dev alias:

diff --git a/vcl/source/gdi/outdev3.cxx b/vcl/source/gdi/outdev3.cxx
index f3f5a77..6e142fd 100644
--- a/vcl/source/gdi/outdev3.cxx
+++ b/vcl/source/gdi/outdev3.cxx
@@ -5301,9 +5301,12 @@ void OutputDevice::DrawWaveLine( const Point&
rStartPos, const Point& rEndPos,
long nWaveHeight;
if ( nStyle == WAVE_NORMAL )
{
-        nWaveHeight = 3;
+        nWaveHeight = 5;
         nStartY++;
         nEndY++;
+
+        nStartY++; //Shift down additional pixel for hidpi screens
+                   //TODO: Probably should be done above, before rotation happens
}
else if( nStyle == WAVE_SMALL )
{
@@ -5320,7 +5323,7 @@ void OutputDevice::DrawWaveLine( const Point& rStartPos, const Point& rEndPos, nWaveHeight = pFontEntry->maMetric.mnWUnderlineSize;


ImplDrawWaveLine( nStartX, nStartY, 0, 0,
-                 nEndX-nStartX, nWaveHeight, 1,
+                 nEndX-nStartX, nWaveHeight, 2,
                  nOrientation, GetLineColor() );

if( mpAlphaVDev )
         mpAlphaVDev->DrawWaveLine( rStartPos, rEndPos, nStyle );

Note this code couldn’t possibly get checked into LibreOffice as-is because it changes the behavior for all screens, but it allowed me to test. Eventually what is needed is a way to know when to use the different values but there was no easy way to get that information. And so Michael Meeks introduced me to Kendy (Jan Holesovsky) who added a member to the low-level OutDev class containing the DPIScaleFactor. In principle, that information is based on the DPI of the screen / window, but sometimes the OS just returns 96 for compatibility reasons and therefore may need additional logic based on the system font size.

With the toolbar and spelling underlines looking great, it made the remaining problems more noticeable, and motivated me to keep going. I decided to next look at the status bar. The first control was zoom, the one I use the most. Eventually, I found the class: SvxZoomSliderControl. In this case, I was able to just read through the 400 lines until I saw the problem areas. For starters, it had hard-coded the bitmap size via #defines, so even if you plugged in a bigger bitmap, it would have mis-behaved. And so I first fixed the control to work based on the bitmap size, and made it draw bigger if necessary. The change was quite simple. From there, I went to the other status bar controls, the navigator, and various other places, and have so far submitted a first and second round of fixes.

The process was straightforward. First I’d find a problem visual area I wanted to work on, and then I’d trace through the code in opengrok till I found the right place, make a change, and see how it looked. I saw plenty of code I didn’t understand, but my changes were mostly simple and safe. A couple of times I had to first put in printf diagnostics to understand what was going on at runtime because there is no supported IDE with debugging, auto-complete, etc.

Sometimes hours of searches turned up nothing! I felt like Indiana Jones digging in the desert. After lots of sweating and effort, you find things, but it can take time to figure out if an artifact is valuable, how it fits with other pieces, etc.

The hardest piece of code to find was the black triangle in the toolbar font color dropdown, and many other places. I eventually resorted to commenting out drawing logic that looked like it might be handling it. At one point, random controls all over the product weren’t visible anymore, but the dropdown indicators were still there. It was the Terminator of Triangles. The LibreOffice codebase is overall very reasonable compared to Microsoft Office, but it does have a lot of widget logic because it tries to draw native visuals in a cross-platform way.

I emailed the dev alias and heard nothing. That wasn’t a surprise because the people who wrote the toolbar code are long gone. This happens more often in the proprietary world. Eventually I gave up and decided to work on the toolbar double-arrows which show up when some of the buttons don’t fit. That routine I found quickly by reading the code that laid out toolbars, figured out how it represented the situation of controls not having enough room, and then found the drawing routine which looked at that data.

And so I changed the arrow code, and started a build. While waiting, I decided have a look around, and found that triangle drawing code was the next function in the file. I about pooped my pants. I had spent at least 4 hours looking for it.

Reading through the code to find the right place to make the change took an unknown amount of time. Once the right place was found, the diff was mechanical, but usually there was something interesting about each case. Once you find the right function, the rest is usually easy, assuming what you want is possible via existing primitives. So far I’ve spent about 40 hours, working a few hours per session, and have fixed the most noticeable places in the product including the Sidebar.

Complexity and community

HiDPI is quite far from finished, in spite of the many changes I’ve made. Most of the code I’ve worked on will eventually be replaced! We shouldn’t be doubling bitmaps, we should have better ones. The toolbar dropdown triangle logic should eventually be replaced with code which calls into the toolkit / OS so it also looks more native. That is a better fix, but also bigger and riskier, and could have implications for toolbar layout. And I have no idea how to do that. I’m much better at telling LibreOffice to double integers and bitmaps.

There are more issues, but also more and more people who will want LibreOffice (and other apps) to work well on their new screen. I’ve been updating a wiki page as I go along and it also lists some of the remaining places. One isolated fix is the tab-drawing (and hit-testing) code in the ruler. The drawing logic starts on line 874.

On the Mac, someone needs to turn off the compatibility mode and see what happens. In my old life, I would have expensed a Macbook Retina to test it myself, but for now LibreOffice needs to find someone else to carry the work further. Unfortunately it seems there is no current developer with a high-res Windows or Mac. I hope someone who is passionate about helping make LibreOffice look good for those others OSes will soon show up. If you are a Mac or Windows capable developer and have a HiDPI screen and you would like to jump in like I did, contact me and we can work on it together.

This feature did not make it into LibreOffice 4.2.0, so it won’t get any of the automatic publicity that would come with that process, but I believe it can get into 4.2.1 and be announced then. LibreOffice make a minor release every month, and Arch automatically grabs them within a day or 2, so it means I’ve only got to wait an extra 32 days. That I could write some code in late December and get it on my machine in early March is quick turnaround compared to the proprietary world.

If I were an undergraduate in college, I’d take a bug somewhere in free software and work on it. LibreOffice is one of the most needed and relatively easy to get into. You have to look pretty hard to find ways to improve the Linux kernel, but with LibreOffice, that is not the case. A number of the things I worked on in school were writing toy programs. It made it easier to grade the assignments, but the code wasn’t realistic or useful to anyone. There also wasn’t as much great free software out there to learn from or contribute to. I joined Microsoft to learn more about programming because I knew there was a lot more to it than the toys I was making. LibreOffice is a very rich program to fully understand, but you don’t need to know very much to be productive. Most diffs are 5 to 50 lines.

One of the ongoing challenges for LibreOffice is to remove as much extra complexity as possible. It has already paid down a large amount of technical debt in the makefiles and other places, but this process will continue for a long time. I think that parts of Base should be re-written in Python, and the VCL replaced with WxWidgets or Qt, but these are very large tasks no one is working on, and therefore just pipe dreams. The good news is that they can be done incrementally. Porting the status bar to another toolkit could be done in a summer by a college student. The toolbar would be a bigger challenge that could take two summers. There is little work happening in VCL or the Writer layout engine. One of the downsides of the Symphony / Apache Sidebar is that it gave LibreOffice a new UI, but not a simpler and better way of building pretty UIs.

I received excellent “customer service” from Kendy along the way. The LibreOffice community is great and patches from first-timers get accepted all the time. So while we wait for IBM to change course, LibreOffice can keep doing the best that it can. LibreOffice has 100 dev contributors per month. Even if people show up to fix one real bug, it collectively adds up.

I plan on doing more, but first I want to help in the process of releasing the code already checked in. I wrote about some of the remaining issues in the wiki, but people might find others. For example, the first batch I submitted changes the width of the SvxPosSizeControl for normal and high-res screens. It is broken on current builds as the text doesn’t fit, and the bitmaps drew on top:

The status bar is one of the few places in LibreOffice that does layout based on pixels, and the resource XML files are not a place you can currently change at runtime. The good news is that there isn’t a lot used in the Draw / Impress status bar so wasting 200 pixels on low-res screens might not be a real problem. I also changed the width of the style and font size dropdown for all screens, shrinking one by 33% and the other by 25%. I wait to hear of complaints!

A competitive office suite is critical to the success of a free desktop. There are billions of Office files out there and people who haven’t even heard of the best free alternative. LibreOffice is an extremely rich codebase, but it is more underfunded for its needs than the kernel, Firefox, systemd, etc. If someone can understand my diffs, they can contribute some code to LibreOffice. I note that the author of the article saying that LibreOffice is “ridiculously easy to build” never actually submitted a patch! People can contribute out of duty or selfishness like me, but it is best to find an isolated area or an issue they care about. There are plenty of Easy Hacks and bugs to read through, and various ongoing projects such as dialog format conversion.

Response from Synaptics regarding Linux drivers

Here is the mail I received this afternoon from Synaptics regarding the petition:

The open source Linux kernel currently has a PS2 touchpad driver from the third party open source community which was not developed, reviewed, or tested by Synaptics.  Based on industry and customer pull, we do not support PS2 interfaces for Linux as it is not a strategic fit for our roadmap or support model. In general, the trend in the PC industry is HID/I2C, for which we do offer driver support.  Our intent is to submit our HID/I2C driver to Kernel.org as time, resource allocation and customer project prioritization allow.

Best Regards,

Nick

———–

Here is what I sent in response:

It would be great if you were to get working on submitting a driver to the kernel. People will help you find bugs in the code as part of this process, and improve it in ways you hadn’t considered. It occurred to me over Thanksgiving after your last email that writing a Linux driver but not freely releasing is like cooking that big meal and not eating it. Proprietary software was one of the big mistakes of the baby boomer generation.

I will pass this information along to the petitioners with some questions / comments. When you make something public, I hope you make a public announcement as well because it will be read more than your average one. People care about companies that share their values. Here is a video of Steam’s recent announcements regarding Linux that I thought you might appreciate: http://www.youtube.com/watch?v=g8PMUvuHK4g

Unfortunately, I’m not knowledgeable about the benefit of supporting the PS2 interface vs HID/I2C. However, I can say that not every trend in the industry is a good one. IPv6 is a trend that still hasn’t arrived. I think UEFI as implemented is of dubious benefit. Can you please explain in more detail why HID/I2C is better? I may do some more research on this topic and ask other people, but I can say that the existing venerable standard has gotten us a long way including multi-touch and gestures. So I wouldn’t recommend supporting only the new trends which are sometimes fads. The Linux kernel supports 60 file systems. Can you consider to support the extremely popular as well as the latest?

When you think about getting code into the kernel, I want to emphasize making good default configuration values. You didn’t mention that aspect so I raise it again in hopes of a response. I’m sure your driver will have interesting knobs people will want to tweak, but you are the best people to know what they should be by default for your devices. So can you please commit to providing a great out of the box experience by including tested and tuned configuration data along with code?

2. Getting 10,000 lines of code into the kernel could take a year. For example, it took quite some time for Google’s wakelocks code to make it upstream. In that case, there were big disagreements about the design which may or may not apply to your code, but unforeseen things like that do happen. Also, your lawyers end up taking time and I’m sure your kernel developers are already busy and possibly understaffed. As this is your first big contribution it will take even longer.

So in addition to the question of whether supporting only one standard is good enough, there is a question of timeframe. That is why I created the #2 item. I realize you didn’t write or review any of the existing code. However, because you hadn’t released any code publicly in the past other people have written it for your customers. The kernel is filled with code written by other people but this module has your name on it and is depended on by millions of people.

Here is the key point. By fixing that code, you will help many customers today. A 10 line fix to a driver would quickly be accepted and deployed to millions within a few weeks and the entire community within 6 months. It would possibly be backported to older releases as as your hardware is so popular and important.

I read a line that a good way to judge an airline is by what happens *after* they lose your luggage. If they were to tell you that yes, they’ve lost your luggage, but they are implementing a new baggage tracking system that will give it to you in a year, you might not be so happy with that answer.

I believe if you were to ask your Linux customers whether they would want you to fix 20-30 bugs in the existing already-upstreamed code or work on some big chunk of new code whose date for delivery is very uncertain, they would want you to fix the existing code first. So this strategy is faster for us and cheaper for you and could even be a useful learning process on the way to the HID/I2C utopia. Once all your customers are no longer frustrated you can run the process for your new driver at your leisure. So can you re-consider your priorities in light of the situation that currently exists?

3. Also, you didn’t mention the #3 issue about providing a UI to configure your devices. Can you please think holistically about the end-user experience and realize that as your hardware gets more sophisticated, a UI to configure the device becomes more necesssary? You could provide a good value for palm detection by default, but a UI allows me to test and change it if the default did not work for my situation. It is quite difficult to tweak a knob without a UI.

In this code as well, people will help you here, by for example suggesting gestures to be added or turned off by default. By not working on freely-available configuration code, you are missing feedback loops everyone will benefit from. It is overall easier to write UI code on Linux versus Windows. There are more Environments / toolkits, but only 3 are used by 90% of users, and the code will be debugged for you, compiled, deployed and translated for you, etc. You don’t need to start from scratch as these environments have code, but it just needs to be improved. Can you please reconsider this end-to-end aspect as well?  It is also easier to find people qualified to write UI code versus drivers and I think most people would judge this to be more important than a HID/I2C driver.

Thanks again for your mail. I hope you will consider these issues and respond soon. It is great to hear of a new path, but I can recommend you verify it is what your customers actually want as opposed to what some people say they ought to want. I will post a request for feedback on your ideas, but these are my initial thoughts.

——

Please post your thoughts about this, preferably on the petition page.

Arch Linux HiDPI wallpapers

Here are a couple 3200×1800 wallpapers I rendered in Blender using the new Cycles engine based on a file I found.

Default:

Change to glass material:

Enjoy!!

Manjaro, Arch, and Debian

I’ve been running Debian-based distros for my first 8 years with Linux, but I decided to try Arch for my new Lenovo Yoga 2 Pro. Most recently, I was running Mint-Debian aka LMDE, but with this new hardware, I wanted to be more up to date with the latest software. For example, LMDE is currently using Gnome 3.4, whereas the latest is 3.10. LMDE also isn’t really rolling in the sense that Debian-Testing or Arch is, as new things only arrive about 3 times per year. Therefore, between the time to get into Debian, and the time to get the new batch of updates from LMDE means it can be many months after I’d read a news article about the new LibreOffice features before those bits would actually arrive on your computer. In fact, by then I might have forgotten why I was excited about their work in the new release.

I installed Arch but considered Manjaro also. The big benefit is a graphical installer:

Not everyone is comfortable in a command-line, but I believe everyone should be. Today, however, many people, especially those who came from Windows or the Mac, have not yet learned it, and so Manjaro is doing a valuable service to improve the initial out of box experience. There is plenty of time to learn more about how your computer works once it is installed and configured. It is surely possible to take the necessary steps, and the best practices in the wiki, and put them together into a friendly and beautiful installer.

I haven’t tried Manjaro so I can’t say how it would have worked for me. However, given the hardware problems I ran into such as needing to use rfkill, setting the acpi_backlight kernel command-line, needing to tweak the Synaptics configuration to make the mouse close to usable, etc., it would have been almost as difficult for me even with the work they are doing to lower the barrier to installation on this new hardware.

I had to know how to tweak GRUB, systemd, and other configuration files, so I may as well set it up. I hope Linux one day doesn’t require anything tricky to get it working out of the box, but that is not today.

Manjaro has taken the mature Ubuntu installer and ported it over to the Arch system. This is valuable to help new people getting into Linux, but assuming it works, the benefits over running Arch on a daily basis drop to near zero. In fact, running it is possibly less useful than Arch, a topic I’ll discuss later. Note that you can’t switch to the Arch repos after you install Manjaro because your computer might not boot.

Manjaro does provide benefits after installation, but they are of much less value. For example, they have added graphical notification of new updates:

Arch by default handles updates only on the command line, and has no GUI to prompt users when new updates are available. This is a nice but very optional feature as the alternative is just running the trivial command pacman -Syu.

Some things I disagree with about Manjaro. I don’t understand why they have their own repositories with a copy of all the packages when they are mostly focused on the problem of installation. Shipping a different installer entry point into the Arch world doesn’t require making a copy of everything. Manjaro might claim to gain benefits from having having separate repositories, but they are changing only a few packages so the improvements are tiny.

Arch is a bigger team, and so I trust them more to have the latest versions available with important security fixes, etc. Keeping up to date with the latest and best software is an ongoing and time-consuming process. It is the lifeblood of a rolling release. I don’t want to run something that isn’t a part of Arch in that important way. Manjaro should enable users running out of the Arch repository to not lock people in.

Manjaro takes released versions from Arch and runs additional tests on them. By doing this, they can claim to be more stable. However, if the underlying problem is that Arch is sometimes unstable, doesn’t that mean it just needs more people running the test builds? Why is waiting to start further testing a good idea?

Even consider the case that sometimes useful features don’t get enabled in the Arch packages. For example, if Manjaro wanted to offer x86 kernels with PAE support for machines with more than 4 gigabytes of RAM, they can setup their own respository with just that package. The cost to mirror and distribute a few packages is much smaller.

Or, people interested in doing the PAE work could just help out the Arch kernel developer who is probably very busy. It is easy to see problems as a reason to create something better and new. But a fork is social engineering and to be taken with care. I also don’t understand why Manjaro have their own wiki. It seems like it should just be a a few pages on the Arch wiki. The wiki is one of the best features of Arch, why would you start over rather than improve what is already there?

As a side note, Antergos is another (top-30) Arch-based distro with its own simple graphical installer but at the end points to the Arch repositories. For some reason, this team and community is less well-known than Manjaro. I don’t understand why so many small groups of people think they should make an OS. At Microsoft, a team of 4 developers would be maintaining a codec.

Mike Conlon has written papers about forks in software. In short, he said they should happen when people can’t get along. While we celebrate the diversity that comes from freedom, we shouldn’t always consider forks as a sign of something positive. Imagine living in a world that celebrated divorce instead of marriage. The benefits of community come with size and division of labor.

I don’t know much about the community of Arch. I have long respected Debian for its good people, generality, stance for freedom, sense of community, and vast institutional knowledge. The DPL election process itself encourages civil dialog about how things can improve. However, at some point with the kernel, systemd, Firefox, etc. the OS itself is just the tool to compile and distribute all the other components.

How Long Here?

I have run Arch for 6 weeks. I don’t know how long it will last, or whether I will find sufficient reason to leave. My review documented all the problems I ran into, but none are Arch’s fault. Put another way, I found no real problems in their package manager, compile and install scripts.

Some have left Arch and before I installed, I read reviews and articles to understand what the reasons were.

I discovered various different causes but I don’t think any of them will be a problem for me. One of common reasons was that there have been some transitions in the past that were disruptive for users: for example, the switch to systemd broke some systems, but there were others as well. Even if the users can fix the problems, the sum of the inconveniences of the years can leave a mark.

The switch to systemd was cited several times, and some thought it was bad idea being forced on them by a cabal. I personally believe it is important technology modernizing low-level usermode aspects of the OS. The kernel is a great piece of technology, but at the end of initialization, it starts one user-mode process and then waits for work. The kernel doesn’t even write its error messages to disk, just to a memory buffer. Such policy decisions as where to store the files and what format are left to usermode.

There are multiple free init systems out there, and so it is hard for groups to come to consensus about whether to switch and which is the right one. Arch because of its simpler social structures and smaller team, doesn’t have the open debate process like what Debian is currently going through. However, it doesn’t really matter how you came to that realization as long as it was the right result!

In most cases, Arch can provide multiple choices such as whether to run OpenBox, KDE, or Gnome. Supporting multiple init systems was a choice they did not provide, but that was also a good decision. Via the configuration files and command-line utilities, you can tweak a system in innumerable low-level ways. The idea that you gain any further useful customizability via multiple init systems is incorrect.

Arch was relatively late (mid-2011) in supporting package signing. (The benefit of this feature is that you can independently verify that a package came from Arch, no matter whether you download it from their servers, or a mirror, or any other place on the Internet.) Some left Arch, or threatened to, because it didn’t sign packages.

But at the same time, you need to be the type that wears a tinfoil hat to believe that anyone would secretly disrupt Arch or its mirrors. If the NSA wanted a secret hook, they’d put a gun to Linus’s head. They wouldn’t bother hacking Arch as it would take too long to accomplish anything. It was one of those features that is a good idea because it adds end to end security, but was only ever a theoretical problem for Arch users.

In any case, Arch now has signed packages. It was simply a lack of resources and more people complaining than programming. Arch is run by people who see what other distributions do and can learn from them. You don’t need a formal social organization to be able to learn from the good ideas of others.

Because Arch doesn’t have elections, release parties or yearly conferences, it doesn’t have as much a sense of community. Some people left Arch because they found too much elitism and rudeness. I experienced it personally already from one of the maintainers.

Upon filing a bug, I was told that it would have been easier for me to create the package I wanted than to keep discussing whether it even should be packaged. Of course, making your first package is much harder than installing it, just like making a cake is much harder than eating it. I told him that he should have just added the package like every other distro has, rather than keep coming up with invalid reasons not to, but that wasn’t convincing ;-)

On Reddit, Arch users commented of my previous review that it was my fault for expecting it to work like Ubuntu, and that I deserved no “sympathy” for any issues I ran into. One fan of Arch even told me I shouldn’t be running it! I have been running Linux for 8 years and working with the command line since he was in diapers.

Of course, judging a community by individual emails and Reddit comments is scientifically invalid. Insanity in various forms is currently a pre-existing condition of society, and I don’t have reason to believe it is worse in Arch than Washington, DC or Los Angeles.

But the distros that have a mentoring process and such will find that it removes a distinction between users which are to be looked down upon, and contributors who are the masterminds. Every contributor was initially a clueless user and some distros even have a mentoring process.

Arch has a process for moving from user to contributor as well, but it is much more about demonstrating past contributions, and eventually given more upload permissions. It also has a user repository area (AUR) that requires no special rights to become a part of the system.

And in the AUR, they have a mechanism which allows people to vote on packages, and the most popular can be included in the main repository. These feedback loops are great forcing functions to make sure the developers listen to users. Bugs can also be voted on, to help developers prioritize and make the system more democratic. Even though Arch is missing some of more refined social structures, it has its own.

I’ve also read about how some people left Arch because they had multiple breakages over time and lost trust. However, there are wiki pages that explain how to handle the situation and best practices for system stability. Almost always the problem is a mis-configured system, or a bug in the upstream code. I have two kernels installed, the mainline (3.12.3) and the LTS (3.10.22) version. So if a kernel upgrade breaks, I can always boot to a known-good one. Because Arch doesn’t have the resources of the big vendors to debug kernel problems, it can be nice to have an extra as backup.

That doesn’t solve many other ways the system can break before it finishes booting, but it is nice to have. It is a fact of life that it is very easy to break an OS. Death is the norm in this world. I’ve read stories of people who accidentally configured their computer to sleep in response to the system start event.

The community of Arch is large enough that users run into all kinds of problems that more slowly moving distros would never see. Of course, Arch isn’t large enough to debug every problem but they have the resources to fix the bugs that “everyone” runs into very quickly. There is safety in numbers.

Many of the hardest transitions in Arch are in the past. I expect it to be smooth sailing now. I can think of little ways to improve it, but given that it is running all the latest software, I have no reason to install anything else right now.

Arch is under the radar compared to Ubuntu, Fedora, Mint, Debian, Suse, etc. It gets less than 1% of the news that those others get. It doesn’t have benevolent dictators making hardware deals and memorable pronouncements. It doesn’t have billion-dollar companies supporting them. Somehow, they’ve managed to survive.

The free software community is still surprising to me. 50% of computer users know of Linux. 10% of them know Ubuntu. 10% of them know Debian. 10% of them run Arch. 1% of them contribute to Arch. There are many good distributions and working on them. When you consider it is only on 1% of desktops, it is astounding how many solid choices there already are. Arch is the secret anarchic distro that works perfectly.

Petition to Synaptics Corporation regarding flakey Linux drivers

I am writing this on my new Lenovo Yoga 2 Pro which comes with a beautiful screen and your Synaptics Clickpad. I have created this petition because there are a lot of problems with your hardware on Linux that are frustrating, time-wasting, and distracting. Right now, the pointer moves when it shouldn’t, and doesn’t move when it should, which is pretty close to the worst-case scenario for a mouse.

I’ve been using your Trackpads happily for years, but with your new model, things are such a mess that they get in the way of my ability to work. What if we users added up all the time we have collectively wasted and sent you the bill? Given it already has about the same marketshare as the Mac, the total number would have a lot of zeros. With the time I’ve wasted, I’d sign up for $200 / year in any class action lawsuit.

I know you spend most of your effort working on the Windows drivers, and that is fine because it is cheaper to maintain Linux drivers. The community will help you. You’ve written a proprietary Linux driver, but you never made it available. Why write something almost no one uses? There is a free Synaptics Linux kernel driver written for your users, but it is buggy and you haven’t ever helped on it.

I wrote a full review of Arch on my new hardware talking about the problems with your mouse in more detail. The point is that your new devices with an integrated button require smarter software to be usable. Well, now that you’ve pushed this new hardware of dubious benefit and definite downsides, what are you going to do?

Synaptics should respect our freedoms, and:

  1. Release a copy of your proprietary Linux driver under the GPL (or public domain) and publish it somewhere where it can be used as a reference to minimize the need for any further reverse engineering.
  2. Find a kernel engineer from somewhere in your 550-person company and have them help maintain the existing in-kernel Linux driver. Aside from the obvious bad behavior you will notice when you install any distro, there are also a number of old bugs in the kernel database regarding Synaptics.
  3. Help ensure the Gnome, KDE, and XFCE user interfaces allow customers to configure the gestures and other advanced features. This configuration UI will be shared with other hardware devices so you won’t have to do that much. While you wait for the magic that is Windows 9, you must have some free time.

Windows might be the majority, but a big part of that is because the laptop manufacturers expend little to no effort on the alternative. Meanwhile, 50% of their customers would be happier running Linux if it was well setup! We can wonder why it hasn’t arrived on the desktop like it has for cellphones and servers, but in the meanwhile, it would be nice if the effing mouse worked well out of the box.

Linus can’t go around saying F-U to every hardware company although surely Synaptics deserves it. if you are frustrated with Synaptics hardware, take 1 minute and sign this petition where you can also add your comments! https://www.change.org/petitions/synaptics-corporation-help-maintain-linux-drivers

Review of Arch Linux on a HiDPI Lenovo Yoga 2 Pro

Warning: this is a 4,700-word review, so read when you have something to drink and some time. I tried to break it up into two posts, but I didn’t find a good place.

I have been running Debian-based distributions of Linux for the last 8 years, but I’ve decided to try Arch for a new laptop to replace my 2008 Thinkpad. I bought:
Lenovo Yoga 2 Pro: Intel Haswell 1.6/2.6 GHz, 8 GB RAM, 256 GB SSD, 13.3” QHD+ 3200×1800. I bought it for a special launch price of $1200, my model is now $1400.

I was running a 5-year-old laptop because it still worked well other than the old CFL bulb which was very dim. Lenovo was also not adding much value in their newer models, and was even taking it away. For example, one of the great things about Lenovo is that it was user-serviceable, but this is becoming less true.

They are also producing an unending stream of indistinguishable models of laptops, (143 at last count, compared to Apple who offers 5 MacBook Pros, and 4 MacBook Airs), while making fewer that are user-customizable. Nike can make a fully custom shoe, whereas Lenovo appears to be heading in the other direction.

Screen

It has been interesting researching a HiDPI laptop. They are still hard to come by. Half of the models offered on the Dell and Lenovo websites are 1366×768. Mostly the rest are 980p or top out at 1080p. Compare that to the Nexus 5, which runs 1080p on a 5″ screen.

Most pictures on the web are made for 96dpi screens so the only thing you can do is scale it up and they don’t look better. The only noticeable graphical improvement is with wallpapers. I searched on Google for some nice ones such as this picture of a Maserati that actually looks like it’s in my computer. (Here are a couple Arch wallpapers.) In the end, I’ll be happy with this machine because the 5.7 million pixels make reading text like looking at a color laser printer.

Keyboard

The most controversial thing Lenovo did since 2008 was change the layout of their keyboards. It was apparently done by people who hadn’t used their previous model. They didn’t show a lot of respect towards the designs they inherited, nor to their customers whose fingers have been trained for years on them. One simple example is how they’ve swapped the left Fn and Ctrl keys. There is no possible benefit from that. The Linux kernel has a policy: “don’t break userspace”. Keyboards are one of the places in computing where backward compatibility is important.

Of course on smaller machines compromises must be made, but Lenovo used to put great keyboards into even their 13” machines. This laptop has almost an inch of empty space on the left and right edges of the typing area. If they had better used it, they could have built a more compatible layout. The chiclet keys work fine, but the idea of talking about an “efficient layout” for a keyboard means that inmates are running Lenovo.

Mouse

So in addition to the muscle-memory re-learning required for the new keyboard, Lenovo (and others) are in the process of removing the mouse buttons. I worry we are heading towards the world of Idiocracy. If I wanted to re-learn the keyboard and mouse in an unmaintainable machine, I’d have bought a Mac. I almost expect them to next copy the Macbook Wheel:

Fortunately, they’ve got a compromise for now with the one-button ClickPad. Apparently it will still have mechanical wear issues so I’m not sure if it is actually an improvement.

By default, the left and right bottom areas are reserved for virtual mouse buttons. With a Clickpad, it can sense where the fingers are, and can guess whether the user meant a left-click or right click:


It would cause confusion for the software if you had fingers in both the left and right areas, but for me at least, I never did that. The problem arises because the button areas are now live and sending touch and movements signals, which can completely confuse the software.

It is possible to live with a Clickpad, but it requires smarter drivers. The laptop shipped with a Windows driver written by Synaptics that worked fine. It was overkill with some of the extra gesture features they provided, but it was customizable and not flakey.

Whereas on Linux, out of the box, the driver is almost unusable. I will talk more about the problems in the kernel section below. There is a free Synaptics Linux driver out there, but it isn’t built with any help from Synaptics the Corporation. It supports gestures, but it has a number of bugs. Synaptics has apparently written a proprietary driver, but you can’t download it unless you are an OEM and make laptops! Since no one uses it, it is probably buggy. Synaptics is perfectly happy to write drivers that don’t actually get into their customers hands. The issue of proprietary drivers on Linux is still a problem for video cards and a few other places, but in general most hardware companies realize it is inefficient and buggy to write closed drivers

(UPDATE: I’ve created a petition to Synaptics Corporation about Linux drivers. Please sign it!)

CPU / GPU

The new CPU is a bit faster, but because of its smaller size (45nm versus 22nm) it uses less energy. My old “Penryn”-class Intel 2.5 GHz dual-core CPU required 35 watts while the newer uses 15. The Yoga is hyper-threaded and runs at 900 MHz to 2.6 GHz, depending on load. Note that Windows thought it was a 2.3 GHz processor so there is a question as to whether these computers have been given an unintentional lobotomy by a bug in Microsoft’s code. The machine feels a lot faster mostly because of the SSD.

The Intel 4400 graphics card is much better than the GMA 965 I had in my old laptop. 2-D performance is plenty fast, even with the 4x more than 1600×900 pixels to move. It will run free little games like SuperTuxKart with 60 fps, but Second Life was unplayable at 2 fps.

SSD

The speed of an SSD is great. I am getting 500MB / sec reads whereas I would get about 50 MB / sec on my old 7,200 rpm drive. I was not usually waiting on the computer, but when re-booting, starting big apps like Firefox / LibreOffice, etc. I was sometimes I/O bound, so it is nice that this is gone. With an SSD, the computer is snappy.

I was paranoid about reliability, but it depends on factors such as the size of the silicon, the size of the drive, how many bits per cell, usage patterns, etc. My 22nm MLC drive should get 3,000 write cycles.

The typical use is when I’m working on a 1MB document. I’ve adjusted LibreOffice’s autosave to be every 30 minutes, which should be fine because it never crashes for me. With wear-leveling across a 256 Gig SSD and 3000 write cycles, I could write that file 768 million times. Given that I might write 20 revisions per day the drive should last 105,000 years.

The next biggest write-heavy app for me is Firefox which out of the box was poorly optimized for SSD drives. It usually wrote about 4 megabytes per click, and on one page wrote 70 MB! The following are the fixes I applied:

1. Go to about:config and set browser.cache.disk.enable to false. I also adjusted the cache size to be 50 MB so as to not let Firefox fill up memory with web pages I don’t typically go back to.

2. I next turned off thumbnail generation which was writing about 300KB per click. Sometimes it even generated two! It seems Firefox should only generate a thumbnail when it decided it actually needed it for the start screen, but this silliness can be worked around. Even worse, after turning off the cache and the thumbnails, I was still getting a couple of MB of writes / click.

3. So finally, I installed a tool called profile-sync-daemon, which sets a link to point your Mozilla profile to the /tmp directory which is actually just RAM. It lets Mozilla write as much junk as it wants with no wear to the drive. Every hour, it will do an update via Rsync, which does an incremental update of only the bits that changed. With those 3 fixes, Firefox performs well.

When adding in system maintenance, media, etc. I should write about 200 MB / day. With that, the drive should last 10,500 years.

More good news is that SSDs are quite smart about fixing problems transparently. It has reserve blocks, and when it detects a problematic cell, it moves the data over to a spare and keeps going. Of course, when one cell goes, because of wear-leveling, it means all the other cells are also old and the drive should be replaced. One big difference between SSDs and HDDs is that old flash cells can lose their data entirely if turned off for days, so backups are still valuable.

Linux has a utility called smartctl which can run diagnostics, and tell you information from the controller such as the number of times the cells have been erased, how many spare cells are in use, etc. This way you can monitor it. I’m sure there is some GUI for Windows, but I prefer a text-mode app anyway.

Part II: Arch

Now to the OS: I decided to try Arch because there while there are many good desktop distros, I wanted to try a rolling one that updates nearly all of its packages within a day of the component shipping a new version. Describing why not all the other distros is hard so I will just mention why I decided against Debian Unstable which is also rolling. (I wrote a separate article about why not Manjaro you can read here.) Debian used to have a reputation of reliable staleware, but this is less true:

Current Package Version

Debian Unstable Version

Arch Version

alsa-lib (1.0.27.2) 1.0.27.2 1.0.27.2
Ati-driver (13.15.2) 13.4.3 13.4.14
firefox (25.0) 24 25
gimp (2.8.6) 2.8.6 2.8.6
glibc (2.18) 2.17 2.18
gnome-shell (3.10.1) 3.8.4 3.10.1
gtk+ (3.10.2) 3.8.6 3.10.2
httpd (2.4.6) 2.4.6 2.2.25
inkscape (0.48.4) 0.48.4 0.48.4
kdelibs (4.11.2) 4.10.5 4.11.2
libgnome (2.32.1) 2.32.1 2.23.1
libreoffice (4.1.3) 4.1.2 4.1.3
linux (3.11.6) 3.11.6 3.11.6
lxde-common (0.5.5) 0.5.5 0.5.5
mariadb (5.5.33a) 5.5.33a
mate-desktop (1.6.1) 1.6.1
MesaLib (9.2.2) 9.2.2 9.2.2
mysql (5.6.14) 5.5.33
NVIDIA (325.15) 319.6 325.15
qt (5.1.1) 4.8.5 4.8.5
samba (4.1.0) 4.0.10 4.1.0
systemd (208) 204 208
Thunderbird (24.1) 17.0.9 24.1
vim (7.4) 7.4 7.4
vlc (2.1.0) 2.1.0 2.1.0
xfdesktop (4.10.2) 4.10.2 4.10.2
xorg-server (1.14.4) 1.14.3 1.14.4

Debian is generally close to upstreams, but there are still some problem areas: their integration of Gnome is 6 months behind. It is an interesting question why Debian, which is a bigger and older team than Arch, can’t keep up. The good news is that Debian is generally on a good trend with more people joining:

Debian is doing better than it used to, but Arch is close to perfect in regards to keeping up with the thousands of Linux applications.

Another downside of running Debian’s rolling release is that the official goal is to make the next stable version, so there isn’t the social culture that exists in Arch where everyone is depending on the build to always work.

I found lots of bugs while running Arch, but none that exist in their code, only in the kernel and the other applications they integrate. The faster I can get the latest code, the faster I can have a better Linux experience.

Arch’s primary assets are build / install scripts (here is the one for LibreOffice) and a wiki. The wiki is superb. I never considered to use the non-existent one for Mint. When I ran Ubuntu, it sort of did everything for me such that I never needed to read up on anything. I followed Arch’s Unofficial Beginner’s Guide, and after that spent some hours reading various articles about how to use and enable fancier features in the software. Using the command line is necessary for Arch, but the set of utilities required is not very large, and you can learn as you go. The wiki and the community assume people are comfortable with the shell which is good because it also keeps the instructions simple and fast. Some people with little familiarity of computers sneak through because the wiki docs are so good that any literate person can run Arch.

Installation

One of my big decisions was to just forget about the UEFI support, which makes installing Linux twice as complicated. Lenovo doesn’t seem to offer any other OS for purchase on their laptops, but fortunately their BIOS still has the ability to boot in “Legacy” mode, which allowed me to ignore all the stuff I don’t care about.

In order to switch back to the old (MBR) partition table, I had to wipe my hard drive and couldn’t dual-boot into Windows 8.1 again. But after spending a few hours with it, I decided it wasn’t worth the 40 gigs of space. Linux on the desktop still seems far away, but there are lots of coders out there so it is useful and empowering and interesting to watch. My install is currently using only 4 gigabytes, so it seems a waste to reserve 10x the space on something bloated and inferior.

Legacy BIOS boot mode worked fine and I set things up the way I was used to with two partitions: a 30-gig root partition for the OS and applications, and the rest for “/home”.

I couldn’t just copy over the entire home directory from my old machine. I tried, but Gnome 3.10 got completely confused with missing icons, ugly widgets, etc. So I started fresh and copied things over selectively.

The install process was straightforward but a little tricky. I needed to run:

# rfkill unblock wifi

on every boot to get the network card to function. And when I run that command, wireless doesn’t wake up and start working, so I need to use:

# wifi-menu wlp1s0

Another tricky issue was that when I setup the machine, I didn’t know to have the package manager install the rfkill command. Rfkill comes on the Arch installer, but it didn’t actually install it by default on the drive. And so when I booted into (text mode) for the first time, I couldn’t connect to the network to get the GUI and the rest of the packages I needed.

And so I went back to the USB install stick. I had to follow the instructions from the top, but I was able to skip many of the steps such as the need to actually copy over all the bits. When I got near the end, I installed the package:

# pacman -S rfkill

and then I was ready to reboot into the OS, connect to the net and install everything else.

Another little issue I ran into is that on most operating systems, when you install Gnome, you also install the GDM, but on Arch, this isn’t necessary, nor does it prompt you. I needed to run:

# systemctl enable gdm.service

Fortunately, that line was at the top of the Gdm wiki page, so it didn’t take me long to fix my problem. I’ve had to fiddle a bit with systemd while using Arch, and it seems to be a simple and reliable way to manage low-level aspects of OS.

I like how the software in Arch is so granular. Gnome is broken up into two meta-packages, an essential one, and a collection of extras. I installed the main one, and some of the extras.

With LibreOffice, I installedWriter, Calc, Impress, and the English proofing tools. Before I used to blindly download and install all the DEBs on every release, but with Arch I took the time, and only installed the pieces I care about. Only they will be selectively downloaded and updated every few months as new versions come out.

Eventually, I did decide to fully re-install the OS. The first time, I put the 32-bit version on but I discovered later that there is a kernel 3.11 bug (https://bugzilla.kernel.org/show_bug.cgi?id=61781) where the machine won’t resume from suspend properly. I saw it on a 2008 Lenovo, and 2013 Lenovo, so it was probably a regression for a large number of computers.

The subsystem maintainer has since stepped in and reverted the patch because the underlying maintainer hasn’t had time to find a proper fix. (Presumably the change was made for a reason.) Resume was only broken on x86 so switching to 64-bit Linux worked around that problem.

Also, the Arch kernel doesn’t enable PAE in their builds, unlike most other distros, so it could only see 4 gigabytes of memory. My old laptop had 2 gigabytes, which was typically more than enough, but I decided that the extra memory could be useful as a RAM drive. There are PAE builds in Arch’s user repository, but they are not widely used or supported, so switching to x86-64 solved that problem as well. I’m guessing a number of kernel developers are running 64-bit nowadays because otherwise the resume bug would have gotten fixed a lot faster, so perhaps 64-bit is more reliable.

I generally ran into 2 classes of problems, hardware bugs and HiDPI issues:

Kernel bugs

1. The screen’s backlight doesn’t work unless I put “acpi_backlight=vendor” into the kernel command line. Fixing this issue involves changing the GRUB bootloader configuration, but the wiki was clear and explained how it could be done. As a side-effect of this fix, the laptop buttons to adjust the brightness no longer work but I decided the tradeoff was worth it. I can do:

# echo 850 > /sys/class/backlight/intel_backlight/brightness

to tell the LEDs to back off a bit, but I don’t mind full brightness. I should get about 5 hours on a full charge, which is more than enough for my typical uses.

2. As I mentioned in the install, I need to unblock my wifi card to get wireless working.

3. The Synaptics Clickpad is almost unusable. In fact, for a while I was sorry I didn’t keep Windows around because a mouse on crack can prevent you from being able to focus on your work. The Synaptics driver built by the community is quite rich, but it is a pain to use:

  • By default, the mouse button areas of the Clickpad were live. I almost always move my finger a bit while pressing down,which meant that it would usually click next to the choice I wanted, but not the one I actually wanted. There is a fix for that that can be put into the xorg.conf file which tells the driver to ignore any movement in the bottom portion of the trackpad, but it took me hours to discover and isn’t enabled by default.
  • There is a driver bug because of the above feature where if I have my left finger resting in the click-only area, then any cursor movements with my right finger don’t register. It would be nice if this were fixed because ClickPads are quite common now, but in the meanwhile, I can workaround it. Click and still drag works, because when you click the mouse, the pointer starts registering in the live area again, but it is a pain.
  • The mouse pointer jumps around randomly while using it.
  • The Gnome option to disable the mouse while typing doesn’t seem to work, so my mouse wanders around a lot. The edge of the trackpad is just 1 cm from the space bar and it is so wide that you are forced to rest your hand on it a bit while typing. On my old Thinkpad, the trackpad was thinner, recessed, and far enough away from the keyboard that it wasn’t much of a problem. Again, this requires smarter software to detect your palm. Fortunately, as light touching doesn’t register any mouse clicks, so the movement is a bit distracting, but not disruptive.

4. The system still doesn’t always work when coming out of resume. Things work better on the x86-64 bit builds, but the mouse pointer and wireless doesn’t always work and so I need to reboot sometimes.

5. The driver leaves artifacts with the Intel driver under SNA mode:

6. The cpupower comand never shows the processor getting down to the minimal 800MHz to save heat and power even with the Intel Powersave governor which is supposed to be able to do that.

7. My logfile fills up with USB (and other) error messages.

usb 2-7: unable to read config index 0 descriptor/start -71

8. Even though I mount my hard drive with the “discard” option for SSDs, if I run fstrim after a reboot, it shows me this:

# fstrim -v /

/: 26.2 GiB (28103872512 bytes) trimmed

# fstrim -v /home

/home: 204.8 GiB (219841617920 bytes) trimmed

9. There is an option to disable hyperthreading by booting the kernel with maxcpus = 2, but it appears to also disable the ability to adjust the CPU frequency.

HiDPI Bugs

In general it is beautiful, but I ran into several problems.

1. Sometimes Gnome uses big cursors and sometimes it reverts to old habits of small ones:


2. Some parts of the UI use bigger fonts, but don’t increase the window size to make room for the bigger text:


3. Some windows have big close buttons and controls, and some have small ones. The scrollbar in all applications is too narrow to easily grab. You need the steady hands of a sober heart surgeon to use the UI.

5. Many apps like LibreOffice, Gimp and Audacity have toolbar buttons that are too small to easily click and even recognize. LibreOffice’s dialog boxes and text everywhere look fine, even pretty, it is just the buttons that are not scaling. Even using LibreOffice’s large icons setting doesn’t make them big enough. Interestingly, Apache OpenOffice seems to do a better job as it sets the toolbar button based on the size of the system font. All apps should do that.

6. Cinnamon and Xfce cannot detect that the screen is high-resolution and so need lots of tweaking to become usable.

7. Firefox needs the No-Squint plugin to make the websites display in a reasonable size. Apparently nothing has been done in the product to officially notice the dpi of the screen, but at least there is an easy workaround. There are still some problems such as the tiny buttons on the Youtube player but it is manageable.

Fortunately, as laptops with high resolution screens become more popular, these bugs will get noticed. There is a lot to be done. I’m surprised that Gnome has so many issues as their 3.10 release announcement makes it seem like they had fixed them, and I don’t see any mention of further work.

I did try Windows 8.1 for a few hours and it had mixed results with regards to handling such a high-resolution screen so I’m not necessarily any worse off with Linux. Teams in the free software community release code 2-4 times per year, so I’m sure it will get smarter relatively soon.

The good news is that Gnome’s Classic mode is almost good enough for someone who was happy in Gnome 2 but there are still various annoyances: most people in Gnome 2 used the left and right arrows to scroll through their workspaces, but in Gnome Classic, you have to use up and down.

I may try Ubuntu’s Unity GUI, as there are builds for Arch, but Unity requires patches to so many upstream components that I may not want to risk it. I’m surviving with Gnome Classic plus some extensions, and I’ll be able to use this machine full-time. So, in spite of all these problems, I don’t think any are Arch’s bugs. The state of Arch is really the state of the Linux desktop. I’m surprised it isn’t in the top-5 most popular distributions on Distrowatch.

Linux on the desktop

It seems not one person at Lenovo installed Linux on this new Yoga before they released it. There are a number of bugs they could easily have fixed. The average patch to the kernel is about 20 lines of code. Unlike with Windows, they can make fixes anywhere they find problems. They also don’t need to build a multi-lingual installer to distribute drivers, they just improve the actual code, and let the other parts of the community distribute it and provide the UI.

If they had put in the same amount of effort towards Linux that they put into their fluff Windows UI nagware, it would be more than enough to make sure the Linux desktop worked well out of the box. There are even Lenovo-specific modules in the kernel that they aren’t contributing to. Other hackers are expected to figure out how to make every new model work by reverse-engineering.

I see lots of Linux users using Lenovo hardware, but because they don’t offer it, they have no idea how many users are running it, or would be happier if it came pre-installed. Lenovo has 33,000 employees and Dell has 100K. They could easily enable a great out of box experience with a relatively small team. If I ran Lenovo, everyone would be required to dual boot. It would be nice if they saw the trends in computing on cell-phones to servers and realize that they should be a part of a better future.

They could also customize it more nicely and more cheaply rather than building a bunch of junk outside and on top of Windows. The community would even help them. When I first started using Linux in 2005, I thought that Linux on the desktop would soon happen, but even in 2013, things haven’t improved much. I guess we Linux users need to complain more loudly to the employees of the laptop manufacturers.

Kernel bugs

The good news is that at least on the hardware side, the kernel has thousands of people, and so they in theory have the resources to fix these issues quickly. I’m looking into filing bugs, but there are already 25 active against the Synaptics driver alone so I’d need to do research and make sure I’m not filing duplicates.

One challenge for the kernel is they sometimes let bugs languish in their database for years. It was recently Halloween, so check out these 400+ scary bugs marked as regressions: http://bit.ly/LinuxRegressions. The Linux kernel currently has 2,194 bugs: http://bit.ly/LinuxBugs. Even scarier, a messy and unprioritized buglist discourages people from even entering bugs.

Linus and the other top maintainers see their job as making sure the hordes of programmers don’t screw things up, and that is a big key to its success. It requires a lot of work to watch over the massive rate of change. The problem is that the team mostly focus on evaluating the stream of patches going in that they sort of ignore users. The Linux kernel has a policy of no regressions, but there is no enforcement.

One of important things I learned at Microsoft was to try to get down to zero bugs as best as you can. An early internal memo recommended developers stop adding new features when they had 10 or more assigned to them. Some people might laugh at the idea that Microsoft tried to write reliable software, but the poor results are because they have richer but more complicated and older codebases. Peter Drucker had a line: “What’s measured improves.” If you never looked at yourself in the mirror, or weighed yourself, you might get heavier than you realized. Linux runs on millions of machines, but there are billions of them.

Another issue is that the kernel dev community is not equally well-staffed in every part of the codebase. There are a lot of ARM contributors, but they can’t fix the random laptop issues. In a few cases, it seems like the hardware will be phased out before it works with Linux. The benefit of a bug list is that it helps you find the areas that need extra brainpower.

In many teams in the free software stack, people are working on their bugs as fast as they can. Groups that have 1000s of active bugs like LibreOffice are simply understaffed and need more money and people. In the case of the kernel, it is partially an uneven resource problem, but also a cultural issue.

The worst part about the current large active bug count is that it is the fans of Linux on the desktop who are getting hurt. There is a world of people out there who are inspired by Linux but can’t make kernel patches. Everyone who files a bug in the kernel appreciates this amazing thing that has been built. They would like it to be used everywhere, including on their computer.

In spite of the problems, I’m happy in Linux. With applications of every kind, it changes how you think about an OS. If I ran Windows, I’d still be using Firefox, LibreOffice, Audacity, VLC, Gimp, Python, Git, Tomboy, and other free software. Windows succeeds because of laziness and inertia. There are some applications that don’t run on Linux: games, Ableton, Solidworks, iTunes, etc. but for most most today it is better code. Looking forward to the next few years of Linux on my laptop.

Note: I’ve created a petition to Synaptics Corporation about Linux drivers. Please sign it!

LibreOffice 4.0 And The Power of Brands

LibreOffice 4.0 was launched last week, and the news reports and activity on social media were massive, more than any release of LibreOffice or OpenOffice before, with better coverage than many of Microsoft’s well-funded introductions. There were numerous links sent around to the usual sites like LinuxToday.com, but also TechCrunch, VentureBeat, Time Magazine, etc. A fair amount of the chatter was people wondering what the difference is between the two versions. Some have basic questions like whether LibreOffice can import their OpenOffice documents.

LibreOffice is introducing their new name and community to the world. All the major Linux distros are already aware, but there are many Windows and Mac users who don’t understand what is going on. People even become attached to names for emotional reasons. Brands are powerful. If you were in a remote village in India on a hot day, you’d quite likely grab a Coke to cool your thirst if that was the only one with letters you recognized. Even people who like to travel and try new things might not want to take a risk on something that looks like carbonated, used bathwater with funky characters when they are tired, hot and thirsty.

Picture by Hari Kishan & Shardul Pandey

In the realm of software, the considerations are different but related. Many are afraid to try new things because technologies so frequently come and go. People have been burned by Farmville, Zune, Tweetdeck, iTunes, Nvidia, Comcast, AT&T, Sprint, Sun, Adobe, Gnome 2.x, Microsoft, IBM, etc.

Some people look down on the LibreOffice / OpenOffice codebases because the user interface is more clunky than Microsoft’s Office, but many who spent time in it saw how it handled their files, has many features, and is generally stable, fast, portable and free. People became attached to “OpenOffice” during the hours they spent expressing their creative ideas. Many attach greatness to the name rather than to the people who built it. This makes people uneasy to try LibreOffice.

If you were to explain to OpenOffice users that Oracle laid off all the programmers before handing the trademark to Apache, and their new team is legally unable to accept changes made by LibreOffice, they might realize they should try the newcomer. That disclaimer is currently not on the Apache website. It would also be a useful warning if they listed all the features missing from LibreOffice. The current full list is already mind-blowing (4.0, 3.6, 3.5, 3.4, 3.3), and they are just getting started (Easy hacks, GSoc).

The biggest issue to consider is the opportunity cost. Instead of enhancing the existing OpenOffice brand, the community is forced to rebuild a new one. That is especially unfortunate because there are many people in LibreOffice who contributed to OpenOffice, and made the brand worth what it is today. As Apache OpenOffice is unable to accept LibreOffice changes, the brand is being squandered. And instead of adding resources, Apache are playing catchup, mandating an inferior license for this codebase, and inferior tools.

Because Apache OpenOffice has the brand, and a handful of full-time employees working on the codebase, they can find always ways to report good news and give the illusion of progress: “There have been 35M downloads, which saves the world $21M per day.” “Who wants to help with the wiki?” “We’ve now got 6 workitems tagged as Easy Bugs.” “Can someone dig up the documentation of our SDF format?” “It would be great to get someone to package OpenOffice into Fedora and give users choice.” “We found 50 naive^Wnew volunteers to help with QA in our recent call for help.” Etc.

This was an exchange that took place during Michael Meeks’ interesting Fosdem 2013 talk:

Question: I don’t know if I’m the only one, but I’d love to see peace between LibreOffice and Apache OpenOffice. Is that in the works?

Michael Meeks: Okay. Well, I think there are lots of opportunities for code sharing. We provide code under a license that Apache can incorporate, into their binary releases at least, so I think there’s a lot of scope for that. And, I think you’d have to go and ask them. I’m not really interested in talking so much about that. I might say something rude.

People in the Linux community are aware of the situation, but many don’t realize that there is very little LibreOffice can do to improve things. LibreOffice cannot prevent new forks from being created, and no one inside was threatening to fork. LibreOffice couldn’t prevent Oracle from giving away the trademark to anyone. LibreOffice couldn’t prevent Apache from creating a project that doesn’t accept their code. LibreOffice can’t prevent new people from getting confused when they see Apache, OpenOffice, and a pretty website, not realizing this is basically the “pet project” of an IBM employee.

It seems like people inside Apache could do something, but many of them liked the idea of having two “cores”. They see themselves as the upstream with the more open license, and LibreOffice is free to grab whatever code they find useful. Unfortunately, they don’t realize that as these codebases diverge, this becomes harder. LibreOffice no longer uses the SDF format for localization. So between the confusion, and the illusion of progress by a stream of money, we could be here a while. IBM has been around for 100 years. Perhaps they’re happy to wait until everyone is dead and hope the next generation of LibreOffice representatives is more amicable to their plans. As far as for things getting better, the best sign to look for would be if IBM were to send their representative new directives from the Home Office. You do see comments stating they’d like to end the fork. If only they had that wisdom before they created one. However, it appears they have no ideas what to do next. More wisdom is yet required.

LibreOffice is doing very well for such a young team. The free software community is jumping in and improving the codebase in many ways. However, the community could easily use millions of dollars to hire more people to work full-time and mentor volunteers. Perhaps the greatest concern is a lack of people who understand the Writer layout code, which is the most complicated piece of logic in the entire suite. Code and people are valuable, but people who understand code even more so.

Note: I write about LibreOffice / OpenOffice because I don’t like to see brands and volunteers wasted.

Dissent on Gnome's Javascript decision

Yesterday I read a blog post / announcement about how Gnome is moving to Javascript and I wanted to write some feedback.

It is great that they are trying to use a garbage-collected language for as much code as possible. For a component-based shell UI, Javascript is surely better than C, C++, or Java. I realize they started down this road towards Javascript years ago, but I think it is worth re-considering whether they are on the right path.

With big decisions, it is nice to have a paper trail. I can find no supporting documentation backing up the decision other than one blog post written after the fact, which doesn’t give very much information.

It appears the decision was made in a meeting. It is great to have meetings to discuss things, and it is great to make decisions in meetings, but oftentimes the best results are about moving the decision-making process forward, not actually committing to big things. Even if there were many in that room, there are surely facts they didn’t have, and other interested parties who were not there. There is the risk of “tyranny” by a self-selected cabal. Hopefully the decision wasn’t made at a bar ;-)

I believe the best choice for Gnome is Python, not because the language is necessarily a lot better than Javascript, but because it is already close to the standard scripting language of the Linux desktop. My Mint-Debian repository has 1809 Python packages, and only 301 for Javascript. (Vala has 68.) Was data like this taken into account during the decision-making process? Without a paper trail, there is no way to validate the analysis or circulate it for feedback. There are many reasons to use Python, but the most important is that it is already so popular on the free desktop. The best reason to use Javascript is if you plan on running inside a web browser. Anyone writing code for the Linux desktop and not constrained to a web browser can do better.

To be clear, some of the reasoning is explained. Here are my responses:

1. Our language of choice needs to be dynamic and high level.

Python and many languages fit that description.

Next, it says:

2. There is already momentum in the GNOME Project for JavaScript — it’s used in GNOME Shell and GNOME Documents.

Unfortunately, that is not really much of a reason. In fact, it could be perpetuating a bad plan with this logic.

Next:

3. There’s a lot of work going into the language to make it especially fast, embeddable, and framework-agnostic.

Every language works to make itself fast. There are lots of efforts to make Python fast such as Cython and PyPy, and as many Gnome libraries will remain in C, this is hardly an issue even with the standard CPython implementation.

I’m not sure what the benefit of being embeddable is for a desktop UI. And Python is embeddable as well, inside apps like LibreOffice. I don’t undertand what the benefit of being framework-agnostic is. Every language needs libraries, and a rich set of libraries is a good thing.

4. JavaScript is increasingly being seen as a first class desktop programming language — it us being used in Windows 8, mobile platforms, and for local web applications.

Aren’t Windows 8, mobile, and local web applications supposed to be a worse experience than a Linux desktop? I imagine living exclusively in any of those platforms and shudder at the thought. They also aren’t planning on sharing code with any of those groups. Please don’t try to convince people the Gnome future is bright by using those three examples!

There is another post by John (J5) Palmieri who appears to be a fan of Python but who nevertheless endorses Javascript for Gnome. In it, he says that Python has a lot of baggage. Unfortunately, he doesn’t describe or link to a document describing what he’s talking about. In addition, it is important to weigh the good (the existing libraries and free software programmers) against the baggage. Every language has some baggage. For example, there is a book called Javascript: The Good Parts, which says something about Javascript’s. Furthermore, Python is setup to evolve and periodically break backward compatibility via things like the Python 2.x / 3.x branches. Can Javascript ever remove its cruft?

My day job is trying to finish a movie (trailer) endorsing Python as part of math literacy. Changing how math is taught to children could take a generation. But if Gnome get going now, they will be ready, and hopefully also be better than Gnome 2.x by then ;-) (I’m stuck in MATE. I believe the decision to remove Gnome 2.x is as good an idea as LibreOffice removing DOC import. This decision can be revisited also, but given how long ago it was made, I’m sure people are tired of the topic, so I will end here.)

Update: Some suggest that Javascript is more popular than Python as being a reason. Google trends show that Javascript popularity is dropping dramatically and the two are close today. Furthermore, most of those people are building random websites, not writing code that ships in Debian. And as pointed out in the comments, a recent TIOBE survey shows Python as more popular than Javascript. Python is also replacing Java in introductory CS classes.

Open Letter to fellow ex-Microsoftie Steven Sinofsky

Congratulations on leaving Microsoft. Unless you have bills to pay, you won’t regret it. I left at the end of 2004, and have since studied a vast and amazing — but still flawed – world of computing out there.

For example, I discovered that we should already have cars that (optionally) drive us around and computers that talk to us. And that Linux on the desktop is powerful and rich but failing because of several strategic mistakes. Google claims to be a friend of Linux and free software, but most of their interesting AI code is locked up. Programming should be a part of basic math literacy for every child. The biotechnology world is proprietary like Microsoft, which is stunting progress in new medicines and safer devices.

The most important lesson is that the free software world outside Microsoft is much bigger and richer. No matter what aspect of technology you want to work on, there are codebases and communities out there. Even the large companies who write proprietary software like Amazon, Apple, Facebook, and Twitter use free software as their base. So you first find out what you want to work on, and then you find the existing codebases and communities to join. In some cases, the are multiple, so you need to decide which best meets your needs.

The good news is that there are already millions of smart people working on any aspect of technology you’d like to work on. That is important because now that you have left Microsoft, you greatly lose the ability to control your own destiny using their technology.

When I first left Microsoft, I took on a consulting job helping a team build a website which used Microsoft Passport as the authentication mechanism. However, as I ran into problems, even Google wasn’t able to help because the knowledge and ability I needed to fix my problems was locked up behind the Microsoft firewalls. Fixing a problem in proprietary software can sometimes feel like performing witchcraft — you have to try lots of random incantations because you can’t know what is really going on. In the free software world, the code, buglists, specs, discussions, etc. are public, and anyone is welcome to contribute. A warning though, it can be like herding cats.

I read you have a Microsoft Surface. I recommend getting another machine and installing Mint-Debian Linux. You’ve probably heard of Ubuntu, but Debian is the 1000-person team that provides the rock Ubuntu builds upon. Mint is a very popular re-spin that adds mp3 playback and other features that have patent risks and can’t be part of the free Debian system. The Windows app store is a Potemkin village compared to what Linux offers. I remember you have a Unix background, I recommend refreshing your knowledge of the command line and reading some new books. I felt like a stranger in a strange land for the first couple of months, but it became perfectly comfortable to me, and has numerous advantages such that now I am as interested in using Windows as I am in using DOS.

I don’t recommend you bother with Apple. They have a proprietary walled garden even smaller than Microsoft’s. If you find a problem with Apple’s technology, your best option is to wait. If you find a problem anywhere in the free software world, you can file a bug, talk to a person, (usually) find a workaround, write some code, hire someone — or wait.

The other nice thing about this global community is that you don’t have to go anywhere to join. You can write code in your pajamas from Seattle and send it to Linus Torvalds in Portland who works from home in his. The Linux kernel alone has 3,000 programmers, scattered all over the earth, some of whom live in countries that are officially at war with each other.

Enjoy your new-found freedom. I have written a book about much of this you can read for free. It contains many things I didn’t know until I left. There are many news sites to learn about what is going on in Linux. I personally use LinuxHomePage, but every community has blog aggregators