PPD - Pimp your Printer Driver

I recently got myself a new printer, the HP Officejet Pro X476dw. A very nice and powerful machine, it can not only print double sided but also scan, copy and send faxes.

And of course it has very good Linux support, thanks to the HP Linux Printing and Imaging Open Source project. On my Ubuntu 14.10 desktop everything is already included to use the printer.

However, the first printouts where very disappointing. They looked coarse and ugly, much worse than prints from my old HP LaserJet 6 printer. After overcoming the initial shock I realized that only prints from my Ubuntu desktop where bad while prints over Google Cloud Print where crisp and good looking.

So obviously something has to be wrong with the printer drive on Ubuntu!

After some debugging I was able to trace this down to the fact that by default CUPS converts the print job to 300 dpi PostScript before giving it to the hp driver, as it shows in the CUPS logs:

D [Job 261] Printer make and model: HP HP Officejet Pro X476dw MFP
D [Job 261] Running command line for pstops: pstops 261 schlomo hebrew-test.pdf 1 'finishings=3 media=iso_a4_210x297mm output-bin=face-down print-color-mode=color print-quality=4 sides=one-sided job-uuid=urn:uuid:c1da9224-d10b-3c2f-6a99-487121b8864c job-originating-host-name=localhost time-at-creation=1414128121 time-at-processing=1414128121 Duplex=None PageSize=A4'
D [Job 261] No resolution information found in the PPD file.
D [Job 261] Using image rendering resolution 300 dpi
D [Job 261] Running command line for gs: gs -q -dNOPAUSE -dBATCH -dSAFER -sDEVICE=ps2write -sOUTPUTFILE=%stdout -dLanguageLevel=3 -r300 -dCompressFonts=false -dNoT3CCITT -dNOINTERPOLATE -c 'save pop' -f /var/spool/cups/tmp/066575456f596

I was able to fix the problem by adding this resolution setting to the PostScript Printer Definitions (PPD):

*DefaultResolution: 600x600dpi

As a result the print job is converted at 600 dpi instead of 300 dpi which leads to the expected crisp result:

D [Job 262] Printer make and model: HP HP Officejet Pro X476dw MFP
D [Job 262] Running command line for pstops: pstops 262 schlomo hebrew-test.pdf 1 'Duplex=None finishings=3 media=iso_a4_210x297mm output-bin=face-down print-color-mode=color print-quality=4 sides=two-sided-long-edge job-uuid=urn:uuid:83e69459-c350-37e5-417d-9ca00f8c6bd9 job-originating-host-name=localhost time-at-creation=1414128153 time-at-processing=1414128153 PageSize=A4'
D [Job 262] Using image rendering resolution 600 dpi
D [Job 262] Running command line for gs: gs -q -dNOPAUSE -dBATCH -dSAFER -sDEVICE=ps2write -sOUTPUTFILE=%stdout -dLanguageLevel=3 -r600 -dCompressFonts=false -dNoT3CCITT -dNOINTERPOLATE -c 'save pop' -f /var/spool/cups/tmp/0666d544aec68

Isn't it really nice that one only needs a text editor to fix printer driver problems on Linux (and Mac)?

On github.com/schlomo/HP_Officejet_Pro_X476dw I maintain an improved version of the PPD file with the following features:
  • Set printing resolution to 600dpi
  • Use printer for multiple copies, not CUPS
  • Default to duplex printing
The corresponding Launchpad bug is still open und unresolved. Apparently it is not simply to submit improvements upstream.


Comparing Amazon Linux

Since ImmobilienScout24 decided to migrate to a public cloud I have been busy looking at various cloud offerings in detail. Amazon Web Services (AWS) has a special feature which is interesting: Amazon Linux is a fully supported, "RHEL like", RPM-based Linux distribution.

While not beeing a true Red Hat Enterprise Linux clone like CentOS or Scientific Linux (which is the standard OS for the ImmobilienScout24 data centers), it is derived from some Fedora version and comes with a nice choice of current software. To me it feels like "RHEL +" because so far all our internal stuff worked well but a lot of software packages are much newer than on RHEL 6 or RHEL 7. The 2014.09 release updated a lot of components to very recent versions.

On the other hand, we also found packages missing from Amazon Linux, most notably desktop-file-utils. This package is required to install Oracle Java RPMs. I found a thread about this on the AWS Forums and added a request for desktop-file-utils in September 2014. In the mean time the package was added to Amazon Linux, although the forum thread does not mention it (yet).

To find out in advance if there are any other surprises waiting for us on Amazon Linux, I created a little tool to collect RPM Provides lists from different Linux distros on AWS. github.com/ImmobilienScout24/aws-distro-rpm-comparison takes a VPC and one or several AMI IDs and spins up an EC2 instance for each to collect the list of all the RPM provides from all available YUM repositories.

$ ./aws-distro-rpm-comparison.py -h
Create EC2 instances with different Linux distros and compare
the available RPMs on them.

  aws-distro-rpm-comparions.py [options] VPC_ID USER@AMI_ID...

  VPC_ID        VPC_ID to use
  USER@AMI_ID   AMI IDs to use with their respective SSH user

  -h --help            show this help message and exit
  --version            show version and exit
  --region=REGION      use this region [default: eu-west-1]
  --type=TYPE          EC2 instance type [default: t2.micro]
  --defaultuser=USER   Default user to use for USER@AMI_ID [default: ec2-user]
  --verbose            Verbose logging
  --debug              Debug logging
  --interactive        Dump SSH Key and IPs and wait for before removing EC2 instances


* The AMI_IDs and the EC2 instance type must match (HVM or PV)
This list can then be used to compare with the RPM Requires in our data centers. To get a better picture of Amazon Linux I created such lists for Red Hat Enterprise Linux 6 and 7, CentOS 6 and Amazon Linux in the Github project under results. For online viewing I created a Google Spreadsheet with these list, you can copy that and modify it for your own needs.

At a first glance it seems very difficult to say how compatible Amazon Linux really is as there are a lot of RPM Provides missing on both sides. But these lists should prove useful in order to analyze our existing servers and to understand if they would also work on Amazon Linux. The tools can be also used for any kind of RPM distro comparison.

In any case, Amazon Linux is exactly that what RHEL cannot be: A stable RPM-based distribution with a lot of recent software and regular updates.


DevOpsDays Berlin 2014

Update: Read my (German) conference report on heise developer.

Last week I was at the DevOps Days Berlin 2014. This time at the Kalkscheune, a much better location than the Urania from last year. With 250 people the conference was not too full and the location was also well equipped to handle this amount.

Proving DevOps to be more about people and culture, most talks where not so technical but emphasized the need to take along all the people on the journey to DevOps.

An technical bonus was the talk by Simon Eskildsen about "Docker at Shopify" which was the first time that I heard about a successful Docker implementation in production.

Always good to know is the difference between effective and efficient as explained by Alex Schwartz in "DevOps means effectiveness first". DevOps is actually a way to optimize for effectiveness before optimizing for efficience.

Microsoft and SAP gave talks about DevOps in their world - quite impressive to see DevOps beeing main stream.

My own contribution was an ignite talk about ImmobilienScout24 and the Cloud:

And I am also a certified DevOps now:


EuroPython 2014

One full week of Python power is almost more than one can take, but I missing it would be even worse.

This was my first EuroPython and with 1200 participants a big upgrade compared to the previous 2 PyCon.DE events in which I participated. The location (Berlin Congress Center) deserves kudos, along with the perfect organization.

The Wifi worked really well (except for a WAN problem on Tuesday which was fixed quickly) and everybody loved the catering. They even had kosher, helal and vegan food (preordered), which is highly unusual for German conferences. Most amazing was the video crew who managed to upload all videos in about one hour after a talk was given.

I managed to give three talks:

  • DevOps Risk MitigationHow we use Test Driven Infrastructure at ImmobilienScout24 as part of our general automation to reduce the risk of giving everybody access everywhere. (Access Slides or Watch Video)
  • YAML ReaderLightning Talk about the yamlreader Python library, which provides a wrapper for the yaml.safe_load function that merges several YAML files. yamlreader is the base for most of the modularized configuration in our Python software. (Access Slides or Watch Video)
  • Open Source Sponsoring
    About why your company should invest into Open Source projects instead of into proprietary software. I did not plan this talk, but a speaker did not show up and I jumped in. (Access Slides or Watch Video)
I very much enjoyed the international public at the conference and hope to be able to also attend next years event.


iPXE - The Versatile Boot Loader

iPXE is a lesser known Open Source PXE boot loader which offers many interesting features:

Talk & Article

Since iPXE plays a role in the ImmobilienScout24 boot automation I gave a talk about it at the LinuxTag 2014. The talk is half an hour long and gives a quick introduction into iPXE. It covers build, configuration & scripting and shows how to develop boot scripts in iPXE with a very short feedback cycle.

Download the slides to the talk and the audio recording as a podcast.

At the conference the German Linux Magazin became interested in the topic and asked me to write an article about iPXE:

Der vielseitige Netzwerk-Bootloader I-PXE
Linux Magazin 08/2014

Demo Scripts

For the article I created a bunch of demo scripts that are available on Gist. To try them out follow these steps:
  1. Install QEMU, usually part of your Linux distro but also available for other platforms.
  2. Download my pre-built iPXE boot kernel ipxe.lkrn
  3. Start QEMU with ipxe.lkrn and the URL to the demo script:
    qemu -kernel ipxe.lkrn -append \ 'dhcp && chain http://goo.gl/j8MbXI'
  4. Try out the various options. The login will accept any password that is the reverse of the username.
This demo script looks like that:

And the QEMU boot looks like that:

Try it out

Anybody struggling with PXELINUX should most definitively check out iPXE to see if it provides a better alternative to their needs.


automirror - Automate Linux Screen Mirroring

I do a lot of pair working and many times I connect a large TV or projector to my laptop for others to see what I am doing.

Unfortunately the display resolution of my laptop never matches that of the other display, and Linux tends to choose 1024x768 as the highest compatible resolution. This is of course totally useless for doing any real work.

My preferred solution for this problem is to use X scaling to bridge the resolution gap between the different screens.

Since none of the regular display configuration tools support scaling, I ended up typing this line very often:

xrandr --output LVDS1 --mode 1600x900 --output HDMI3 --mode 1920x1080 --scale-from 1600x900

Eventually I got fed up and decided to automate the process, the result is automirror, a little Bash script that automatically configures all attached displays in a mirror configuration. automirror is available on https://github.com/schlomo/automirror.

Typical Use Cases

Connecting a Full HD 1920x1080 display via HDMI to my 1600x900 laptop. In this case automirror will simply configure the HDMI device with 1920x1080 and scale the 1600x900 laptop display. As a result I stay with the full resolution on my laptop display and it also looks nice on the projector.

Another case is where I work with a 1920x1200 computer monitor and add the 1920x1080 projector as a second display. Again the common resolution offered by both devices is 1024x768. automirror will recognize my 1920x1200 display as primary display and scale it to 1920x1080 on the secondary display, which is not really noticeable.

It is recommended to configure a hot key to run automirror so that one can run it even if the display configuration is heavily mwessed up. In rare cases it might be neccessary to run automirror more than once so that xrandr will configure the displays correctly.


Granting root access in a DevOps world

At the 2014-06 Berlin DevOps Meetup this week we had an interesting fish bowl discussion about

What is the risk of giving DEVs root access in production?

Since I suggested the topic I was asked to give a short introduction into the topic:

The discussion that followed was suprising in several aspects:
  • A major concern is safeguarding the production data, but nobody had a really good solution for that. Many people have more problems with Developers seeing live customer data than with Develops changing something in production.
  • "Nobody should have root" was proposed by a security specialist, but he had no practical working example for this approach.
  • The question is tightly coupled to the degree of automation. The more automation you have the less need for anybody (Dev or Ops) to use their root privileges.
  • Not everybody having root access knows what to do with it, Developers are sometimes afraid of using their power if granted root.
  • This is mostly a question for larger companies and classical IT organizations. Small companies and start ups just give root to everybody who knows what to do.
For me that was the first time having this discussion when nobody tried to prove that Developers should in principle not get root access. The Test Driven Infrastructure fish bowl at the Berlin DevOps Meetup 2013-12 last year also touched upon this topic and the discussion was much more against giving root access to Developers.

My personal opinion is that in a DevOps world people are in the focus of our interest. The official title or organizational position should matter less than what the people are doing. We should therefore
give root access to people based on
  • Trust to act in our common interest
  • Commitment to fix everything they brake
  • Skills to tread carefully in our production environment


My SMART TV - Linux For The Win

I love my "smart" TV - it got Linux inside which is the base for a whole range of nice hacks.

TV Router

The most important one is that the TV is actually a wireless router that provides Internet via Ethernet to my TV rack. Usually the Ethernet connection is used by the Playstation or a Raspberry Pi.
The original reason for this hack was simple: The Playstation 3 has a really really bad Wifi reception which made watching Netflix nearly impossible and the unavoidable PS3 updates painfully long. The USB Wifi adapter connected to the TV has a much better reception, sharing it with the PS3 solved all the performance problems.

Samsung Linux TV

And here comes the good part. The TV (Samsung LE32C650) runs Linux inside and there is an Open Source project (SamyGO) that "opens up" the TV firmware and extends this Linux with useful tools.

In my case I only had to enable IP forwarding, configure a static IP on the Ethernet interface (eth0) and start a DHCP server on it. The Samsung kernel already included IP forwarding (thanks!) and the DHCP server is part of Busybox that comes with SamyGO.


Another benefit from rooting the TV is the option to add NFS support. The TV has a great media player that plays almost all file formats, even with subtitles and multiple audio tracks. The player can fast forward/rewind and even remembers the last playback position for each video. But all of these nice features only work when playing videos from USB storage, not over DLNA.

Thanks to SamyGO it is possible to mount a NFS share onto a directory on a USB stick. The TV thinks that the NFS share is on the USB stick and happily plays all the videos with all the fancy features.

Wife Acceptance Factor

Back in 2010, when I bought the TV, this was a really cool solution with a high WAF because both watching TV and videos from our collection work with the same remote control. Nowadays I would probably just attach a Raspberry Pi (with OpenELEC) to the TV and enjoy the seamless integration thanks to HDMI CEC. But is is still nice to know that I can extend my TV to better serve our needs.

I can only hope that the next TV will be equally hack friendly.


Win-Win: Employer Branding and Corporate Social Responsibility

Does your company care about employer branding? Probably yes.

Does your company care about corporate social responsibility? Probably yes.

Does your company combine these two to create a win-win situation? Most likely not!

Take my employer ImmobilienScout24 as a typical example: The about us page mentiones that ImmobilienScout24 is a great place to work (4th in our region) and the CSR team talks about the social engagement, e.g. blood donations or the social day where all employees donate their work time to non-profit organizations.

However, there is no obvious connection between these two things.

I would like to suggest a simple way how to combine both employer branding and corporate social responsibility:

A company should make it a priority to support charitable organizations and social projects related to their own employees.

  • Sponsor non-profit organizations or neighborhood/community projects that employees are involved with.
  • On social day, go to schools and kindergartens where employees are parents.
  • Involve employees who are in the red cross or similar organizations to organize the annual blood drive.
  • Support local or neighborhood charity organizations instead of global ones.
Basically the idea is that CSR related activities should be geared around the employees private life and activities.

This will create a win-win situation and especially help to retain employees because they get additional fulfillment and satisfaction from their employer supporting their social engagement.

There is no added costs involved, it is enough to change the way how CSR budgets are spent.

I mostly hear these arguments against this idea:
  1. CSR spending must be charitable beyond doubt, employee projects could be too narrowly orientated to count as generally charitable.
  2. Employee-oriented sponsoring would lead to envy between colleagues.
  3. The danger of personal enrichment or employees taking personal advantage is too high.
  4. Niche projects and small target groups would get too much funding compared.
  5. Employees who are less outspoken or less engaged would be disadvantaged.
All these arguments are most certainly valid and represent the fear that "something could go wrong". Of course sponsoring a large and well-established institution is much easier and safer, but also much less gratifying. And much less worthy of press attention and less outstanding.

I believe that all these concerns can be adressed by establishing simple rules related to funding:
  • Communicate the concept of employee-oriented CSR funding to all employees so that everybody understands the value of making CSR spending more personal and more related to the people.
  • Make CSR funding very transparent - from the internal application through the reasons given till the detailed spending report.
  • Publish follow-ups on past fundings to ensure sustainable spending and to give positive examples.
  • Make a very visible call for participation to invite all employees to suggest organizations and projects they care about.
  • Not every single project must be charitable for the general population - all projects taken together should have a sufficiently wide spread.
With these rules a company can easily resolve the concerns preventing the benefical combination of CSR spending and employer branding.

The following links discuss this idea in part without drawing the obvious conclusion that smarter CSR funding could improve employer branding for free:
Image: © Can Stock Photo Inc. / ribah2012 and / mindscanner


Adding Custom Menus for Linux Desktops

The "Start Menu" of a Linux Desktop usually comes with a predefined set of categories that make up the sub menus. If you have a lot of custom applications then you might want to group them under a dedicated sub menu instead of having them spread out over all the menu categories.

Adding sub menus and new categories on Linux Desktops is defined in the Desktop Menu Specification in Appendix C. It turns out that it is really simple and the following example from ImmobilienScout24 can serve as a base for your own custom menu.

You will need the following parts:
  1. A Desktop file using a custom category
  2. A Directory file defining the icon and description for the new sub menu
  3. The icon for the sub menu
  4. An XML file describing how to integrate the new sub menu into the menu structure and which categories of Desktop files to show in the new menu
The Desktop file describes the menu entry, in this example the VPN client:
The important part here is the Categories entry which specifies a generic category (Network) and a new custom category (X-IS24). The Desktop Menu Specification states that custom categories must start with X-. The Desktop file usually goes to /usr/share/applications.

The Directory file also conforms to the Desktop Entry Specification but is of Type Directory:
The XML file is placed usually in /etc/xdg/menus/applications-merged and extends the menu structure with the new sub menu, tying together the categories and the Directory file:
In this case we also exclude the X-IS24 category from the Network category so that our menu entries will not show up in several sub menus.

KDE, Gnome Classic, XFCE and other desktops with a regular menu all follow the same standards and show the new sub menu. Unity and Gnome 3 seem to have a fixed set of build-in categories and don't show the new sub menu as a new category.