twincode.net - Technical

zsh Color Woes (Changes for 4.3.9)

Tue, 24 Feb 2009 09:25:50 +0100

Dearest fellow Z shell users.

The customize freaks among us - who additionally run Debian squeezy might have discovered that our hand-crafted prompts from older versions (namely 4.3.6 and before) do not work any more.

After running a dist-upgrade, my formerly colourful prompt was a bland grey. And adding insult to injury, quite some applications (including VIM) had troubles with some sequences.

So, I went through the provided 'zsh prompt themes'. Lo and behold, there have been some changes that I actually like. There is no 'custom' hash of colour codes anymore, referred by '''$fg_bold''', '''$fg_nobold''' and the like, but the whole thing has changed and reminds me quite a bit of good old tcsh:

To extract some information from the manual page:

Visual effects
   %B (%b)
          Start (stop) boldface mode.

   %E     Clear to end of line.

   %U (%u)
          Start (stop) underline mode.

   %S (%s)
          Start (stop) standout mode.

   %F (%f)
          Start (stop) using a different foreground colour, if supported by the terminal.  The colour may be specified two ways: either as a numeric
          argument, as normal, or by a sequence in  braces following  the  %F,  for  example %F{red}.  In the latter case the values allowed are as
          described for the fg zle_highlight attribute; see Character Highlighting in zshzle(1).  This means that numeric colours are allowed in the second format also.

   %K (%k)
          Start (stop) using a different bacKground colour.  The syntax is identical to that for %F and %f.

From: ZSH Manual, Prompt Expansion (Visual Effects) http://zsh.sunsite.dk/Doc/Release/zsh_12.html#SEC45

So, this means:

%{$fg_bold[green]%}%n becomes %B%F{green}%n%f%b.

I quite like it.

Hope that helps anyone that looks for a solution.

When Agile Fails? (Methods and techniques misunderstood?)

Wed, 19 Nov 2008 14:43:41 +0100

Lately, I had the same experience over and over: people are rejecting certain agile concepts, methods and techniques - seemingly without or little reason. Examples are teams rejecting BDD or pair programming, line managers rejecting Scrum, ...

I tried to dive into that question and interviewed a lot of people, as well as reading a lot of posts and blogs that showed such behaviour. I found out quite some interesting things - some of them logical and clear, some of them not so much.

Now, the problem in my line of work and interest is that people working, no, fighting for better working environments and better products are hemmed if people are rejecting their ideals and working material. More to the point, if people start rejecting concepts outright, there's not much we can do in order to create win-win situations. And we usually get called when things have already gotten to a point where little hope is left ...

But let's go into details now:

Among the most common things I heard and experienced is

Every Miss Hurts

In short words: every time a sprint fails, a method becomes a tedious standard procedure, people are seeing that the applied method failed, and start to reject it - with good cause. If then the process is forced again and again on the team members and line managers, the notoriety of the applied method is becoming worse by the minute. Now, if that process or the whole method is abandoned, the naysayers (which by then encompasses as good as everyone on the job) will be proven right and will in future reject any notion of moving in that direction again.

Also, another common thing is

The Opposite Of Well Done Is Well Meant

A lot of teams have to pick up agile methods or concepts because a line managers insists. This is not necessarily a bad thing - sometimes people don't know what they need until they have tried it. Of course, it would be better if everyone supports that move, but this is not Utopia. The problem starts if people think that everything written in books or blogs is the recipe they need, the silver bullet. As we all well know, nothing is. No method is perfect.

The problem starts when people can not abandon the thoughts that everything is plannable and measureable. XP methods and Scrum are quite far from that - they help you to develop the best possible environment, and uncover failures that should be addressed. What they are not is a board full of spices where you pick the ones you like most. They are also not complete - any agile method must be introduced and then given ample time and space to develop.

Which brings us now to the attitude of

Do Unto Told, Else

A lot of misunderstandings appear in teams and for line managers because a absolute conceptual misunderstanding exists. Often the managerial advice "Do what I tell you" cannot be fulfilled - because neither the developers do know what their manager actually means, and the manager has no means to check if the team actually understood the advice (even if it is slightly more detailed than the example given).

This problem is particular to a lot of organizations in growth, but also applies for very old and crusty, bureaucratic organizations. It is a habit that is hard to break, because the lack of communication is not something that can be solved by recipes - again.

Another big problem of viewpoint and attitude is

The One For The Job

If you look into many organizations, the call for a 'cross-functional' team is one that sounds unheard. I do know that in many organizations this is not wanted or possible, even if the basic thought of 'Matrix' organizational structures for projects would support just that.

There is much more to that, though. There are the elititsts on one hand that swore never to touch the filthy base of the code again - they dabble with their playthings, thinking grand thoughts ... and then there are the worker bees, having a hard time understanding what those elitists babble about, but doing their utmost not to learn anything, in order to be able to do their work calm and peacefully. Noone is going to go out of their way to learn to work together, and most of the time this attitide is supported by the line management: they know whom to detail which task, and also know whom to blame if everything fails.

And the last (for this article) is

Where Change Is, There Fear Lurks Nearby

A manager might seem to be forced to change part or all of his organizational structure that supported him well enough until today. A developer will be expected to learn new methods of work. A designer is pulled from his comfort zone and thrust into a development team. A solo player suddenly should take up pair programming. A couple of serious engineers suddenly play silly card games.

The reason that agile methods are introduced are that someone thinks that there is room for improvement. Some people will not be convinced that easily, especially when confronted with unfamiliar methods. Most will naturally react with fear, followed by rejection.

But How To Cure That?

I think the first steps are being sincere and true.

First, we have to become clear in our terminology and teachings. Whether it is for managers or for teams, we should try to make it crystal clear that any wisdom gained from books alone is just a skeleton that has to be fleshed out by experience.

What is easily forgotten, or maybe never learned, is the fact that we are working with complex adaptive systems within complex adaptive systems. If we try to understand and control them in detail, we fail. If we try to control the flow and the results, we might win.

Then, it should also be clear that all methods have been created to get rid of dysfunctional counterparts. APM is for getting rid of dysfunctional project management, Scrum for getting rid of dysfunctional product creation methods and dysfunctional teams, XP at large for getting rid of dysfunctional development processes, etc.

Next, make clear that every change will necessarily mean change in some other part of the organization. I cannot introduce Scrum and ignore the uncovered impediments - I cannot introduce T(B)DD and not improve contact with my users or at least user representatives. I cannot introduce any agile method without any change to the processes and very possibly I even need change to the structure of the organization. Some of those changes may be minor, some aren't.

Most of the time we will be met with rejection in those cases. We will have to take our sweet time and explain what the reason behind it is - and motivate people not only to let them be taught and follow guidelines, but to partake and help shaping said guidelines.

And last: improve communication. Everything about agile methods is being in the very first word: agile. Being able to react without long preparations, long-hauled, expensive meetings and decisions made by a comittee of project leaders, that's the goal! But we can only do that if everyone knows about the vision, everyone knows about agile methods and her role in it, and everyone does the best to succeed.

At that point, a lot of naysayers pointed out that people are basically lazy, do not want to improve, and will reject everything. Also, they will not improve, because they do not have the intellectual capacity. Huh - I must have had the best and most docile teams, because after initial resistance, everyone, especially the scepticists were suddenly developing, supporting and carrying the processes.

I also heard that the roles, artefacts and processes are always set in stone and may not be changed howsoever. WRONG! (It sadly shows how little people understand sometimes, even people interested in the subject, and officially working 'agile'.

To tackle the technical communication issues between managers and developers, there's also only one answer: communication and trust. Both sides have to learn to talk to each other - sometimes it's just the matter of starting a project, planning, implementing and presenting the first functionality, and moving from there. Make it clear that one of the goals must be to still have specialists for everything - but to have other people being proficient with the same tasks in order to be able to mitigate temporay unavailability of said specialist.

For me, communication is the grease - it makes everything run smoothly. The fear will subside, the specialists teach others, the others will be teaching the specialists, the line managers will understand their role, the designers, developers, engineers will understand their role, there will be no need for reprimands, and trust in every direction will be built. To spare the worst message for last: the people that cannot adapt to the 'new way' will be leaving, and will be replaced by people that can adapt.

And last, an appeal: stop cherry-picking framework supports as methods or methodologies, unless you really really know what you are doing. Don't say you're doing TDD if 90% of your code still arrives untested in production. Don't say you're doing Scrum if you don't track and solve impediments and/or lack any product vision (among all the other points that usually show up).

Everything is about building excellent products with excellent people working on them.

Even more it is about building enjoyable work environments - which will in turn improve quality and performance.

Moving Pictures (Compiz and Co)

Thu, 10 Jul 2008 01:45:45 +0200

YAY! 2 boxes, 2 nvidia cards, 2x Debian (stable/testing and testing/unstable), 3 monitors .... behold the beauty!

nvidia-xconfig and nvidia-settings along with nvidia-glx and nvidia-kernel-src, module-assistant (m-a), a run-off-the-mill kernel, compiz*, and now I have rotating desktop cubes with nice pictures and and and and ...

without a lot of work. Just building the module, letting nvidia-xconfig do the rest - and now I have XFCE4 backed up by compiz and gtk-window-decorator.

WHEE!

X.Org Composite and xfce4 Compositor (Success!)

Sat, 28 Apr 2007 19:41:01 +0200

Long story short: an nVidia card is the better solution (I am biased, though). After enabling the composite extension in xorg.conf via

Section "Extensions"
  Option "Composite" "true"
EndSection

I can set the Compositor settings in Window Manager Tweaks. And it works, just the xfce4-terminal can't be convinced that I want a transparent background. But for typing, that's not very nice even ...

How to use mail-notification over secure imap (When License Issues strike ...)

Sun, 04 Jun 2006 00:33:48 +0200

From /usr/share/doc/mail-notification/README.Debian:

mail-notification for Debian
----------------------------

I had to disable SSL because of a license issue.
Hopefully, this will be solve soon.

Now, how to work around this?

Easy: use stunnel

First step: install it:

aptitude install stunnel

Second step: test it

As root, run: stunnel -c -d 9993 -r [targethost]:993 (replace "[targethost]" with your target host).

Then, configure mail-notification (I won't outline this here, just point the IMAP-host to localhost, port 9993).

It should work already. If it does, configure stunnel:

Again as root, run editor /etc/stunnel/stunnel.conf.

Change the lines:

cert = /etc/stunnel/mail.pem

;cert = /etc/stunnel/mail.pem

Comment out the not-needed blocks after the line

;Use it for client mode

and add the following

[imaps]
client = yes
accept = 9993
connect = [targethost]:993

(again, replacing "[targethost]" with the corresponding hostname or IP address).

First, kill the running stunnel process, then run the following:

# cd /var/lib
# mkdir stunnel4
# chown stunnel4:stunnel4 stunnel4
# invoke-rc.d stunnel restart

Voilà.

Transparency Woes II (Can't we play nicely together?)

Sat, 03 Jun 2006 23:12:43 +0200

System: Debian SID
Hardware: ATI Radeon 9700pro
X System: X.org
Program: xfce4
Target: allow visibility.
Status: bleh

Unless I specify '--compositor=off' in the .xinitrc for xfwm4, the system is just giving me the notice 'XRender too slow' and then goes haywire.

I want a new fglrx installer.

Or an nVidia card.

Benchmark Beats (F|S)CGI (When the cause is out of scope.)

Mon, 15 May 2006 00:58:55 +0200

I admit it, sometimes my 'haste' gene gets the best of me, and makes me do stupid things. This time, Jürgen and me worked hard at bringing an old server (namely this one) to it's knees - via ab2.

Sadly, this worked surprisingly well. Symptoms for the wary: Memory footprint was small, the database was idle, the apache processes few and far between. SCGI seemed to be the wrong thing for - and that's the puzzling part - fast serial requests, while it coped comparatively well with concurrent request.

Even moving it to another machine helped little ... it just let the load skyrocket. Cursing and steaming, I reconfigured the whole thing to use fcgid. This was the proverbial hop from the pan into the fire: the whole server was unresponsive, the fcgid processes crashed. Yikes.

So, I took the scenic route and tried to profile/benchmark the application. No mean task, but I went from checking the database (still idle) and models (blazingly fast) to the render component - which was still fast, but the wait time seemed awkwardly long.

I checked the views, and Bingo! the culprit was found. A innocent call to markdown() made the poor computer run in circles. Yes, my naricsm made me run everything 'dynamically', storing the markdown in the page and rendering on demand. With my affinity towards long articles, this lead to glacial loading pages. D'oh.

The first solution, truncating the text, lead to even more confusion, it wasn't pretty, yadda yadda. Then, a friend, confronted with my whining about the whole thing, meant: "Why are you rendering it on demand, why not when saving?". Duh. Yeah. Stupid me.

Another field in the database and a handful of changes later, it was done. Now, the text is rendered to html via markdown() into another field before the article is saved, the pages display the rendered content, the textarea on the admin pages displays the raw content.

Note to self: don't render on demand unless absolutely necessary. Lesson learned.

NB: Thanks for the help!

Nota Optime: By the way, SCGI outperforms fcgid by factor three (after the change). Go go go!

Ruby and Rubygems on Debian (Love, Hate or - Freedom?)

Wed, 10 May 2006 19:23:04 +0200

The Situation

Debian is a Linux distribution that comes with a great package manager, a good policy, a big supporter base and different flavors. For the normal user or server, it is advised to use the version called 'stable', while the 'Power-user' or 'Bleeding Edge Adopter' will want to try her hands at 'testing' or 'unstable'.

Ruby, my favorite programming language, has adopted a packaging/versioning hybrid system of it's own called rubygems.

For the attentive reader it is not hard to see that there might be clashes.

The Problem

There are a few, therefore

Facts First

Debian has been following a strict policy, including adhering strictly to open licenses and the file system hierarchy standard (FHS). For a programmer/sysadmin hybrid like me, this is invaluable, a Good Thing™. Some things seem like a corset at first, but in the end there are a very few things that you need in everyday use which only seem awkward at first. These are:

You must not install software which doesn't come in Debian package format (.deb).
You should use the Debian helpers to manipulate run control files and executable alternatives.
You must honor the FHS, eg. use /var for variable files.

I will explain later on why this is important.

Ruby itself is packaged as Debian packages for all distribution versions. But while in testing/unstable there is version 1.8.4 available, at the time when the current stable version was created, Ruby made it in with version 1.8.2. This is rather bad, because most Ruby software and tutorials nowadays are based on 1.8.4, and there have been a lot of changes and bug-fixes. Yes, I will explain backports, too.

Observations Second

The current Ruby Maintainer adhered to the policy of 'create a modular system' - a little too hard in my opinion. This led to ruby being split up in quite a handful of packages. If you do not find articles like http://www.debian-administration.org/articles/329, where installing Ruby on Debian is explained, it can be quite hard to get all the packages right at first try. It is not hard to fix that, though, and the Ruby packages are under superior maintenance. You should be able to google for 'Install Ruby on Debian', maybe.

A lot of ruby programs are not available on Debian. The reason is that gem packaging is too easy, and forgives that the provider of the gem ofttimes isn't able to cater to the whims of every system. So, the Debian maintainers are every now jumping from the fry into the fire, as they have to patch the gems heavily to make them work together nicely with dpkg (the Debian Package system). Thus, a lot of packages are not very recent, because it's a lot of work.

The solution seems to be easy: install rubygems. But that's something a true system administrator will cringe over. Why? Well, you don't mix package managers. No, really, don't. It's a bad idea. Especially when both package managers are writing in the same directories. That leads to race conditions, like both think that a certain file is belonging to their package, and remove it on updates, or on removals. If the file belonged to the other manager, you're pretty much screwed.

If you want to try the outcome, run setup for a box (or VM) with Debian Sarge, install 'rails' from Debian, install 'rubygems' from source, run 'gem install rails' and 'dpkg -r rails'. That can render rails unusable, even if it's still there!

Boiler Plate

Now, the people working with Ruby and the Debian maintainers are all intelligent, headstrong and opinionated people. This can be seen in discussions on the Ruby on Rails and ruby-lang mailing lists where certain people are fighting open-handed and close-minded about who has the longer ... whatever.

But is it really necessary to state things like 'I feel that rubygems belongs in /usr/lib' when there's the valid complaint that rubygems can hardly be coerced to install somewhere else? Is there a reason that authors of great libraries cease to support people because they are gasp using Debian? Or, from the other side, is there a reason to resort to name calling or belittling the authors of software?

Reasoning

We all are called to resolve conflicts like those. Emotions, animosity and alienation have never brought anything good. But what can we do?

The Debian people can provide tutorials to use recent ruby packages on all Debian flavors, or even run package mirrors for backports (Here's the explanation: a backport is a package originating from Debian testing or unstable, which has been ported back for being used in Debian stable). They can also provide templates and tutorials for people who are creating ruby programs, so they can easily be integrated into Debian.

The rubygems people can provide a means to have fine grained control over the place where rubygems has to be installed, or even discuss with Debian people where they files should go, or provide insight where patches could help to generate this missing functionality. And please read Debian/Ruby Extras - Position on RubyGems. It's not mean, it's facts and to the point.

And me?

Well, with help from my friends who want to have a smoother integration from ruby and ruby programs into Debian, I will provide:

Tutorials
Links
Unwanted advice

Or even dedicate one of my valuable pages to the solution of the whole conflict.

Believe it or not, we have the same goals in the end.

Feeble Forges (Does having 'forge' in the URL make things go amiss?)

Fri, 05 May 2006 14:29:35 +0200

What's a Forge, anyways?

According to Merriam-Webster:

Main Entry: ¹forge
Pronunciation: 'fOrj, 'forj
Function: noun
Etymology: Middle English, from Old French, from Latin fabric, from fabr-, faber smith
1 : a furnace or a shop with its furnace where metal is heated and wrought : SMITHY
2 : a workshop where wrought iron is produced or where iron is made malleable

Well, in our case this has started out as open source libraries, namely sourceforge and it's pendant, rubyforge. While being a good idea, I have a few words to say:

Signal/Noise

The signal/noise ratio is bad. At both sites. I have been searching for interesting projects on both, and more than once ended up on a dead end, with the project, while sounding interesting, being in Alpha or Planning or Pre-Alpha state - with the last comment dating back to before the discovery of the wheel. Releases? Naaah. The worst part was one project which has been taken over by another person just to seemingly finish it's days as a less than half functional application.

Don't get me wrong: I'm not angry. There is a multitude of applications which have been developed and released with the support and convenience provided by these sites. The community and collaboration options are outstanding.

But it makes me sad that so many cool projects are abandoned because of the missing second rule: Release Often.

CMS Soup (Rails against Fails?)

Thu, 04 May 2006 11:40:10 +0200

Content Managements as a whole

A path of thinking brought around by the essay why content management fails by Jeffrey Veen has been in my thoughts as of late: The summary of the essay is as follows: the content - as in "gathering the information", "observing law and policy rules" and "writing nicely" - and the design are more important than the management system. The usual approaches - letting programmers design the pages is like hitting them with the ugly stick (sorry fellas), letting a lawyer write about his work is challenging at least, and letting anyone but the legal department decide what goes on the pages is asking for trouble, really - are hurting the company in various ways.

Rails as solution?

At least part of. Ruby on Rails takes the importance of the system away, because it is so ridiculously (in the best sense!) easy to write a new CMS ... and another ... and another. There, don't focus on the CMS, focus on the content and processes!

But let's not forget the valiant people at http://radiantcms.org/, who seem to put a really nice piece of work together for everyday use. (Sidenote: My friend and programmer Andi likes the DSL Radius, I don't agree, because the repeat features and things are again writing a seperate language where ERB does it's job already. But tastes differ ...)

X.org Composite (Oh, please, please, please, please ... I want my eyecandy!)

Wed, 03 May 2006 21:03:00 +0200

I'm running Xfce4 on a Debian Sid, and I've been reading about X.Org Composite extensions ... you have no idea how much I want those gorgeous real transparent windows - the drop shadows are just adding to the beauty!

But, alas, there's no gain without pain: the Debian package for xfwm4 has been compiled without compositor. Bleh.

So, I tried xcompmgr and transset, and while it worked, the performance went down the dumps. So, with a heavy heart, I decided to let the plans rest until the performance is better.

I've been waiting for years, so another won't matter, and I guess it won't be that long until everything works.