F* Yeah! 2 + 1 * 3 = 5

I spent some time lately just reading about compilers a little bit. Nothing too in depth but I wanted to explore the idea of a basic compiler. So I worked my way through some Wikipedia link jumping. Reading about lexical analysis, parsers, grammars. abstract syntax trees, stack machines, register machines, some of everything. I'm surprised I didn't end up on Kevin Bacon's page.

The Idea

Anyway, after all of that I realized something about an exercise we did in school. Take an arithmetic expression, something like 2 + (3 - 1), parse it into a tree data structure, then evaluate the tree. This was basically a simple compiler. We didn't have to describe the lexical and grammar rules, nor generate an intermediate or executable representation, but it was still the same idea. Given this, turn it into something else.

So that sounds like a perfect project, so lets do it in Erlang.

The Lexer


I know the tokens I want: +, -, *, /, number, (, )

After a bit of research I find that Erlang has a lex implementation call...wait for it... leex. Given some lexical rules (defined with a bit of regular expressions and Erlang code) it'll generate me the Erlang code for the lexical analyzer.

Well that's easy. Here's are rules to turn a string representation of an expression into a series of tokens:

[0-9]+ :
  {token, {number, TokenLine, list_to_integer(TokenChars)}}.

[0-9]+\.[0-9]+ :
  {token, {number, TokenLine, list_to_float(TokenChars)}}.

\+ : {token, {'+', TokenLine}}.
\- : {token, {'-', TokenLine}}.
/ : {token, {'/', TokenLine}}.
\* : {token, {'*', TokenLine}}.
\( : {token, {'(', TokenLine}}.
\) : {token, {')', TokenLine}}.

[\000-\s] : skip_token.
[\n] : {end_token, {'$end', TokenLine}}.

The Parser


Ok, I have my lexer. I can turn "2 + 1 * 3" into [{number, 1, 2}, {'+', 1}, {number, 1, 1}, ...
That first 1 on all the tokens is the line number. In my current situation I'm only worrying about a single line expression.

Working directly with that list of tokens would be about as fun as fighting a cool monkey. I want to be friends and he wants to look like a badass while throwing poo. This is where a grammar definition and a parser comes in.

And guess what... in the butt? No, Erlang has a yacc implementation called... wait for it.... yecc. So here's my grammar:

Nonterminals expression.
Terminals number '+' '-' '/' '*' '(' ')'.

Rootsymbol expression.
Endsymbol '$end'.

Left 300 '+'.
Left 300 '-'.
Left 400 '*'.
Left 400 '/'.


expression -> expression '+' expression : [ '$2', '$1', '$3' ].
expression -> expression '-' expression : [ '$2', '$1', '$3' ].
expression -> expression '/' expression : [ '$2', '$1', '$3' ].
expression -> expression '*' expression : [ '$2', '$1', '$3' ].
expression -> '(' expression ')' : '$2'.
expression -> number : '$1'.

Basically I'm saying that every list of tokens should result in an expression (a nonterminal). Which could be a number, an expression inside parentheses, or two expressions joined with one of my supported operators. Those "Left " is the associativity, precedence, and token to aid the parser in getting things correctly.


The Generator


I have a lexer and a parser, sweet. What does that mean? I can turn "2 + 1 * 3" into

     [{'+',1},
          {number,1,2},
          [{'*',1},
              {number,1,1},
              {number,1,3}]]}



I know that is a terrible representation, but's an abstract syntax tree represented as a list (brackets) and tuples (curly braces). The '+' token is the head with only a number to its "left" and a subtree to its "right". Since the tokens come out as a list of tuples, we need another method to represent the tree. Lists will do just as well. This probably looks strange to an OOP guy, but structured data is structured data when you are processing and translating.

Well I'm not done. I have an abstract syntax tree, but how do I use it. Lets turn it into a made up stack based machine language:

-module(generator).
-export([generate/1]).

generate([Root, Left, Right]) ->
  LeftExp = generate(Left),
  RightExp = generate(Right),
  RootExp = generate(Root),
  LeftExp ++ RightExp ++ RootExp;
     
generate({number, _Line, Value}) ->
  [{push, Value}];
    
generate({'+', _Line}) ->
  [{add}];

generate({'-', _Line}) ->
  [{subtract}];

generate({'*', _Line}) ->
  [{multiply}];

generate({'/', _Line}) ->
  [{divide}].

That's the whole generator. The top generator() function just visits each node of the tree while the others rely on Erlang's pattern matching to turn a token into an instruction ({number..} into {push}). You are probably thinking why didn't the lexer generate these instructions as the tokens? Well you don't want to combine these truly separate concerns do you? Also the straight list of tokens as they come out from the lexer wouldn't work with a stack machine where you need to push the data on a stack and pop it off to operator on it. Oh, my operator instructions do the popping.... anyway.


The Virtual Machine


Now I'm turning "2 + 1 * 3" into [{push, 2},{push,1},{push,3},{multiply},{add}]. I can go from a line of "code" to a list of machine instructions. Hot damn, but I still don't have the evaluated value.

At this point you might need to know how to read Erlang code, so I'll skip the gist embedding and provide a link to my git repository for the curious:

https://github.com/copenhas/mathexp

Ok, holy crap. What else do we have to do just to evaluate some math? You start the virtual machine (which runs in it's own process), you send the VM your program (list of instructions), then asynchronously it runs the program and finally will send you the return value in a message.

I'm afraid I'll also leave understanding the basics of a stack machine to the reader as well, but the example of a program above should give you a good idea. I don't think I'm strictly following it (I believe they are usually a 0 operand instruction set not a 0-1), but it works pretty well and the code is pretty straight forward.

Nothing impressive, but the title expresses my feeling when I saw it work.

The Absurdity

Here is the absurdity as a series of Erlang code to get that magical (high) 5.... he he he.

{ok, Tokens, _EndLine} = lexical:string("2 + 1 * 3").
{ok, Ast} = grammar:parse(Tokens).
Program = generator:generate(Ast).
vm:start().
vm:load(Program).
vm:stop().
receive {return, Value} => Value end.

Erlang's assignment does data unpackaging (pulls the data out and sets the variable) and pattern matching (the right side has to match the format of the left). That last line would actually print out the value in the Erlang interpreter. This last bit might have been confusing but I'm going to leave you hanging.

ASP.NET Web Forms vs ASP.NET MVC: The Middle Ground

Ok, so it is about time I contribute to this blog as I have been listed as a contributor since it started and have yet to write anything. I apologize in advance for grammatical errors. This is something that has been bothering me for a while and I am seeing more and more argument about this online and among fellow developers and while both sides have their points, I find that there is a nice middle ground that most ignore.

Preface: This is about the .NET MVC framework, not the MVC design pattern. With the exception of one point in the article where I mention the design pattern specifically, any reference to MVC is related to the framework.

ASP.NET Web Forms vs ASP.NET MVC: The Middle Ground

I have heard one problem on both sides "It creates spaghetti code". For MVC it is hard to find where the actual work is being done, where it is posting, etc. For web forms it is the IsPostBack check in the page_load of every page to handle different events and lets not forget the ViewState. Both are valid points in my opinion and I am not a fan of either.

When I started writing websites, I used straight HTML for my pages. My first server side language was Cold Fusion (ugh.) and I never used the built in widgets for rendering. I followed that with ASP 3.0 and that didn't have any (that I knew of). When I started with .NET, I wrote my pages the same way. I never used the web controls provided by .NET, I used normal code. For/While loops instead of repeaters, Response.Write rather than labels, etc. I would use the normal HTML form tags rather than .NET's. I never used .NET's event model, relying instead on javascript and I didn't even know the ViewState existed for the first few years of using .NET.

I would make all of the data I needed for rendering page available to the view after the page_load was complete and that was it. .NET was no longer involved once the page was rendered, and when another page was called I would pull from the Request object for query string or form values and pump data back into the Response object if necessary. If I needed something to persist in the app or session, I used the Application, Session, and Cookie objects provided.

I always felt that this gave a better page lifecycle and user experience anyway, users could use the back button without messing up the flow, and a page only knew how to do one thing, two at the absolute most. I have heard MVC called an advancement of this methodology, and while I agree to a point, I feel it overcomplicates things is some ways.

The first thing that MVC is supposed to provide better than web forms is true separation of concerns. The model does the work, the controller acts a gate keeper, and the view just does the rendering. While all of this is true, I don't see it any better than how web forms CAN be used. Now, I have been told that the way I use web forms is practically the MVC design pattern, so I wonder why is there a need for a completely different set of libraries and handlers if that is the case. What am I really gaining from having the advanced routing methods? Prettier URLs? That is only an advantage for the end user because the views are still hard tied to controllers. Yes, a controller can return something else (such as json) without a view, but that is a marginal improvement at best. The only other advantage I see is less files in the code base, but the controllers are large if they handle a number of actions and can be a bit of a pain to search when you are looking for a specific one. In web forms, I know exactly where to look for my files based solely on the URL. The code behind can (and should) act as a "Controller" with one action. It will talk to the "Model" and handle flow control and redirection based on the request data or the "Model's" response. With the exception of smaller websites that have very little business logic to them, the "Model", whether in in MVC or Web Forms (in a perfect world) is separate from the website anyway. Whether it is a one or two-tier app with business logic in separate libraries/classes, or it is a separate entity (WCF, Windows session, etc) in a n-tier environment, the "Model" is already separate. So once again, MVC is not providing much.

I think the key thing with Web Forms has stirred up so much ire from the development community is the "pluggable widget model" (as D2 calls it). The intent behind the Web Forms design pattern is to act like a WinForm app, and since the early days of VB, WinForm development has been very much plug and play. Developers just drag and drop objects where they want them then wire it up in the back end. In this "controlled" (I use that term lightly) environment, it worked pretty well, but the web world is not as controlled. Programmers don't have the luxury of controlling every action of the user(without some crazy javascript, and that is only if javascript is enabled), only the browser does. After a web request is complete, there is no longer a connection to the server until the client does something else. That is the nature of the web. Embrace it.

When many developers tried to make the shift, from Win to Web, they tried to follow that ideology, not remembering that users, for the most part, are dumb. They don't read your warning not to use the back button, and for the most part, don't care about how you want it to be used, they will use it way they think it should be used, and this causes a lot of headaches. The result was spaghetti code.

The other cause of spaghetti code was the RAD model that Web Forms allowed. People with little to no development experience could create a website that did all kinds of flashy things and provided the end result they wanted without having to worry about silly things like security or maintainability.

In either case, these developers carved out a nice little niche and business types liked their quick turn around and usually low prices so they were hired to write apps that later needed to be maintained by others. Sometimes those others were similar developers that made the code base worse, and other times they were real developers that spent much of their time beating their heads against the wall and sending snippets to The Daily WTF, all while being told by the client that they don't understand why it is so difficult, "It's just a web page".

Like I said earlier, I don't like the Web Forms design model, and I don't see any real reason to use .NET MVC. Both have their advantages and disadvantages, but if you write code with the mindset of how the request/response model works and write your own html, why not use what .NET provides by default? This makes deployment easier as the boxes that house the web app will not need another set of libraries installed on top of .NET, and maintainability is a lot better because of the better codebase structure. This also ensures that your JS and CSS will act the way you expect them to and that you get a lot more control over what is happening at all stages.

Regular Expressions and HTML

In lieu of writing something interesting or thought-provoking (at least to me) or making any kind of point, today I would instead just like to share something for posterity.  This was in response to the never-ending stream of questions on Stack Overflow regarding parsing HTML with regular expressions.

---

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the trangession of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

---

In other words, it can't be done. Stop trying. Use XML libraries and HTML DOM parsers, there is no shortage of them.

Originally posted as an answer by bobince on Stack Overflow.

Abstractions: Where do you live?

You know software development and computer science are extremely overwhelming fields. I believe the saying goes something like "the more you learn the more you realize you don't know". That might not be perfectly quoted but you get the idea. We luckily take advantage of previous ideas and build up abstractions and layer to help manage all this overwhelming complexity.

Lets just take a quick look at a set of related abstractions:
electronic signals (yeah that's right go fuck yourself)

Ok seriously, I'll try to list what's involved with just data access:

• Set and Relational Theory
• Relational Database Topics (schemas, functions vs stored procs, views, transactions) 
• Sql Server
• SQL
• OLEDB
• ADO.NET
• NHibernate

I think that hits the big ones involved for a general .NET data access stack. Now working at the NHibernate level that abstracts away a lot of ADO.NET (provider creation, connections, transactions a little, generating some sql), but you aren't exempt from knowing about it. Beyond that you can't ignore that fact you have a real database underneath that speaks SQL and has its own way of doing things (Sql Server vs Oracle for example).

That's something everyone has to keep in mind: working at one level of abstractions does not mean you can be ignorant of lower levels of abstractions. I do differentiate between the required knowledge to work at a level of abstraction and potential knowledge.

Taking my previous example if you are working at the NHibernate level you are required to know basic NHibernate and some of it's abilities. You are also required to know SQL and something about relational databases (a little design mostly). Now there is the potential (and advantages) to knowing the inner workings and reasoning behind NHibernate (such as Unit of Work, advanced relational mapping), how to work directly with ADO.NET (providers, core interfaces, managing connections), plus all the things you can learn about how your database of choice works.

So what does this mean for people coming into this field... I hope someone is looking out for you. Seriously, you should try to get in at some level of abstraction with a mentor. Realistically you'll have to put some effort into finding a mentor. They can guide you into understanding the level of abstraction you'll be living at and figuring out how it fits in.

What does this mean for people already in the field? You should figure out the potential knowledge in your stack of abstractions you work with (both above and below where you are). Figure out where in that stack you live and try to tackle at least 2 levels down to increase your understanding. I would also suggest going up a level to keep in mind how something is used, but the important thing is to not be ignorant of what's powering your level.
 

Typing - I'll take both

I have seen my share of dynamic vs static articles and they have raised my eyebrow enough times I'm thinking I need to distill my current thoughts in a post. Let's see where this goes...

Firstly, WTF? Seriously, everyone seems to get really defensive about typing. I'm just getting annoyed by the whole thing. I see tons of posts that start off great turn aggressive either due to the author or (usually) via the comments from very opinionated readers. Then sprinkled between those are another annoyance I'm seeing on blogs and project comments. Yeah I think we know you have to make educated technical decisions based on your requirements, but seriously I don't need someone chiming in with nothing to add except "best tool for the job". Ok, I think now that I can try and put some thoughts about the debate down.

I personally think that dynamic typing feels little more closer to how object-oriented programming is suppose to feel. By that I mean the fact that duck typing makes you only concerned with what an object is capable of. You can't think about types when really writing code with duck typing. You are thinking about the object and not the class. Either it can perform the action you want (or process the message) or it can't and you'll get some delegated response or an error. Of course it's that runtime error that gets people twitchy, but seriously not all your code is controlling the space shuttle flight control systems.

Of course static languages give you some explicit self documenting code that the compile can do light correctness checking on, but you do sacrifice some flexibility. Also I find that design artifacts start to pop up from the usually static type system. Things like extra methods on interfaces (as in public members) that are only used if certain properties are a specific value (I know there are ways to design around this). Or heavy use of generics (as in the C# feature), interfaces, design patterns to try and get some extra flexibility out of your code. So there is certainly some trade offs as you would imagine.

Now I have messed around with F# some in the past and I have to say that the type system most common (such as the class system in C# or Java) doesn't do a lot for you. You have to fully define your types and declare them for use everything. It's very explicit on your part and the compiler doesn't do a lot for you. C# did introduce a very light type inferencing with the 'var' keyword but it's abilities are miniscule. If you take a seriously look at F# (I have heard that Haskell is mind blowing) you'll see that it infers types for you as much as possible. You can declare functions without any type information and the compiler will make sure you are passing in types that can fulfill the needs.

I need to spend more time in an advanced type system so I can understand how far type inferencing can take you, but I would like to see languages have a focus on dynamic typing with an optional static system. Even though I love dynamic languages there are still areas you want static typing such as external facing interfaces or touch points between components even inside the same system.

Anyway, that's enough for now.

Why I Don't Like Microsoft's Widgets

I previously pointed out that .NET questions on Stack Overflow seem to be degrading to nothing more than beginners asking about how to plug together Microsoft's widgets to produce a desired effect.  That tide seems to be subsiding, though it is still clearly present.  I guess it's just the nature of the beast these days.  The market is flooded with entry-level developers (many of whom are no longer considered entry-level because of their experience, whether or not they've actually learned from that experience) who use frameworks and tools and, ultimately, pluggable widgets.  I see Microsoft as the king of these widgets, hence my generalization.

But I was thinking today about the whole picture and wondering why I specifically don't like this.  Basically, I was helping a few people on Stack Overflow with some ASP .NET controls and had to put in some serious effort not to just rant about how they're going about the whole thing all wrong and that they shouldn't be afraid to write some code themselves instead of using Microsoft's controls.  (I particularly cringe whenever I see remnants from .NET 2.0 web controls when Microsoft went control-crazy and hooked up some provider frameworks to make web development as plug-and-play as possible.  People still use that stuff.  A lot.  And it really bothers me.)

It occurred to me what the root of my concern really is.  It's not entry-level developers, it's not frameworks and tools, it's not even me being some stodgy old know-it-all because in my day we used to program in bare text editors and used terminals and get off my lawn!  It's because these widgets address the wrong side of the equation.

Writing software is easy.
Supporting software is hard.

It's that simple, really.  (And reminds me of one of my favorite quotes... "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian Kernighan)

Microsoft creates clever widgets which a novice can, with minimal training, plug together to produce a desired result.  This makes the step of writing software easier, but at the cost of supporting the software.  And that doesn't make the entire process any easier.  In fact, when you look at the whole picture of the software life cycle, it ends up being more costly in the long run.  Small changes to requirements or small additions to business logic that exist just outside the scope of the pluggable widgets require significant effort to implement.  Custom code, on the other hand, isn't difficult to change at all (if written well).

We're really reminded of this on an almost daily basis when developing in .NET.  Think of a .NET stack trace.  At the top is where the exception ended up (caught somewhere useful, I hope), and as you descend the trace it goes through your code so you can see where it was thrown and the path it took down the stack.

Many times, however, by the time you get to the bottom of the trace you are in Microsoft's code.  This isn't really a bad thing, after all it is all running on top of Microsoft's libraries.  So this is bound to happen.  And the more experience we have on the subject the more we can identify what's wrong by looking at Microsoft's part of the trace as well.  But the boundary is still clear.  Once you cross over from custom code to System.* then you officially enter the black box.  There are some things you can do, but it's just not the same as code you can actually step through.

How do you step through a DataGrid entirely defined in HTML controls and bound to a data source also defined in HTML controls?  You didn't actually write any code.  Do you even know what it's doing?  Maybe you do, maybe you don't.  Maybe the next guy who has to support it does, maybe he doesn't.  That's a problem.  But when something breaks, what do you do?  Your stack trace begins with code you didn't write.  For an experienced developer this may not present a huge problem, albeit an annoying one, but for that novice who was hired because he demonstrated an ability to plug widgets together it could be a serious stopping point.

I'm not saying that we shouldn't use tools that are available to us.  But we need to know what those tools are doing.  You can't make a career out of "I press a button and this happens, so as long as people want this to happen they can pay me to press this button."  Don't let Microsoft do the thinking for you.  Writing the software is the easy part.

This VIM is a bomb!

For our CodeDojo presentation meeting a month ago, I did a presentation about VIM. I didn't go into any details about basics of getting around or how to install it. Those are too intro. What I tried to talk about is the immediate annoyances of VIM and a couple of concepts to keep in mind during your adventures. I thought I could mention a couple of things from the presentation and talk about my profile I'm building.

I really can't go on without mentioning the power of normal mode in VIM. VIM's modes are actually quite the advantage when you get used to them, but you should keep in mind that you want to be in Normal mode. It truly is where one of VIM's strength comes out, and that is text surgery.

# Language of Text Editing #
### `vimsentence := [count]vimexpression`
### `vimexpression := vimcommand | <operator><motion>`
### `vimcommand := <action> | <motion>`

### _That is while in NORMAL mode_

# WTF? (examples) #

### Delete next 3 words: 3dw

### Paste above line: P

### Move cursor down 2 paragraphs: 2}

As you learn some of the different keys you can apply them in pretty interesting combinations to build up sort of sentences of what you want to do. It takes a little while to get efficient at this, but sort of a rhythm starts to occur as you dispatch lightning ninjas from your Zeus fingers....hmm.. anyway.

The other Zeus like feature of VIM is it's plug-in and profile abilities. It's absurd how much you can customize VIM itself, but then on top of that you get python, ruby, perl, vimscript, shell, etc scripting to enable even more. When a plug-in and a developer love each other they, the developer lets the plug-in in to play with the others. This is where profiles are born.

Now to a newbie, straight out of the box you get a confusing text editor where you'll spend the first 5 minutes wondering why you can't type in text and the next 5 figuring out how you exit without killing the process. After a couple of tutorials and some VIM'ing, you probably have a profile pieced together by snippets that sort of work and that you don't understand. Well that's where my profile is trying to come in.

Here's a snippet where I setup some stuff with folding blocks of text:

" When we fold or unfold a block of text determine the block
" delimiters via syntax. You can use 'za' to toggle a fold.
" There are several other commands as well.
" 'zM' to close all folds
" 'zR' to open all folds
    set foldmethod=syntax

" Lets map 'za' to spacebar while in NORMAL mode
    nnoremap <space> za<space>

" By default I like to see all the code, but the first time
" you try to fold something this will get toggled and all folding
" well be on.
    set nofoldenable

As you can see I'm trying to document and organize things. The whole idea is that someone who knows a little about VIM can use my profile to learn a hell of a lot more. The profile is also to include several plug-ins in an attempt to create a ready to go development environment for a few programming languages. Right now I have a fairly sophisticated setup for Python (refactoring, test running, error detection, auto-completion) and something very usable for Erlang (syntax, error detection, auto-completion, compile a single file). Now I also have some built-in plug-ins and a couple of other things for other languages but I just haven't spent a lot of time tracking down, configuring, and testing plug-ins for other languages besides Python and Erlang.

I'm welcome to any help or suggestions in tracking down more to flesh out a Mono/C#, Ruby, C, or just about anything else. Anyway anyone interesting in using VIM please go ahead and check out my profile.

https://github.com/copenhas/dotfiles

My presentation is also available but the slides are pretty skimpy. You also need to use showoff (ruby gem) to actually see the slides in full.

https://github.com/copenhas/presentations

Of Horses and Carts

As developers, we like to code.  We want to write code.  It's what we do.  So, naturally, when a project begins what we want to do is dive in and start writing code.  From a proper planning perspective, this is generally frowned upon.  And for good reason.  When you're just starting to plan and haven't flushed out the details and don't have a firm grasp on the actual requirements (not just the documented requirements that some business user wrote down) is precisely when you shouldn't be etching into stone the logic to be used in the software.

But this reality can easily be (and often is) misconstrued as a mandate to not write any code just yet.  This is a fallacy.  Writing code isn't the problem.  Writing code that's etched in stone is the problem.  And overlooking the actual problem by mandating against what is essentially a means to the problem very easily leads to not solving the problem, but instead just moving it somewhere else.  Somewhere sinister.  The data model.

We've been writing software for years, and we generally know how it goes.  Almost every developer still does this just out of habit.  First you build your database and model out your tables, then you write your code to sit on top of that.  Right?  That's how everyone has always done it, so it must be the way.

Sadly, and at the cost of untold man-hours, it is not the way.  But it's just such common practice that people continue to behave in this manner out of nothing more than habit.  It's what they know, it's how they think, and it's a tried and true approach that management understands so it's the safe route.  (Safe for the developer, not for the ongoing maintenance of the software.)

What is essentially happening here is that the early attempt at solidifying the requirements is being etched in stone in the database instead of in the code.  And raise your hand if you think that re-factoring a database later in the life cycle of the software is significantly more difficult than re-factoring the code.  That's what I thought.

It all comes back to my favorite of favorites... separation of concerns.  You may be using proper IoC, you may be putting in hard assembly or even service boundaries between your layers.  But you haven't flushed out all of those dependencies.  The overall structure, in every direction, still depends on its core.  And when you first begin designing the software you are essentially designing its core.  The choice is yours... Should the core be the data model or should the core be the domain model?

Let's go with the common approach, the data model.  You build your ER diagram, create your tables, map your keys, create your association tables for those pesky many-to-many relationships, etc.  You now have a core database upon which your software will sit.  Essentially, you now have this (pardon my crude diagrams):

Your layers are separated, and that's all well and good.  But notice a subtle dependency there.  The overall shape of your software is governed by its core.  There's no getting around this, not unless you do what will likely amount to more abstraction than you need in a highly de-coupled service architecture.  (Get ready for tons of DTOs and "class explosion" for that.)  Even if these are broken apart by assembly and dependency-injected and all that happy fun stuff, there's still the underlying fact that your software's core is its data model.  What happens if that data model ever needs to change, or if you need to move to a different data store entirely?  A lot of work happens, that's what.

Consider instead shifting your core a little bit.  Imagine for a moment breaking that cardinal rule that "thou shalt not code first" and actually begin the design by creating your domain models.  In code.  What about the database?  You can figure that out later.  Or, at least in my case, hopefully a trained data modeler can help you figure it out later.  (Developers like to think we're also data modelers, but most of us just aren't.  A lot of that comes from the fundamental differences in design and function between object-oriented thinking in code and relational thinking in an RDBMS.)  Now, you have this:

The structural dependency is still there, but the core has shifted.  Your data model was built to accommodate your domain model, instead of the other way around.  By this approach, the data persistence is simply an interface which interacts with the domain, no different than the UI or anything else that hooks into the central domain.  The idea here is to be able to re-factor things more easily, especially in the data model (where significant growth can lead to unforeseen performance problems and scaling issues not evident in the original design), without impacting the entire system.

Many times this boils down to a cultural problem, really.  Businesses have spent decades with the understanding that "the data is paramount."  While there is generally truth to this statement, it should not be extended to believe that everything about the database is the core of your system and all that matters.  After all, the engine which drives that data plays a fairly critical role in your business.  Unless you're dealing with simple forms-over-data applications and simple rails-style interfaces, you would probably do well to consider the importance of all that business logic.

A common analogy in the English language is "putting the cart before the horse."  And you know how developers love analogies...  The cart is your data.  It's the payload that's being transported.  The horse is your engine.  It drives the data to and fro.  In the age-old struggle between cart-makers and horse-breeders there is a debate over which is the more important part of the system.  Without the horse, the cart doesn't move.  Without the cart, the horse has nothing to do.  Both are valid points to be sure, but when designing the system which natural construct ends up being the core?  No matter how well you abstract your horse-to-cart interface, there's still a natural architectural dependency in the system.  And it's a hell of a lot easier to build a cart that fits your horse than to breed a horse that fits your cart.

Don't Forget To Track Your Hours

I was entering my hours worked into my employer's time tracking system today and it got me thinking about that whole process from a developer's perspective.  Now, it generally goes without saying that we as a breed don't like doing that.  We're here to work with code, not tell you how long we spent working with code.  But it occurred to me as I was entering my time that I didn't entirely mind doing it.  I didn't feel inconvenienced or annoyed at the prospect, and most of all didn't need to be reminded to do it.

We're intelligent people, we know why management wants and needs this information and how valuable it is to the company's bottom line.  But that knowledge alone isn't enough to capture this information and successfully report on it.  The process itself must be scrutinized and tailored to the actual daily needs of the employees.  Otherwise, you're going to spend extra effort trying to get your employees to enter their time, they're going to spend extra effort listening to you and finally entering it, and the numbers just aren't going to be good.

Let's take a look at some of the time tracking methods I've used over the years...

Many moons ago I worked for a small company in a small town that primarily set up computers and small networks for small businesses.  As the company grew, myself and eventually another developer were added to expand into small websites and custom applications.  This company had a home-grown (developed by the only other guy there who knew a little VB before I came on board, I think) time tracking system.  Basically, it was a little application with a list of "open projects" and functionality to clock in and clock out.

I never used it.  Well, I used it a couple times at first, but that quickly faded into not using it at all.  It was silly.  I (and the other developer when he joined the team) could not be bothered with tracking project time.  The application reported to a spreadsheet and, upon the manager's request for specific project times, I would just manually send a single project's time to the manager.  It was a rough guess.  How do I know how long I spent on that project?  I was doing 10 different things that day.

The system worked well for the network guys, because more often than not "clocking in" to a project was done before they actually went to the client site, and "clocking out" was done when they returned to the office.  Made sense.  But as for the developers, we saw it as pointless.  (As the company grew we also brought on board a PC tech guy and gave him a workroom to perform various machine maintenance.  I don't think he was even asked to use the system.  If he was, he promptly ignored the request.  I mean, when he has 5 open computers on his bench, 2 are formatting, 1 is installing something, one is booting up, and one he's actively using... what "project" is that under?  Is he expected to "clock out" and "clock in" each time he wheels his chair from one machine to another?  Didn't think so.)

Fast forward through some various other endeavors in my career to a more recent example.  Two jobs ago we used a system called Rally.  And, although this sentiment wasn't universally shared by every last one of my co-workers, I actually really liked it.  Sprint planning and task break down was a pain, and I'm fairly convinced there's no way to ease that.  But actual time tracking was quick, efficient, not inconvenient in the slightest, and actually a joy to do.

Logging work hours was really streamlined in this system.  I look at a grid of my tasks for the current spring, I mass-edit a few numbers of how many hours I spent per task that day, and I save.  Takes maybe 30 seconds of my time.  The UI was clean and intuitive.  And burndown charts are just pretty to look at, naturally providing incentive to take those 30 seconds.  Now, I'm sure the system could be horribly abused, and may have been for the co-workers who didn't enjoy it as much as I did.  (It didn't account much for distractions, so "putting in time" during a day where one's time was wasted by a dozen other people just isn't going to sit well.)  But, all in all, the numbers were good, up to date, and once we got used to the system we didn't need constant reminding and fighting from management to enter our time.

Step forward into another job, where we used a system called Remedy.  It was awful to say the least.  The system itself is highly configurable, so perhaps it can be made to be better.  And I'm certain we were on an old version, so maybe it's improved since then.  But, where the rubber hit the road, it was an absolute pain in the ass to use.  Entering information into the system or retrieving it from the system was akin to a scavenger hunt through Hell.  Needless to say, I patently refused to use it.  Early on in my time at that job there are a few entries that I was persuaded to put into the system, but for most of my stay there the reports had me flat-lining across the board.

There was absolutely no incentive to enter my hours.  The system was bulky and awkward and served no purpose to my daily work other than to explicitly get in my way and prevent actual work.  It may have been perceived that I felt that I was above such pettiness and wouldn't belittle myself to track my hours.  Giving some thought on the subject, I'd be lying if I denied such a sentiment.  It wasn't in any arrogant way, really.  It's just that, as a professional, it was a waste of my time and effort.

Most of the employees there had been there for quite some time.  Many had come up through other groups and other departments and perhaps this was all they'd known for a long time, for a good number of them perhaps even all they'd ever known.  They'd gotten used to it, I suppose.  It had beaten them.  "That's the way we do things here" was a common utterance.  (On a side note, I never understood how maintaining the status quo so staunchly made any sense in a company plunging into bankruptcy, but I digress.)  The bottom line was that I wasn't going to use it.  My time is valuable enough that I'm going to spend it doing the work I was hired to do.  If you don't think my work is valuable, then fire me.  (Heh, funny story about that, but I again digress.)

Step forward again to my current job.  Here we use a system called Jira.  I'm pretty happy about this, actually, because I'd wanted to use Jira for some time now and learn more about it.  Perhaps we may even begin using some of its companion products, which would be sweet.  Anyway, entering time is once again a clean, quick and simple process.  It's not quite as streamlined as Rally, so that one is still my favorite to date.  But it is quick and simple and the UI provides enough direct incentive to make it happen on a nearly daily basis.

Logging work still prompts for work descriptions and various other nonsense that I continue to leave blank.  Honestly, the description is in the task.  What did I do for those 3 hours?  I did what was already described.  Hopefully they won't grill us for more information and more tracking.  I honestly doubt they will, it's a pretty casual and extremely efficient place here.  If something holds up progress in any way, it's not going to last.  There is no "status quo" here other than getting the job done.

So, looking back, it would seem evident that it's not really in a developer's nature to avoid time tracking entirely.  It's not beneath us or a waste of our time, provided that it's done properly.  The time tracking system should be tailored to the work, not the other way around.  (And a fantastic example of that is the first example above, at least for the client-site network guys.)  If you're trying to fit the square peg of work into the round hole of the time tracking system your purchased, don't expect good numbers.  But if you, as a manager, take some time and actually understand how your employees think and work and act then you can get that useful business information from them without an ongoing battle simply by adjusting the system to accommodate the work.

Developer must reads

I had mentioned this a couple of times to people about how a book would fit into my must reads or where it would fit in. I figured I would try to dump that list here while trying to list them in a suggested order.

• Code Complete: 
• gets you thinking about how you structure and write code on the line or function level.
• talks about developing good coding habits (comments, naming) and skills (debugging)
• Pragmatic Programmer:
• starts talking about basic principles such as DRY, goes a step further then Code Complete
• talks about learning your tools and introducing some ideas of craftmanship
• Apprenticeship Patterns:
• establishes some ideas of how you can attack your pursuit of mastering your craft
• some ideas of how to stay organized and on track as well
• Refactoring:
• how to make your code better when you start to see the pitfalls of your approach
• can make it easier to know how to tackle or see/smell the possible problems and code smells
• Mythical Man Month:
• start introducing you to some on going issues with managing software teams and projects
• contains some timeless papers that introduce Brook's Law and No Silver Bullet
• Design Patterns:
• now getting up to lower system interactions and design
• starts to develop some common design terminology
• help you understand categories of problems and possible solutions
• Applying Domain Driven Design:
• takes design and development approach a step further
• ties in some design patterns, test driven development, and refactoring together
• starts to introduce you to architecture patterns
• you get to see the workflow and thought process of how a system comes together
• Patterns of Enterprise Application Architecture:
• now we get to how to establish components and layers of a system
• getting to applying the same ideas of Design Patterns but to a higher level of design

Now I need to finish a couple of those books and figure out new books to read (probably something on XP or Agile, TDD, continuous delivery, and DSLs), but that is a series of books that should probably be stretched out over the beginnings of your career (say first 2 or 3 years) because some of them will require a mentor or experience to start to grok (I'm still trying). 

It's probably debatable about where some of them appear (maybe Applying DDD before Design Patterns?), but these are all books that help regardless of technology. You'll probably enjoy having them on your self after your done as well (for reference or the reminder). 

Besides these books you should strive to master your primary programming language and dig into core technologies you use. After that you should also try to explore other languages and technologies even if they don't apply directly to your work.

I'm still trying to accomplish some of those last points but I imagine I'll always be going after something.

To learn is to abuse

I know sort of a strange title but I feel like there is some order of operations I'm following in my growth as a developer.

Learn thing -> abuse use of thing -> actually start to understand thing (hopefully) -> use thing appropriately -> learn new thing -> ...

My thoughts are interfaces are starting to change...mature? I think the .NET convention of an 'I' (ex: IEncryptionAlgorithm) prefixed defeats part of the point. I think that encourages us to see it as an interface code construct and not the required object description that your module/class/method wants. Perhaps Java's '*able' (ex: Runnable) is a better one. I hope that makes sense but I'm starting to think a lot of developers have focused on "this is a class" or "this is an interface" when both are actually an object description and when you take that in as a parameter you are saying I require these members. When I start thinking that way I start thinking that class methods should be virtual by default and that in general we probably don't use interfaces correctly (hell a lot of people don't use them at all I'm sure). Just having a class by default have virtual methods means we could mock a class the same as an interface. This does mean that you couldn't rely on a class's implementation at runtime but you shouldn't be programming against the implementation anyway.

Of course then I think about how it's easier to reason about a class if it's sealed up nicely. It's also easier to reason about code interactions if types are explicit, but then again it's easier to write tightly coupled code if types are explicit. Perhaps since class members are by default non-virtual, proper use of things like TypeMock should be encouraged. 

My biggest objection is if it allows you to open up the class and mock out a hidden internal dependency. If you needed to mock an internal dependency to properly test a class then perhaps it should be an explicit dependency instead (pass in through a parameter). 

Beyond mocking, interfaces should be encouraged as well since they are what provide real modular and reusable code. Type hierarchies only project shared/related implementation reuse which I don't think helps an application on a whole, just a piece of a subsystem or layer. They are best at describing related types, perhaps that's all they should be used for and not for passing around or interacting between subsystems.

I think this has been influenced by my interest in dynamic and message passing style languages. I also think this is just my understanding of object-oriented design hopefully increasing (you never can judge your own aptitude). So I'm thinking I'm leaving behind a phase of my understanding where I probably overly used classes and probably about to enter one where I overly use interfaces. Of course this only really matters in static languages where you have to worry about types. Maybe I'm over thinking things...or maybe I should just go back to Ruby or Python.

The Times, They Are A-Changin'

Ya, I've been pretty quiet lately.  Turns out I've been pretty busy lately.  See, I started this blog when I started a new job and a new phase in my career.  I just figured it was time to write some stuff and generally cultivate the idea of writing as an important part of my overall career.

Well, I've recently started at a new new job.  We don't need to get into all of the ugly details about how the old new job ended, suffice it to say that it was a career learning experience.  But we can all readily agree that I definitely wasn't happy there, and everybody knew it.  There was friction.  A lot of friction.  I wish them well and all, but it's just better for everybody that I'm not there.

So here I am at a new job.  This one is kind of an interesting mash-up of the industries of my last two jobs, but leaning more towards the former than the latter.  But the overall culture of this place is definitely a far cry from either of the previous two jobs at every level of the organization.

Anyway, being the new guy at a busy place, and trying to pick up various psychological pieces that have fallen over the past few years, you can imagine that I'm quite busy these days and that would explain the lack of recent posts.  (The 3 hours per day of driving doesn't help either, but that should change at some point.)  I was hoping originally that there would be more active contributors here by now, but it is what it is.

I can, however, take a moment to jot down some initial impressions of the new job.  A pros and cons list, if you will...

Pros:

1. It pays more.  That's always a plus.
2. Casual day every day.  Shorts and flip-flops are even acceptable.  (Though I despise flip-flops and would never be caught wearing such things, but shorts are a nice option on hot days.)  I'm enjoying sporting the t-shirts on my first week, but I imagine I'll mix it up with some polo shirts just so I'm not always wearing t-shirts.  Besides, why would I want to wear out my best t-shirts?
3. Software best practices.  None of this "we're a Microsoft shop" nonsense, but rather a corporate culture from the top down to use what works for good long-term reasons.
4. Good talent pool.  I especially hear great things about the lead architect here.  Basically, I enjoy working with peers from whom I can learn things and refine my craft.  This seems to be the kind of place where they focus on building a good team and let the team figure out how to handle the projects, rather than tightly defining individual roles and ending up with vaguely qualified people who don't actually work well together.
5. Great corporate culture and environment.  The people who work here seem to genuinely enjoy working here.  I don't hear any complaining, I don't see anybody slacking off or taking breaks to vent, none of that.  Everybody's busily working away at things they enjoy doing.
6. Venturing outside of my comfort zone.  One of my main reasons for leaving my previous previous job was because I was becoming too settled into my role.  I didn't want to build a career on being the guy who supports one legacy product.  Furthering that, I continue to not want to build a career based on one technology or one platform.  I love .NET, and I enjoy furthering my skills as a .NET developer, but at this job I'm currently doing a good bit of PHP (which we'll be converting over to .NET eventually, but for now changes to the existing PHP product need to be made).
7. Working with a friend again.  I've really missed daily interactions with the friends I made at my previous previous job, and it's good to be working with one of them again.
8. Change of scenery.  For one thing, I like working in a proper office park again (the variety of lunch choices alone is a welcome change).  But this is also 90+ miles from home.  It's in a whole new market.  If it works out, we can relocate here permanently (so far this is a 3-month contract gig with a possible hire, but if that doesn't happen I can just slip into another contract gig or something, no big deal).  But overall it's good to make contacts in another market.  If you spend your whole career in a single (especially small) market, you have to keep the bad contacts along with the good.  Old bridges are burning, and even if we didn't start all of the fires it's still best to avoid trying to cross them.
9. Individual accountability.  As Tony puts it, "giving you enough rope to hang yourself."  Basically, initiative and project ownership are welcome here.  If you want to do something that will be good for the company, then do it.  Just make sure you do it.  With initiative comes responsibility.  They don't expect us to be perfect, so some room for mistakes is acceptable.  But overall I like the idea that it's not all surrounded with a vast sea of red tape, nor are we begging at heels of the on-high masters for a little bit of leeway to do our jobs.  We're allowed and expected to be leaders here.
10. No offices.  Seriously, this is kind of cool.  The cubicles are very nice and very spacious, and everybody short of the CEO is in one.  The only office is for the CEO, because he clearly needs a place where he can close the door and be on the phone or hold a private conference without having to book a room or anything like that.  And even his office is all glass with full view of everything going on.  Even his monitor is facing outward so he's not hiding anything.  But everybody else is in a nice cubicle.  Some have better window proximity than others, but there's no avoiding that.  Generally, though, I find this brings a nice sense of flattened management structure to the whole game.  We're all peers, we're all members of the same team, we're all in this together.

Cons:

1. Long commute.  90 minutes each way is pretty exhausting.  It's likely that I'll get a cheap apartment in the area, but that still feels like money down the drain.  It still comes out to a net increase, but doesn't feel right.  (Though the added sleep available in a local apartment would be more than welcome.)  Maybe I can work out some remote VPN time instead.  The lead architect suggested that, and I definitely like the idea.  But until I get fully entrenched in tasks and heads-down coding work, I need to maintain a presence in the office.
2. The lead architect is hard to read.  He's clearly a nice guy, that much is obvious.  But it's hard to tell if he's being nice or if he's upset at something or if something else is wrong.  To be fair, the guy is probably tired as hell.  It's a demanding job.  And let's also be fair on my end of this interaction... I have below average body language and situational awareness and I'm paranoid as hell.  Not a good combination when dealing with subtle interpersonal interactions.  So that's more a con for me than for the job.
3. The carpet texture is uneven.  Seriously, this bugs me on a daily basis.  It feels like the soles of my shoes are unevenly worn or that I have something in my shoe or something like that.
4. There's no secret garden.  In the building there are two communal sets of men's/ladies' rooms, that's it.  This means that, on a not-so-fresh day, I don't have a private place to poop.  I'm thinking about the greater good of the community here.  You don't want to be around when this bomb goes off.  At my previous two jobs I had sought out, found, and retained designated safe zones for accomplishing this task.  Here there simply are none.  Sorry guys.

That's about it for now.  I'm sure I'll think of more stuff later, but I think this sums up the experience so far quite well.

Is .NET On Its Way Out?

I can't help but notice that the frequency of C# (or .NET in general) questions on Stack Overflow is dwindling.  And the questions which are asked tend to be asking how to use one of Microsoft's pluggable widgets to do some specific task, usually exposing entry level code with no understanding of what the code is doing short of what the widget's instructions said to do.

It's a little unsettling, to tell the truth.  And it makes me wonder if the C# enterprise developer's days are numbered.  Java still seems to be going strong in the enterprise, so it's not much of a jump to that.  But to be honest I really do love .NET and where it's gone in the past few years, but it seems like Microsoft is steering it back to the old days and enterprise developers just aren't going to follow it there.

I've been thinking about this for a while, actually.  My own personal view on the timeline of Microsoft development may be skewed by my own experiences, both positive and negative, but this is just how I see it.  (Consider that a disclaimer, and if you have any insight contrary to my following assessment then please present it.)

Back in the day (pre-.NET), "Microsoft development" generally consisted of VB programmers, fluent in and ins and outs of COM and ActiveX and other such things, rapidly developing applications.  Whether it was in Access, ASP, or the more proper flavors of VB (do you remember VB5?  blech).  This wasn't really "enterprise development" so much as it was rapid application development performed sometimes in enterprise environments (or start-ups, or other small environments, etc.).

"Enterprise development" was going strong, but didn't get a lot of press.  C++ developers are a reclusive breed, and they don't tend to hang certifications on their cubicle walls to validate their skills.  Their software spoke for itself, and the enterprises in which they worked listened.  (Sure, that's painting a general picture.  Bear with me, this is just an allegory.)

Along came .NET, and things began to change.  Microsoft was ambitious with this project, and rightly so.  They wanted to "unify" things, in no small part because Java was already doing that.  Java was at the time nowhere near the enterprise language that it is now (remember "applets"? do they still have those?), but it was a clear and present threat to Microsoft's hold on the developer market.  After all, Microsoft sells the widgets, the tools that use those widgets, and the classes that teach those tools.  They wanted to protect that.

I remember the early days of .NET when C# was the cool new kid on the block, and I remember hordes of VB6 developers up in arms demanding that VB remain strong in the emerging .NET ecosystem.  That sort of made me wonder at the time... What kind of developer ties himself to a specific language?  Back then my background was almost purely academic (university), and different languages meant little to me beyond the syntactic expression of the core concepts.  VB, C#, it made no difference to me.  But the framework, now that was cool.

Continue forward through the life of .NET.  Microsoft's own people loved it, and treated it as best they could.  1.0, 1.1, 2.0, 3.0, 3.5... each new version presented wondrous new ways to apply design principles.  Though, to be honest, I really didn't like the explosion of web controls and providers in 2.0, and I guess that can be seen as a foreshadow for what I'm discussing here.  The web controls and providers were "pluggable widgets" where Microsoft was continuing its previous ways of trying to remove the developer from the equation.  Remember their business model all along.  They're not interested in helping you make better software, they're interested in helping your manager make the software himself.  Or have his secretary do it.

But .NET seemed to be changing this.  Enterprises were using it as a serious development platform, and it accommodated them well.  3.0 and 3.5 really moved it forward, and 4.0 continues to push it.  New forms of expressiveness, easier ways to apply known design patterns and build robust, maintainable code.  Sure, there was still room for the rapid application developers, the widgets still existed.  But the environment was much more than that.

And open source projects like Mono get into the game and have made their own enhancements and improvements.  Overall the community contributions have grown over the years, and some cool stuff has come out of it.  IronPython and IronRuby just to name a couple.

But what's going on now?  Have you noticed that some of Microsoft's own people have been jumping ship?  IronRuby has been pretty much abandoned by its overlords, though I'm still holding out hope that the open source community will pick up that slack.  (This might be something for the Code Dojo to look into, actually.)  Microsoft, and by extension Visual Studio (which is a damn good IDE, for all your IDE haters out there), have recently been moving back to the pluggable widget model.

Why?  I guess it's their business model, and that's their right.  But the effect I'm beginning to see is that they're being taken less seriously in enterprise development.  Sure, there are plenty of enterprises who have bought in to Microsoft's widget mentality, and don't see a difference between rapid applications and enterprise development.  But there is real development going on out there, and it's being done by real developers.  Microsoft's own people know this, and some of them are sticking with it despite Microsoft's turn in the other direction.

To visualize this, I tend to think of enterprise development and Microsoft development as two parallel lines.  Those lines have converged more and more throughout the life of .NET.  But it appears that they're once again pulling away from one another.  And there is no shortage of developers from the Microsoft line who have jumped over to the enterprise line.

It occurs to me that developers who are enjoying the enterprise line would do well to rise above the Microsoft stack and become more tool-agnostic in their development.  Otherwise, we may easily find ourselves in the same boat as the VB programmers who were called to arms ten years ago to protect their invested skill sets.

Windows Wizardistrating

Today I'm getting my first real taste of administrating a Windows server and I have to say WTF?

There must be a better way to go about changing and adding configuration settings in Windows. What I feel like is that I'm naturally lead to menus and GUIs off of the Admin Tools or Manager windows. Sometimes this leads me through little wizards (inspiration for the title of this post) that don't seem to do anything except split up a few real options across several mouse clicks. I must just be assuming that this is the natural way to do these things in Windows from prior experiences, because I do find that some of this can be managed by command line. Of course even those commands don't seem very useful sometimes.

I think a lot of this frustration comes from the fact that I have more experience administrating a Linux box then a Windows one. In Linux you are naturally led to the command line. If you need to get a little dirtier (which I also find is easier sometimes) you find out what daemon (service for you Windows people) controls what you want and find it's configuration file located in a fairly consistent why under /etc directory. I should add that these configuration files are generally plain text and in a human editable and readable format (imagine that a configuration files meant for us). The mature ones (most are) will have excellent documentation on how to work with these files, whose formats are generally consistent as well.

There are some interesting side effects from the Linux way of server administration. You don't necessarily need a GUI at all, all tasks can be automated, finding configuration files is easy, and backing up or restoring a server's configuration is extremely easy. I have even seen using a source control tool such as Bazaar to keep your server's configuration change history for easy reverting or tracking how the environment has been changed. In Windows I have no idea where most of these configuration settings are being stored nor if it is easily backed up or rolled back without a system backup/restore.

Another thing that bugs me is that there seems to be no easy way of searching and scanning the Windows Event Logs. The searching/filtering abilities in that GUI are pretty lame and it turns out that the log isn't stored in a series of simple plain text files. It's a bit irritating that I can't just do a multi-file grep to find the lines I'm interested in, awk them into a different format, grep for the awk output in another log, and see how different events in the system are linked together.

I suppose I need to just spend more time with Windows and Powershell so I can get past in mismatch between how I expect things to work (Linux) and how they actually work.

Anyway I just wanted to vent a bit of frustration.

ORMs for all?

I believe we have done a discussion post on IoC and Repositories. For IoC it sounded like we were all in agreement that once you get into the pattern it's hard to not use it, the complexity fades away, and the extra upkeep saves you code in other ways. For Repositories I believe hit on trying to properly use the term and trying to restrict their abilities. Now I'm thinking we should do a post to discuss the use of Object-Relational Mappers or ORMs.

I personally am finding the use of an ORM almost always worth it. Your code takes a completely different shape and a lot of worries you had to deal with yourself almost vanish and are replaced with much easier to maintain ORM configuration code. That is once you learn the ORM.

The majority of my experience is with NHibernate and I am still learning this beast, but it has some extreme power at your figure tips. You sometimes almost feel like an evil conjurer summoning the magic of powerful dark wizards (this is natural because you are). The problem is of course that NHibernate adds it's own sort of spells and rituals to the mix that you have to learn. These spells and rituals also can blowup in your face in a different manner then what you might be used to from ADO spell casting. So there is a learning curve in order to gain easy connection/transaction managment, object change tracking, query batching, non-magic string statically typed queries, sophisticated mapping maintained in a single location, lazy loading, and the ability to have a fuller more meaningful object model.

Now this is with NHibernate which is trying to support the abilities of mapping a fuller object model that is completely independent of the database schema.  If you are learning an ORM that went along with a more ActiveRecord pattern then it's probably much easier to use. Generally the easier it is to use though the less flexible so you could run into problems down the run, but you application may never need that. Going with straight ADO will probably be easier to write and get traction when you don't know an ORM, but now you are managing everything yourself. Even if the code is generated it's still code you have to maintain. Naturally your application's DAL and database may never change so managing all that manually may never come into effect.

There is of course the loss of performance and the added overhead to using an ORM. I really don't think this is a problem in our world. ORMs can help optimize database access, in all seriousness, but then you may be biting the bullet on CPU cycles and memory, but that's what we have the most of anyway. Of course there are  system/architecture designs that take care of most bottlenecks.

You also have less transparency with an ORM over ADO and SQL. The ORM has to interpret your query and generate the SQL in order to take full advantage of its mappings. You can run into problems of debugging  problems because you didn't realize the ORM was going to doing something. This goes back to the learning curve that can be involved for a team, but I rather dislike second guessing myself when I just want to rename or change the mapping for a property.

I think I have hit the majority of concerns with using an ORM and I'll restate my opinion here with a bit more explanation.

I personally find it hard to think of a project where I wouldn't be using an ORM of some kind. This does not mean that you get to avoid learning about databases and SQL. That complexity is there and constant, but it does mean that you get to avoid writing mapping code, casting the spells around ADO (or whatever framework library you have), have to write your own change tracking for dynamic updates, lazy loading, working with dummy objects and more procedural style code, and others.

That last one about procedural code could be a whole other post, but I think that when it's difficult to create and map a fuller object model, developers will create a dummy object model, and that leads to procedural code. I think most of us understand the side effects of medium+ amounts of procedural code (hard to maintain, easy to make mistakes).

So what do you guys think about ORMs and their use? Certainly we all want to be pragmatic, but, unless the project is very small,  I find it hard to not think about using one. ORMs seem to becoming like an IoC container for me. This might make it hard for me to gel well with some teams though.

On Benchmarking Skills

A common topic on Stack Overflow (and possibly many other places) is developers looking for some measure of their skills.  Many ask what they should learn next (or learn first), some ask what they should do for a job search or an interview, some simply and directly ask how they can know how good they are at what they do.  (My personal favorite, and I'm currently having no luck finding a link, was when somebody asked if it was possible to completely master a programming language.  Most of the responses were similar to what I am about to write, but one was simply the word "Yes" linked to Jon Skeet's profile.)

It's a natural question, really.  Humans have a psychological need for positive reinforcement.  We want to know how good we are at something, most specifically in relation to how good other people are at that something.  We want to be weighed and measured in the hopes that we can pin a badge on ourselves demonstrating that we're experts at this and that.  Hell, there's an entire industry of certifications and courses built around this.  And I've certainly met a fair amount of people who cling to those certifications as ironclad proof of their superiority in the field.  (After all, if Microsoft says you're good at something, then you must be, right?  Why else would they be willing to sell you the course materials over and over again every time they release a new version of something that they also sell you?)

But it's relative.  We all know the old saying: "The more you know, the more you know you don't know."  It's a popular saying because it's essentially true.  So true, in fact, that the most common answers I see to such questions as above are that as long as you're striving to improve, you're good.  It's only when you believe that you've mastered something and that you have no room for improvement that you should worry.  No matter how good you get at something, you should always be aware of your limitations.  It's not entirely necessary that you have the ambition to overcome those limitations, so much as it's necessary that you be aware of them.

We've met the people who thought they were the top dog.  Hell, I've been that person.  Back when I worked at a state job, I was the man in terms of software development.  I knew my stuff, I was up to date on technologies, I was the hot shot codeslinger.  I had what I now refer to as "big fish in a small pond syndrome."  It was a state government job, it presented no challenges or growth.  There was no evidence that my skills were lacking or reason to improve.  The job I took after that corrected this syndrome.  The pond, the size of which being measured by the overall skill and talent of my fellow developers, grew and grew and grew.  (It eventually tapered off and I found myself needing to expand and take upward flight under my own strength, which led to my seeking a new job again, but that's another story.  Still seeing how that's playing out.)

I've discussed this with Tony a few times as well.  He's mentioned that the team at his current job as a big fish in a small pond.  Some developer who is "the senior developer" for no other reason than he's unopposed.  He's not really skilled, but until Tony got there he had no means to measure his skills against anybody else.  Recognizing this in the professional world, Tony now finds himself wanting to be the smaller fish in the bigger pond.  This is because our skills are relative.  You only know how good you are at something when standing next to someone who's better.  (American Idol auditions notwithstanding.)

So, to really answer the question of "how do I know how good I am at something?" is to find someone who's better at it.  Learn from them.  There is no certification or online test that can measure you quite so well as you can measure yourself when you work with someone more knowledgeable or more experienced.  Keep in mind that the business model of certification courses isn't to make better developers, it's to sell certification courses.  Online tests (the kind I loathe when required by a hiring manager) don't actually test your ability to perform on the job, they test only your ability to take online tests.  Unless you're interviewing to be an online test taker, they're not particularly applicable.  (Though all too often they're a necessary evil to get past the first line of defense in an interview process.  Even though the code seen in such tests is generally the kind of code a qualified candidate would run from screaming rather than choose to support as a career.)

If you believe yourself to know all there is to know on a subject, that's a bad sign.  The more you know, the more you know you don't know.  Or, as I once saw it comically expressed online somewhere:  "When I graduated high school I thought I knew everything.  When I graduated college I realized I didn't know everything.  When I received my Master's Degree I realized I didn't know anything.  And when I received my PhD I realized that it's alright because nobody else does either."

Someone Doesn't Like Me

I must have offended somebody's delicate sensibilities on Stack Overflow:

Some quick research on meta.stackoverflow.com revealed that they have a daily script that checks for this sort of thing and remedies it.  (They have a number of daily scripts that look for a number of things, actually.  It is the internet, after all.)  So we'll see if that does anything.

It's just 10 points, so no big deal either way.  I just hope whoever this is doesn't continue to have nothing better to do.  People are just strange.

Intro to Programming with Python

About a month or two I did a video for SummeryPyGames to teach entry level programmings (or those very new) about programming in Python. I just recently uploaded the video to my vimeo account so I figured I would share it. My first screencast and it came out pretty good I thought.

http://www.vimeo.com/14223348

I have thought about doing more maybe delving into some details about different parts of the language, or slowly building a library to teach someone Python moving onto classes and basic scripting after the video above. I'm sure I would learn a lot in the process.

On Business Priorities

Raise your hand if you've been told "this application is very important and critical to the business and absolutely must work."  Now raise your hand if you've ever called them out on that bunk.  Maybe I'm just a little more cynical or blunt about these things, or maybe I'm a bit of an ass, but I think it's important that business users be made aware of the level of bunk often found in this statement when it comes to their applications.

For all the times I've heard similar statements made, they can all fit neatly into two distinct categories of "highest priority" (notwithstanding the fact that constantly pushing the panic button does not build a constructive sense of urgency):

1. Real business priority
2. "Squeaky wheel" priority

The former is simple.  There are systems that are critical to a business' core functions.  At my company, for example, our core database is of the utmost importance.  Millions of dollars have been spent, teams of people surround and maintain it at all times, there are detailed policies and procedures that have been created by experts and have stood against rigorous testing, etc.  It's a business priority, and the business knows it.

The latter, however, is nonsense.  And it's nonsense that affects the business' bottom line often without the business even knowing it.  It wastes everyone's time and effort (and, ultimately, the business' money) all because someone somewhere is being a "squeaky wheel" and demanding attention.  More often than not, we as support personnel are simply told to just go along with the demands.  That is, after all, part of our job.  (And that's fine, just don't complain to me when the real work suffers as a result.)

But let's talk for a minute on what this "priority" really boils down to.  This business user has an application that was created for a specific business purpose.  For this user (and this user alone) this application is critical.  They need it in order to do their job.  And their job relates to other business functions which are also critical and so on and so on (which this user will gladly explain in an email, since justifying their job is clearly more important than describing the actual problem).

But if this application is so critical to the business, why then was it not given any design or architectural attention?  Well, that's "just how we do things here."  Our development group isn't so much a "team" as it is an assortment of people who work individually on their own projects and support their own projects until they eventually leave and everyone else inherits their mess.  But that's another story for another time.  If the application is important to the business, then why was the business not willing to put any effort into creating it.  It was given to the cheapest "resource" (person) and done as quickly as possible.  That sounds pretty low-priority to me.

But the user thinks otherwise.  The fact that the business as a whole consciously decided that this application was not important enough to merit any effort and that the resulting application was designed to fail doesn't matter.  Evidence and historical data are unimportant if that wheel can get squeaky enough.  Now we're into a political battle, not software support.  Now we get to play a game that managers and children alike play.  Complain loud enough and you will be heard.  (It's amazing what business professionals have in common with my two small children.)

It gets especially interesting when the user begins invoking names and business terms as part of their squeaking.  Toss around the term "HR" or cc: an executive and the squeaking gets louder.  Sorry, but I don't believe in magic.  Stringing together words of power to form an incantation that will rouse and frighten others into doing your bidding is, I'm sorry to say, witchcraft.  Voodoo.  And it has no place in a professional environment.

So what ends up happening?  Amid all my ranting and complaining (sorry about that) it all comes down to one simple truth.  All we're going to do for that user is placate them.  We're going to apply some quick fix to get them up and running again.  Not even get them up and running, but just get them to shut up.  No effort, no thought, no expertise at all.  Just do whatever it takes to stop the wheel from squeaking.  Again, does this sound like a business priority?

So why are we maintaining this charade?  This charade costs money.  It directly affects our bottom line.  We know that user will be back, we know that application will fail again.  It all comes down to a very simple business decision...

Is this application a priority or not?

While this post was a whole lot whinier and rant-ier than my last, the message is essentially the same.  Care about your software.  As a business (either a whole company or just a department or even a single user, whatever entity "owns" a particular application), if a piece of software is critical to your business then put forth the effort required to make sure it meets your needs.

I've said it before and I'll say it again... If the person who actually cares about this application doesn't really care about it, why should we?

Interconnecting The Tubes

I was bored at work today, so I figured I'd add Stack Overflow flair to the side of the blog.  I also started a personal blog (since too much personal stuff would be noise here) and added my Twitter feed to that.  There's also a Twitter list for this blog's contributors and peeps, but it seems that the Blogger widgets don't have a way to add that as a feed as well.  At least, not one that I've found.

Thoughts?

On Quality Software

Sean sent us this link today to an interview with Dave Thomas and Joe Armstrong on the subject of software quality and craftsmanship.  (I highly recommend downloading the MP3 available on the link and keeping it in one's archives in case the link ever goes stale.  The interview is full of good points and insights that should be retained for the ages.)  It really got me thinking about software quality in the enterprise environment.

It's a conversation a lot of us have had with our managers, our fellow developers, etc.  And it occurs to me that the response is often the same.  It comes across differently, but the message is clear.  Writing quality software in "the real world" is "too hard."  Sure, it's nice for open source projects or academia or anywhere else that money doesn't matter, but in a real business where costs are important it's just not worth all the extra effort vs. just doing some good old rapid application development and whipping up some application to get the job done.

I find that this point of view is based on two core assumptions:

1. Bad software costs less than good software.
2. Good software is more difficult to write than bad software.

The former of the two assumptions confuses, downright baffles me.  We've been in this business for a long time now.  Some of us are a little newer to the game than others, but the industry as a whole has been around for some time.  And as such, there are certain lessons I would think we'd have learned by now.  After all, we're talking about businesses here.  Groups of professionals who's sole purpose in the office day in and day out is to concern themselves with "the bottom line."  Money.  Costs.

Can anybody in any position of management over a software development team truly claim with a straight face that supporting bad software is in any way cheap?  That it can just be written off as a justified cost and pushed aside and forgotten?  Teams of highly-paid developers spend a significant portion of their time troubleshooting legacy software that was rapidly whipped up to just get the job done.  We as an industry have enough of this old software laying around that we know full well at this point how difficult and, at the bottom line, expensive it is to support.

Now, what's done is done.  I understand that.  This software has already been written and invested in and it's "business critical" so we need to support it.  But it's not like we're done.  We're still writing software.  We're still creating new features and new applications and new projects.  So why is "the way we've always done things" still the status quo?  You know how much it's going to cost in the long run, that math has already been done.  Hasn't it?

You're either going to pay for it now or pay for it later.  Either way, it's going to cost you.  But there's a critical difference between "now" and "later" in this case.  "Now" is brief.  It's measurable.  You know when it's over.  "Later," on the other hand, is open-ended.  You don't know how long "later" is going to last.  You don't know how much it's going to cost, other than potentially a lot.  Let's express this with a little math...

• 100 * 3 = 300
• 50 * n = 50n

So, assuming for a moment (and we'll get to this in a moment) that creating the "quick and easy" software costs half as much as creating the "good" software (seriously, remember that this is hypothetical just for the sake of simple math here... we'll really get to this in a minute), can you tell me which final amount is higher or lower?  Quick, make a business decision.  300 or 50n?

Now the latter assumption is something towards which I can be more sympathetic, mostly because the people who make that assumption don't know any better.  If you haven't written both bad software and good software over the years and really grasped an understanding of the two, then I can't really expect you to truly understand the difference.  Whether you're a manager who just isn't really into the technical details or an inexperienced developer who hasn't really learned yet, the assumption is the same.

But the thing you need to understand about this assumption is that it's just patently untrue.  Writing good software doesn't cost a lot more than writing bad software.  It doesn't even cost a little more.  Maybe this assumption is based on the generality that in most industries good costs more then bad.  Yes, this is true.  But it's true for a different reason than you think.

Manufacturing quality goods costs more than manufacturing cheap goods because the manufacturing is mechanical and commoditized.  It's an industry of tools, not of people, and better tools cost more.  But software isn't manufacturing.  It isn't plugging together a series of pre-defined widgets on an assembly line.  (Microsoft would have you drinking their Kool-Aid and believing that it's all about the tools, but that's because they happen to sell the tools.  But we'll cover that another time.)  It's a craft, and it's performed by craftsmen, not assembly line workers.

Does good art cost more than bad art?  Sure, the classic good art costs a fortune.  But that's because it's rare and can't be truly reproduced.  Software doesn't really have that problem.  But art being created right now has nothing to do with the financing of the work.  Good or bad is a measure of the craftsman, not of the financial backing.

And a good software craftsman can often be found at roughly the same cost as a bad one.  This is because that cost is usually based on how long the craftsman has been doing the craft, not how good he or she happens to be at it.  We've all met "senior developers" who are only in that position because they happen to have been writing mediocre software for a long time, or because they're over a certain age, or because they've been with that company for a long time in various capacities such that when they got into the software gig they were considered "senior level."

But ask any craftsman in any industry.  Ask a seasoned carpenter or electrician or anybody.  Not taking into account the tools, is the actual act of doing something well inherently more difficult or, at the bottom line, more expensive than doing it poorly?  No, absolutely not.  It seems that way to laymen because the professionals "make it look easy."  But the fact is that, for those who know what they're doing, it is easy.  It actually takes a professional more effort to do something in an amateur way than to just do it the way they know how to do it.  It's not right, it's not natural, it forces them to stop and un-think every once in a while.

Do yourself a favor, do your company a favor, do your bottom line a favor and actually care about the software you manage.  Your developers care about it, because craftsman like to produce good things.  (If they don't care about it then perhaps you should re-think your team.)  Produce better software so that, in the long run, your support efforts can be lower and you can have the time and resources to produce even more good software.  Your business end users may not know the difference, they may not know you've done anything at all.  They shouldn't have to, their software should just work.