Regular Expressions and HTML

In lieu of writing something interesting or thought-provoking (at least to me) or making any kind of point, today I would instead just like to share something for posterity.  This was in response to the never-ending stream of questions on Stack Overflow regarding parsing HTML with regular expressions.

---

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the trangession of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

---

In other words, it can't be done. Stop trying. Use XML libraries and HTML DOM parsers, there is no shortage of them.

Originally posted as an answer by bobince on Stack Overflow.

Abstractions: Where do you live?

You know software development and computer science are extremely overwhelming fields. I believe the saying goes something like "the more you learn the more you realize you don't know". That might not be perfectly quoted but you get the idea. We luckily take advantage of previous ideas and build up abstractions and layer to help manage all this overwhelming complexity.

Lets just take a quick look at a set of related abstractions:
electronic signals (yeah that's right go fuck yourself)

Ok seriously, I'll try to list what's involved with just data access:

• Set and Relational Theory
• Relational Database Topics (schemas, functions vs stored procs, views, transactions) 
• Sql Server
• SQL
• OLEDB
• ADO.NET
• NHibernate

I think that hits the big ones involved for a general .NET data access stack. Now working at the NHibernate level that abstracts away a lot of ADO.NET (provider creation, connections, transactions a little, generating some sql), but you aren't exempt from knowing about it. Beyond that you can't ignore that fact you have a real database underneath that speaks SQL and has its own way of doing things (Sql Server vs Oracle for example).

That's something everyone has to keep in mind: working at one level of abstractions does not mean you can be ignorant of lower levels of abstractions. I do differentiate between the required knowledge to work at a level of abstraction and potential knowledge.

Taking my previous example if you are working at the NHibernate level you are required to know basic NHibernate and some of it's abilities. You are also required to know SQL and something about relational databases (a little design mostly). Now there is the potential (and advantages) to knowing the inner workings and reasoning behind NHibernate (such as Unit of Work, advanced relational mapping), how to work directly with ADO.NET (providers, core interfaces, managing connections), plus all the things you can learn about how your database of choice works.

So what does this mean for people coming into this field... I hope someone is looking out for you. Seriously, you should try to get in at some level of abstraction with a mentor. Realistically you'll have to put some effort into finding a mentor. They can guide you into understanding the level of abstraction you'll be living at and figuring out how it fits in.

What does this mean for people already in the field? You should figure out the potential knowledge in your stack of abstractions you work with (both above and below where you are). Figure out where in that stack you live and try to tackle at least 2 levels down to increase your understanding. I would also suggest going up a level to keep in mind how something is used, but the important thing is to not be ignorant of what's powering your level.
 

Typing - I'll take both

I have seen my share of dynamic vs static articles and they have raised my eyebrow enough times I'm thinking I need to distill my current thoughts in a post. Let's see where this goes...

Firstly, WTF? Seriously, everyone seems to get really defensive about typing. I'm just getting annoyed by the whole thing. I see tons of posts that start off great turn aggressive either due to the author or (usually) via the comments from very opinionated readers. Then sprinkled between those are another annoyance I'm seeing on blogs and project comments. Yeah I think we know you have to make educated technical decisions based on your requirements, but seriously I don't need someone chiming in with nothing to add except "best tool for the job". Ok, I think now that I can try and put some thoughts about the debate down.

I personally think that dynamic typing feels little more closer to how object-oriented programming is suppose to feel. By that I mean the fact that duck typing makes you only concerned with what an object is capable of. You can't think about types when really writing code with duck typing. You are thinking about the object and not the class. Either it can perform the action you want (or process the message) or it can't and you'll get some delegated response or an error. Of course it's that runtime error that gets people twitchy, but seriously not all your code is controlling the space shuttle flight control systems.

Of course static languages give you some explicit self documenting code that the compile can do light correctness checking on, but you do sacrifice some flexibility. Also I find that design artifacts start to pop up from the usually static type system. Things like extra methods on interfaces (as in public members) that are only used if certain properties are a specific value (I know there are ways to design around this). Or heavy use of generics (as in the C# feature), interfaces, design patterns to try and get some extra flexibility out of your code. So there is certainly some trade offs as you would imagine.

Now I have messed around with F# some in the past and I have to say that the type system most common (such as the class system in C# or Java) doesn't do a lot for you. You have to fully define your types and declare them for use everything. It's very explicit on your part and the compiler doesn't do a lot for you. C# did introduce a very light type inferencing with the 'var' keyword but it's abilities are miniscule. If you take a seriously look at F# (I have heard that Haskell is mind blowing) you'll see that it infers types for you as much as possible. You can declare functions without any type information and the compiler will make sure you are passing in types that can fulfill the needs.

I need to spend more time in an advanced type system so I can understand how far type inferencing can take you, but I would like to see languages have a focus on dynamic typing with an optional static system. Even though I love dynamic languages there are still areas you want static typing such as external facing interfaces or touch points between components even inside the same system.

Anyway, that's enough for now.

Why I Don't Like Microsoft's Widgets

I previously pointed out that .NET questions on Stack Overflow seem to be degrading to nothing more than beginners asking about how to plug together Microsoft's widgets to produce a desired effect.  That tide seems to be subsiding, though it is still clearly present.  I guess it's just the nature of the beast these days.  The market is flooded with entry-level developers (many of whom are no longer considered entry-level because of their experience, whether or not they've actually learned from that experience) who use frameworks and tools and, ultimately, pluggable widgets.  I see Microsoft as the king of these widgets, hence my generalization.

But I was thinking today about the whole picture and wondering why I specifically don't like this.  Basically, I was helping a few people on Stack Overflow with some ASP .NET controls and had to put in some serious effort not to just rant about how they're going about the whole thing all wrong and that they shouldn't be afraid to write some code themselves instead of using Microsoft's controls.  (I particularly cringe whenever I see remnants from .NET 2.0 web controls when Microsoft went control-crazy and hooked up some provider frameworks to make web development as plug-and-play as possible.  People still use that stuff.  A lot.  And it really bothers me.)

It occurred to me what the root of my concern really is.  It's not entry-level developers, it's not frameworks and tools, it's not even me being some stodgy old know-it-all because in my day we used to program in bare text editors and used terminals and get off my lawn!  It's because these widgets address the wrong side of the equation.

Writing software is easy.
Supporting software is hard.

It's that simple, really.  (And reminds me of one of my favorite quotes... "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian Kernighan)

Microsoft creates clever widgets which a novice can, with minimal training, plug together to produce a desired result.  This makes the step of writing software easier, but at the cost of supporting the software.  And that doesn't make the entire process any easier.  In fact, when you look at the whole picture of the software life cycle, it ends up being more costly in the long run.  Small changes to requirements or small additions to business logic that exist just outside the scope of the pluggable widgets require significant effort to implement.  Custom code, on the other hand, isn't difficult to change at all (if written well).

We're really reminded of this on an almost daily basis when developing in .NET.  Think of a .NET stack trace.  At the top is where the exception ended up (caught somewhere useful, I hope), and as you descend the trace it goes through your code so you can see where it was thrown and the path it took down the stack.

Many times, however, by the time you get to the bottom of the trace you are in Microsoft's code.  This isn't really a bad thing, after all it is all running on top of Microsoft's libraries.  So this is bound to happen.  And the more experience we have on the subject the more we can identify what's wrong by looking at Microsoft's part of the trace as well.  But the boundary is still clear.  Once you cross over from custom code to System.* then you officially enter the black box.  There are some things you can do, but it's just not the same as code you can actually step through.

How do you step through a DataGrid entirely defined in HTML controls and bound to a data source also defined in HTML controls?  You didn't actually write any code.  Do you even know what it's doing?  Maybe you do, maybe you don't.  Maybe the next guy who has to support it does, maybe he doesn't.  That's a problem.  But when something breaks, what do you do?  Your stack trace begins with code you didn't write.  For an experienced developer this may not present a huge problem, albeit an annoying one, but for that novice who was hired because he demonstrated an ability to plug widgets together it could be a serious stopping point.

I'm not saying that we shouldn't use tools that are available to us.  But we need to know what those tools are doing.  You can't make a career out of "I press a button and this happens, so as long as people want this to happen they can pay me to press this button."  Don't let Microsoft do the thinking for you.  Writing the software is the easy part.

This VIM is a bomb!

For our CodeDojo presentation meeting a month ago, I did a presentation about VIM. I didn't go into any details about basics of getting around or how to install it. Those are too intro. What I tried to talk about is the immediate annoyances of VIM and a couple of concepts to keep in mind during your adventures. I thought I could mention a couple of things from the presentation and talk about my profile I'm building.

I really can't go on without mentioning the power of normal mode in VIM. VIM's modes are actually quite the advantage when you get used to them, but you should keep in mind that you want to be in Normal mode. It truly is where one of VIM's strength comes out, and that is text surgery.

# Language of Text Editing #
### `vimsentence := [count]vimexpression`
### `vimexpression := vimcommand | <operator><motion>`
### `vimcommand := <action> | <motion>`

### _That is while in NORMAL mode_

# WTF? (examples) #

### Delete next 3 words: 3dw

### Paste above line: P

### Move cursor down 2 paragraphs: 2}

As you learn some of the different keys you can apply them in pretty interesting combinations to build up sort of sentences of what you want to do. It takes a little while to get efficient at this, but sort of a rhythm starts to occur as you dispatch lightning ninjas from your Zeus fingers....hmm.. anyway.

The other Zeus like feature of VIM is it's plug-in and profile abilities. It's absurd how much you can customize VIM itself, but then on top of that you get python, ruby, perl, vimscript, shell, etc scripting to enable even more. When a plug-in and a developer love each other they, the developer lets the plug-in in to play with the others. This is where profiles are born.

Now to a newbie, straight out of the box you get a confusing text editor where you'll spend the first 5 minutes wondering why you can't type in text and the next 5 figuring out how you exit without killing the process. After a couple of tutorials and some VIM'ing, you probably have a profile pieced together by snippets that sort of work and that you don't understand. Well that's where my profile is trying to come in.

Here's a snippet where I setup some stuff with folding blocks of text:

" When we fold or unfold a block of text determine the block
" delimiters via syntax. You can use 'za' to toggle a fold.
" There are several other commands as well.
" 'zM' to close all folds
" 'zR' to open all folds
    set foldmethod=syntax

" Lets map 'za' to spacebar while in NORMAL mode
    nnoremap <space> za<space>

" By default I like to see all the code, but the first time
" you try to fold something this will get toggled and all folding
" well be on.
    set nofoldenable

As you can see I'm trying to document and organize things. The whole idea is that someone who knows a little about VIM can use my profile to learn a hell of a lot more. The profile is also to include several plug-ins in an attempt to create a ready to go development environment for a few programming languages. Right now I have a fairly sophisticated setup for Python (refactoring, test running, error detection, auto-completion) and something very usable for Erlang (syntax, error detection, auto-completion, compile a single file). Now I also have some built-in plug-ins and a couple of other things for other languages but I just haven't spent a lot of time tracking down, configuring, and testing plug-ins for other languages besides Python and Erlang.

I'm welcome to any help or suggestions in tracking down more to flesh out a Mono/C#, Ruby, C, or just about anything else. Anyway anyone interesting in using VIM please go ahead and check out my profile.

https://github.com/copenhas/dotfiles

My presentation is also available but the slides are pretty skimpy. You also need to use showoff (ruby gem) to actually see the slides in full.

https://github.com/copenhas/presentations