Web development in 2017 – A Journey part II: VSCode and Git

Leave a reply

The second part in this series is about what and how to install the tooling you need nowadays. No more Notepad, I’m afraid…

Please note: work in progress. This article will be updated to reflect new insights.

To install

 
Remember the previous article? Well, here are the first two tools to install. Just click on the link to jump to the relevant section if you’re impatient:

Make is no longer on the list, because Make isn’t worth the bother of installing Cygwin. Yes, I agree that if you do not install Git, you need to install Cygwin for the GNU Core Utilities. But as they come with Git who has its own version of them, I think that all the hassle you have to go through to get Git and Cygwin working together just isn’t worth it. However, for those foolhardy enough to want to experiment, I’ll explain how to run them side by side in a separate post.

Node.js and related tooling will be discussed in the next post. First, we discuss VS Code and Git.

 

Visual Studio Code

back to top
First, download and install Visual Studio Code . Just click on “Download for Windows (stable build)” and install the download. If for some arcane reason you’re not working on a Windows system, just click on the dropdown icon next to the download button and select a relevant installer or package.

During the installation of the fairly small (37.2 MB) package, you get a number of questions eventually that ask you whether to add context menu’s, register the editor for the relevant filetypes and add the path to Code to the Windows path. Something similar will likely happen on other platforms. My urgent advice is to check all the boxes, unless you already have another development IDE installed (such as Visual Studio). I’d still register Code for everything, but afterwards restart the other IDE and make sure you register the filetypes for that IDE again. Or don’t register and do this manually. I just register the editor for everything, because nothing is as annoying as clicking a .js file and starting an interpreter or worse, Visual Studio itself.

Once installed, verify that all is working by starting Code. If all went well, this can be done from the commmandline, shell or whatever you use. Type “code” and the editor should start.

It is possible that you get a warning about a typescript compiler. In that case install the correct typescript compiler in NPM (using the indicated version) with the command “npm install -g typescript@2.3.2”. This will install version 2.3.2, replace it if Code needs a different version. If there is an older version of typescript already installed, you can remove it with “npm uninstall -g typescript”.

But we will assume that Code starts just fine. In that case we will first set our preferences. Go to File/Preferences and select the Color Theme (“Tomorrow Night Blue” for me) and File Icon Theme (I use the VSCode Icons but Seti is the popular choice and it’s easy to see why). Just select File / Open Folder and open a folder with source code to check what your icons look like.

Then, we add extensions. Open them with the menu on the left side, or with Ctrl-Shift-X. I installed the following extensions:

  • Git Lens

    Git Lens helps you to easily see who changed what in your source code and also get some graphical information on Git commits. It gives you the ability to open older versions of a file, or the same file on Github, or compare it. It shows commits and annotations and a whole host of other items. More even than I currently know. So just install it.

  • Gitignore
    This plugin downloads gitignore files for your specific project. Very helpful, but usually only once.

  • Languange-stylus

    If you use Stylus for CSS, this add-in makes sure you get syntax coloring and checking.

  • Auto-Open Markdown Preview
    A very useful extension that just opens the preview of any given MarkDown-syntax file. Especially useful when editing open-source packages that almost always require a README.md file.

  • ESLint

    ESLint promotes best practices and syntax checking, is very flexible and can include your own rules. However, I found it to be pretty annoying to set up and get working without a gazillion errors (or none at all). If you do this, best follow the instructions on the website. It’s really quite good, but JSHint works out of the box, and ESLint doesn’t provide much value without changing the configuration file. See https://github.com/feross/eslint-config-standard/blob/master/eslintrc.json for an (overly complex) example. That said, it’s rapidly becoming THE linting tool of choice. So for futureproofing it might be your best bet.

    “To sum up, JSHint can offer a solid set of basic rules and relatively fast execution, whereas ESLint offers all that one can want from a linter and then some more as long as he’s willing to put an effort in setting it up.” – Quora

    An alternative option for ESLint is JSHint. This will give very good warnings about JavaScript issues in your code. However, you will also need to install the npm module as well (we’ll get to that later) with the command “npm install -g jshint” which will install the actual syntax checker globally as a commandline tool. It could be installed per project as well, see the website for more details.
    When using it, insert the following line in functions where you use var declarations:
    'use strict';
    If you use import and export commands in for instance d3 plugins, use
    /*jshint esversion: 6 */
    as your first line in any javascript file.

    If you use JSHint then you better add the JSHint default config extension as well: using the command palette in VS Code (Ctrl+Shift+P) you can type “generate” and then generate a JSHint configuration file. Very nifty!

That’s it for the VS Code plugins. If you need more plugins, visit the marketplace and type “@recommended” to see recommended plugins.

 

Git

back to top

Pfew. Git. The mammoth of version control. If you need documentation, here is an extremely nice and well done tutorial. I’m just going to put down some basic points and then leave this topic alone.

First, install Git after downloading it. It installs itself as both a commandline tool, and comes with its own shell. If you’re into Unix shells, Git BASH is nice, and compatible with many open-source projects out there that use shell commands in their taskrunners (like Make). Personally I just use the CMD from windows, or PowerShell for special projects. Whatever you choose, after installing Git you have access to an updated version of the GNU core utilities, giving you tools as wc, grep, cat, less, vi, rm -rf, and many more.

Each project has its own repository, because Git works per repository (and separating them prevents accidents). Creating one is easy: just type “git init” in a commandline in the folder you want to have in Git. Git will create a subdirectory where it stores the repository. With a .gitignore-file you can tell Git to ignore files and folders. The syntax for that file is all over the web, but for firebase projects this is my .gitignore:

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release

# Dependency directories
functions/
/node_modules/
jspm_packages/

# Typescript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# Yarn Integrity file
.yarn-integrity

# dotenv environment variables file
.env

# firebase stuff
*.firebaserc

You can also get this from the Gitignore plugin in VS Code. Remember: .gitignore goes into the standard folder, not the .git repository folder…

Git commands

There are some very good Git manuals out there. A nice PDF to have is the Atlassian Git cheat sheet PDF. Atlassian also has a list of basic Git commands.
I recommend reading at least the basic manual if you haven’t worked with Git before, otherwise it will be difficult to understand what’s happening.

GitKraken

Something that will make Git easier to use is GitKraken. Once downloaded and installed, you can use this tool to visualize the Git branches and maintain them. For instance, you can combine a large number of commits into one single commit, to make your commit history much clearer. You can also operate on branches and create pull requests for open source software. In general, once you get into publishing open source software on GitHub, you really want to use this. Yes, you can do everything on the commandline, but it’s a hassle and GitKraken makes it much easier. You will still need to know the Git commands in order to use GitKraken, though.

Web Development in 2017 – A Journey part I

Leave a reply

A few weeks ago, after publishing my Collatz calculator, I decided I was going to develop a small web application to practice modern web development. And along the way I quickly discovered that a WebApp in 2017 is not nearly the same as that same WebApp in 2007, or even 2015.

Please note: work in progress. This article will be updated to reflect new insights.

Choices, choices… and Firebase

Web Development in 2017 is not your father’s web development anymore. For one thing, it’s now completely dominated by JavaScript. There are applications where even the static HTML and CSS are rendered through three different frameworks. This even extends to the backend with Node.js – even thought it may not be optimal for your needs.

But that’s just the beginning. There are a ton of choices to be made – we are drowning in tools and frameworks, a surfeit of riches in fact. So we have to make choices fast and be prepared to change things around as we gain experience with the choices we made. The important part is to try and not paint ourselves into a corner. But… there are some choices that will have quite an impact.

The biggest impact is created by something I wanted to try for a while now, which is having my back-end not only hosted by another party, but developed and maintained as well: this is called Backend-As-A-Service (BaaS). With BaaS you don’t host your own back-end. You don’t even host it somewhere else. No, someone else is hosting a back-end for you somewhere, and you are allowed to use its standard functionality. This will usually include authorization and storage.


Facebook used to have a very nice BaaS-solution called Parse, but that one has been shut down because it was no longer part of Facebook’s strategy. They did offer a migration path to the open-sourced version though, and you can deploy that server on AWS or Heroku, so it is still a viable option. But I chose to go with a different platform.

Google is still in the BaaS business, with an offering called Firebase. I’m not going to detail Firebase, because extensive documentation is available on the Firebase website. I will however say that, just like Parse, it has (amongst others) the following functions:

  • Authentication
  • Database (with authorization rules)
  • Filestore
  • Message queues
  • Events

In the beginning I will limit myself to the use of Authentication and the Database.

Having made the choice for Firebase, we are now stuck with some others as well. Developing for the web in 2017 needs suitable tooling. And you cannot just buy Visual Studio and expect it to work. Firebase is based on Node.js, JavaScript and Web API’s. You need suitable tooling for that.

JavaScript, Typescript and ES6 compliancy

Funny as it sounds, we have to discuss the language we use first. We can chose TypeScript or JavaScript, and in JS we can choose ES6/ES2015 or ES5. The ES stands for ECMAScript, which is the actual name of JavaScript but noone calls it ECMASCript. If that sounds confusing it’s because it is, but here is a good explanation.

Typescript?

Typescript is nice. It checks your datatypes at compile time which prevents bugs. If you do back-end development in JavaScript, you should probably do TypeScript. But it also adds another compiler to an already unholy mess of libraries and supporting crutches. And it opens the door for things like the aptly named Babel. Before you know it, you start targeting ES6 JavaScript and you need *another* compiler, Babel, just because you wanted to use templating and arrow functions. But you have a lot of work to do before you can actually display “Hello, world!” on a page now. Getting that investment back is pretty hard on small applications. So I avoid TypeScript for now.

ES6/ES2015?

ES6 gives us all kinds of nice things like arrow functions, templating, and a whole array of syntactic sugar. But using it means using Babel to first compile the ES6 JavaScript into ES5, which can be understood by browsers and Node.js. It’s a dependency I can do without.

So my choice is standard ES5 JavaScript. Having settled that, we move on to the tooling to support this choice.

Mandatory tooling

Some tools are so important, doing without them means drastically reducing or even completely negating your development speed or ability to even develop anything at all. These are what I call the mandatory tools. Like a compiler and IDE for C++ development.

Text editor

So, you’re going to write a webpage. VI, notepad or notepad++ is what I used back in 1997. Actually, in 2015 as well. For application development in 2017 the choices are different though. It will be either Visual Studio Code, Atom or Sublime (paid software). Sure, you can try alternatives (try typing “code text editor” in Google for a pretty neat custom list), but chances are it’s one of those three. They all have integrated support for Git, syntax checking and coloring, starting terminals and debuggers from the editor, and extensive customization and plug-ins. I believe VS Code has about 12000 plugins and the other two are not far behind.

Node.js

You may not consider this a tool, but once you install Node.js you also get Node Package Manager and access to a gazillion extremely useful other packages. Like CSS precompilers, plugins, Firebase deployment software, task managers, source control software, and of course Node.js itself: a pretty powerful backend webserver. Install it, because frontend development in 2017 is impossible without it. And all web development is made easier once you have it.

Source Code Control System

Once you develop, you need source code control. You literally can’t do without. And while I do not particularly *like* Git, it’s so ubiquitous and integrated in almost any toolset nowadays, you have to have a really pressing argument to use something like Mercurial (which I like a lot better than Git, but sadly had to let go along with my teddybear when I grew up). Let’s not discuss TFS or Subversion – they’re dead except for special use cases. Web development is not a special use case. So, install Git.

Front-End JavaScript Development Libraries

One word: jQuery. Whatever you do, you probably want to include this. A lot of frameworks include this out of the box, but if they don’t you’d still want this. Tons of utility functions, loads of functions for manipulating the DOM and they work as fast as the current browser will allow, without having to worry about what browser you run on. Absolutely essential for fast development.

CSS Framework

To make page layout easier, you can use a library that will give you easy ways of making a page responsive to where it runs and on what media it runs. This might look like an optional choice, but given the amount of different browsers and mobile media nowadays, it is quite impossible to handcode everything for every platform yourself. That’s why choosing one of these frameworks is a must.

The classic package for this was bootstrap.js, but you can also choose foundation.js. They both provide widgets such as buttons, sliders, cards, dropdowns, tabs etcetera but also responsive and columnar layout, and often styling as well. Bootstrap is the most used and best supported library, but Foundation is a strong contender. Currently I will go with Foundation.

Noteworthy is that both options support Google’s new vision on how to design for the new internet, called Material Design. Material Design is a design philosophy that ties the styling for all components you can use on the web together in one design philosophy. Google has changed all its applications over to this design, and also has its own implementation to showcase how this works, called Material Design Lite. This can be used as a lightweight layout framework, but is limited in application and styles. Since it is simple to use and looks very good, however, this is becoming quite popular. You can see it in action on the standard login-screen of Firebase applications that use the default UI. For now, I go with Foundation when I need layout, because Material Design Lite is a bit *too* simple.

Optional tooling

There are also some tools you can live without, but have the potential to make your life a lot easier.

CSS precompiler

A CSS pre-compiler gives you the ability to write CSS in a slightly different language, that gives you smaller CSS that’s easier to understand. If you have just one small stylesheet, you can do without. But once your styles get more complex, a CSS precompiler is very helpful. They provide loops, conditionals, functions for cross-browser compatibility and usually a more readable CSS. Choices here are Less, SASS and Stylus. All can be installed using NPM. Personally, I think Stylus provides the best and cleanest syntax, so I have chosen Stylus.

Task runner

A task runner is software that can take care of the precompiler step, then combines files as needed, minifies them, uglifies them, uploads them to the server and opens and refreshes a browser window. While this can be done with (Gnu)Make or Node Package Manager scripts, it’s easier to do in tools like Grunt and Gulp. Tools like Bower and Webpack also serve slightly different purposes, like combining files into one big JS include, but with HTTP/2 this may actually hurt performance more than it helps. This means there is a whole zoo of task managers and no clear winner in sight.

At the moment I use Gnu Make (from the Cygwin project) to compile stylus files and deploy and run Firebase. NPM Scripting wasn’t powerful enough without serious JavaScript coding, so I can’t recommend it. And yes, what I do could all be done by just starting the tools with the right options, but I find Make easier to use. Should I disover that I need something more powerful, I’ll try that and update this section.

Even more optionally optional tooling

And then we have the section with tools you don’t want or need to install unless you suddenly have a pressing need. And even then you should reconsider this until you have run out of alternatives. For most applications these are overkill. Come back when you are supporting something as complex as Facebook.

JavaScript libraries

jQuery is often combined with Underscore.js or Lodash.js for utility functions. Lodash seems to be faster and more agile. However, I consider it an optional library and you can chose whichever you like.

Another potentially useful library is Immutable.js. This provides you with enforced immutable datastructures, that eliminate accidental side effects from functions, preventing errors and improving performance. However, I don’t use it currently.

Testing Frameworks

Mocha and Chai are frameworks that provide you with the ability to do easy unit and integration testing, with good reporting. However, I’m not developing a library used by dozens of people. And neither is my game in any way going to be mission critical for anyone. So while breaking things while fixing others does look unprofessional, I can live with that for now. My application will likely remain small and easy to bugfix, so I am not going to invest in these frameworks at this time.

Templating Libraries

Templating libraries help us with HTML-templates that we can fill with data. Very useful if you want to display a list, for instance. However, I will skip this subject for now. Mustache.js and Handlebar.js are great libraries for this, but we already have templating in jQuery. If we ever get to a framework like Angular2.js, React.js or Vue.js, things will have to change again anyway. For now, I think jQuery will be fine. For more information, you may want to look into this overview.

JavaScript Frameworks

I haven’t yet discussed the elephant(s) in the room: Angular2.js, React.js and Vue.js. These are very popular frameworks that bring you everything from design to state machines, and the kitchen sink as well. The choice however, can be difficult. I have not yet decided whether to actually use one, because it’s probably overkill for my needs. I do not currently intend to build a Single Page Application. However, it may well turn out to be a better option than building a lot of separate pages. In that case I intend to go with Vue.js. This is because Angular2.js has a Model-View-Controller architecture I don’t think meshes particularly well with my application or Firebase. I’m much happier with a Model-View-ViewController type of architecture with one-way databinding (updates flow from the model to the view, not vice versa). This would mean either React or Vue since both support the Flux architecture with Redux and VueX. React is a bit heavier than Vue and renders the HTML from JavaScript, which is something I’m not particularly fond of, so if it comes down to it, I’ll go with Vue. For now though, I will stick with Foundation and jQuery for layout and templating.

My choices

As this is a journey, I’m going to travel a bit. Currently I have packed the following tools for my journey:

  • Development environment: Node.js (+ Node Package Manager) + Cygwin on windows
  • Language: JavaScript/ES5
  • Text editor: Visual Studio Code
  • Back-end: Firebase
  • Source code control system: Git
  • Front-End JavaScript Library: jQuery
  • CSS pre-compiler: Stylus
  • CSS layout framework: Foundation.js
  • Task runner: Make

That concludes my first post in the journey for now. My second post will detail my setup, including installation and configuration.

How to learn JavaScript

Leave a reply

I’ve been busy with JavaScript for some time now – with various degrees of succes – and I thought it would be nice to list a few resources that I found both quite helpful, and accessible.

Highly recommended, but not used by me because I only found out about it after the fact:

Once you know a bit more about JavaScript (or ECMAScript, as it is properly called) you probably want to use it in something interesting. I’ve built a few things with the JavaScript graphics library D3 that give immediate results in just a few lines of code, which is a great motivator.

If you have any suggestions for improvements or additions, feel free to let me know in the comments!

Collatz calculator

Leave a reply

By Pokipsy76 - English wikipedia, Public Domain, https://commons.wikimedia.org/w/index.php?curid=7188269

Grundsätzlich IT B.V. would like to publish a small javascript tool: the Collatz Calculator, on the occasion of the publication of a new proof of the Collatz Conjecture.

Two days ago Peter Schorer published a new proof of the famous mathematical problem named the Collatz Conjecture. This doesn’t mean it has been proven, though: new proofs for this particular problem appear with depressing regularity, only to be invalidated in a few weeks. But we can always hope for the best!

The problem itself can be stated in simple terms:

Take any positive integer n.

  • If n is even, divide it by 2 to get n / 2
  • If n is odd, multiply it by 3 and add 1 to obtain 3n + 1

Repeat the process (which has been called “Half Or Triple Plus One”, or HOTPO[7]) indefinitely.

The conjecture is that no matter what number you start with, you will always eventually reach 1. This seems easy to prove, but the Collatz mapping above exhibits chaotic, even fractal behaviour. Thus, a proof has long been sought but not yet been found. Due to the relationship to several other longstanding mathematical problems, this problem has occupied mathematicians for at least a century, possibly much longer than that. Leading to frustration in some quarters.

Used with permission.

To get some idea of the complexities of the mapping, feel free to play with this Collatz Calculator (javascript): Start the Collatz Calculator

 

A jQuery cheat sheet

Leave a reply

jQuery cheat sheet

A few days ago we received an e-mail from Robert Mening at WebsiteSetup.org, who kindly pointed us to his jQuery cheat sheet. There are of course more than a few cheatsheets for jQuery out there, but this one at least has the advantage that it all fits on one page.

 Ronald Kunenborg | march 2017.

The cheat sheet can be found at the bottom of this page. Note that clicking on the image will take you to a webpage where you can also find a PDF-version of the cheat sheet. Alternative cheat sheets can of course be found with a quick search on Google, and from various sources that integrate others (for instance at https://www.sitepoint.com/10-jquery-cheat-sheets/ ). The ones we found most useful though, are the following:

We do have some issues with the cheat sheets in question: there is usually no license on the sheet itself, and the version of jQuery for which it is relevant isn’t always mentioned. These small failings also apply to the cheat sheet shown below. However, if you’re doing some jQuery work now and then, you could do worse than just putting up a copy of the cheat sheet displayed here.

jQuery Cheat Sheet

Image source: websitesetup.org – free to link with attribution

… or not migrating to Cloudflare

Leave a reply

www.cloudflare.com

Several days ago I wrote a blogpost about my Cloudflare migration. I have done a bit more research since then and unfortunately, there are a few snags that could really hurt your website if you fail to consider the implications of the way in which Cloudflare and similar services work.

 Ronald Kunenborg | october 2016.

Main issues

There are two main issues, as described by an interesting blogpost on the blog of Sven Slootweg. Sven is the former administrator of AnonNews.org, and a security researcher. He was also suspected of being part of LulzSec, a very high profile group of hackers. I’ve read most of his posts now and I take them very seriously.

What he states is basically that

  • CloudFlare offers services that are very insecure by default [1];
  • CloudFlare has a business model that depends on them being the “Man in the Middle” with access to a lot of traffic [1];
  • CloudFlare does not mitigate DDOS attacks, unless they are quite small and only affect your webserver [1];
  • Some other minor issues, which I’ll skip. You should read the references for things like issues with Tor [2], SEO rankings [3], website impersonation [4], etcetera, to determine whether they apply to you.

The main problem with the security is that while the user may consider his or her experience to be encrypted (by SSL), the path from CloudFlare to the origin server is not. Now, consider an enduser in a country with an oppressive government who uses your CloudFlare-fronted website. CloudFlare may have a local server in that country. Suppose the user visits a page on the governments blacklist and leaves a nasty comment. Normally, SSL would encrypt all traffic and the government couldn’t intercept the traffic and look inside it. Well, not unless they were using a bit more resources than a simple scanner. But with CloudFlare the traffic between the CloudFlare local server and the webserver hosting the incriminating page, that traffic goes over the border *unencrypted*. Ouch. In some countries, your user will not survive this experience. And the user isn’t even warned: the browser tells the user that everything is fine. Even an SSL test (https://www.ssllabs.com/ssltest/analyze.html?d=www.grundsatzlich-it.nl) will say so. But it’s not. Oh, and if you have a front-end that accepts credit card information, that information will *also* travel the entire internet unencrypted. Not as dangerous as the first scenario, but probably not something you’d like to see as someone using a webstore.

Now, you can actually use the option to also encrypt the traffic between CloudFlare and your server. Let’s just say it’s not the default and requires a bit more expertise than just “point and click”. However, it’s not that difficult. That still leaves CloudFlare as the single point where a lot of web traffic goes through – entirely unencrypted. Sven Slootweg comments that:

This may not sound that bad – after all, they’re just a service provider, right? – but let’s put this in context for a moment. Currently, CloudFlare essentially controls 11% of the 10k biggest websites, over 8% of the 100k biggest websites (source), and almost 5% of sites on the entire web (source). According to their own numbers from 2012(!), they had more traffic than several of the most popular sites and services on earth combined, and almost half the traffic of Facebook. It has only grown since. And unlike every other backbone provider and mitigation provider, they can read your traffic in plaintext, TLS or not.

And finally, the DDOS issue. CloudFlare uses a method that mitigates against DDOS attacks by putting up a front page that asks you to enter a CAPTCHA. Apart from the fact it blocks bots, even backup bots, this doesn’t actually stop any big DDOS attack against something other than your web pages since it doesn’t block attacks by inspecting the packets but just relies on the CAPTCHA and stands in front of you. Dedicated DDOS mitigation works by making sure the packets never reach the real servers – CLoudFlare actually lets the packets reach their servers and relies on having “enough servers”. Given the new threat environment with DDOS attacks now going up to 600 Gbit/sec thanks to the “Internet of (apparently quite insecure) Things”, this may not be enough. Certainly, for the money you have to pay for the mitigation service it’s probably more effective to pay a dedicated DDOS protection service. Since CloudFlare is also responsible for hosting many of the DDOS-service provider websites, paying them for DDOS protection feels a bit like paying protection money to the Mafia.

The verdict

If you use CloudFlare to avoid buying a real certificate to secure your blog on “Human rights in Russia”, or run a webstore that accepts creditcard payments on your own webpages instead of using a payment processor page, you’re opening your users up to huge risks. And pleading innocent is less and less an option now that this information is out there. You can actually get better protection but CloudFlare will always be a Man in the Middle. Sven’s blog lists a number of alternatives if you can’t accept that.

However, for websites like mine, it’s not a big deal to accept less security. After all, before I migrated to CloudFlare the entire site was in cleartext. It’s slightly more secure now than it was, and thanks to this information I will make it more secure in the near future by using a better certificate.

If you move to CloudFlare like I did, you need to carefully weigh the pro’s and cons – better than I did, at least – before moving. But for a lot of blogs it may still be a very good option. If however your webshop or political blog is hosted by CloudFlare, you’d better do some checking before you post or pay. When in doubt, do not enter.

References

2016
[1] http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-have-a-problem/
[2] https://blog.torproject.org/blog/trouble-cloudflare

2015
[3] https://salt.agency/blog/seo-rankings-cloudflare/

2014
[4] https://scotthelme.co.uk/tls-conundrum-and-leaving-cloudflare/
[5] https://scotthelme.co.uk/cloudflares-great-new-features-and-why-i-wont-use-them/

Server-test
[6] https://www.ssllabs.com/ssltest/analyze.html?d=www.grundsatzlich-it.nl

Integrating Twitter in WordPress

Leave a reply

twitter large logo

Last year Twitter decided to change the way Twitter interacts with the rest of the world, by making it more difficult to integrate its twitter-streams with your own website. While you can get around this if you can deploy server-side software and go through the hassle of signing up for a developer key, a lot of folks run websites without being interested in having to program just to get their own tweets to display.

Twitter does have a solution, but this just dumps the stream on your site with the lay-out and styling of Twitter. While this is understandable from a branding and marketing point of view, it’s incredibly annoying to have your website look like a hash of different styles just because Twitter doesn’t like you changing the lay-out. So there are a lot of people looking for alternatives.

The best alternative I’ve found for my purpose is http://jasonmayes.com/projects/twitterApi/. Jason Mayes twitter API just takes the formatted twitter-feed, removes the formatting and provides the stream with normal tags to the page. Using standard CSS you can then style the stream and presto, you have a nice looking twitter feed.

How it works in WordPress is as follows:
– Download the software from http://jasonmayes.com/projects/twitterApi/
– Upload the javascript file “twitterFetcher_min.js” to your website. This could be as media but I chose to use FTP to upload it into a theme. As long as it’s on your website it’s okay though, the location is unimportant.
– Add a Text widget to the page where you want the tweets to show up.
– Include the following text in the widget:


<script src="/{path}/twitterFetcher_min.js"></script>
<div id="tweet-element">Tweets by Ronald Kunenborg</div>

<script>
var configProfile = {
"profile": {"screenName": '{yourtwittername}'},
"domId": 'tweet-element',
"maxTweets": 10,
"enableLinks": true,
"showUser": true,
"showTime": true,
"showImages": true
};
twitterFetcher.fetch(configProfile);
</script>

Replace “{yourtwittername}” with your own twitter name (of that of someone whose timeline you wish to show), and the {path} with the path of the uploaded javascript and you’re good to go. However, this looks pants. So we need to style it. In order to do that, include the following text in the widget before the script:
<style>
/*
* Tweet CSS - on Jason Mayes tweetgrabber (http://jasonmayes.com/projects/twitterApi/)
*/

div#tweet-element ul {
list-style: none;
}

div#tweet-element h2 {
clear:both;
}

div#tweet-element p {
font-size: 9pt;
margin: 0 0 0 0;
}

div#tweet-element ul li {
list-style:none;
overflow:hidden;
border-top:1px solid #dedede;
margin: 5px 0 10px 0;
padding: 0px;
}

div#tweet-element ul li:hover {
background-color:#f0f3fb;
}

/* tekst of tweet */
.tweet {
clear: left;
}

.user {
clear:left;
float:left;
}

.user a {
}

/* hide the @twittername, which is the 3rd span in the user class */
.user span:nth-child(3) {
display: none;
}

.user a > span {
margin-left:2px;
}

.user a > span {
display: table-cell;
vertical-align: middle;
margin: 5px;
padding: 5px;
}

.widget-text img,
.user a span img {
display: block;
float:left;
max-width: 40px;
margin: 2px 2px 2px 2px;
}

div#tweet-element p.timePosted {
clear: left;
font-style: italic;
}

div#tweet-element p.timePosted a {
color: #444;
}

.interact {
float:left;
margin-top:-7px;
width: 100%;
}

.interact a {
margin-left: 0px;
margin-right: 5px;
width: 30%;
}

.interact a.twitter_reply_icon {
float:left;
text-align: center;
}

.interact a.twitter_retweet_icon {
float:left;
text-align: center;
}

.interact a.twitter_fav_icon {
float:right;
text-align: center;
}

/* show media on front-page - hide it with display:none if you don't want to show media included by others. */
.media img {
max-width:100%;
}

#linkage {
position:fixed;
top:0px;
right:0px;
background-color:#3d3d3d;
color:#ffffff;
text-decoration:none;
padding:5px;
width:10%;
font-family:arial;
}
</style>

Make sure the <style> part is first in the Text widget.

Of course you can also put the style (without the <style> tags) in a stylesheet (.css) file, upload it and then refer to it, instead of pasting the stylesheet in the Text widget. In that case use the following command:

<link rel='stylesheet' id='twitter-css' href='/{path}/twitter-style.css' type='text/css' media='all' />

And please replace {path} with the desired path.

I hope this helps you as much as it helped me.

Migrating to CloudFlare

Leave a reply

www.cloudflare.com

I recently added CloudFlare as cache in front of my website. Not only does it provide worldwide local caching of my website, it also improves security by adding in an easy manner all kinds of features you’d otherwise find hard to arrange. It’s still a hassle, but not as much as it used to be.

The standard CloudFlare plan is free. Yep. And you can’t beat the value. The following features are part of it:

  • Caching at a server that is local to your visitors, improving their browsing speeds. Pretty nice and worth the price all in and of itself.
  • Analytics in an easy dashboard. You can get this by incorporating Google Analytics on your pages, true, but here it’s already in the product. However, CloudFlare also has a nice button that allows you to add Google Analytics to all of your pages, if you really want it, without changing your website in any way.
  • Safe browsing over SSL for people visiting the CloudFlare cache, at no charge.
  • DNSSEC can be turned on, securing your DSN entries (DNS translates the name of your website into the IP-address you need to actually get there) against rogue DNS-servers that change the IP-address to their own sites, so they can intercept the traffic or just spoof your website and change pages around. Could be quite embarassing if you are a dissident or well-known political figure, or a bank.
  • A “web firewall” that tries to catch spambots and scrapers before they even reach your website. The more advanced options are paid, but the free option is pretty nice. It has, for instance, the option of asking suspect browsers to authenticate their “humanity” before allowed to access your site. This is enabled by default.
  • IPv6 to IPv4 translation. If you’re on a provider that does not provide IPv6 website hosting you should move ASAP anyway, but suppose you can’t? In that case you have the option of pretending to your rather outdated server that the request is actually an IPv4 type request. Could be useful.
  • An option called “Apps”. Apps are small features you can enable that are provided by 3rd parties. One of these for instance is “A Better Browser” which warns users of older browsers that they should upgrade. Once again, no code on your website changes and you can turn it on and off quite easily. Other apps provide analytics, more security and monitoring but almost all of these are paid options.
  • Email-address obfuscation. For the truly paranoid, you can turn all your emailadresses on the site into addresses that cannot be harvested by the scrapers they said they would stop. I don’t bother with this, but feel free.
  • Hotlink protection. This is pretty nifty if you have a site with a lot of images, and people blogging about them link directly to your site from their article. That means their pageviews count against your bandwidth. With this option you can prevent those requests from being served.

These options are all easily accessible through a set of buttons as displayed here: cloudflare-buttons

Pretty nice all by itself. But I’ll discuss the setup and two main features in more detail.

Setup

The setup is easy. Just sign up and add your website. The main thing you need to get working is the nameservers. If you cannot change the nameservers for your website, things will get really tough because that is how CloudFlare works. If you cannot change them, contact your provider. If your provider does not allow nameserver changes, move away to another that does support it. Otherwise none of the newer features of the internet will work unless your provider agrees to provide them to you. That won’t be cheap.

After you get the nameservers changed, you have to log out and wait a few hours. By then the change will have been recognized by CloudFlare, and now you can actually use its features. The two most useful features are of course caching and encryption, which I explain below in a bit more detail.

Caching

The caching features of the CloudFlare platform help you in the sense that small DDOS attacks won’t bring down your website or hurt your direct provider. Big ones will mean you have to pay up (a lot), but it’s better than your direct provider shutting down your website for a minor DDOS assault, right? They also have the option named “always online(tm)” that provides a cached copy of your website, if it is offline on your own side. Note that this only goes for the popular (cached) pages but these are the most important ones anyway. Of course, caching can be disabled (temporarily) by turning on “development mode”.

Encryption

Encrypting the website gives you the option to have browsers come in over SSL. And this is very interesting because browsers are now signalling by default that your site is untrusted if it is not protected by SSL. The CloudFlare option provides SSL for your website from visitor browser to CloudFlare, but if you don’t add something more, it will still be unencrypted between CloudFlare and your original website.

If you trust the channel between your website and CloudFlare, this is still pretty safe. For most websites it’s a major improvement because they go from no SSL at all, to SSL between visitor and cache. But if you want more it’s pretty easy. Most website hosting companies provide you with the ability to place a self-signed certificate on your website, and CloudFLare can be set to acknowledge that certificate. You could also set CloudFlare to acknowledge only certificates signed by a trusted authority, increasing the security of your channel either further, or reducing it to zero, depending on who you trusted as certificate provider. In my case, I go with the self-signed certificate.

DNSSEC is however a bit more involved. I was unable to get this working because my hosting provider does not provide me with the ability to add a “DS” record to the DNS-server. I’m still looking into it. HOWEVER… my provider has automatic DNSSEC as long as I use their nameservers… This effectively means that I am going to have to do without DNSSEC *or* CloudFlare. Given the risks involved (minor) I’m going to stick with CloudFlare for a while, but I may be returning to the provider I have. I would really like them to have this though.

Summary

All in all, I can highly recommend CloudFlare. It’s free, it’s easy and provides immediate benefits for most websites. If you’re big enough to already have most of this it may be less interesting, but for 90% of the internet this is a step forward.

Update

Update 09-okt-2016: I’ve written a new article about why you should be careful when moving to CloudFlare, as it is not quite as suitable as I thought for websites that require actual security and encryption.

Encryption is not a silver bullet

Leave a reply

Have I been pwned?Recently, well-known security researcher Troy Hunt, responsible for the website Have I been pwned? described how someone lost 324000 records with full creditcard details, including security codes, by posting them on a public server. There were two parties suspected of the data breach, but neither could find any breach at first. So both parties stated categorically that there was no breach, all data was 100% encrypted and completely secure on their servers so the problem had to lie elsewere. And they were right, all the data was encrypted.

Now, encrypted data should be safe. And to be honest, encryption is more and more the mainstay of securing your data. Firewalls can be breached, servers and companies infiltrated, but if the data is encrypted it should remain secure even if you publish it on the internet. This is somewhat correct – barring adversaries like national intelligence services, who are very likely to be able to decrypt most schemes at the moment. It’s well known that the Dutch National Intelligence and Security Service (AIVD) is investing heavily in quantum computing research, for instance, which means that the NSA probably has one working right now. But apart from those entities, it’s still quite hard to crack decently encrypted data.

That is why in the new SQL Server edition, SQL Server 2016, it is now possible to keep the data encrypted all the time. Only the client can decrypt the data with their own keys. Barring vulnerabilities in the implementation this is a huge step forward: it is impossible for the database administrators to access data they aren’t allowed to see and the loss of a key only affects data stored for that client. Both are very important steps forward to enable clients to trust databases in the cloud. Which is one reason why Microsoft is pressing forward on this, because they will become entirely dependent on Azure in less than a decade, according to their own predictions. This means that trust in Azure will be a make-or-break issue for the company and their focus on improvements in security reflects this knowledge.

And let me be clear: this is a huge leap forward. The old situation could encrypt some data with server-side keys, but when you made a backup it was decrypted. And in several other scenarios it didn’t work if your data was encrypted. But now it works all over the database, you can set it up quite easily and even choose whether columns are encrypted in a deterministic way that gives the same result every time you encrypt the same value, which enables searching and joining, or random: every time you encrypt the value is different. The latter gives more protection from attackers who encrypt “likely values” and see if they match, which is a classic attack against password-files (see: rainbow tables / dictionary attacks).

In the picture you can see how it works by storing the keys on the client:
Always Encrypted SQL Server 2016

This means we can now store creditcard information and sensitive information in the cloud while not having to rely solely on the goodwill of the Azure database administrator.

There is unfortunately also a downside. The fact that data is now safer does not mean it is safe in all circumstances. The way “always encrypted” works has consequences for your implementation that could blow your encryption scheme right out of the water if misused. So while the temptation to store sensitive but potentially very interesting data because hey, “it’s encrypted” and thus safe, can overcome common sense and even regulations, we should still firmly ignore that temptation.

Because the case I linked in the beginning showed everyone that even if data is encrypted, it is not always safe. In the case which I quoted at the start of the article, the data was encrypted too, and it still leaked. The reason was that the encryption keys were known to the organisation involved and used to decrypt data for analysis. That decrypted textfile was then stored on a publicly accessible server. Encryption cannot mitigate that scenario if the keys are part of the webapplication and the owner of the application can also access the data. Anyone who can get to the keys, can decrypt the information. After that, the security of the data once again depends on what that person does with it – such as putting it on a public server.

This is the reason that if you want to process creditcard information, for instance, you need to be PCI compliant. This is a set of regulations drafted by the financial industry that tell you what data you can store and how. Very sensitive details such as the security code should NEVER be stored. They don’t give security regulations for the storage of the security code: storing it violates all the rules, no matter what you do. The case with Regpack shows that this is still true. What you store will eventually leak, even with encryption. Once quantum computers become available widely, all current encryption schemes are broken and that nicely encrypted data on the internet that wasn’t a problem… is suddenly readable text.

So while “always encrypted” is a step forward, you still need to be very careful about what you store and it still needs to be secure – processing encrypted data on an insecure platform means your data is just as insecure, as the data can be intercepted in memory. While solutions are in the works (Philips, IBM and others are working on homomorphic encryption schemes) this is currently not an option.

recommendations

My recommendations on this subject are as follows.

  • Do not store any data you are not allowed to store.
    If you do this anyway and lose the data, you will get fined or even shut down when this comes to light.

  • Do not store any sensitive data you do not have to store.
    Everything you store is a security risk, if you don’t store anything there are no risks. Being smart about what data to store is a big part of any security strategy.

  • If you do store sensitive data, let the owner of the data hold the key to that data if at all possible.
    After all, a file where every line is encrypted with a different key you don’t have, is a file that will be pretty hard to decrypt and certainly can’t be decrypted by accident by one of your employees.

  • If you cannot do even that, and your application does the encrypting, make sure the decryption key is locked in hardware like a smart card that is NOT reachable on any computer without physical presence.
    Violating this simple rule was what destroyed the Dutch Public Key provider Diginotar.

Some companies prioritize time-to-market and lower cost over data security. But eventually, those companies will be destroyed over that practice. The current digital environment is just too hostile to survive such practices for very long.

Certified Anchor Modeler

Leave a reply

As of today, I am certified as Anchor Modeler. My thanks go to Lars Rönnbäck (UpToChange.com), the best teacher you could have, as well as Juan-José van der Linden for inviting me and to Essent for hosting the course.

While the community of Anchor Modelers is still quite small, it will likely expand as the concurrent-reliance-temporal model is extremely interesting. The notion of positors and reliance combined with the positing and changing time is quite advanced. I’m looking forward to combining this with Martijn Evers’ notions about timeline choices with respect to Consistency/Accuracy/Availability.