Software Craftsmanship by Marcin Kliks

Tips for Software Engineering Managers in BigTech

Thu, 29 Dec 2022 00:00:00 +0000

Don’t ask for permission; ask for forgiveness

If it’s a good idea, and it doesn’t break anything - do it. It’s faster to do it and learn from mistakes.
Systems are inflexible by design. But are the only way to drive the organization. If the system accepts it - do it. If the system is broken - ask me to change it. Not everything can be improved right away. Be patient.
I forgive people trying to do things and making mistakes. I don’t forgive avoiding things.

Release early release often

Shipping early without getting feedback is stupidity. But shipping early just for listening to the customer’s feedback is the art of wisdom.
Be bold and ship an unpolished version; you are fighting with the market. Don’t criticize your colleges for shipping MVP.
Always share your proposals early to get initial feedback, before you invest more energy into finishing it.

Focus on things that give results

Working software beats the best proposals.
Written proposal beats every meeting.
Building prototypes beats thinking, if it’s worth doing it.

Meetings

Review your calendar every month.
Avoid being at meetings only for “just in case” / FOMO
Leave meetings where you don’t contribute to.
Only attend the meetings with clear goals. Be brave to give polite feedback on meetings which are waste of time.

Culture and Strategy

The culture of people beats executive strategy every day.
If you want to know the culture of the company, listen to the jokes, it tells the truth under the surface.
The only way to change the culture is to empower good parts of it.

People are not resources

The best team is the effect of training, not recruitment.

Don’t assume you will just get the best developers from the market. As a leader, your job is to develop and grow your team. Hire potential, and invest into people.
People will remember you for how you grew your team and what kind of teacher you have been. People remember managers, but they love great teachers.
Build a plan for the development of Junior, Mid, Senior, and Principal engineers. Most people don’t build plans not to specify points of failure.
If you want to make everyone happy, don’t become a leader. Good leaders make decisions that are good for the team in the long term, which can differ from individual goals.

Communication

Communication is such a crucial skill for the developer as the seven layers of the network stack are for microservices.
If you feel, that you’re good at communication, think again. Communication skills cover so many aspects of your job that it’s a never-ending skill to develop.
Invest in developing skills on how to do the best PR review, how to create good commit messages, take a course on how to write tech proposals, ADR, write concise documentation, make screencasts, and even the art of convincing management to secure the budget.
Most of the “the product were great, but management has killed it” problems might come from not understanding how to communicate its features, progress, and problems.
Invest every year in building up your communication superpowers.

Swift on Raspberry Pi 400

Tue, 19 Jan 2021 16:45:00 +0000

Raspberry Pi 400 has sufficient power to run full-featured microservices written in Swift. Running Swift (together with MongoDB) on such a small gadget is serious fun!

I’ve tested this setup on Raspberry PI 400 which has a quad core processor, 4 GB of RAM and a speedy USB 3.0 SSD, however the instructions below are not limited only to this device.
Even though base configuration was not overclocked, I’ve managed to run simple Vapor-based microservice, and achieve over 10 000 req/sec on this little piece of silicon!

Installing Swift on Rasperry PI

If you want to use the latest Vapor 4, you need to upgrade Raspberry PI OS to 64 bits. Otherwise, this step may be skipped.

Ideal setup, as of 01/2020 is:

Swift 5.3.1
MongoDB 4.9.0
Nginx

For IDE we will use:

Visual Studio Code 1.52.1 (latest current version)
sourcekit-lsp trunk (revision:6eb17c9a7bc00bec83d57d2399ba9f5ab14d3bcc )

Upgrading to Rasperry PI OS 64 bit

Swift ARM builds are available here: https://packagecloud.io/swift-arm/release. Unfortunatelly, the latest version for Rasperry PI OS (default distribution) is 5.1.3 and it’s not usable with the latest Vapor 4 version. We need to upgrade the operating system to 64 bits.

Install the beta version of Rasperry PI OS 64 bit here if you need the latest Swift.
Install swift package from packagecloud.io https://packagecloud.io/swift-arm/release/packages/debian/buster/swiftlang_5.3.1-3-debian-buster_arm64.deb with dpkg -i
Swift REPL is not usable at the moment, but it’s not a big deal so don’t worry.
Swap file. Swift needs a lot of memory during the compilation, so let’s find a spare SSD drive, and create some fast swap files, at least 5 GB.
```
swapoff -a
dd if=/dev/zero of=/media/[yourssd]/swap ibs=1M obs=1M count=5000
mkswap /media/[yourssd]/swap
swapon /media/[yourssd]/swap
```
That’s it. Test if your code compiles. Vapor 4 should run just fine!
```
swift package init --type executable
swift build
```

IDE on Raspberry PI with Visual Studio Code, LSP and Swift

Install ARM64 DEB package from Visual Studio Code https://code.visualstudio.com/download
You need this package: https://github.com/apple/sourcekit-lsp to use the Auto-Completion functionality in Visual Studio Code, because version 5.3.1 is unusably slow. The new version is 10 times more performant. We will compile it from sources using this documentation here. This will work for arm64:

swift build -c release -Xcxx -I/usr/lib/swift -Xcxx -I/usr/lib/swift/Block
cp .build/aarch64-unknown-linux-gnu/release/sourcekit-lsp /usr/local/bin/sourcekit-lsp

apt-get install npm
cd Editors/vscode
npm run createDevPackage
code --install-extension ./out/sourcekit-lsp-vscode-dev.vsix

After installation, every feature should work well, including: autocompletion, fix-its, documentation, go to definition. If not, it’s probably because of the internal card speed - see below how to improve the performance by a factor of 10.

Optimization

SD cards tend to be slow. I recommend to either boot from an SSD entirely, or boot from an SD card, but in this case move the essential data to faster drive. It makes a huge difference during development.

Directories to move to external faster drive:

~/.config/Code (for Visual Studio Code)
~/.cache  (for various developmnt tasks)
/tmp (e.g. lsp cache)

You can tune your system for maximum performance:

fs.inotify.max_user_watches=524288
cd /sys/devices/system/cpu
echo performance > cpu0/cpufreq/scaling_governor 
echo performance > cpu1/cpufreq/scaling_governor 
echo performance > cpu2/cpufreq/scaling_governor 
echo performance > cpu3/cpufreq/scaling_governor 
echo performance > cpu4/cpufreq/scaling_governor 

MongoDB on Raspberry PI

Ok, this is an optional step. Chances are, your microservice needs a database, such as MongoDB. Yes, it works just fine, but you need to compile it from sources.

You need at least 20 GB of free space for compilation. Compilation will take at least 5-10 hours.

We will compile as usual using https://github.com/mongodb/mongo/blob/master/docs/building.md, but have to provide additional flag since there are problems with crc32 because of missing instructions. More information here

Using flag --use-hardware-crc32=off will compile it fine. Full script should look similar to this:

git clone https://github.com/mongodb/mongo.git

python3 -m vevn virtualenv
source virtualenv/bin/activate

python3 buildscripts/scons.py install-mongod --disable-warnings-as-errors --use-hardware-crc32=off

Congratulations, after 5-10 hours, you will get the 4,2 GB binary file called mongodb. It’s time to strip it to just ~40 MB with the instructions below:

root@raspberrypi:/media/pi/rpi/mongo# ls build/install/bin/mongod
build/install/bin/mongod

root@raspberrypi:/media/pi/rpi/mongo# strip build/install/bin/mongod
root@raspberrypi:/media/pi/rpi/mongo# cp build/install/bin/mongod /usr/local/bin

Now it’s time to prepare a storage space for a database and logs. Use your external drive and mount it somewhere. E.g. create mongo-data and mongo-logs directories, and use the following configuration files as examples. I used the simplest configuration possible. I recommend changing the default user pi, for security reasons.

root@raspberrypi:/media/pi/rpi/mongo# cat /etc/systemd/system/mongodb.service

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=pi
Group=adm
#EnvironmentFile=-/etc/default/mongod
ExecStart=/usr/local/bin/mongod --config /etc/mongod.conf
PIDFile=/var/run/mongodb/mongod.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

And this is the counterpart /etc/mongod.conf file.

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /media/[yourssd]/mongo-data
  journal:
    enabled: true
#  engine:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /media/[yourssd]/mongo-logs/mongo.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Now, it’s time to run it!

systemctl daemon-reload
service mongodb start

Check your logs directory for troubleshooting with journalctl -u mongodb.service.

Congratulations!

I hope you will have as much fun as I did with this. Feel free to leave any comments or suggestions below. Thanks for trying it out!

Arctic vault - say hello!

Sun, 02 Feb 2020 00:00:00 +0000

If you read this, Github probably already did “it”.

Github is going to archive all, world-wide public github repositories in a secret “Vault” located somewhere in Arctic mountain. Today. Including this page.

The process begins in 2 hours, on 2/02/2020. More info

It seems, this blog is going to be archived to, and will be stored underground for about 1000 years. Well, it’s great oportunity to write, something. Something that lasts.

What is it like to live in 2020?

I think, I’m grateful to be living in 2020.

Living in 2020 is understanding much more than ever, but living in social “bubbles”.
Living in 2020 is to buy everything you’d really imagine, having free shipping worlwide.
Living in 2020 is having private chauffeur available everywhere (we used to call it “Uber”).
Living in 2020 is to have supercomputer with 20 sensors, just in your pocket(smartphone).
Sadly, living in 2020 is to forget about God, and avoiding taking responsibility.
Sadly, living in 2020 is giving privacy for stupid pictures.

Being grateful

I’d like to thank all previous ~ 10,000 generations of people, including my nearest family, to work on foundation for everything we have today.

I’m grateful for:

my wonderful family, wife, doughter
oportunity to live in a world of programming, and fantastic work experience
piece of mind, piece of soul

See you in 1000 years!

Anet A8 3D Printer - 1 year later

Wed, 19 Dec 2018 00:00:00 +0000

A year with Anet A8 3D Printer

Today, you can buy 3dprinter just under 200$ (Anet A8 self assembly required in this case). This is price territory of generic “desktop printers”. I’ve used my 3D printer more often than desktop printer! After 1 year, I’m really happy with the purchase, and don’t regret it. Some parts such as: filament heads, and bearings needed to be replaced, but it was easy to do, and the parts are ready available.

How I used my Anet A8 printer in a 1 year

Dishwasher

My IKEA dishwasher door broke one day. The problem was with one small plastic part, which is quite often material used by IKEA furniture. I decided to give my 3D printer a try. Tinkercad was chosen as the simplest app to quickly prototype part. I had no previous 3D-modeling experience whatsoever.

It took some time to measure the part but it was really quick and easy to print it out. I took 3 tries to improve it further to make it slightly bolder than original to not broke anymore.

Lifehacks

Small life hacks here and there. Bathroom improvements.

Boxes

I’ve used wood-based filament, quite expensive but worth it: https://3dreaktor.pl/filament-WOOD/Fiberlogy-FiberWood-750g-175

I had to increase printing head diameter to 0.5, to not constanly get stuck. But it made the difference in overall outlook of the product! Looks like real wood!

Figures

Toys for my daughter

Printer self-improvements :-)

Power supply cover

Tools

Iphone Cover

IPhone stand

Toy basketball

Książki warte przeczytania

Mon, 17 Dec 2018 00:00:00 +0000

Książki warte przeczytania / 2018

12 Rules of Life / 12 życiowych zasad - J. Peterson

Dlaczego strukturami społecznymi ludzi rządzą te same zasady co społeczności … krabów? W jaki sposób silny kręgosłup moralny pozwala odnieść sukces? Książka znakomitego profesora psychologii o 12 pięknych zasadach, które nie ułatwiają, ale na pewno nadają nowy sens życiu.

Głaskologia

Książka - zaskakująco twarda, pomimo “miękkiego tytułu”. Każdy powinien ją przeczytać, ponieważ 65% pracowników nigdy nie usłyszało od swojego pracodawcy pochwały na temat wykonywania obowiązków zawodowych. Powoduje to spadek motywacji. W Polsce ten wskaźnik pewnie jest jeszcze wyższy. Gdyby każdy menedżer przeczytał tą książkę, pracowalibyśmy o połowę lepiej.

Siła nawyku - Charles Duhigg

Nawyki nami rządzą - jak sprawić by działałby na naszą korzyść? Kilka prostych tricków - “ustaw buty koło łóżka”, które skutecznie walczą z alter-ego. Pozycja dość znana, ale polecam do niej wracać co pewien czas.

Homo deus - Yuval Noah Harari

Futurystyczna acz ponura wizja przyszłości w której nie ma chorób i błędów, nie jednostka podejmuje decyzje, tylko “system” - co da doskonałą przyszłość nadludzi. Książka polecana przez Billa Gatesa, Zuckerberga ;-)

Dlaczego Zalando? - Hagen Seidel

Ciekawa historia startupu, który nie boi się ekspansji i planowania długoterminowego zamiast szybkiego zarobku.

Labirynty Scruma - Jacek Wieczorek

Scruma znamy wszyscy - co można jeszcze wymyślić? Ano, jeśli uważasz, że twój Scrum (daily, retro, planning) działa dobrze, to przeczytaj książkę. Odczarujesz złudzenia i samego siebie. Książka bardziej skuteczna niż katechizm przed spowiedzią!

The pyramid principle - Barbara Minto

Jak precyzyjnie niczym laser wyrażać myśli na piśmie, aby docierać do odbiorców. Jeśli szykujesz pitch do CEO - nie możesz tej pozycji pominąć.

Coders at work - Peter Seibel

Okazuje się, że programista programiście nierówny. Nawet Ci najlepsi programiści programują zupełnie inaczej. Brendan Eich lubi programować w stylu eksploracyjnym, a twórca Linuksowego XSavera - pisze w niskopoziomowym C ciurkiem, od góry do dołu. Obaj są legandami. Wspaniałe wywiady z wieloma znakomitymi programistami, które inspirują i dodają nadzieji, że każdy może odnieść sukces.

Swift 4 Cookbook - Linux & MacOS, part 1

Wed, 11 Oct 2017 00:00:00 +0000

Swift is powerful, fast and safe programming language. From version 2 it’s open source, and officialy supported on Linux. Its native performance, type safety, interactive (REPL) playground and scripting posibility, is a joy to use.

The main goal for Swift on Linux is to be compatible with Swift on macOS/ios/watchos, and there is an official Apple documentation for Foundation. Howerever, this quick cheat-sheet gives you quick glance what this language+standard library is capable of.

Setup

There are 2 ways to install Swift 4 for server side applications:

Standard installation:

Download Swift 4.0 from: swift.org for your Ubuntu distribution, following manual
You can also use Docker instead, (it runs on all distros)

docker run --rm --privileged -ti swiftdocker/swift:latest /usr/bin/swift -I /usr/lib/swift/clang/include/

What’s so special about Swift 4 ?

expressive, high level language with generics
safe by design(type safety, optionals, error handling, undefined behavior detection)
native code compilation with high runtime performance
seamless, zero cost C language interoperability without ha ck s
efficient memory management arc and ownership
functional paradigms included

How to use this tutorial

Make sure you read this great intro to Swift 4
This tutorial is about using Foundation API’s, it’s not a language guide.
This tutorial doesn’t cover writing web app in swift, though - it’s coming soon!

To play with examples in this tutorial, you can use swift REPL. You also compile the code with swiftc [file.swift]

To run REPL:
1. On Mac - just type swift
2. Linux - type: swift -I /usr/lib/swift/clang/include/
You can also use XCode playground - fire up Xcode, then File -> New -> Playground

More capabilities of Swift REPL here

Printing text

>print("Hello world!")
Hello world

>print("""
Hello\nworld!
""") 	
Hello
world

>print("one", "two", "three")
one two three

>print("one", "two", "three", separator: ",")
one,two,three

>print("one", "two", "three", separator: "\n")
one
two
three

>print(1, "and", 2)
1 and 2

Printing to stderr

import Foundation
FileHandle.standardError.write("test".data(using: .utf8)!)

Printing to stdin

    
import Foundation

FileHandle.standardError.write("test".data(using: .utf8)!)
FileHandle.standardInput.write("test".data(using: .utf8)!)
FileHandle.standardOutput.write("test".data(using: .utf8)!)

Executing shell commands

Run process with arguments and wait for completion

	
import Foundation

let p = Process()
p.launchPath = "/usr/bin/wc"
p.arguments = ["-l", "/etc/passwd"]
p.launch()
p.waitUntilExit()

Suspend and resume process in background

import Foundation

let p = Process()
p.launchPath = "/bin/cat"
p.arguments = ["/var/log/system.log"]
p.launch()
p.suspend()
sleep(1)
p.resume()
sleep(1)
p.terminate()

Monitor process running in background

This example prints system logs for 10 seconds, and then exits.

    
import Foundation

let p = Process()
p.launchPath = "/usr/bin/tail"
p.arguments = ["-f", "/var/log/system.log"]
p.launch()

var elapsedSeconds = 10
while p.isRunning && elapsedSeconds > 0  {
  print("Running")
  sleep(1)
  elapsedSeconds -= 1
}

Redirect stdin / stdout

    
import Foundation

let p = Process()
p.launchPath = "/bin/cat"
p.arguments = ["-"]

let pp = Pipe()

p.standardInput = pp

p.launch()
pp.fileHandleForWriting.write("test\n\u{4}".data(using: .utf8)!)
p.waitUntilExit()

Networking

Download HTTP document

    
import Foundation

let downloadURL = URL(string: "http://www.google.com")!
let contents = String(contentsOf: downloadURL, encoding: .utf8)
print(contents)

Download HTTP document and autodetect(sniff) encoding

     
import Foundation

var x: String.Encoding = String.Encoding.ascii 
let url = "https://www.ecma-inational.org/publications/files/ECMA-ST/Ecma-128.pdf"
try? String(contentsOf: URL(
            string: url)!, 
            usedEncoding: &x) 

print(x) 

Download binary file

    
import Foundation

let fileContents = Data(contentsOf: URL("http://filea.x")!)

Note: It uses RAM memory to store entire file in it, it can be handy for smaller files / quick scripts.

Download in parts(chunks)

    
import Foundation
import Dispatch

let config = URLSessionConfiguration.default
let s = URLSession(configuration: config)

let task = s.downloadTask(with: URL(string: "http://google.com")!) { url, response, err in
    count += 1
    print(count)
}
task.resume()

Filesystem operations

Read binary file

    
import Foundation

let binaryData = Data(contentsOf: URL(fileURLWithPath: "/tmp/a.x"))

Read text file

    
import Foundation

let textFile = String(contentsOf: URL(fileURLWithPath: "/etc/passwd"))

Write to text file

import Foundation

"hello world".write(toFile: "/etc/xxx", atomically: true, encoding: .utf8)

Write to stderr

import Foundation

FileHandle.standardError.write("test".data(using: .utf8)!)

Write to stdin

    
import Foundation

FileHandle.standardError.write("test".data(using: .utf8)!)
FileHandle.standardInput.write("test".data(using: .utf8)!)
FileHandle.standardOutput.write("test".data(using: .utf8)!)

Read data in chunks

    
import Foundation

let f = FileHandle(forReadingAtPath: "/etc/passwd")
f!.readData(ofLength: 10)

// all: 
f!.readDataToEndOfFile()

Check if file is readable

    
import Foundation

if FileManager.default.isReadableFile(atPath: "/etc/fstab") {

}

Remove file

    
import Foundation

FileManager.default.removeItem(at: URL(fileURLWithPath: "/tmp/x")!)

Copy file

    
import Foundation

FileManager.default.copyItem(atPath: "from.txt", toPath: "to.txt")

Copy directory

import Foundation

FileManager.default.copyItem(atPath: "/etc/", toPath: "/etc.backup")

List directories

    
import Foundation

// Shallow list without folders
for entry in FileManager.default.contentsOfDirectory(atPath: "/") {

}

// deep list
for item in  FileManager.default.enumerator(atPath: "/etc")! {  
  print(item) 
}

Recursively list files

import Foundation

FileManager.default.subpathsOfDirectory(atPath: "/root/")

Create directory with permissions

    
import Foundation

let permission = 0o700 as NSNumber

FileManager.default.createDirectory(
  atPath: "/tmp/my/brand/new/directory", 
  withIntermediateDirectories: true, 
  attributes: [.posixPermissions: permission]
)

Coroutines / threads /

Swift comes with LibDispatch which uses threads and queues for dealing with asynchronous tasks.

Concurrent requests example

        
import Foundation
import Dispatch

let config = URLSessionConfiguration.default

let s = URLSession(configuration: config)

var count = 0
for i in 1...100 {

let task = s.downloadTask(with: URL(string: "http://www.wp.pl")!) { url, response, err in
  count += 1
  print(count)

}
task.resume()
print("Started")

}
dispatchMain()

Scripting with Swift

You can use Swift as a interpreted language, too with pure text files!

Just create swift file:

    
touch /tmp/hello && chmod a+x /tmp/hello

With contents:

    
#!/usr/bin/xcrun swift
  
import Foundation

print("Hello, what's your name?")
let name = readLine(strippingNewline: true)
print("Hello, \(name)")

Run it with /tmp/hello

Voila! You can execute any of examples using this technique.

Swift 3 on Linux tutorial - Zewo - part 2

Sun, 04 Sep 2016 00:00:00 +0000

Swift 3 Linux tutorial - part 2

In this tutorial we will continue exploring more advanced parts of Zewo framework, including:

concurrency
JSON handling
communication via channels

Coroutines

Unlike Go, Swift 3 does not come with coroutines out of the box, but uses separate library called VeniceX. Worry no, the usage is quite similar to go, and performance is great, scaling up to milions coroutines on a single host.

Zewo follows CSP model, similar to Golang read more model coroutines incorporting wonderful libmill C library.

Example coroutines

This code runs in the “background” as a lightweight coroutine. Unlike threads, you can run literally milions of these on the modern hardware, it’s so performant.

import Venice

for i in 1...100 {
	co {
		print("Spawned \(i) coroutine! I'm so asynchronous!")
	}
}

To comunicate with the coroutine, we use channels, for sending / recieving values of specific type(for example Int):

import Venice

let channel = Channel<Int>()

co {
	nap(for: 1.second)
	channel.send(1)
}

print("Waiting for the result...")
let value = channel.receive()
print("Got \(value) !")

Run it and:

Waiting for the result...
Got Optional(1) !

Keep in mind, that recieving a value from the channel is a blocking operation. To check buffer for presence of data without blocking, you can use select function described later on.

DIY - worker

In the first part of the tutorial, we’ve built synchronous http service which immediately returned a value. In this tutorial, we will try to make prime numbers web-service! Computing of prime numbers takes a lot time, and we don’t want to have API HTTP timeouts. That’s why we cannot do it synchronusly. We will defer computation tasks to the background instead.

How to do it? Do we need some worker solution, similar to Python RQ, Celery, threads, Redis or something? The answer is - not at all! Zewo with Swift does the trick without any other components!

Let’s begin with the endpoints:

/task/new endpoint will schedule long-running computation in the background. The only thing returned synchronusly is task id as a JSON. We handle POST requests here, thus “creating” task resource.
/task/[uid] - this endpoint will be used by the clients polling the result.

The code

File main.swift:

import Axis
import HTTPServer
import Foundation

var messages:[String:Channel<Int>] = [:]


let app = BasicRouter { route in

    route.post("/task/new") { request in 
        var body = request.body
        let data = try body.becomeBuffer(deadline: 30.seconds)
        let content = try JSONMapParser().parse(data)
        guard let value = content?["value"].string else {
            return Response(status: .badRequest, body: "Missing value")
        }
        guard let intValue = Int(value) else {
            return Response(status: .badRequest, body: "\(value) is not a number")
        }

        let taskId = UUID().description
        let channel = Channel<Int>()
        messages[taskId] = channel

        co {
            let computedPrime = nextPrime(intValue)
            channel.send(computedPrime)
        }

        return Response(
            headers: ["Content-type": "application/json"],
            body: try! JSONMapSerializer.serialize(["taskId": .string(taskId)])
        )
    }

    route.get("/task/:taskid") { request in
        guard let taskid = request.pathParameters["taskid"] else {
            return Response(status: .internalServerError)
        }
        guard let channel = messages[taskid] else {
            return Response(
                status: .notFound,
                body: "Routine not found"
            )
        }
        var response: Response?
        select { when in
            when.receive(from: channel) { message in
                response = Response(body: "Result: \(message)")
                messages[taskid] = nil
            }
            when.otherwise {
                response = Response(body: "Still computing.")
            }
        }
        return response!
    }
}

try Server(host: "0.0.0.0", port: 8080, reusePort: true, responder: app).start()

Create file computation.swift:

import Venice

/// Returns true if number is prime number, false otherwise
func isPrime(_ number: Int) -> Bool {
    guard number != 1 else {
        return false 
    }
    var prime = true
    var i = 2
    while i < number {
        if i % 1000 == 0 {
            yield
        }
        if number % i == 0 {
            prime = false
            break
        }
        i = i + 1
    }
    return prime
}

/// Returns the first next prime number greater than given number
func nextPrime(_ previousPrime: Int) -> Int {
        var currentCandidate = previousPrime
	var found = false
	while found == false {
	    currentCandidate += 1
	    if isPrime(currentCandidate) == true {
	        found = true
	    }
	}
	return currentCandidate
}

Both files(main.swift, computation.swift) share common namespace, without a need for importing files each other.

Now, compile the code and let’s try our computation.

> curl -XPOST  "http://127.0.0.1:8080/task/new" -d '{"value": "198491329"}' -v

{"taskId":"7398EFF1-838A-495E-AB9C-AC7C5C73BE3"}

> curl "http://127.0.0.1:8080/task/7398EFF1-838A-495E-AB9C-AC7C5C73BE3"

Still computing.

[after a while]

> curl "http://127.0.0.1:8080/task/7398EFF1-838A-495E-AB9C-AC7C5C73BE3"
Result: 198491369⏎                      

Explanation

The essence of our program is the usage of this coroutine:

co {
    let computedPrime = nextPrime(intValue)
    channel.send(computedPrime)
}

nextPrime() function tries to behave nice, yielding the control of program to other coroutines, with the yield function. Without it, it would preempt entire CPU time for one coroutine.
using guards to unpack Optionals makes a program really readable
we used select() function which doesn’t wait until channel is filled up, so we can always answer HTTP requests as fast as possible

Summary

Congratulations! You completed part 2 of the tutorial. Don’t miss part 3 with Filesystem functions!

Swift 3 on Linux tutorial - Zewo - part1

Mon, 01 Aug 2016 00:00:00 +0000

Swift 3 Linux tutorial - part 1

This tutorial shows how to write HTTP micro-service with Swift on Linux, thanks to the powerful Zewo framework.

Zewo

Zewo is an extremely modular set of server side libraries for server side development. It includes everything that’s needed to write full featured microservices in Swift, as well as commandline applications, including:

HTTP and WebSocket servers
HTTP/S clients
couroutines (library “Venice” based on libmill)
database drivers, and ORM mappers
templating engine(“Mustache”)
Zeromq, Redis support
dozen of helper libraries

Beware - it’s neither stable, nor have documentation yet, but the Sources are really readable. The components are slick, boundaries well defined, but the real deal is - it’s really easy to play with, customize, hack something working quickly. Thanks to Swift generics and protocols it’s a joy to use.

SPM - swift package manager.

SPM is official package manager for Swift, which is really simple and useful.

It follows convention over configuration, so there is no need to configure directory paths, or versioning schema. SemVer is the only versioning schema you can use, so there is a strict standard around packages to follow. An example could be: 0.1.2 version. 0 is the major version, 1 - the minor, and patch level - 2.
Packages are stored in the generic GIT repositories, so no central - hub, or packages index is needed. Just use generic http+git url for package location.
SPM is the part of swift distribution itself, so there is no need to install anything.
Some extra features:

can compile c code, integrates with apt-get and homebrew
manifests are written in pure-swift, and are easly readable
can generate xcode project file (snapshot version needs workaround, because it has a bug)

Using SPM

Overview:

OVERVIEW: Perform operations on Swift packages

USAGE: swift package [command] [options]

COMMANDS:
  init [--type <type>]                   Initialize a new package
                                         (type: library|executable|system-module)
  fetch                                  Fetch package dependencies
  update                                 Update package dependencies
  generate-xcodeproj [--output <path>]   Generates an Xcode project
  show-dependencies [--format <format>]  Print the resolved dependency graph
                                         (format: text|dot|json)
  dump-package [--input <path>]          Print parsed Package.swift as JSON

OPTIONS:
  -C, --chdir <path>        Change working directory before any other operation
  --color <mode>            Specify color mode (auto|always|never)
  -v, --verbose             Increase verbosity of informational output
  --version                 Print the Swift Package Manager version
  -Xcc <flag>               Pass flag through to all C compiler invocations
  -Xlinker <flag>           Pass flag through to all linker invocations
  -Xswiftc <flag>           Pass flag through to all Swift compiler invocations

NOTE: Use `swift build` to build packages, and `swift test` to test packages

Start with “Hello world” app!

Enough theory! Now, let’s build simple HTTP app.

Create your project dir:

> mkdir hello && cd hello

Now, let’s generate barebones of the package:

> swift package init --type executable

Creating executable package: hello
Creating Package.swift
Creating .gitignore
Creating Sources/
Creating Sources/main.swift
Creating Tests/

If you need generic library, choose “–type library”.

Sources dir is currently opinionated name of all Sources directory. At the same time, when swift compiler detecs main.swift file, it assumes, executable is desired. Let’s see how main.swift file looks in practice:

> cat Sources/main.swift
print("Hello, world!")

Now build the hello world app -

> swift build 

Compile Swift Module 'hello' (1 sources)

Hurray! Hello world app is ready. Output executable, with debug symbols is located here:

> ls .build/debug
  
ModuleCache       hello.build       hello.swiftdoc    hello.swiftmodule 	helloTests.xctest

Run it:

> ./.build/debug/hello
Hello, world!

When ready to release, you can compile it without debugging symbols(faster, and smaller executable)

> swift build --configuration release

Writing HTTP server

We will use Zewo framework, which contains over 50+ reusable, server-side components such as HTTP Server, JSON handling, storage-drivers (mysql, postgres), posix functions and much much more.

> cat Packages.swift

import PackageDescription

let package = Package(
        name: "hello"
)

We will use various Zewo components in separation. But remember:

Try to use the same version (0.7 for example) or Zewo modules used in your project. Mixing different versions of Zewo components might create a dependency hell.

We will create simple http server app, using JSON, routers, and creating some files.

Now, update your Package.swift file with:

import PackageDescription
	
let package = Package(
    name: "hello", 
    dependencies: [
      .Package(url: "https://github.com/Zewo/Axis.git", majorVersion: 0, minor: 14),
      .Package(url: "https://github.com/Zewo/HTTPServer.git", majorVersion: 0, minor: 14)
  
    ]
)

Now let’s write full-featured http app with routing. We use only 2 modules from Zewo - HTTPServer and Router. We expose parametrized endpoint /hello/[whatever] to demonstrate full - featured url parameters handling, rendering http response, and returning http status codes.

import HTTPServer
    
let app = BasicRouter { route in
    route.get("/hello/:name") { request in
        guard let name = request.pathParameters["name"] else {
            return Response(status: .internalServerError)
        }
        return Response(body: "Nice to meet you, \(name)!")
    }
}
    
try Server(responder: app).start()

Read more about routing and HTTP server for more details.

To build your packages, type: swift build. Dependencies are downloaded into Packages directory, as a full-featured git repositories, checked out for given git tags.

ls Packages/
C7-0.8.1                      HTTPParser-0.7.4              POSIX-0.5.1                   
CHTTPParser-0.5.0             HTTPSerializer-0.7.2          POSIXRegex-0.7.0              
CLibvenice-0.5.0              HTTPServer-0.7.2              PathParameterMiddleware-0.7.0 
CURIParser-0.5.0              IP-0.7.0                      RegexRouteMatcher-0.7.0       
File-0.7.5                    JSON-0.9.0                    Router-0.7.2                  
HTTP-0.7.2                    MediaType-0.8.0               S4-0.8.0     

String-0.7.3
StructuredData-0.8.3
TCP-0.7.4
TrieRouteMatcher-0.7.1
URI-0.8.1
Venice-0.7.2

Tip: You can modify any of the Packages/ repository, and push it back to the repo. It simplifies development a lot.

Now point your browser to: http://localhost:8080/hello/zewo

Hurray, it’s your first Swift HTTP app with routing and status codes handling.

Packaging

To share your app/library with others, just push the repo somewhere (to the github for example) and make ordinary git tag (following the SemVer schema).

For example:

git tag 0.1.0
git push origin master
git push origin --tags

You can use this repo in other projects with simple:

.Package(url: "https://github.com/user/hello.git", majorVersion: 0, minor: 1)

You can specify also patchLevel as an argument, but it’s not recommended. When ommited, SPM always uses the latest version, which is backward compatible.

Tuning performance

Zewo framework is very performant, it uses the same concurrency strategy as GO language, using CSP coroutines.

But you can find, that by default, it uses only one core. You can change it very easily with ‘reusePort’ parameter. It informs operating system to balance incoming socket requests, around many application instances. Remember only to run as many processes as needed, operating system will take care of the rest. You can run multiple http servers and/or run multiple processes of our app with configuration below:

try Server(reusePort: true, port: 8080, responder: app).startInBackground()
try Server(reusePort: true, port: 8080, responder: app).startInBackground()
try Server(reusePort: true, port: 8080, responder: app).startInBackground()
try Server(reusePort: true, port: 8080, responder: app).start()

Part 2

Don’t miss part 2 of tutorial! We will use files manipulation, coroutines, operations on strings, and JSON.

Swift vs Go vs Python battle

Sun, 22 Feb 2015 00:00:00 +0000

Sweet spot

Swift is the new programming language brought to you by Apple last year. It brings a refreshing air of novelty to the world of C, Python, Ruby, Rust and Go. The language is neither as beautiful as Python nor as fast as raw C, and the compilation speed cannot compete with Go. But the motivation behind Swift was to hit the sweet spot between C, Python and Go. I strongly recommend reading the Swift Book for example here.

Apple prepared a thorough comparison of Swift and Python during the presentation. Much as they are similar, they are different by two yet essential characteristic features:

both languages share the same language simplicity (for example, print is the keyword, OOP programming is optional)
both are great interactive interpreters( swift vs swiftc)
both can be used for scripting purposes (#!/usr/bin/xcrun swift)
but the performance differs between these languages and
Swift is not Open Source yet. Booo!

Technology underlying Swift is based on LLVM, which handles all of the data structures, all compiling and optimization tasks on its own. LLVM is used also by PyPy, CLang, and other languages as well.

Is it really as fast as advertised?

Last week Swift 1.2 was released so I decided to write a small benchmark game for:

Swift 1.2
Google’s Go 1.4.2
CPython 2.7.9
PyPy 2.5.0 - Python 2.7.8

It was delightfull to simultanously write the code for 4 platforms in multisplit code editor window :-) The alghorithm is fairly easy - creating 1M array and filling it with the incremented integers(64 bit ints). Then, creating another array, which consits of sum of every 2 elements from the first array. De facto it tests memory read / write access time. Then comes the next step: summing up every 100th element of the array - it should be very difficult to inline it properly, but who knows what Swift can do ? :->

Note: I haven’t used the list comprehensions, not to favor any language. I will prepare this variant later.

Code differences

Go

As you can see, the Go code is the longest one, but not much longer that the others. The source code is fairly short whatsoever. But that mandatory package() and main functions requirement is enforced. The weird part of Go is that Rob Pike decided not to make Print a keyword, but rather a library function. That’s why you should import fmt module. Appart from that, array functions are quite limited in comparison to Python and Swift. By default, Go’s arrays are not resizable - as in C. Go language provides append keyword for resizing arrays but the creator of Go couldn’t make up his mind how to implement single vectors. For example, you can use Slices pattern. WTF. After 30 years language designers should provide one efficient generics implementation! But let’s move on. The way we declare the variable is bizzare - inside for loops you declare local variable without var. On the other hand, outside the loop you can still use two options interchangeably, a := 1 or var a = 1. As you can see, it is pretty incoherent. The good part is that the compiler is fast - it’s the fastest compiler I’ve ever seen. Basically, it feels like an interpreter(but it’s not interactive). You can even run go run to hide the compiling step.

Swift

Swift has type inference everywhere as well, hence the short syntax for type definitions. append() is a method, not a keyword, just like in Python. The thing I love is that an optional OOP is not enforced on you. No package, no module, not even main function required. Furthermore, adding a header #!/usr/bin/xcrun swift followed by chmod +x test.swift allows us to run ./test.swift without compilation, just like in Python. Wow. What else? Println and print are keywords. Your fingers will appreciate it for sure. I really like the Range operators, which are lazy evaluated, and handle custom iterations steps in performant manner. Arrays work as expected and live interpreter has nifty editing capabilities, which is so much better than in Python (multiline handling). Editing capabilities are even better thanks to excellent type inference.

Python

I really love Python, but the lack of optional typing, type inference, and compilation is definitely lacking here. But here is where the language shines - the source code is the shortest of all and the cleanest of many. var y: Array<Int> = [] looks really iffy in comparison to the simple y = [] in Python. But in Swift we can emulate that writing var y : [Int] = [], which is slighty better on the eyes. I really love the Python clarity here. xrange operator is here with a similar purpose as Range<Int> in Swift.

Performance

I’ve run loop 30 times. The results are mixed. CPython required 35x more time to finish the benchmark. Yes, it was 35x slower than Swift! On the other hand, Swift and Go come so close in results that it doesn’t matter - but in this particular microbenchmark Swift was 60% faster than Go. It trully amazed me how performant PyPy was - more than 5x faster than generic CPython implementation. Wow.

The run times for all implementations:

Let’s put the results together:

In this particular test - Python is 34x slower than Swift.

Conclusions

It’s just one simple benchmark, real life is another story, BUT:
Swift performance looks like a good fit not only for scripting, but high computing purposes too.
I liked its type inference.
PyPy’s performance is better than expected. It’s just 6x slower than Swift, but also 6x faster than generic CPython. If it is so fast perhaps we should ask why it’s not the default Python interpreter ;-)
Swift being faster than Go is quite surprising because Go is 3 years ahead of Swift in public development.
I can’t wait to see ![Chris Lattner][http://nondot.org/sabre/] open-source Swift.
Swift is so young (6 months) that it still has some nasty bugs and rough edges. I will write about it another day

Detalizm popłaca

Fri, 02 Jan 2015 00:00:00 +0000

A więc nazywasz siebie web-developerem, programistą, webmasterem a może nawet inżynierem? Zapytam więc Ciebie - jak działa web browser? Co dzieje się przy wpisaniu w przeglądarkę wp.pl i wciśnięciu entera. Proste pytanie, skomplikowane odpowiedzi.

Zazwyczaj programiści odpowiadają na dwa odrębne sposoby:

Pierwszy typ programisty “To proste. Przeglądarka sprawdza DNS, łączy się z serwerem i wyświetla stronę w formacie HTML”
Drugi typ programisty “To dość skomplikowane. HTML czy XHTML? Nawiązanie samego połączenia sieciowego to sporo pracy pod maską, nie mówiąc o headerach. O jakim żądaniu mówimy GET, PUT, czy POST? Czy mam wyjaśnić też cache?. Do tego wchodzi jeszcze parser… jaka przeglądarka? Czy strona była wcześniej odwiedzona? Czy mam też omówić jak Javascript działa? Layout engine? Multiprocesowość? Czy w grę wchodzi Websockets ?

Często spotykam się z argumentem, że po to technologia została stworzona, abyśmy nie musieli zastanawiać się jak działa stack pod spodem. Po 11 latach zabawy z kodem uważam jednak, że:

Kariera każdego dobrego programisty przypomina proces obierania cebuli

Pokrótce opiszę dlaczego.

Wszyscy zaczynamy podobnie. Zaczyna się od fascynacji programowaniem, obiektowością, bazami danych, grafiką. Wszystko wydaje się takie przemyślane i spójne - myślisz sobie, “ten świat naprawdę jest zbudowany na ramionach olbrzymów”. Następnie fascynacja ładnie opisanymi i zrozumiałymi design patternami. Chcąc jednak robić więcej zaczynasz poznawać detale działania frameworków i zasad ich funkcjonowania. Niepokoi Cię myśl, że pod maską frameworków ktoś ukrywa coś ciekawego, użyte biblioteki ograniczają i krępują ruchy.

Virtual Machine / Language internals

Zaczynasz dostrzegać minusy języka programowania w którym byłeś zakochany. Niekompletne lambdy w Pythonie? Threading którego nikt nie chce używać, bo jest broken by design? Widzisz czego Twój język X mógłby nauczyć się od języka Y. Jednak trwasz dalej i wykorzystujesz idiomy dla przejrzystości i kruczki implementacji żeby wycisnąć jeszcze trochę wydajności po czym …

DB

Twoja firma, twój serwis szybko zyskuje popularność i przestaje się skalować. Co robisz? Zaczynasz pozbywać się abstrakcji. “Obierasz cebulę” z pierwszej napotkanej warstwy - zapewne warstwy ORM i skupiasz się na mikrooptymalizacjach zapytań SQL. Debugujesz outputy z ORM-ów, zaczynasz tune’ować bazę (in memory) a później wręcz optymalizować JOIN-y w zmaterializowanych widokach. W dyskusjach z kolegami przestajesz śmiać się(śmiałeś sie?) z gołych zapytań. Zmieniasz ENGINE’y bazy danych i dowiadujesz się dlaczego InnoDB nie potrafi defragmentować się online. Romans z PostresSQL też masz już zaliczony, miało być pięknie, a jednak limity zapisów dały w kość

Middleware and Cache

Porzucasz mikrooptymalizacje zapytań i obierasz kolejną warstwę - tym razem bierzesz pod lupę template’y i rezygnujesz z dynamicznych zapytań dla każdego request’a . Poznajesz Varnisha, ale żeby to osiągnąć obierasz kolejną warstwę - Middleware i zgłębiasz tajniki HTTP. Czujesz wiatr w skrzydłach - wreszcie wiesz co dzieje się pod spodem. Czujesz, że przez wiele lat miałeś na oczach opaskę i nie widziałeś “całej maszynerii” pod spodem.

NoSQL

Po przestudiowaniu co tak naprawdę robi JOIN w różnych engine’ach bazy danych rozumiesz, dlaczego wybierasz Cassandrę, Redisa i flat files w niektórych przypadkach.

TCP/IP

Czujesz ukontentowanie aż do pewnego popołudnia traffic rośnie i osiągasz limity otwartych połączeń w kernelu linuksa, lub limit otwartych plików. Zdejmujesz kolejną warstwę abstrakcji i odkrywasz jak działa stos tcp/ip i interrupts coalescencing WOW.

Frontend

Przeglądarka - końcówka naszej podróży. Wydaje się prosta - ma po prostu ma wyrenderować formularz, stronę.

Czy wiesz, że przeglądarka Chrome składa się z 3 razy większej ilości linii kodu niż np Linux? (16 mln LOC, gdy jądro Linuxa zawiera tylko blisko 5 mln) To naprawdę skomplikowane bestie. Jeśli kiedyś zastanawiałeś się, dlaczego scrollowanie przeglądarki “klatkuje” tzn że jeszcze nie wiesz jak tak naprawdę działa scrolling i jak zarządzać FPS w przeglądarce. Zaciekawiony oglądasz przeglądarkę internetową w ukrytej kamerze i cieszysz się że od dziś rozumiesz już trochę więcej.

PROFIT

Najlepsi inżynierowie decydująć się na funkcje biblioteczne dokładnie wiedzą co “kupują”. Obecnie próg wejścia w pisanie aplikacji jest coraz niższy. Frameworków JavaScriptowych jest obecnie tak dużo(blisko 1500), że powstała nawet strona do ich skatalogowania pod kątem filozoficznym - liberal vs opiniated. Jeśli jako frontendowiec nadal jesteś uzależniony od JQuery to może powinieneś przeczytać to - just in case.

Dlatego moja obserwacja jest następująca: aby rozwijać się jako programista należy drążyć detale do gołej blachy, nie dać się rozleniwiać istniejącym warstwom abstrakcji. Czy TCP/IP jest idealny? Dlaczego Google pracuje zatem na QUIC? Jeśli będziesz w stanie utrzymać swoją dziecięcą ciekawość świata, a przez to zaczniesz rozumieć abstrakcje których używać - to w efekcie - usmażysz Smaczny Software