building our repository (builder)

[ASX]

This post is to keep you informed about what is going on.

Today I started to build some packages on the Canadian (OVH) server, it is worth to note that I did package bullds and updates for the last 3 months on my machine, using synth and practically without issues,

Building packages is an heavy process for itself, in terms of CPU usage, and building with synth (or poudriere) is also disk I/O intensive.

I have spent my time to understand exactly the resource usage, and how to control it, because I want to avoid to overload our server.

For a start, I did it under a newly created /build ZFS dataset, including deploying a clean FreeBSD11 base.
(exception is the ccache directory, sized to 15 GB, which is still under /root/.ccache).

More info to follow. in a couple of hours.

[ASX]

Code: Select all

devel/automake
shells/bash
sysutils/beadm
devel/ccache
mail/claws-mail
mail/claws-mail-fancy
mail/claws-mail-pdf_viewer
sysutils/conky

The above list pulled in 271 dependent packages, for a total of 278 pkgs.
I have configured synth to use 2 builders and 2 jobs per builder (thus imiting the CPU usage up to 4 out of 8 cores), and the build completed in 3:54 hours, more than half of this time was used to build llvm37, llvm39 and webkit-gtk2

Ram usage was contained within 10 GB ... approximately, swap practically was not used.

The global average on this first run is 71 pkgs per hour,

I'm going to perform a rebuild of the same packages which should be faster due to the now populated ccache, and also because distfiles are already fetched (and stored locally).

[ASX]

The second run completed in 1.14 hours, for an average of 221 pkgs per hour, that is mostly the effect of ccache. Additionally the load average was nearly half of the previous run.

That means we can safely increase the total numer of jobs (however, I will increase it gradually, need to assure that there are always enough resources available for the webserver and related).

The two results are respectively the minimum and the maximum we can achieve without changing synth config paramters, on a day by day use we can approximate the throughput around 200 pkgs per hour.

The average desktop, as provided from GhostBSD XFCE or Mate require approximatively 1.000 pkgs, which will require 5 hours of build, or approximatively 2 hours per day if we update daily. But then we will have at least to double it for i386 in addition to amd64 arch.

However I have in my mind that a "reasonable" repository will contain 10.000 packages, so that is going to be hard to achieve with the current setup.

One thing that would help a lot would be the addition of an SSD disk, Eric please consider that for a future purchase/addition if possible.

[ericbsd]

Purchase of SSD for desktop or a server with SSD?

[ASX]

Purchase of SSD for desktop or a server with SSD?

An SSD to be added onto the OVH server, that will separate the I/O from the webserver/mysql I/O, and will allow for faster building of packages.

EDIT: Anything between 128 and 512 GB will suffice.

[ericbsd]

I have look on So You Start doesn't look that is that option I will contact them to see if it is possible to get that upgrade.

[ASX]

I have look on So You Start doesn't look that is that option I will contact them to see if it is possible to get that upgrade.

OK, thanks!

[ASX]

Changed the synth configuration from 2 builder x 2 jobs per builder to 2 x 3:

Total build time: 1:11 hours, throughput 237 pkgs per hour

Load as always been at 2, with a short peak a 5 (less than 5 minutes) there is room for improving.

[ASX]

3 builders, 2 jobs x builder:
total time: 1:01 hours, rate: 271 pkgs per hour

Note: due to the relatively low number of packages, things lined up badly (from a dependencies point of view), and the last 40% of the time there were only one or two builders running, likely the other builders was waiting for some packages to be ready. I would expect better result from a larger list of packages.

[ASX]

4 builders x 2 jobs per builder:

So far this is the better confiuration:
total build time: 1:00 hours, rate: 275 pkgs per hour

Max load was 5.9, and the system responsive at any time, minimum 10 GB of inactive memory, however things could be a little heavier when 4 big packages would line up and build simultaneously.

Beside this test, I plan to keep the number of builders below 4, that allow for a good safety margin, however 4 builders remain a good option if we can tolerate some little overload (say 20 minutes ... or so).

EDIT: as a side note, the first 237 pkgs were build in 22 minutes, at a rate of approx 600 pkgs per hours, the remaining 41 pkgs took more than half of the build time.