building our repository (builder)

[ASX]

I would imagine that the processor won't be top-notch but I think the priorities are: 32 GB RAM and SSD storage (probably mixed with 2 SSDs for building purposes) and 2 HDDs for storage.

The disks proposal looks great, but I'm a little worried about the CPU: while RAM and disks are necessary to achieve a consistent number of parallel builders, CPU remain a requirement to build big packages and packages that do not make use of ccache (mainly those using languages other than C/C++).

My worry increase while thinking that the big packages (say 20%) will use the 80% of the build time.
Sorry about the premature rant, but I though it is better we all understand the complete picture.

I guess. In that case the machine could perhaps also assume a second role: Hosting backups of the Canadian server. This shouldn't put too much extra load on the server, especially if we throttle traffic. IMO this is something that we should consider.

Depending on available storage, probably yes.

I'd like Synth to grow a feature like this: Recognize the "biggies" (webkit*, chromium, ...) from a list and use a different profile for them.

I can tell you it is not going to happen, I have clearly understood that the author decided to use swap to deal with the extra requirements from big packages, and admittely it make sense considering that is nearly impossibile to predict what packages will require more resources.

Just to let you understand, each "job" (C/C++ compiler) usually consume 100/200 MB of RAM, but I have seen enough often some use of 500 MB, and a peak (ok it was a single case) of 2 GB.

So, forget about.

This is because I've often all packages built except for Chromium - and that one took around 9 hours to build with just two jobs. At the same time 5 other builders have already reached the "shutdown" state and if Chromium is build with e.g. 12 jobs instead, it's done much, much more quickly. Any thoughts on this? Perhaps this could be a future feature request for Synth.

Unless the improvement while using 12 jobs is coming from ccache ... i.e. first time you build using 2 jobs and second time using 12 jobs ...

1) make sure you ccache is working properly, upon rebuild chromium will build in much less time.
2) next time you will update, try to use something like:

Code: Select all

synth just-build www/chromium  # build chromium and dependency first using 2 builder and 8 jobs
synth prepare-system                  # build the remaining pkgs
pkg upgrade -r Synth                   # upgrade

I wrote 2 builders and 8 jobs, depending on your ram and swap, you could use more builders, use a much jobs as your CPU allow (upon rebuild using ccache the process will be I/O bound, do not fear to overload the CPU).

[kraileth]

Ok, here's the current situation:

My colleague took a look around what hardware we can spare for a rather cheap price. Most of that stuff was... rather dated and most of the boards could not even take 32GB RAM! The worst ones even were DDR2 only. However my boss talked to him afterwards and allowed using a way better machine.

So it looks like we could get this server:

CPU: Intel Xeon E3-1230 v2 @3.30 GHz (4 Cores / 8 Threads)
RAM: 32 GB DDR3 ECC @ 1333 MHz
Disks: 2 x 1TB SATA HDD

Two other goodies:
It has IPMI (which means that I can access the server at any time even when SSH is no longer possible and fix things!)
It is actually a 2U case that can house up to 8 drives! Without additional controllers the board is limited to 6 connectors, though.

And we could get that for 70€ (100 CAD) monthly! Normally my company charges 49€ (70 CAD) just for housing a 1U or 99€ (140 CAD) for housing a 2U server provided by a customer. I haven't checked exactly but a root server (normally with backup, though, that we'll do without) like that would probably be priced around 300€ (430 CAD) per month. So I'd say that this is probably close to the best that we can get - a pretty decent 2U machine for less than the usual housing price for 2U. Yes, the Canadian hoster is even cheaper but I have no idea how they do that (if it's really dedicated hardware and not actually an equivalent in form of a VM - because in that regard it's very common to aim for cheap prices due to massive over-commitment). But here we'd have IPMI and we can extend the server if we want. Since I have both network and physical access, I can replace parts if needed.

The only thing that we should probably buy separately is an SSD (do we actually need it mirrored and buy two? Isn't it primarily used for a fast caching drive that stores replaceable files and redundancy is optional?). I can add that or any more drives any weekday.

So what do you think?

[ASX]

I think it is a good startig point.

(well, I hope we can add a small SSD in the future, 128 GB would be enough).

For me is OK, the final word to Eric.

May I prepare the disk layout plan ?

By the way, thanks kraileth! And thank to your boss too!

[ASX]

The only thing that we should probably buy separately is an SSD (do we actually need it mirrored and buy two? Isn't it primarily used for a fast caching drive that stores replaceable files and redundancy is optional?). I can add that or any more drives any weekday.

No, we don't need and don't want mirrored SSD. (in fact a mirrored disk would be slower), generally speaking we don't need mirroring anywere (package building related), all data can be considered temporary and can easily be rebuilt as required. (ok it will cost some time to reinstall/rebuild, but that's all).

[ericbsd]

What is the ram limit of that machine?

[ericbsd]

I much that machine could be expandable?

I mean is 32GB of RAM the machine limit of the machine? Because at that price I can get 6 core which is not much important, but also 64GB of RAM and if we can expend that machine RAM and Disk I have no problem with this machine.

kraileth I know that you will have access to the machine and it is a plus, my only concern is to spend more money then other option out there it is why if the RAM can be expended this something I have no problem to go with. If we can buy more RAM and shove it in I have no problem with that.

It is mostly my big concern.

[kraileth]

Eric: Looked up the spec. It seems like the board is indeed limited to 32GB RAM in total. However we could add up to 4 SSDs in addition to the two HDDs. And having nice fast swap space spread across multiple SSDs leaves us enough room for improvement IMO. I'd also say that if we should really need to upgrade to 64 GB RAM in the future (e.g. if we decide to support another platform such as ARM and build packages for that, too), by that time chances are that we can get a newer board or something. Since this would be a ton of work I wouldn't expect it to happen anytime soon.

Do you expect that we might need >32 GB this year or do you just like to be on the safe side in case of future need in general?

For me the biggest plus is that I would have IPMI access. So even if we'd lock ourselves out while tuning the firewall (who hasn't done that before??) or if we crash the server or accidentally "halt -p" it or something stupid - I can allways remotely fix the firewall, look into what happened, power it back on and things like that even from home. In general we'll probably work on the server mostly on the weekend and being able to fix things is quite nice. Also IPMI means that I can remotely change BIOS settings, see the boot process, use single user mode without SSH, etc.

Another thing to consider is that a lot of hosting companies won't let you install the OS yourself. You get a server with their image of FreeBSD put on it; maybe you can decide if you want ZFS or not but that's it. We'd have all the flexibility and can completely choose how to layout the system. We could even install HardenedBSD or something if we want.

But of course the decision is yours to make.

[ASX]

I'm splitted:
I understand Eric point (price/performance ratio), and I also understand kraileth point (phyical access and other pluses).

From a price/performance point of view, the Canadian hoster is the winner:
https://www.soyoustart.com/ca/en/offers/e5-sat-2.xml

From a management/maintenance point of view kraileth proposal looks better.

I'd also say that if we should really need to upgrade to 64 GB RAM in the future (e.g. if we decide to support another platform such as ARM and build packages for that, too),

RAM would be ideally needed now, of course we can live with 32 GB. The support for additional arch is irrelevant because only one instance of synth can run at any time, thus we can only build one arch at a time.
That means we will build serially amd64 first and i386 immediately after that.
Synth doesn't run in jails (althogh some hack exists to make it run inside a jail, this is discouraged from synth' author).

More RAM means we can run more builders in parallel (hopefully to achieve better throughput), while doing so we will need also more CPU and more disk I/O.

You can also look at the whole, by evaluating the the picture from "bottlenecks" point of view:
Having a large RAM and not enough CPU to make use of it is meaningless. (cpu bottleneck)
Having a large RAM and/or a powerful CPU and not enough disk performance (disks I/O bottleneck)
Having a powerful CPU and performant disk but not enough RAM to make use of it. (RAM constraints).

I'm sorry, I have no enough data to state exactly which one will be our bottleneck, the tests I was able to perform so far were been severely constrained by lack of swap, the contextual running of our webserver and the underlying ZFS based setup.

What I can add is that with 32 Gb we can probably build the 80% of the packages without swpaping, at the same time the remaining 20% of packages will be built using the 80% of the build time and heavily swapping too:
just building the two openoffice packages in parallel will use 30 GB of RAM, add a few more very light builders and your system will be however swapping for the entire duration of the openoffice build.
That's why 64 GB would make a difference now, not in the future.

That said, as I wrote above, I'm spitted because I can recognize the added value of kraileth proposal.

[ericbsd]

I would add this it is not that am against going to this server but I am split like ASX. Not only the solution that ASX just linked is a 6 core.

The only part that I ca say no is having you access to that server.

This is why I am split.

[kraileth]

Well, if RAM is the most important issue here, that's something where I unfortunately cannot do anything. In that case we should probably consider the external option if there's really a lot of benefit to it (is there if we're building weekly?).

(Still I'm not sure how a company like soyoustart can actually continue to exist. The small fee that they charge will probably pay the power costs and air-conditioning. No idea how it will ever suffice to pay off the server that the company has to buy (even if they are getting massive discounts). Even if they hope that the server price can be paid back over the course of several years, I still remain sceptical. Unless the servers assemble themselves and fly into the racks they would also need employees. And then no harddrive must ever die (which is purely utopic, they die rapidly in large datacenters)... But that's just my perspective. Maybe they found the philosopher's stone or something.)