infrastructure

[ASX]

Yea scenario 3 would be easy but, the performance is really poor.

I disagree, performance are in line with dragonfly results using nvme disks
32 core -> 1200 pkgs/hour
ours: (with rotational disks)
12 core -> 386 pkgs/hour

I'm under the impression you have too much high expectations from this hardware, I also think that we are giving to much emphasis to the pkg rate because we are late, but the real thing will be that we will do at most two complete builds for each arch per month. (at least for the next one or two months).

What can really change if we can achieve 600 pkgs hour instead of 400 ? ok, we will save one day, and that's all.
Our real job, after the build, will be testing, testing, testing. And most likely we will do that on amd64 only.

TrueOS also do bi-weekly to montly updates, the important thing is that we do it right, not that we do it often, that is what our users are expecting from us.

Dragonfly users too are waiting, from more than one month right now.

We could go with your subjection for 2 servers it will show us how much SSD play a role in performance.

I'm thinking that the SSDs will not improve the situation that much, surely not up to the level you are expecting, I would be happy if we can reach the 300/350 pkgs on an E3-1245v2.

Now, and for the upcoming month only, if you agree:
- we keep the current e3-1650
- we rent the e3-1245 with 3x120 SSD.
we will revise the situation when we know the results from the SSD based machine.

That will spare us some time, allow to test an SSD builder, while continuing the job to complete GhostBSD-11

[ericbsd]

Yea scenario 3 would be easy but, the performance is really poor.

I disagree, performance are in line with dragonfly results using nvme disks
32 core -> 1200 pkgs/hour
ours: (with rotational disks)
12 core -> 386 pkgs/hour

I'm under the impression you have too much high expectations from this hardware, I also think that we are giving to much emphasis to the pkg rate because we are late, but the real thing will be that we will do at most two complete builds for each arch per month. (at least for the next one or two months).

I don't expect more of an old 6 core 12 thread with old ram and HDD.

Now, and for the upcoming month only, if you agree:
- we keep the current e3-1650
- we rent the e3-1245 with 3x120 SSD.
we will revise the situation when we know the results from the SSD based machine.

That will spare us some time, allow to test an SSD builder, while continuing the job to complete GhostBSD-11

I think it is a good idea.

[ASX]

we have rented the e3-1245, with 3x120 GB SSD, already up and running and this is the preliminary result:
(the decision about the second builder has been postponed 2 days)



I just realized that I used the same config that was using for the e5-1650 (9x5) but that is too much for this smaller machine. Going to reconfig (6x4) and continue.

[ASX]

Thinking at measuring synth performance, I realized that measuring the pkgs per hour is not a precise indicator about the performance, because of the large variability of packages (some build in seconds, some in many hours).

I though to measure the "size" of built packages, reasoning that you can build one large package in 1 hour, or build 100 small packages in the same amount of time.

Therefore I wrote a little script the periodically (20 mins) measure the size of the repository, what I have found is that this is not precise either, (my aim is to compare synth results after a short amount of time, say 2 hours), but do offer some relative indication.

A full repository, single arch, use approx. 75 GB, can can be built very approximately in 75 hours, that means roughly 1 GB per hour, on average.

Below a prelimiary result:
e5-1650, after 8 hours of build:

Code: Select all

Elapsed 466 mins         Repo size 3515 MB       impulse 215 MB/hour     Rate 450 MB/hour
Elapsed 486 mins         Repo size 3805 MB       impulse 871 MB/hour     Rate 467 MB/hour

and the result from our new e3-1245 with SSDs:

Code: Select all

Elapsed 460 mins         Repo size 3096 MB       impulse 229 MB/hour     Rate 400 MB/hour
Elapsed 480 mins         Repo size 3152 MB       impulse 165 MB/hour     Rate 393 MB/hour

I guess that we all can agree about the fact that the e3-1245 has a better performance/price ratio and that the e5-1650 doesn't perform as good as expected.

Also. we can compare the result after building the same amount of packages, say 3200 MB:

e5-1650:

Code: Select all

Elapsed 386 mins         Repo size 3289 MB       impulse 255 MB/hour     Rate 509 MB/hour

the e5-1650 reach this size after 380 mins, while the e3-1245 reach the same size after 480 mins.

* note that the e5-1650 does use ccache, and this result come form a second build, thus is taking advantage from the existing ccache.


The measure script, if you are interested to make test on your own:
(customize the repo path to fit your build environment)
EDIT: this script only work when building repos anew, doesn't work if you are updatting.

Code: Select all

# cat measure32 
#!/bin/sh
elapsed=$1
elapsed=${elapsed:-"20"}
last=`du -sk /builder/jails/gbsd-repos/build/repositories/i386`
last=`echo $last | cut -d ' ' -f 1`
prev=$last
sl=1200
while true
do
	last=`du -sk /builder/jails/gbsd-repos/build/repositories/i386`
	last=`echo $last | cut -d ' ' -f 1`
#	echo last=$last" MB"
	rate=$(($last / elapsed))
	rate=$(($rate * 3600))
	rate=$(($rate / 1024))
	impulse=$(($last - $prev))
	impulse=$(($impulse * 3))
	impulse=$(($impulse / 1024))
	prev=$last
	last=$(($last /1024))
	minutes=$(($elapsed / 60))
	printf "Elapsed %s mins\t Repo size %s MB\t impulse %s MB/hour\t Rate %s MB/hour\n" $minutes $last $impulse $rate
	sleep $sl
	elapsed=$(($elapsed + $sl))
done

[ASX]

This new machine doesn't work like expected, not at all: going to change something ...

[ASX]

Today we have to decide about what to to with the e5-1650.

This machine never performed well, it is barely above the e3-1245v2, additionally the 3 TB disks were slower than the 2TB disks we had with the e3-1245v2. (160 MB/s vs 210 MB/s, this was unexpected but that is).

The amd64 repo has been successfully transferred to the webserver, hopefully today we will complete the webserver setup, and the amd64 test repo will become a fact.

The e5-1650 is currently building i386, but will not complete before 30 hours. (we can loose this at any time).

The e3-1245v2 with the SSDs, is also building i386, but so far it has the same performance of the same machine with rotational disks, or may be a little better, not that much really.

I'm of the idea to dismiss the e5-1650.


Our build performance results so far are always bound to the CPU performance,

RAM speed: this is something I think can improve the performance, because all builds are performed entirely in RAM, and swithing from 1333 Mhz to 2333 Mhz might help.
32 GB RAM is the minimum required, 64 GB would be be better but is not strictly necessary.

SSDs helps, but with small CPUs there is no need for fast disks, I suspect SSDs can help more with more powerful CPUs.

Considering all the above, I would suggest to move to something newer, if budget allow:
https://www.ovh.com/ca/en/dedicated-servers/enterprise/
SP-32-S. SP-32, SP-64 are using CPUs on pair with the e5-1650, RAM is much faster, they can have nvme disks as an option.

This doesn't need to be done today, only the dismissal of the e5-1650 need be done today, unless for time related reason we decide to keep the e5-1650 another month; but considering that we can setup the new builder in a couple of hours, and that the prices are similar, I would eventually move to the SP32S or SP32 immediately.

~~~

FYI: the e3-1650 is in RAID0 stripe, each disk show a maximum transfer rate max/min 160/120 MB/s, I have performed a simple test:

Code: Select all

dd if=/dev/zero of=file-on-stripe-disk, bs=1M count=10000  # 10 GB

dd show a trasfer rate around 460 MB/s (correct from the app point of view, but not real).
zpool iostat 1 # show a real write speed around 220 MB/s, that's look correct considering that the stripe was made up using the second half of the disks (slower).

I consider the ZFS Stripe performance nearly equivalent to an SSD.

[ericbsd]

Considering your thoughts I think it is a good idea. This month software we made over 416$.
I am confident that we can look for good server.

I'll look in to that today.

[ASX]

Considering your thoughts I think it is a good idea. This month software we made over 416$.
I am confident that we can look for good server.

Great, I'm also confident that if we do it right we will see an increase in donations.

I'll look in to that today.

Thanks, my post is mainly about: you can leave the e5-1650 expire ... all the rest can be done later.

~~~

By the way, and for your info, ZFS send / receive works "atomically", it will show out the filesystem on the destination machine only when the transfer is complete. And that's very good.

[ericbsd]

When it comes to a 4 core do we really need 64 Gig of Ram?

[ASX]

When it comes to a 4 core do we really need 64 Gig of Ram?

No, we don't require 64 GB.

We can configure less builders, yet increase the jobs per builder. That's what allow to maximize CPU usage while keeping low the requirement about RAM. (approx rule: 5 GB per builder)

So we could use something like 9x4 or 9x5 on a 64 GB machine, and 5x4 or 5x5 or 5x6 on a 32 GB machine.

However, there is to say that it is easier to work with 64 GB than with 32, and also that ZFS make great use of the available memory. While not strictly required, 64 GB are a plus, but if you have to keep down the costs go for 32 GB.