builder setup

[ASX]

This is a tentative to document the builder setup.
(I will later edit this first post to have all the info in the same place)

updated the server to:
root@buildserver:~ # freebsd-version -uk
11.0-RELEASE-p8
11.0-RELEASE-p8

- added kern.hz=100 in loader.conf

- enabled ssh root access: changed /etc/ssh/ssd_conf: PermitRootLogin = yes
and copied our pub-keys under root user .ssh dir

- installed pkgs: smartmontools, synth, ccache, tmux, bash, rsync

- added wrapper commands ccache32 and ccache64 in /root/bin
they are used to query and set option respectively for /ccache32 (i386) and /ccache64 (amd64), because of the non-standard location of ccache. Usage is the same as original ccache command.

- note also, that because of that I have added two files for synth:
/usr/local/etc/synth/GhostBSD-amd64-environment
/usr/local/etc/synth/GhostBSD-i386-environment
they are needed to make synth use the proper ccache location, based in synth profile name.


- loaded a couple of modules in rc.conf (coretemp, linux, linux64)
(linux support is needed to build linux pkgs)

- switched repo to latest (need synth > 1.66 because of bug while building lang/sbcl)

addressing this, because we are using a very large swap:

Code: Select all

# dmesg | grep warning
warning: total configured swap (33554432 pages) exceeds maximum recommended amount (32550960 pages).
warning: increase kern.maxswzone or reduce amount of swap.

I had added a sysctl in loader.conf, but that was not enough ... apparently FreeBSD can't manage a swap larger than approx 120 GB, therefore I resized the swap partition and now there is a 8 GB hole.

Code: Select all

root@buildserver:~ # more /boot/loader.conf 
kern.maxswzone=1200000k
root@buildserver:~ # sysctl kern.maxswzone
kern.maxswzone: 1228800000

root@buildserver:~ # gpart show -p
=>        40  1953525088    ada1  GPT  (932G)
          40        1024  ada1p1  freebsd-boot  (512K)
        1064   251658240  ada1p2  freebsd-swap  (120G)
   251659304    16777216          - free -  (8.0G)   <--------------------- hole
   268436520   268435456  ada1p3  freebsd-ufs  (128G)
   536871976  1416653152          - free -  (676G)

disks sanity check: (more or less all 3 disks are in good health):

Code: Select all

root@buildserver:~ # smartctl -l selftest /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p8 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      5848         -
# 2  Extended offline    Completed without error       00%      5684         -
# 3  Extended offline    Completed without error       00%      5516         -
# 4  Extended offline    Completed without error       00%      5348         -
# 5  Extended offline    Completed without error       00%      5180         -

root@buildserver:~ # smartctl -l selftest /dev/ada2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p8 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6016         -
# 2  Extended offline    Completed without error       00%      5853         -
# 3  Extended offline    Completed without error       00%      5685         -
# 4  Extended offline    Completed without error       00%      5517         -
# 5  Extended offline    Completed without error       00%      5349         -


root@buildserver:~ # smartctl -l selftest /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-RELEASE-p8 amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF READ SMART DATA SECTION ===
SMART Self-test Log not supported  <------------------------<<---------------<<--------------<<-----------

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours_and_Msec 0x0032   000   000   000    Old_age   Always       -       922780h+26m+14.020s
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       19
170 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000    Old_age   Always       -       19
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x000f   099   099   050    Pre-fail  Always       -       2646884
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       19
225 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       2910140
226 Workld_Media_Wear_Indic 0x0032   100   100   000    Old_age   Always       -       65535
227 Workld_Host_Reads_Perc  0x0032   100   100   000    Old_age   Always       -       50
228 Workload_Minutes        0x0032   100   100   000    Old_age   Always       -       65535
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   099   099   000    Old_age   Always       -       0
241 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       2910140
242 Host_Reads_32MiB        0x0032   100   100   000    Old_age   Always       -       2970749
249 NAND_Writes_1GiB        0x0013   100   100   000    Pre-fail  Always       -       23869

disks transfer rate: (not the best benchmark, I know):

Code: Select all

root@buildserver:~ # diskinfo -t /dev/ada0
/dev/ada0 (SSD)
Transfer rates:
	outside:       102400 kbytes in   0.291247 sec =   351592 kbytes/sec
	middle:        102400 kbytes in   0.288381 sec =   355086 kbytes/sec
	inside:        102400 kbytes in   0.274665 sec =   372818 kbytes/sec

root@buildserver:~ # diskinfo -t /dev/ada1
/dev/ada1 (HGST 7200 rpm)
Transfer rates:
	outside:       102400 kbytes in   0.790371 sec =   129559 kbytes/sec
	middle:        102400 kbytes in   0.874438 sec =   117104 kbytes/sec
	inside:        102400 kbytes in   1.515178 sec =    67583 kbytes/sec

root@buildserver:~ # diskinfo -t /dev/ada2
/dev/ada2 (HGST 7200 rpm)

Transfer rates:
	outside:       102400 kbytes in   0.770728 sec =   132861 kbytes/sec
	middle:        102400 kbytes in   0.824977 sec =   124125 kbytes/sec
	inside:        102400 kbytes in   1.373707 sec =    74543 kbytes/sec

the TZ appear incorrect, going to fix that:

Code: Select all

root@buildserver:~ # date
Sat Mar 25 14:28:55 UTC 2017

The current disk layout:

Code: Select all

root@buildserver:~/.ssh # cat /etc/fstab 
# Device	Mountpoint	FStype	Options		Dump	Pass#
/dev/ada1p2	none		swap	sw		0	0
/dev/ada1p3	/		ufs	rw		1	1
# distfiles: newfs -b 16k -f 64k, softupdates
/dev/ada1p4	/distfiles	ufs	rw,noatime	0	0

# ccache for amd64: newfs -b 4k -f 16k, softupdates
/dev/ada2p2	/ccache64	ufs	rw,noatime	0	0

# ccache for i386: newfs -b 4k -f 16k, softupdates
/dev/ada2p3	/ccache32	ufs	rw,noatime	0	0

#repositories: newfs -b 16k -f 128k, softupdates
/dev/ada2p4	/repositories	ufs	rw,noatime	0	0

#root-amd64, rooti386, ports, options: newfs -b 4k -f 32k, softupdates
/dev/ada2p5	/build		ufs	rw,noatime	0	0

[ASX]

Hmm ... need help to setup the TZ correctly, I don't know if the machine BIOS clock is set to UTC time or localtime.

Also, take in mind I have transfers in progress, please do NOT reboot the machine.

[kraileth]

Hmm ... need help to setup the TZ correctly, I don't know if the machine BIOS clock is set to UTC time or localtime.

Also, take in mind I have transfers in progress, please do NOT reboot the machine.

Most likely local time GMT+1 - that should be the standard that our servers are setup as far as I know. I generally prefer UTC, though. What do you think?

Just tell me when the machine can be rebooted so I can enter the BIOS and have a look and adjust it if we can agree on whatever setting.

[ASX]

Most likely local time GMT+1 - that should be the standard that our servers are setup as far as I know. I generally prefer UTC, though. What do you think?

Nothing, fact is it say UTC but time is set to UTC+1, so can't be right.

Just tell me when the machine can be rebooted so I can enter the BIOS and have a look and adjust it if we can agree on whatever setting.

Thanks, I will let you know.
I'm currently transferring distfiles (100 GB), amd64 ccache and i386 ccache, unlikely they finish before midnight, however it is not a very urgent thing, tomorrow will be OK, or even later. I will post here and/or will PM you.

[ASX]

Hmm ... nothing to worry much, but may be the machine is in need of a fan-cleanup
(observed under heavy load)

Code: Select all

root@buildserver:/build # sysctl -a | grep temper
hw.acpi.thermal.tz1.temperature: 29.9C
hw.acpi.thermal.tz0.temperature: 27.9C
dev.cpu.7.temperature: 73.0C
dev.cpu.6.temperature: 73.0C
dev.cpu.5.temperature: 73.0C
dev.cpu.4.temperature: 73.0C
dev.cpu.3.temperature: 70.0C
dev.cpu.2.temperature: 70.0C
dev.cpu.1.temperature: 73.0C
dev.cpu.0.temperature: 73.0C

[ASX]

A quick update: run a few tests, mainly building amd64 pkgs with several different configs and look like there is some noticeable improvement in using a dedicated UFS based machine.

So far we were able to reach and overcome the same rate packages build rate as on the Canadian server, without using the SSD at all.

~~~

I have since configured the ccache for i386 on the SSD, and started a complete i386 build just now, we will see in 12 hours how much gain will give us the use of the SSD.

[ASX]

Just tell me when the machine can be rebooted so I can enter the BIOS and have a look and adjust it if we can agree on whatever setting.

kraileth, for now please the build(s) going on.
There is no way to reliably measure the performance, other than a complete repo build, let it going on so that we can have some real result in terms of pkgs/hour.

[kraileth]

Just tell me when the machine can be rebooted so I can enter the BIOS and have a look and adjust it if we can agree on whatever setting.

kraileth, for now please the build(s) going on.
There is no way to reliably measure the performance, other than a complete repo build, let it going on so that we can have some real result in terms of pkgs/hour.

No problem, we're not in a hurry to correct this, I guess. Also you didn't tell me if you also prefer setting the time to UTC or not. I'm currently trying to rebuild everything on my laptop and hopefully fix VirtualBox. Then I can fix things tomorrow evening or so. Just tell me when the build run is complete. I'm very much looking forward to compare the outcome to what we had so far!

BTW. what do you think about sharing some statistics in the end? I'd like to see some data made public as we're probably the biggest Synth users (on FreeBSD) since all the other projects seem to prefer Poudriere.

[ASX]

performance look like improved so far (500 pkgs/hour) but that is only a preliminary estimate, and considering that it will take no less than 2 + 2 days, and good part of the improvent come from having the ccache on the SSD.
(for i386 only right now, there is no room to have both, most likely that will change and I will put the amd64 ccache there).

About the clock: yes the hardware clock should be set to UTC, (that is always the case for Unix systems, and certainly for FreeBSD), after setting the TZ correctly (CEST / Europe-Berlin) timezone) all should be fine.

(you can safely use the tzsetup utility).

About the build statistics, I see no problem in making them publicily available.
And I'm not sure we are the biggest synth project-users, even less sure the vast majority use poudriere, true that poudriere "was" the most used, but that is mainly because poudriere is older than synth.

[ASX]

I decided to stop and restart the builder at this stage:

http://i.imgur.com/fUoENMD.png

Although apparently we have plenty of free swap, the machine degraded a lot, you can see below an excert of iostat and vmstat output, and you can clearly see the cpu is idle most of the time (last column at right).

The machine show a bad responsiveness, typical about a machine running out os swap, even if it is not the case. Puzzling.

Therefore I'm going to reduce the number of builders and restart.


[code^Croot@buildserver:/build # iostat 1 100
tty ada0 ada1 ada2 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 2446 19.80 213 4.11 17.98 276 4.85 37.16 102 3.69 59 0 16 0 24
0 234 0.00 0 0.00 18.24 1220 21.73 0.00 0 0.00 0 0 1 0 99
0 74 0.00 0 0.00 14.48 1038 14.68 128.00 8 0.94 0 0 3 0 97
0 1504 16.00 4 0.07 4.36 1218 5.18 112.87 49 5.39 9 0 8 0 82
0 78 17.45 11 0.18 31.38 709 21.71 100.14 55 5.35 35 0 17 0 49
0 81 0.00 0 0.00 29.25 725 20.70 0.00 0 0.00 0 0 1 0 99
0 77 0.00 0 0.00 20.70 893 18.06 0.00 0 0.00 0 0 1 0 99
0 77 0.00 0 0.00 30.03 788 23.12 0.00 0 0.00 0 0 2 0 98
0 79 0.00 0 0.00 31.46 593 18.21 0.00 0 0.00 0 0 1 0 99
^C 3 241 0.00 0 0.00 15.34 598 8.96 0.00 0 0.00 0 0 1 0 99

root@buildserver:/build #
root@buildserver:/build # vmstat 1 100
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id
0 19 111 4.6G 67M 28422 0 119 128 46823 35283 0 0 767 34675 29839 59 17 24
0 19 111 4.6G 67M 2 0 13 861 17 4123 0 845 908 369 5667 0 2 98
0 19 111 4.6G 68M 0 0 16 522 239 3541 0 494 556 397 3649 0 1 99
0 19 111 4.6G 68M 0 0 18 737 362 2911 0 726 778 368 4966 0 2 98
0 19 111 4.6G 69M 0 0 123 1297 409 1992 0 1365 1431 366 8882 0 2 98
0 19 111 4.6G 71M 0 0 42 1092 433 3540 0 1094 1160 358 7250 0 2 98
0 6 131 4.6G 84M 0 0 1422 302 6306 3452415 0 1655 1790 388 10956 0 9 91
19 0 107 4.5G 115M 41925 0 529 102 77405 46340 0 590 719 256662 4674 41 40 20
0 18 105 4.4G 31M 13631 0 107 719 9518 3166 10 793 887 134623 5440 27 23 50
0 18 105 4.4G 50M 0 0 17 732 0 4375 0 709 763 364 4822 0 2 98
0 18 105 4.4G 62M 0 0 17 699 402 5987 0 689 732 361 4590 0 1 99
0 18 105 4.4G 66M 0 0 16 824 868 3466 0 805 859 365 5309 0 1 99
0 18 105 4.4G 67M 0 0 18 827 528 3416 0 786 881 367 5416 0 1 99
^C
][/code]