it took more time that what I initially though to describe and write down how and why a specific layout is desirable.
To late right now, I will write that tomorrow.
Builder machine disks layout
Builder machine disks layout
for now this is only a placeholder:
it took more time that what I initially though to describe and write down how and why a specific layout is desirable.
To late right now, I will write that tomorrow.
it took more time that what I initially though to describe and write down how and why a specific layout is desirable.
To late right now, I will write that tomorrow.
Re: Builder machine disks layout
How synth build packages: a little analisys of the various phases, from a disk usage point of view.
For performance reasons builds are performed on tmpfs disks, which is in fact RAM+SWAP
in this context, assuming some high load, when you read read or write to tmpfs it imply read or write to swap.
0 - evaluating source tree, options and repository state:
this is mostly a one time evaluation, performed at the beginning of the build and we can consder that irrelevant from a performance poing of view
1 - fetch phase: internet -> distfiles
source files are fetched from their respective master repositories and saved in "distfiles", considering the relatively low speed of the download the location of the distfiles is also irrelevant.
2 - building the local base and work area
This is a phase where many packages are read from the "repository" and written to "tmpfs/swap", this is probably one of the most intesive disk I/O phases.
Therefore it make lot of sense to have "swap" on a disk different from "repository"
3 - build phase:
in this phase we can have two different scenario:
a) building a package for the first time will require mostly CPU work (write cache)
b) rebuilding the package will make use of ccache (read from ccache)
in both case the result is written to tmpfs/swap and in both case there is some ccache I/O.
What make sense here is to have ccache on a different disk than the one were swap is located.
4 - package
in this phase the resulting files are packaged, read from tmpfs+swap and written to the repository,
same requirement as before: "swap" on a disk different from "repository".
To recap, 3 areas are subject of relatively high disk I/O activity:
- repository itself (read often for build dependencies, and run-dependencies)
- ccache
- swap
the other areas are irrelevant or however have a negligible impact from a performance point of view.
A next post will follow.
For performance reasons builds are performed on tmpfs disks, which is in fact RAM+SWAP
in this context, assuming some high load, when you read read or write to tmpfs it imply read or write to swap.
0 - evaluating source tree, options and repository state:
this is mostly a one time evaluation, performed at the beginning of the build and we can consder that irrelevant from a performance poing of view
1 - fetch phase: internet -> distfiles
source files are fetched from their respective master repositories and saved in "distfiles", considering the relatively low speed of the download the location of the distfiles is also irrelevant.
2 - building the local base and work area
This is a phase where many packages are read from the "repository" and written to "tmpfs/swap", this is probably one of the most intesive disk I/O phases.
Therefore it make lot of sense to have "swap" on a disk different from "repository"
3 - build phase:
in this phase we can have two different scenario:
a) building a package for the first time will require mostly CPU work (write cache)
b) rebuilding the package will make use of ccache (read from ccache)
in both case the result is written to tmpfs/swap and in both case there is some ccache I/O.
What make sense here is to have ccache on a different disk than the one were swap is located.
4 - package
in this phase the resulting files are packaged, read from tmpfs+swap and written to the repository,
same requirement as before: "swap" on a disk different from "repository".
To recap, 3 areas are subject of relatively high disk I/O activity:
- repository itself (read often for build dependencies, and run-dependencies)
- ccache
- swap
the other areas are irrelevant or however have a negligible impact from a performance point of view.
A next post will follow.
Re: Builder machine disks layout
SWAP
Using a large swap amount will require some kernel parameter tuning: I noticed that while trying to add 64 GB swap previosuly and got some "complaint" in dmesg, at the time I stepped back and limited the swap partition to 32 GB, this is just a memo that it need to be done.
If we put the swap on a dedicated SSD, I would consider to partition the disk, to leave some free storage so that the disk firmware could properly perform wear leveling. However such requirement is debatable and it is not consistent across the wide variiety of disk available.
CCACHE
ccache store the result of C/CC++ files compilations, every time ccache detect that a source file has already been compiled and is unchanged and the compiler who generated to output is unchanged, will reuse the generated files.
Those files are many (I collected approximately 3.000.000 files) after building the whole repository, and by default .are stored on /root/.ccache.
The average size of the files is around 24~28 Kb.
I had previously changed the ccache default "dir_levels", from 2 to 3: it means that the 3 millions files will be stored in 16^3 directories instead of the default 16^2.
In practice, using 3 levels each dir will contain ~700 files, while using 2 levels each dir would contain ~12.000 files.
That probably going to make some performance difference (in favor of three levels) but I've not measured that.
(the problem is to make a measure I would have need to have a dedicated machine and also the time to make two complete repo builds).
I already wrote about the fact that ccache was performing poorly on ZFS and after that what I did was building a UFS based ccache oon top of a ZFS volume.
While at that, I tried it with and without lz4 compression, with 4kb and 16kb sized blocks (on zvol):
compression was negligible when using 4kb blocks, on average 1% or 2%
instead it was very consistent when using a 16Kb block, saving approx. the 50%.
ccache also offer built in gzip compression but I have not tested it.
REPOSITORY
This is the third thing I would put on SSD, if available.
The requirement are around 80 GB for aech repository, also considering it will contain packages I would be inclined to use some large block size.
A next post will follow.
Using a large swap amount will require some kernel parameter tuning: I noticed that while trying to add 64 GB swap previosuly and got some "complaint" in dmesg, at the time I stepped back and limited the swap partition to 32 GB, this is just a memo that it need to be done.
If we put the swap on a dedicated SSD, I would consider to partition the disk, to leave some free storage so that the disk firmware could properly perform wear leveling. However such requirement is debatable and it is not consistent across the wide variiety of disk available.
CCACHE
ccache store the result of C/CC++ files compilations, every time ccache detect that a source file has already been compiled and is unchanged and the compiler who generated to output is unchanged, will reuse the generated files.
Those files are many (I collected approximately 3.000.000 files) after building the whole repository, and by default .are stored on /root/.ccache.
The average size of the files is around 24~28 Kb.
I had previously changed the ccache default "dir_levels", from 2 to 3: it means that the 3 millions files will be stored in 16^3 directories instead of the default 16^2.
In practice, using 3 levels each dir will contain ~700 files, while using 2 levels each dir would contain ~12.000 files.
That probably going to make some performance difference (in favor of three levels) but I've not measured that.
(the problem is to make a measure I would have need to have a dedicated machine and also the time to make two complete repo builds).
I already wrote about the fact that ccache was performing poorly on ZFS and after that what I did was building a UFS based ccache oon top of a ZFS volume.
While at that, I tried it with and without lz4 compression, with 4kb and 16kb sized blocks (on zvol):
compression was negligible when using 4kb blocks, on average 1% or 2%
instead it was very consistent when using a 16Kb block, saving approx. the 50%.
ccache also offer built in gzip compression but I have not tested it.
REPOSITORY
This is the third thing I would put on SSD, if available.
The requirement are around 80 GB for aech repository, also considering it will contain packages I would be inclined to use some large block size.
A next post will follow.
Re: Builder machine disks layout
Now the disks layout, assuming we have two rotational diskss and optionally SSDs
disk 0:
- freebsd-boot 512 Kb
- freebsd-swap 128 GB
- freebsd-ufs 128 GB FreeBSD 11-RELEASE
- freebsd-ufs 20 GB logs
- freebsd-ufs 200 GB distfiles
- free ~500 GB
disk 1:
- freebsd-boot 512 Kb (not used, just in case)
- freebsd-ufs 128 GB amd64 ccache
- freebsd-ufs 128 GB i386 ccache
- freebsd-ufs 200 GB repository
- freebsd-ufs 20 GB root-amd64
- freebsd-ufs 20 GB root-i386
- freebsd-ufs 30 GB port source-tree
- freebsd-ufs 5 GB port options
- free ~450 GB
disk 2 optional
if we can have an additional SSD I would surely put there the swap.
disk 3 optional
that's were I would put the repository and ccache
of course, that's just my idea of that, please comment about,
disk 0:
- freebsd-boot 512 Kb
- freebsd-swap 128 GB
- freebsd-ufs 128 GB FreeBSD 11-RELEASE
- freebsd-ufs 20 GB logs
- freebsd-ufs 200 GB distfiles
- free ~500 GB
disk 1:
- freebsd-boot 512 Kb (not used, just in case)
- freebsd-ufs 128 GB amd64 ccache
- freebsd-ufs 128 GB i386 ccache
- freebsd-ufs 200 GB repository
- freebsd-ufs 20 GB root-amd64
- freebsd-ufs 20 GB root-i386
- freebsd-ufs 30 GB port source-tree
- freebsd-ufs 5 GB port options
- free ~450 GB
disk 2 optional
if we can have an additional SSD I would surely put there the swap.
disk 3 optional
that's were I would put the repository and ccache
of course, that's just my idea of that, please comment about,
Re: Builder machine disks layout
Sounds like a plan! Just one suggestion: Since that leaves plenty of room on both HDDs, what about making one partition on both for a simple ZFS mirror and put the OS on that? The benefit of having the OS mirrored is that if the system disk dies we don't have to reinstall FreeBSD but can simply boot of the remaining HDD, put in a fresh drive and resilver. The other data is irrelevant and does not have to be mirrored as it is rebuildable (or re-fetchable in case of the distfiles). The OS is nothing that we couldn't replace but in case of a failure it would spare us (or rather: meASX wrote:of course, that's just my idea of that, please comment about,
) a reinstall.Re: Builder machine disks layout
This was also my initial idea, but later I though about to save a much RAM as possible and use maximize the RAM usage for synth. As you know ZFS could be resource hungry especially of RAM.kraileth wrote:Since that leaves plenty of room on both HDDs, what about making one partition on both for a simple ZFS mirror and put the OS on that? The benefit of having the OS mirrored is that if the system disk dies we don't have to reinstall FreeBSD but can simply boot of the remaining HDD
However yes, it could be done, especially if we wil have SSDs.
Re: Builder machine disks layout
Ah, ok, makes sense. ZFS is definitely a bit on the heavy side when it comes to RAM... What about using a gmirror then? It doesn't have to be ZFS but I'd say that a mirror would still provide value.ASX wrote:This was also my initial idea, but later I though about to save a much RAM as possible and use maximize the RAM usage for synth. As you know ZFS could be resource hungry especially of RAM.kraileth wrote:Since that leaves plenty of room on both HDDs, what about making one partition on both for a simple ZFS mirror and put the OS on that? The benefit of having the OS mirrored is that if the system disk dies we don't have to reinstall FreeBSD but can simply boot of the remaining HDD
However yes, it could be done, especially if we wil have SSDs.
Re: Builder machine disks layout
For the SSD's could we buy those as soon we have about 600 CAD of budget cor that. Here 2 SSD 480GB would be around 300 CAD.
I have no idea what's is the prices of SSD's out of America.
I have no idea what's is the prices of SSD's out of America.
Re: Builder machine disks layout
I'm OK with a gmirror setup too.
The thing I care about is that, provided we have only rotational disks, the initial part of the disks should be reserved for swap / ccache / repository, those are the three areas for which we need the best performance.
Of course, if we move them onto SSDs, you are free to use the disks as you prefer.
swap required: between 64 and 128 GB
repositoories: approx 200 GB (inclusive of both amd64 and i386)
ccache: approx 128 + 128 GB (100 + 100 would be OK too).
so that we can compare oranges with oranges and not lemons with apples.
The thing I care about is that, provided we have only rotational disks, the initial part of the disks should be reserved for swap / ccache / repository, those are the three areas for which we need the best performance.
Of course, if we move them onto SSDs, you are free to use the disks as you prefer.
swap required: between 64 and 128 GB
repositoories: approx 200 GB (inclusive of both amd64 and i386)
ccache: approx 128 + 128 GB (100 + 100 would be OK too).
brand / model or link ?Here 2 SSD 480GB would be around 300 CAD.
so that we can compare oranges with oranges and not lemons with apples.
Re: Builder machine disks layout
A 480 GB SSD is currently priced around 140 EUR (200 CAD) according to Amazon.ericbsd wrote:For the SSD's could we buy those as soon we have about 600 CAD of budget cor that. Here 2 SSD 480GB would be around 300 CAD.
I have no idea what's is the prices of SSD's out of America.