building our source tree

[NevilleGoddard]

Note: SPS (small p***s syndrome)

[ASX]

Builder running from approx 24 hours, a little more than 10.000 packages already built.
Going to reboot, want to see if rate increase ... the idle time look too high.

[ASX]

Heavy building, notice the load factor:



a side note: erlang-runtime16 built in 10 minutes (not visible)
erlang-runtime17 built in 18 minutes
erlang-runtime19 built in 32 minutes

[ASX]

building the latest 7000 packages, I think it will maintain the impulse speed for the whole task.

[kraileth]

Hey, ASX, just wanted to write that you're doing an absolutely fantastic job in experimentally figuring out how to tune Synth! Just pressing that "thanks" button probably shows some appreciation, but it cannot convey just how much I really appreciate what you're doing. 11.0 is taking painfully long but it's really good to know that at least it does so for a very good reason. Doing things right in the first place is always much, much better than get something "kind of working" out quickly. So let me state it again: Terrific job with the work on the repo!!

BTW: I got a couple of hours (wasting most of the time fighting nginx due to a silly misunderstanding...) to setup a pkg repo at my day job last week. Of course I'm also using Synth and the proposed shell script ("repoctl.sh") + cron jobs. Still missing: A repoctl function to update the port options in the git repo and one to switch to a new quaterly branch. However it's of course an extremely simple case (a couple of hundred packages in total, builds once a week, tracking quaterly) compared to what you're working on. But it works pretty well so far.

[ASX]

Hey, ASX, just wanted to write that you're doing an absolutely fantastic job in experimentally figuring out how to tune Synth! Just pressing that "thanks" button probably shows some appreciation, but it cannot convey just how much I really appreciate what you're doing. 11.0 is taking painfully long but it's really good to know that at least it does so for a very good reason. Doing things right in the first place is always much, much better than get something "kind of working" out quickly. So let me state it again: Terrific job with the work on the repo!!

Thanks, it just happens I have some more time, I'm not doing anything particularly special.

BTW: I got a couple of hours (wasting most of the time fighting nginx due to a silly misunderstanding...) to setup a pkg repo at my day job last week. Of course I'm also using Synth and the proposed shell script ("repoctl.sh") + cron jobs. Still missing: A repoctl function to update the port options in the git repo and one to switch to a new quaterly branch. However it's of course an extremely simple case (a couple of hundred packages in total, builds once a week, tracking quaterly) compared to what you're working on. But it works pretty well so far.

About switching to quaterly repo, I strongly suggest to use svn or svnlite to get the source and then use the command svn switch to get one or the other branch.

example:

Code: Select all

svnlite checkout http://svn.freebsd.org/ports/head /build/ports
cd /build/ports
svn switch ^/branches/2017Q1

Additionally I grabbed this snipped of code from a mail-list, that explaine how to track the latest quarterly branch: (it worked for me when I tried to use "quarterly")

Code: Select all

> Would it be possible to "simply" create a symlink at
> https://svn.freebsd.org/ports/branches/quarterly that always points to
> whatever the current quarterly branch is? That way, everyone who wants
> to use quarterly packages/ports wouldn't have to worry about
> constantly switching the ports branch on every system they manage. A
> side benefit would be that packages and ports can share the
> terminology and meaning for their quarterly branch. Bonus points:
> change pkg and ports to have a CURRENT symlink pointed to HEAD and a
> STABLE symlink pointed to the latest quarterly branch so the names are
> more in sync with the base system's branch names.

Currently, all our tools working automatically with the quarterly
branches do this before any run:

latest_branch=$(svn ls https://svn.freebsd.org/ports/branches/|sed -ne '/^2.*Q./s|/$||p'|tail -1)

All you'd need to do before running svn update would be to run:

svn switch ^/branches/$(svn ls https://svn.freebsd.org/ports/branches/|sed -ne '/^2.*Q./s|/$||p'|tail -1)

About ports options: using synth there could be two methods to manage options:
put the options in <profile>.make.conf
or
put the options in individual files, (by default in /var/db/ports).

I'm planning to use the latter method, because I think it will be easier to push options from multiple devs (presumably each of us will push options related to one specific ports at a time), and additionally synth can recognize the outdated option in case of a port change, but can't do that when using <profile>.make.conf.

Options will be stored on our github repo: https://github.com/GhostBSD/ports-options

[ASX]

the build completed, I was waiting for the last packages to make a final screenshot, but got distracted and thebuild finished:

Code: Select all

The task is complete.  Final tally: 
Initial queue size: 16663              
    packages built: 16222              
           ignored: 324                      
           skipped: 76                        
            failed: 41                           
                                                      
Duration: 1 day and 21:36:57       

The build was performed in three tranche, and the global final rate is: 386 pkg/hour.

Yes, there is a significative increase by using a RAID0 setup, but overall the build rate is CPU bound.

[ASX]

The ports-init script is ready:
- download freebsd-ports (from our github)
- download gbsd-ports
- download ports-options
- merge the two tree
- generate a list of ports to build, excluding unwanted packages.
- TODO: manage own ports that substitute existing freebsd-ports (i.e. gksu from kraileth).

The amd64 repo has been completed, including our ports.
- TODO: either setup a local webserver or transfer the repo to the webserver (need to talk with ericbsd).

[NevilleGoddard]

Thanks, ASX!

Can I now replace my current usr/ports with these ports?
If so, how can I merge the port trees? I suppose you have to use subversion to merge the ports.

Some help would be nice if they are ready for use.

[ASX]

Thanks, ASX!

Can I now replace my current usr/ports with these ports?
If so, how can I merge the port trees? I suppose you have to use subversion to merge the ports.

Some help would be nice if they are ready for use.

Yes and no. At the moment the two trees are merged only locally on the builder, the idea was to make it publicily available along with the already built repositories. thus in sync, thus from the same server that will serve packages.

However we could consider to make the merged tree available from github ... we will have to decide about that.