building our source tree

[ASX]

One problem we have to solve is "how to" build our source tree:

so far we are working with:
- the freebsd port tree (from head/latest)
- a /build/listExcluded, list of excluded packages (opeoffice and a few packages that fails to build)
- one script /build/make_gbsd_list to build a FullList of packages to be built

we still need to add our own packages, those actually hosted on github.
I think that we could / should have them all under a single tree, and then we could mount_unionfs our_tree on top of freebsd port tree;
the resulting tree will be feed up to synth;

one problem to solve is about packages available in freebsd port tree, where we want to add our own patches:
in that case we should really delete that single freebsd port and use our own only.
EDIT: Really not a problem for now, I was thinking at kraileth changes to gksu, either we will make it happen upstream or we can wait, it is not a critical thing.

a second problem is that each added port, need to be listed in the port category Makefile, otherwise it will be ignored.

If you have better ideas, please shout out.

[ASX]

Our final target is to be able to provide users with a port source tree in sync with the packages as available in our "current" repository; seem simple but there are a few implications that need to be managed properly:

I will recap a few things, for better clarity, and to be sure we all underdstand and agree about the flow:

recap from a previous discussion, "quarterly vs. latest":

- build <== temporary repo, feeded from synth, will exists only for the duration of the synth build.
- latest ==> to be used from tester /devs
- current ==> the main ghostbsd repository
- previous ==> the previous repository, to be used in case of undetected issues in "current"

~~~

flow:

1a) get/sync freebsd ports tree
1b) get/sync our own ports (limitation: our ports cannot overlap the existing freebsd ports)
1c) get/sync our own ports options

2a) start synth and build the temporarry "amd64/build" repository
2b) start synth and build the temporary "i386/build" repository

3a) move/copy "amd64/build" to "amd64/latest"
3b) move/copy "i386/build" to "i386/latest"

4a) latest is now usable from developers and testers only.
4b) devs meeting: if "no go", apply changes and continue from point 1 or 2 above, depending on issues.

5a) move/copy "amd64/current" to "amd64/previous"
5b) move/copy "i386/current" to "i386/previous"
(previous is a fallback repository, supposedly will contain the previous valid build).

6a) move/copy "amd64/latest" to "amd64/current"
6b) move/copy "i386/latest" to "i386/current"
("current" is the default repository for all users)

~~~

Considering the above flow, we will have always two source tree, the first that we will named "latest" for developers use, and synced with "latest" repository, and another source tree, named "current", to be make available for general public use by those users who want to further customize their installations.

It is implied that "latest src tree" will be moved/copied to "current src tree" at the same time we will move the corresponding pkgs repositories.

Thinking at that, I would be inclined to change the directory structure so that "latest" and "current" will include
all things that needs to be moved/copied at the same time:

amd64/latest/pkg
amd64/latest/ports
amd64/latest/options

so that we can move both the pkgs repo and the src tree at the same time to:

Code: Select all

# mv amd64/latest amd64/current 

will result in:

amd64/current/pkg
amd64/current/ports
amd64/current/options

~~~

Mostly we need to focus on point (1) and decide exactly how/where/when manage the sources and the options:

Comments welcome!

[kraileth]

Here's some thoughts (deliberately before looking at what is done right now or knowing much about the current internals):

In general we need something as simple as possible but as sophisticated as needed to make the whole thing robust. A good means of a very simple approach is using sentinel files (empty file; just using the filesystem to store data in the form filenames) to indicate the current build phase. It practically works like this: [ -e /path/to/sentinels/name ] && take action.

We obviously have to come up with a means of automating the whole process. Cronjobs executing scripts come to mind. Of course we should try to avoid typical cron problems like long-running scripts that require some kind of locking / checking if already running or use pretty high intervals which then waste time as the build process might be idle without any real reason. Need to balance things out.

I like the idea to break the whole thing down into small, manageable steps. To avoid confusion, I'm going to base my thoughts on the numbering scheme from ASX:

1a) simple script (shell code should suffice):

Check if a Synth process is running. Abort if it is. Check if a sentinel file exists. Abort if it does (which means that some other phase is currently at work). "portsnap fetch extract". Check error code. Create sentinel file for 1a on success or send mail (?) and write special "error" sentinel in case of failure. The "error" sentinel will halt the whole build process until somebody can take a look at what happened and clear it by hand.

1b) script (probably python?):

Check if a Synth process is running. Abort if it is. Check if sentinel file indicating 1a is done exists. Abort if it doesn't or any other sentinel file is found (which means either 1a is still running or we're currently in another phase of the build process). Delete the sentinel and create "1b work in progress" sentinel. Checkout our ports via git. Do some checks if any port in the checkout is invalid (like name is already taken in the FreeBSD ports tree), abort and report error (mai?) + set "error" sentinel. If everything is ok, merge ports (copy them over and add subdir directive to the categorie's Makefiles, etc.). If everything's fine delete the "work in progress" sentinel and set the "1b done" one.

1c) script (python?):

Where will the port options be stored? Another git repo? If so: Check if Synth is running, abort if it is. Check sentinel file "1b done" exists and no other. Otherwise: Abort. Delete sentinel and create "1c in progress". Checkout port options via git. Merge them into the tree. Report error if any. (If we experience any "typical" problems over time this would be the right place to do additional checks before actually running Synth.) Delete sentinel and set "1c done" one.

2a) simple (shell?) script:

Check if Synth is already running. Abort if it is. Check sentinels, continue if only "1c done" exists, abort otherwise. Delete old sentinel, set "2a running". Run Synth and terminate (doesn't make sense to keep a long-running script around the whole build time! Better launch a second one on finish.)

2aa script (python?):

Check if Synth is running, abort if it is. Check sentinels, continue if only "2a running" exists (since Synth is not running anymore 2a should actually be done). Clear sentinel and set "2aa running". Check if the repo was successfully and completely built, report an error otherwise. Clear sentinel and set "2aa" done.

2b) simple (shell?) script:

Like 2a but if sentinel "2aa done" is found.

2bb) script (python?):

Like 2aa but if sentinel "2b running" is found.

This would be the actual package building part - and probably enough to discuss for now.

[ASX]

I must admit that I had head pains after reading my own post, but after reading yours, kraileth, the pain was so intense that I had to get away from my computer.

- I will not even try to automate the process until we will have a well defined and already working flow.

- The posted info was meant to share the goals, and I deliberately tried to avoid implementation details, but I can see how that is going to hijack the thread.

- about the "sentinels", we will see, they aren't as "robust" as it may seems, beside that, it seems you want to serialize things that doesn't need to be serialized: 1a) 1b) 1c) could be very well executed in parallel without no issues at all. (instead 1), 2) and 3) will need to be serialized. Note also that synth doesn't come into play until our source tree is already completed.

But I can see what you are referring to, so yes, we will use something, but we wll define that later.

port options: yes they will be stored on github.
EDIT I can see this can get confused, worth to expand a bit: options will be stored on github, exactly like our own ports are also stored on github; however, must be noted that once a build session has been completed ("latest" repo built), those options along with the source tree will be frozen at side of the repo itself, and that will be on our build-server, similarly later they will be moved to "current".

~~~

Back on topic, this thread was opened to decide which method / technique to use to generate our source tree.
So far we have lightly mentioned a couple of options:

a) forking the freebsd ports tree, and add our changes there (new ports, changed ports, ...)

b) keep maintaining our own ports, and use script to generate a source tree (this is what dragonfly is doing, and also what TrueOS is doing, although their methods are different).

I think that option a) is going to require too much work from us, and also require a "constant day by day" work, I not sure we can afford such task. IMO it would require at least one full time developer.

Option b) seem more easily manageable to me, in that we can decide arbitrarily how much frequently to update the repositories.

[ericbsd]

Not that I have lots​ time right now to give well written answer, but I say that I agree to go for b there is two projects doing like you mentioned ASX. There's​ is code all ready written for that can help us.

[kraileth]

I must admit that I had head pains after reading my own post, but after reading yours, kraileth, the pain was so intense that I had to get away from my computer.

How's that? Good thing I stopped half the way through!

- I will not even try to automate the process until we will have a well defined and already working flow.

Of course not! But thinking about it from the beginning makes it easier to avoid coming up a temporary solution which is later found to be unfit for automation. In other words: No need to write the crontab lines first. I'd of course also start by writing the first script, then the second one, etc. and trigger them by hand so they can be tested.

- The posted info was meant to share the goals, and I deliberately tried to avoid implementation details, but I can see how that is going to hijack the thread.

Didn't know that you wanted to discuss things from a high-level perspective only. In my opinion the process as described by you makes sense and I thought that this was already decided on (you haven't yet answered my question what the current status is, but now I undarstand that it's "draft phase").

- about the "sentinels", we will see, they aren't as "robust" as it may seems, beside that, it seems you want to serialize things that doesn't need to be serialized: 1a) 1b) 1c) could be very well executed in parallel without no issues at all. (instead 1), 2) and 3) will need to be serialized. Note also that synth doesn't come into play until our source tree is already completed.

Never claimed that my suggestion didn't leave room for improvement. But doing stuff in parallel makes it much more complicated to start with. Yes, I'd initially serialize the tasks then optimize the flow. Or do you really intend to start out with the complexity of having that running in parallel? (In that case you're in fact ahead of me and shouldn't complain about me thinking about how this can eventually be automized )

port options: yes they will be stored on github.

Good.

~~~

Back on topic, this thread was opened to decide which method / technique to use to generate our source tree.
So far we have lightly mentioned a couple of options:

a) forking the freebsd ports tree, and add our changes there (new ports, changed ports, ...)

b) keep maintaining our own ports, and use script to generate a source tree (this is what dragonfly is doing, and also what TrueOS is doing, although their methods are different).

I think that option a) is going to require too much work from us, and also require a "constant day by day" work, I not sure we can afford such task. IMO it would require at least one full time developer.

Option b) seem more easily manageable to me, in that we can decide arbitrarily how much frequently to update the repositories.

Definitely b)! Forking the ports tree and constantly merging things sounds like insanity for a small project.

[ASX]

OK, option b) has been decided as the way to go, fine, also good that we all agree about that.

Time to make a little pause, look at dragonflybsd implementation, at TrueOS implementation, evaluate how their solutions may or may not fit our needs, and continue the discussion later.

~~~

to kraileth:

I'm going to answer you here very shortly, to not hijack the thread again, but I would gladly discuss things with you any time you want, and preferably on some IRC channel.

Be sure, I appreciate your inputs. Yes, some things has been already decided, but that doesn't prevent any reasonable improvement or any changes that could improve the result. In fact I already changed my own decisions a few times.

However, an answer about the current status could be helpful:
things we have already decided / implemented so far are:
- we will use synth (and not poudriere) mainly because it is simpler and faster.
- a basic infrastructure is up and running, but is not definitive, we may change hardware again.
- we were also evaluating the option to use two builder machine instead of one, there are pro and cons.
- unlike the forum / website announce, you may have noticed the repos will be 4 instead of 3, the 4th (build) was added to make possible perform tests using "latest", while synth is building in "build", which is going to take days.
- source tree: option b); once this is implemented we can start to make tests.

As far as "automation", we clearly have different views about if/how to do that, do not forget that managing ports options will require some amount of manual work, it is unlikely we can automate this part of the job, and as a result it may be better to not automate at all. The big part of the job will be about generating our ports tree and related options, all the rest is quite trivial and doesn't require additional complexity at all, probably a couple of helper scripts is all we will need.

[ASX]

Adding a note here, just noticed something like a special case:

updated the ports tree:, and noticed:

Code: Select all

...
U    /usr/ports/security/p5-Crypt-LE/distinfo
A    /usr/ports/www/firefox/files/patch-bug1359051 <-----
A    /usr/ports/www/firefox/files/patch-bug1359142 <-----
U    /usr/ports/www/tengine/Makefile
...

Note that two patches has been added to firefox port, synth in this case doesn't rebuild firefox, and of course I can see why synth doesn't detect the change.

That's to say, that at least from time to time it will be safer to rebuild ALL, definitely before any important task, such as a new release.

[ASX]

replying to myself:
on a second though, there should have been a version bump on firefox port ... which clearly didn't happened.

[ASX]

There has been a few discussion on IRC about to keep a single builder machine, or two separate builders, one for i386 and one for amd64.

assuming we will use two builders, there is a small problem that need to be addressed: how to keep in sync the ports tree on both machine.

I think a possible solution is to fork the freebsd ports tree, so that it will be stored under github.com/ghostbsd/ports.

The builders machines will grab the source from our github repo, that will assure the ports tree will be identical on both builders.

The merge of the port tree with our ports will be processed locally, because the result will be slightly different for amd64 vs i386 (vs. arm64).

I guess that, by doing so, we could also "tag" some specific tree, eventually.

The forked ports tree on github can be safely updated manually (every two weeks for now).

In short, we shall have a fork of the freebsd ports tree, ONLY to assure that mutiple builders may make use of exactly the same ports tree.

Ports options too will be stored on github.