Repository server status?

[ASX]

To kraileth:

I have had some discussion with Eric about this builder-server:

- at this point in time the state of the things are unsatifactory.

That alone would justify us to go with the SoYouStart offer, now.

But we also considered that:

- we recognize we made a deal with you and because of that you took some action, among other ordering the new RAM, talking with your company boss, etc.
- we think you are in good faith and you genuinely want to help this project
- by going that way we would put you on some trouble (or at least in an unpleasant position) with your company and your boss.

Considering all the above, we have decided to accept your server, but for 3 (three) months only., which means from today up to 21 June 2017.

We hope that will be enough time clear up the misunderstandings, and to improve the situation.

We expect that you evaluate the whole situation from your side, that you understand our needs, and that you state clearly what you can or cannot do,

You are a member of this project, we want you in possibly, we would like to see things improve, independently from which server we will end up to use.

By june 21 we will evaluate what to do next, to remain with your/you company server or switch to something else.

Hope you will find our decision reasonable. What stated above has been jointly decided by Eric and me.

I'm going to use the buildserver as soon as, but not before, you agree with our decision.

[kraileth]

Got some more bad luck; the first conference day was really nice, but I'm unfortunately missing the second one because I'm sick (being tied to a bed however allows me writing in the forums here, though).

Mmm what is the plan after the conference about the server?

As I see 16G of Ram is really not much, and this will be solved soon, but yeah I know that shipping take times.

It's not meant to remain at 16 GB since that way it wouldn't be of much use for the purpose it was set up for. I was in an extreme hurry on Wednesday, so I didn't talk to my colleague in the datacenter longer than needed. It was basically "Could you by any means get what we currently have into the rack, give me an IP and the internal IPMI IP?" - "Sure, so with 16 GB for now?" - "Yes, that's better than nothing" - "Hm. We might at least add a small SSD in that case." - "If you got one I won't say no!" - "Ok." - "So what about the RAM?" - "I'll have to take another look to find something that's both compatible and fits the budget. If that doesn't work out we might have to replace the board so it accepts 32 GB of ECC REG." - "Alright, thanks a lot!" If the board actually needs to be replaced, I'd be looking at getting something that supports 64 GB at least.

There is a gui who pledge 100 USD on Patreon, the first goal was 100 for a server to build pkgs.

From Patreon we meet this goal.

Incredible! Didn't know that - but it's totally awesome!

Also we have to start doing dev metting on Skype or Hangout that is on of the 100$ rewards and all Patron of 100 will participate on those mething. Never thought someone would pledge 100 per month.

Absolutely! Wow, I'm really amazed right now. To be honest: I always maintained a healthy distance to services like that, but in that case I'll have to think again, I guess. A man should always live up to his word and I knew the bounties on Patreon. Now if somebody actually pays this, I'll have to take a look at Skype, soon! Also this is probably a good thing anyways. All those issues around the server showed that there were way too many misunderstandings that we have to reduce in numbers for the future.

Good lord, you are not being serious, are you?

As hard as it may sound, we are. We must be, we have collected money and donations, among other things explicitly to make things like "our own repos" happens, it is also one of the reasons why I'm pushing hard toward this goal. (I can add this is also part of the reasons why I came back).

I've calmed down since Wednesday and your point is totally making sense. However I was also totally unaware of that fact - the whole situation looked a lot different to me. It was about like this: "We'll have a buildserver but it's not decided if we will have a dedicated machine or do it on the server that we already have." That neither sounded like utter resolve to do it nor did I have a clue how time critical things were. As soon as I knew that we could get some hardware, I told you here and kept you updated through the process (what kind of hardware, etc.). I also told you that there was no 32 GB of RAM available back then and that it would have to be ordered and shipped. Even from your replies I still did not get the feeling that time was of the essence. Knowing that would have been a game-changer. Other companies definitely sell more servers and thus have the hardware ready in at lease a couple of hours. I could not promise that but I didn't knew that it mattered as much.

! I told you that I was going to go to a conference today right from the beginning. Yes, I had hoped to get the whole thing over with on Tuesday so that it wouldn't mean so much pressure. I used my vacation and the hours at night to work on the project, hoping to have Salt ready before the server. I neglected my family and in fact I'm in trouble with my wife over that. And I was supposed to be on the road about 4 hours ago. I don't want to complain, but I did stay so that we can get this done before I leave. Just wrote you a PM with how you can get access.

The whole salt related project is a your own idea, I have nothing against that, but you can't use that as an excuse.
And btw, take care of your wife and your family before of GhostBSD, no one for sure will ask you to endanger your own family relations for this free project, certainly not me and I'm sure Eric would say you the same.

I didn't mean to use it as an excuse and in fact I don't think there is any need for one. In fact it's unrelated; yes, having Salt ready was my personal goal, I missed it. Too bad for my pride but that's about it. It's just that I wanted to point out on Wednesday that I really did anything that was possible for me to drive things forward (even though Salt is not related to the buildserver other than in a "nice to have" fashion). I wouldn't have written anything like that today but on Wednesday I was really in a upset state of mind because things were going wrong and it was all things that were all completely out of control for me. I totally understand that you were disappointed with the situation, ASX, but don't think that I was even a bit happy with it (though unable to fix things - which only made matters worse). And then having to read what I already knew (that all of this was not cool), didn't really help, either. Hope you can understand that. I was really under a lot of pressure and more in need of a nice word or something to reclaim some optimistic thinking. But hey, the world didn't end that day and if a team endures clashes like that we'll be more "battle-hardened" in the future.

Overall I think you need to reconcile your great desire to improve GhostBSD (yes, we can recognize that) with what your free time allow you to do (without endanger yourself).

It's not easy for me to second that, but you're right. I'll have to reorganize things a bit. There's little gain in overdoing things; I'll need to stretch it out a bit.

You (with your company behind) offered us the server, at some price and under the condition it was used. Some things have been explicitly said: physical access as a plus, chance to expand the hardware if required and so on, other things were somewhat implied, overall you let us think that we could have had better support than the one we would have had from SoYouStart. That is why we choosed a somewhat lesser machine proposed from you/your company in place of SoyouStart machine, we traded hw specs in exchange of (timely) support. Am I wrong ?

No, you're right here. I will need to setup a dualboot with Linux on my laptop because VirtualBox failed my on Wednesday (and it still segfaults...), but then I can remotely fix things no matter on what day of the week (or weekend) and no matter what time. Generally if there are issues, it's mostly software or even OS-related. The problems we've faced so far are all hardware-related and that's were I cannot cast magic. If a drive dies, I can replace that in a timely manner. Same thing if the network card should give up (haven't experienced that a lot, though). Things like that which can get an existing server back up and running. And upgrading and extending things is also possible just as I offered. If we decide that we want an SSD (well, we have one now but that's small and thus probably old), I can generally put it in the next day or so.

Also, we understood we were getting support from your company ... or so you told us ... You didn't told us that we will have support from "only" from kraileth if/when possible.

There's two things we have to keep separate: We're a normal (though reduced rent) customer. So if the power supply unit should explode, call them and it will be replaced. This is independent of myself being at work or on vacation or whatever. They'll also reset the machine if the kernel panicked or something like that. What is "kraileth exclusive" is software / OS stuff. It's a root server and anything software related is what we have to deal with as they don't have an account on the machine like they have on managed servers. But that's pretty obvious I think. The special treatment regarding software is that I can do a lot more than that, e.g. the remote installation of the OS. I can change BIOS settings, access a panicked machine and try to debug - things like that. And, most inportant from my point of view, thanks to IPMI I can get console access even when the machine is in a weird state. At some point we'll have to setup a proper firewall, get SSHguard workings, etc. And my experience with that is that you've quite likely to lock yourself out while tuning the settings. This is likely to happen on weekends (when we're working on GhostBSD). In such cases I can disable the firewall or kick out the problematic rule or reset SSHguard. If we never need that - all the better! But it's be surprised if no such things ever happened.

[ericbsd]

There is a gui who pledge 100 USD on Patreon, the first goal was 100 for a server to build pkgs.

From Patreon we meet this goal.

Incredible! Didn't know that - but it's totally awesome!

Also we have to start doing dev metting on Skype or Hangout that is on of the 100$ rewards and all Patron of 100 will participate on those mething. Never thought someone would pledge 100 per month.

Absolutely! Wow, I'm really amazed right now. To be honest: I always maintained a healthy distance to services like that, but in that case I'll have to think again, I guess. A man should always live up to his word and I knew the bounties on Patreon. Now if somebody actually pays this, I'll have to take a look at Skype, soon! Also this is probably a good thing anyways. All those issues around the server showed that there were way too many misunderstandings that we have to reduce in numbers for the future.

I talked with the guy and he did 100$ by mistake it 1$ he wanted to put.

[ASX]

I talked with the guy and he did 100$ by mistake it 1$ he wanted to put.

Hmm ... OK, ... tell him 10$ ... and deal !
Joke aside, better to know that soon than later, and thanks anyway!

[ASX]

To kraileth:
[...]
I'm going to use the buildserver as soon as, but not before, you agree with our decision.

kraileth, the above post need a reply from you.

[kraileth]

Sorry, missed this post because it was on the second page. Just got home and since I need some time to recover from the drive before I can go to sleep, I'll try to get some work done.

To kraileth:
- at this point in time the state of the things are unsatifactory.

There's nothing to debate about. Again: I can't say that I'm happy with the situation.

That alone would justify us to go with the SoYouStart offer, now.

You can, anytime. It's just that I work for a hosting company and that I thought it would make sense to try and get something arranged. That's just me, it doesn't necessarily mean that was a great idea.

But we also considered that:

- we recognize we made a deal with you and because of that you took some action, among other ordering the new RAM, talking with your company boss, etc.
- we think you are in good faith and you genuinely want to help this project
- by going that way we would put you on some trouble (or at least in an unpleasant position) with your company and your boss.

That is true; while I have regretted making that offer to some point, I still hope that it can be beneficial in some way or another.

Considering all the above, we have decided to accept your server, but for 3 (three) months only., which means from today up to 21 June 2017.

We hope that will be enough time clear up the misunderstandings, and to improve the situation.

Thanks for giving me the time to see if this can be fixed, I appreciate that! A lot of unfortunate things came together and just by the law of chance a series of bad luck has to end somewhere... And being pressed for time doesn't make things easier. I expect to sort out things on Monday as good as possible so we can talk Monday evening or so on what to do next. If that works out well, there should be lots of time to evaluate the actual builds instead of worrying about all those hardware issues...

We expect that you evaluate the whole situation from your side, that you understand our needs, and that you state clearly what you can or cannot do,

Sure thing - that's only reasonable.

You are a member of this project, we want you in possibly, we would like to see things improve, independently from which server we will end up to use.

I'm not a friend of making threats and I certainly won't tie my membership of the GhostBSD project to the server question (this in not Linux and we don't want to be childish here, right? ). What you proposed sounds completely reasonable to me: Making a decision "sine ira et studio" is always a sound plan. In the end it's about what fit's the project's needs. I cannot only accept that, I favor that.

By june 21 we will evaluate what to do next, to remain with your/you company server or switch to something else.

Ok, here's what I propose: On Monday I'll try to find out what to do about the RAM or if another board would be needed (or I'll just have a look if I can buy 4x8 GB ECC non-REG myself and donate those). For the SSD I suggest that you, ASX, do some tests first. We have one now and even though it might not be on par with what we could buy today, it should give us an idea of what we can expect from SSD usage. Is there anything else that I forgot that needs to be taken care of ASAP?

Hope you will find our decision reasonable. What stated above has been jointly decided by Eric and me.

I'm going to use the buildserver as soon as, but not before, you agree with our decision.

Go ahead, set the remaining things up! I fully agree to that decision (we had a pretty upset atmosphere on Wednesday and I feared for a much, much less constructive outcome). Let's make the best of what we have and see if it can be improved and if it makes sense to improve it - even if we don't get another 100$ per month!

Oh, and just one request, ASX: Could you document the setup? For one thing I still want to have proper (internal) documentation of the project infrastructure at some point and I also hope to get some ideas for automation purposes.

[ASX]

Go ahead, set the remaining things up! I fully agree to that decision

Great!

[kraileth]

ASX: Looks like you did some more experiments (judging from the other thread). Just tell me when there's a building break so I can do a reboot and take a look at the BIOS settings.

It also looks like you're trying to find out what the best way to use the SSD is. What is your feeling about that right now? Does it seem to help the build process? And what would you suggest could/should be the next step regarding hardware?

Another question: You're currently targeting "synth everything", right? Or did you prepare a list with some of the discussed packages excluded?

[ASX]

ASX: Looks like you did some more experiments (judging from the other thread). Just tell me when there's a building break so I can do a reboot and take a look at the BIOS settings.

Yes, I did. Tried to use the SSD in the following positions, one at a times :
- ccache
- swap
- repository
And it turned out the best use we can do of the SSD is to store the ccache.

It also looks like you're trying to find out what the best way to use the SSD is. What is your feeling about that right now? Does it seem to help the build process? And what would you suggest could/should be the next step regarding hardware?

Yes, I tried different settings, specifically builders x jobs, tried to saturate the CPU and at the same time to not overload the machine from a disk I/O point of view. I cannot say that I succeed, the variability of package sizes make this rather difficult, using a large number of builder *usually* allow for a better average distribution of the load, but also expose the system to very high load factor (30.0), high swap usage (25 GB ...). ...

I'm under the impression that FreeBSD doesn't respond very well under low RAM condition, and may be it need some tuning.

As for your question about hw. improvement: nothing, the point is that we need to grow up all resources (CPU. RAM, disk I/O) and I'm not sure that spending money on the current machine is the right thing to do.

Another question: You're currently targeting "synth everything", right? Or did you prepare a list with some of the discussed packages excluded?

I'm using synth everything, but removed a couple of packages from ports: mainly gcc-arm-embedded and related dependencies, because the build failed (wasting hours of processing).

We will need to setup our ports tree (freebsd ports + our additions/removals/changes).

[ASX]

As for your question about hw. improvement: nothing, the point is that we need to grow up all resources (CPU. RAM, disk I/O) and I'm not sure that spending money on the current machine is the right thing to do.

This need to be clarified: I can imagine that by adding some RAM, or an additional SSD, I can improve the performance,but the fact is that building packages at 520 pkgs/hour or at 550, or 480 ... isn't going to make a significative difference.