Good Bye Terrasync?!

[IAHM-COL]

Hi! By the end of this year (2015), our current hosting of
- the scenemodels database
- the scenemodels website
- the terrasync repository
will no longer be available for use by the FlightGear project.

As a result, we need a new home for those services.
The technical requirements are roughly
- a postgresql database (with postgis extension) of approx. 40GB and growing (scenemodels db)
- apache webserver, php, access to the database and the usual toolkit for hosting the scenemodels website
- a svn server with 140GB+ for the terrasync scenery
- enough bandwith to serve the data
- One ore more reverse proxy serving terrasync requests (not much storage required, just bandwidth)
Is anybody out there able to provide a drop-in replacement?
Any serious idea or offer to help is welcome.

Torsten

[MIG29pilot]

Good-bye current server at any rate

[IAHM-COL]

Dear Torsten

I am a bit on shock with the news of terrasync reaching end of life cycle

Thanks for letting us know thou.


I wanted to communicate to you that a few months ago I have been suggesting the possibility of changing the way that scenery is fetched by FG. Specifically, I see a need of making some changes in the Terrain fetching to allow a modular approach.

The idea takes several fronts, which can be summarized as

1) Scenery becomes modular
2) change SVN to a git method
3) host the modules as 10x10 degrees quads on gitlab or github
4) This also allows making the scenery distributed (as opposed to centralized) making it viable to people to submit patches with newly installed buildings, or any other configurable xml (think of groundnets, jetways, others) via pull requests.
5) if a scenery construction server is established with standardized landclasses and compiled altitudes are used (gdalchop and ogrdecode already pre-performed), then, even we could speed installation of new layouts by local builds of small scenery regions. (as oppose to need for a long wait for new airport layouts and layout updates)

I would love that this idea can be given some serious consideration.

Best,
Israel Hernandez

(send privately because I can't post on the devel.list. Also, attempted to send at the devel list and posted to http://www.thejabberwocky.net forum)

[IAHM-COL]

Good-bye current server at any rate

Say what?

[MIG29pilot]

Good-bye current server at any rate

Say what?

Very funny
But that's what he said--the server is going kaput due to circumstances beyond their control and a new one is in order.

[IAHM-COL]

no. Mig29.
It is not funny,.
It is a serious issue that needs serious treatment. Serious solutions. and serious proposals. Not weird one liners.

The alternative Torsten is proposing deals with finding someone else capable of hosting such server and its massive resource requirements.
My alternative, which I hope reaches fertile ears, due to 1) lack of workmanship, 2) need of modularize the huge database, 3) the fact that the requirements get more and more complex, and 4) a real technical solution to that I am proposing.

We will see.
But a resolution will be found, so your posture is just a caricature, again when a real problem is presented, and real solutions need to be addressed.

[legoboyvdlp]

What the f*** are these people thinking?

[IAHM-COL]

O-o
Heated language!

Who are "these people" lego?

[legoboyvdlp]

I can't see how Israels suggestion helps.

1) Scenery becomes modular

Our scenery is alread modular

2) change SVN to a git method

SVN is just the distribution method, no history accumulates on the client because terrasync uses a striped down svn client.
Does anybody seriously considers keeping more than 50.000 revisions of scenery locally?
Also, you are no longer backward compatible.

3) host the modules as 10x10 degrees quads on gitlab or github

I consider this an abuse of gitXXX. At least github asks to keep repositories under 1GB, creating hundrets or repositories for a single project as a data store does not make it better just because it is not explicitely forbidden.

4) This also allows making the scenery distributed (as opposed to centralized) making it viable to people to submit patches with newly installed buildings, or any other configurable xml (think of groundnets, jetways, others) via pull requests.

We have a way to use distributed scenery by providing reverse proxies. People can submit updates to scenery models to the scenemodel db using the web interface. Those updates get reviewed by a human and applied. Using git patches for the workflow might be possible but does not help in our situation.

5) if a scenery construction server is established with standardized landclasses and compiled altitudes are used (gdalchop and ogrdecode already pre-performed), then, even we could speed installation of new layouts by local builds of small scenery regions. (as oppose to need for a long wait for new airport layouts and layout updates)

As far as my understanding of the scenery building process goes, this is not as easy as it looks. Landcover triangles tend to cross 10x10deg areas unless you are on a small island. Those triangles need to be sliced along the boundary, the vertices marking the edge have to be on the exact same position after slicing.
While you can easily generate scenery for a single tile, it might look wrong at the edges.
Sure, a rolling-release of scenery is desirable. Unfortunately the tool chain is not stable enough to provide such a service.

Torsten

Oh, forgive me.
I now see it is apparently due to Google code shutting down... not because some of those core developers decided to go lunatic.

[IAHM-COL]

Hi Torsten.

I can't see how Israels suggestion helps.

Thanks for your rebutal on http://forum.flightgear.org/viewtopic.p ... 48#p263671

Hopefully that is a path for me to explain my points of view. And hopefully you can consider the situation with a bit openmind.

Our scenery is alread modular

Well. Yes and No. The scenery itself is not modular. You distribute parts of it and compile it in a non modular set.
This would requires lots of confusing paragraphs to explain. So I'll go with 1 image == 1000 words thing here

This is how FG scenery looks like, and the underlying structure coded to be understood by the core



Now. When I mean modular, I mean that inside the Flightgear scenery directory, the content could be not the scenery directly, but a collection of Modules. Each directory inside could be a complete scenery package for a much much more small scenery package (a tile of 10x10 degrees, or a custom scenery, per example) Each of this directories are seen below with the letter M indicating they are actually scenery modules. Very similar as how the Aircraft directory actually contains subdirectories, each of them being one aircraft.



So, although you say FG Scenery is modular, it is not as modular as it could be convenient in order to split it effectively in smaller more manageable chunks (with ease).

Does anybody seriously considers keeping more than 50.000 revisions of scenery locally

well. First of all, that is something to be decided individually. It is not to the core developers to decide for all users how much of the total revisions we can be willing or capable to have. But more importantly, 1) a new refactoring of scenery can start again from "initial commit" (revision 1 if you which). Past is past now. 2) secondly, if every Module as shown above where separated, each of them will have a completely different "history". Frankly, I doubt everyone of them will accumulate 50000 revisions. that will mean our world got REALLY REALLY complete. and thus at the end a reason to celebrate.
3) Git has options to have a shallow clone, which allow reducing the depth cloned at a given time to manage this situation. and 4) More importantly, if we were to use github, an option to fetch the git sceneries via SVN exist.

Also, you are no longer backward compatible.

Yes and no.

In fact, each individual module is (would be) backward compatible. We do it all the time when we create custom sceneries and point flightgear there as --fg-scenery.

I consider this an abuse of gitXXX. At least github asks to keep repositories under 1GB

Indeed I found this to be concerning before. Firstly, Gitlab had grown its individual repository size allowance to 10 G. And Github allows 1G.
As far as I am informed no single 10x10 degree tile in our whole world scenery approaches the 1G limit. Not even remotely.

A few months back I actually addressed github with the exact concern of having lets say 500 additional repos that exceeded 100 GB of content. As you are aware I have hosted the FG aircraft. a total of 700 repos counting for about 25 GB of data, and no concern by Github have been raised, but I worried about scenery being too large of a package. The answer by github was



Which indicated to me that hosting such scenery, if no single repo exceeds 1GB was not a violation of their TOS, and they would be calm about it.

Keep in mind that in terms of hosting, it is better to have 600 repos of half a gig each than to have 1 repo of 200 G total. Modularity pays off a great deal with computing.

We have a way to use distributed scenery by providing reverse proxies. People can submit updates to scenery models to the scenemodel db using the web interface. Those updates get reviewed by a human and applied. Using git patches for the workflow might be possible but does not help in our situation.

I am aware of it. I used it. And it has been O.K.
But I am certain that the possibility of submitting pull requests with scenery improvements will be an step forward from this situation. It gives lots of much flexibility. It also allows for human revisions (git pull requests are not forced writes!).
Furthermore, it will allow us to submit pull request for changes like new groundnets, new tower.xml or jetways.xml configurations, in addition to just add new models. It gives us major power in customizing the scenery. It allows us also to submit scenery that is currently unsubmittable via the db, such as OSM2City outputs.

As far as my understanding of the scenery building process goes, this is not as easy as it looks

Granted it is not. But it is my opinion that enhancing the ability to patch scenery with new layouts "on the fly" is a clever way to go (objective to aim).
And I understand work is being done in this direction.
but, modularizing the scenery will be a major advantage is being able to modify smaller tile sizes as well. Maybe reconstructing a 1x1 degree as needed and submitting the commit via Pull Request, which would be modifying only a 10x10 degree tile scenery repo -- not a whole world repo.

Just food for thought.


Thanks a lot for your attention,

IH-COL