Responding to Richard

[IAHM-COL]

Hi Richard, I created this external topic, becuase you post very important and interesting questions I'd like to 1) answer to you, and maybe 2) It all goes smooth and we can reach a point of agreement, we can even push a door down and enter a new realm forward.

In fact, since terraGIT inception, I visualized such realm I am talking about as a possibility, but I understood the need of Core developers support. Well! you are one!
And if I can convince you to pull with me, we can actually make things happen (or so I hope)

So, lets' give a good attempt to get in the same board, so we can row in synchrony.

I'm probably going to go off topic now and bear with me if I'm wrong because scenery isn't my area of expertise.

Bearing with you. I'll reply per parts as it is necesary to establish a better clarity. Things you mentioned definitely are close to correct but still missing the mark, on ways that are critical.

[IAHM-COL]

TerraSync is not a scenery storage method it is a distribution method.

1. Terrasync Is indeed a distribution method for scenery,
but

2. Terrasync is also the scenery storage method for FlightGear. and
3. Terrasync is the development repository for scenery for FlightGear.

Technically speaking, terrasync is a monolythic 85GB subversion repository hosted in SourceForge. You can actually access all of its content via subversion (or git svn) if you wish. Althought this is NOT encouraged by the core.
Although on its origins it was implemented as "on the fly" scenery fetching; no other "official" or "core" scenery storage actually exists in a manner that is accesible. You can find terrain tiles (no Objects) here: http://ns334561.ip-5-196-65.eu/~fgscene ... 2.0.1.html and objects in the scenery database, as a global or shared snapshots.

So, where is Scenery development in Flightgear take place?

Answer: really no-where in particular.

How? Well,
in spite that terrasync IS a development svn repository, no-one actually uses it as such. No one except an absolutely limited number of developers can svn up to it. (in reality only 2 that I know of; Martin Spott, who left the team, and Torsten Dreyer)

So: who pushes content to terrasync? Normally -- no one
Only the database. But at that we'll come back later.
(the database is like Martin Spott svning up, basically)

So when you said:
"TerraSync is not a scenery storage method it is a distribution method."

You critically miss the mark. Terrasync is actually a development repository for scenery development that is used as a distribution mechanism.

Now, you can get the data from it via diferent methods

• On the fly: As everyone knows
• via terrafs (great method by the way), but requires Linux https://github.com/FGMEMBERS-SCENERY/terrafs
• via terramaster : https://github.com/FGMEMBERS-SCENERY/terramaster
• or using Subversion (svn clone or git svn clone)

[IAHM-COL]

It would be very nice if terraGit supported the Terrasync protocol (via HTTP); as that would avoid having to download the whole lot and the changes could be picked up as you're flying along.

It is very nice indeed.
Not only that it could.

The actuality is that it can be done. And it is utterly simple

If you realize, during FG2016.2 code development, terrasync fetching within Flightgear changed its method of fetching from svn to curl (using https)

Most importantly, the server from which FlightGear fetches terrasync is not hardcoded. It can be 1) configured or 2) implemented at launching with the

Code: Select all

--terrasync-server=https://url/to/terrasync/or/alternative


Literally, this means, we could have all terraGIT content stored in an HTTP apache server, and then distribute the url.
Pilots using the new "terraGIT-server" url there will actually fetch on the fly exactly the content on terraGIT.
Not a mistery there.

For flightgear it would be just as if it would be fetching you good old terrasync.

Why had not we done this?
Vincent explained already.

So far, the major problem we would face to make this a reality, is to have such apache server, with enough capability (at least 500GB storage) and an amazing bandwith to handle usage.
Certainly, Pink Floyd's money would start to play. And that's been a limitation.

So, you miss the mark by saying terraGIT can't be distributed just equally as terrasync. It is technically just there. We just don't own the required infrastructure.

[IAHM-COL]

The standard FlightGear scenery database is stored in a PostGIS database; which was used to generate the data that TerraSync serves and that was the start point for terraGit.

Not really.
That is a misconception of what the scenery database really holds.

The scenery database holds no data EXCEPT 3d models for shared and static objects and its locations post-computed in the form of STG lines


So, let's split what is flightgear scenery for a moment

1. Scenery is the Terrain mesh tiles and its corresponding landclasses contents (what you find in /Terrain)
This is not handled or contained in the database
2. Scenery is the configuration files for airports, including: Jetways, thresholds, groundnets, twr, other xmls, which you find in Airports/I/C/A/ICAO.file.xml
This is not handled or contained in the database
3. Artificial Intelligence traffic files: This is not handled or contained in the database
4. 3d model objects populating the world (your objects may look like little houses, terminal buildings, static airplanes, etc)
The 3d models are in format .ac, and respective png textures. It can contain one xml for additional effects or configurations. Objects can be shared or Static
ONLY this is handled and contained in the database (directories /Models[for shared] and /Objects)
5. OSM2City objects: This is not handled in the database.

The database therefore CANNOT be used to generate terrasync. In reality is only used to manage/administer, or access 3d objects. Terrasync content exists before the database, and then it can be populated using it. Which in reality is a bad design approach (more about that later).

The more important consideration onto this particular misconception is that, if we limit scenery development to what the database can do then we can only put or remove buildings. Everything else becomes inaccesible. This including, new airport layouts (terrain), or configuration files (AI, jetways, groundnets are all very critical), and OSM2City objects become also un-accesible to such model of Scenery development.


https://scenery.flightgear.org/

You can here contribute to FlightGear scenery by adding objects in your favorite place

This is a delicate point.
Why?
because the scenery development has stagnated FOR YEARS due simply to such simplistic model that prevented any development NOT RELATED to models to be placed.

During those years, our upstream community, xplane has concomitantly been very active on update the airport layouts worldwide, and has even created major advances on the airport specifications (apt810 -> 850 -> apt1000) during these exact 3 years.
As a consequence, while the XPLANE community IS enjoying a much more developed world, we are stagnated in Scenery ws2.0 for years.

(and to top the problem, many airports are underground or underwater due to a glitch occuring during ws2.0 generation. Glitch that IS virtually unfixable while we hold the paradigm that scenery can not be fixed by any means except the database)

[IAHM-COL]

I believe changes submitted to the scenery website (except airports) will be picked up and added to TerraSync - I'm not sure if these will get pulled through into terraGit - Israel can advise us about this.

Read my long post above.
Only models can be changed/added via the scenery website. So in reality, many things that need development are locked under that paradigm.

TerraGIT updated from new objects from terrasync a week or so ago. I posted that somewhere here.
Updating the new models from terrasync into terraGIT is very easy to do. Simple add the models and git commit.

Well, unfortunately there are technical caveats, but nothing major. So yeah, for the guys that push a building to terrasync via the database, yup terraGIT will get them./


But terraGIT, at the same time is a git repository and anything can be git committed, and git pushed.
* Terrain
* Config files (jetways, groundnets, etc)
* Shared Models
* Static Models

Now, you would say:
We don't need to get static models, but we could get them from terrasync.

Unfortunately again, a big misconception

Firstly,
Nothing prevents models to be added directly into terraGIT
Secondly,
OSM2City objects can so far ONLY be added to terraGIT (the database does not accept those)
[Note: Stuart with a moderator hat on just pushed core changes that include new scenery specification for OSM2City objects, but this does not mean that the scenery database is open now to accept those. So go figure why Radi never pushed HongKong to terrasync?]

Thirdly, but very importantly


Whomever designed the "rules" for objects to terrasync wasn't a visionary.
The rules: https://scenery.flightgear.org/contribute.php

Things that are not just discouraged, but outright forbidden includes, per example: 1) Texture sharing between models. and 2) joined meshes for objects separated (thing 3 or 4 buildings that are close but are independent units); the database REQUIRES these to be one object at a time.

These to conditions are simply put: WRONG.
But also a big design error. From computational point of view. And from development point of view (one spends hours making changes that are otherwise unnecessary, while at the same time breaking appart computer performance)

I tested this already heavily a couple of times. And ultimately, my development on Edinburgh is a complete proof of concept of what I am talking here about


This is how it goes:

Drawables in the scenery are better kept at a minimun. That is: lets say we have a cargo area with 10 buildings 5 hangars, 1 tower.
We can have 16 buildings to draw. Or 1 object.
The computer preffers 1 object every single time. It does it faster, and it uses less graphics card everytime it tries to re=render.

So? Lump them togehter in 1 object. It takes a bit longer to draw it, but then the fps resulting is much better. (keeping vertex number minimun is still very very important, but this does not change with lumping)

Secondly, how many textures? Textures must be loaded into the Graphics card Ram (or any other accesible ram in lower specs machines).
its better to have the lower amount of textures possible.
So lets say you have 1 big texture per building and 1 big lightmap per building to comply with the scenery database: then you get 32 rather large textures. That way you arent doing any favor to your computer, nor to you as a developer: and we are talking here about only the cargo apron.

Much better, consider those buildings per textures. Imaging you have the texture of the roofs, or the roofs of the hangar. One texture for walls another for hangar tiles. That's about 5 or 6 textures that you can share along ALL of your cargo zone (and even your airport). Use tiling to texturize, and you can get amazing resolutions without the expense of texture per pixel.

And thus?
YOU Could not possible add this better method buildings on the database because you violated rules 1 and 2 together and VicMar is going to go ape-shit on you

(really. VicMar already BANNED me from pushing object to terrasync for "continually pestering him and pushing the boundaries")

So?

I recommend all authors to ignore the scenery database rules. Make buildings and models that better render, more efficiently tiled, with shared textures accross your airport, with lumped/grouped objects to reduce drawables and get amazingly busy well constructed airports at a fraction of a computational cost,
And instead of crying because these can't be pushed to the database in this improved condition, simply, push them to terraGIT

Proof of Concept:
Edinburgh.

[IAHM-COL]

However what doesn't currently work is airport changes; and this is because this bit was never automated. There is a method for manual updates but it's quite complex to prepare the data in the right format.

Again. Nor airport changes, nor much anything you could possibly do to actually develop scenery.
Except misconceptualized objects.

Thus we created terraGIT.

[IAHM-COL]

Ideally to future proof any scenery development most things need to go into the scenery database as otherwise they'll be lost when the world is regenerated for V3

Not at all.
The scenery database and terrasync, if anything are not even past-proof.
(see how world development stagnated for 3 years under such scheme)

This is where I hope I can get you as a core developer in my own board and help me rowing in the same direction.

We need to bring news to FlightGear. (which usually isn't fast to accept news)

The news is that a change into how scenery is developed is impending.
We need a decentralized (as in git) scenery development repository with no boundaries imposed to the developers ability to alter and improve the face of scenery.

TerraGIT is a good starting point. Is a good example. A good mirror.

What we need is find the correct mechanism to incorporate it in a form that globally most users benefit.

The best way I can imagine right now is by connecting TerraGIT and terrasync in the correct manner. This manner implies

1. Scenery is developed in terraGIT
(new content is git committed, git pushed, git pull requested, evaluated, git merged)

2. terraGIT is a development repository for this content and as such can be fetched via git with submodules (as of now)

3. New Buildings as well as config files, terrain etc are pushed via github. (not the database!!)

4. Terrasync is used as an HTTP server to distribute widely via on the fly (or terramaster, or terrafs) to all users of FG
To achieve this, terrasync content is TOTALLY downstream of terraGIT. We can sync as frequently as desired, but nightly sounds like a good frequency.

terrasync therefore will be really a "distribution" mechanism of FG scenery developed in team within terraGIT.

5. The database sits downstream of all this as a catalog. Not as a blocker of new content. Not with the ability to modify and re-update content (this is the only reason why, per example terrasync as is now CANNOT hold OSM2City buildings!!~)

The database is a great resource to evaluate what objects exist. But it is a terrible gateway for content. In addition pushing objects there is painful for the soul. Imaging pushing Project 3000 or any OSM2City content. you 'd cry.
Whereas in terragit all you need to achieve this magic is git commit

The database currently has the ability to overwrite the scenery content. That direction of action is mayhem!

By sitting the database downstream (just reading the content and updating what it founds --think of it as a computer spider), then software like KL666's map could correctly catalog airports with content, and/or people could browse search that content.

If we achieve this, together, Richard, we would be looking at the future. Not at the past. Nor at blockades.

(would you be willing to help me defend such a proposal on the devel list? I hope so. But I noticed you had not even posted Edinburgh in Curtis' Forum... so I fear)

[SHM]

Beautifully said.
Lets hope they will be ready to accept it.

[Richard]

Thanks for taking the time to write up all of the detailed information and splitting the discussion out.

I've never modelled scenery, and although your explanation helps my understanding it's still a theoretical understanding not backed by any practical experience of having built a model. I've also got a very out of date and rusty knowledge of the E&S visual modelling database formats.

Approaching scenery from the point of view of a data modelling exercise it's obvious to me that a database should be used for 2,3,4 in your list above. OSM data should be handled separately as it's an external source; and the database should only contain our own creations.

/Terrain seems to me to be something that doesn't belong either in a database or in a repository - maybe the source files; but the output can just remain on disk and be merged with the rest.

In terms of the models that are in the database - I can understand why the current approach is taken it's effectively a GIS that can be converted to use as scenery. It keeps the identity of each model and ensures a granularity that allows the GIS data to be processed. A large mesh that happens to have a group of buildings loses the definition of what the mesh is and it becomes purely a graphics asset. I'd rather have the GIS. For the runtime performance shared models are preferable, as are shared textures. Importantly the GIS approach means that someone could write a program to combine the .ac models into a much larger model that is a single mesh with a single texture; or maybe even something that built an OpenFlight format database. Something that I've been looking at on and off for a while is using hardware instancing to improve rendering performance (for trees). My expertise isn't in 3d rendering but it seems like it could work and be a big improvement - but it would only work where there are identical models in the visual database that can be instanced.

I've been thinking about how git fits into this and the simple answer is that I don't know. It seems wrong (from a software engineering perspective) to use git for controlling large amounts of binary data. It seems the sort of solution that will work fine at first but over a few years the repository sizes (especially with lots of active development) will probably grow to a point where it starts to become an issue and may require a rebaseline (i.e. drop the history and start again); but again I don't know for sure how it would work. Statistics gathered from TG may prove useful to answer this question with a bit of extrapolation.

So the idea of simply replacing everything with a git repository doesn't really seem immediately to be the best option. It is only solving one problem - which is the difficulty in getting scenery submitted; but I'm not convinced that large meshes are an improvement over individual buildings. Individual buildings seem to me to have the advantage of being neutral to the underlying terrain height, or at least more so than a large mesh over the same area which will need to better fit to the terrain.

However to follow this through to the logical conclusion and taking what we know about the V3 format

- it will have levels of details
- new terrain meshes which may have different heights
- airports that are probably draped over the terrain rather than being cut into it
- landclasses implemented via a shader

I know it's hard to say, but how do you forsee merging this into all of the work that will be in terragit by the time that the V3 is available? That to me is the question that needs to be answered before deciding what the future of scenery could be.

[IAHM-COL]

Thanks for taking the time to write up all of the detailed information and splitting the discussion out.

I've never modelled scenery, and although your explanation helps my understanding it's still a theoretical understanding not backed by any practical experience of having built a model. I've also got a very out of date and rusty knowledge of the E&S visual modelling database formats.

I think something that has been broken for a long while now is the ability to communicate properly ideas and projects up and down the "two" groups. Lack of proper communication has led the project to unnecessary strains.

A natural consequence of this has been the lack of understanding what each other party "needs" are, and how we can find a win-win situation, where those needs are bilaterally met.

Such efforts of communication had been sometimes been lost with mis-conceptualizations, and fears. Example: "The hostile Fork".

For the good of the community, it makes sense to brake some walls down and build some bridges, but that only can be achieved via the so-lost communication. Which implies honest explanations of one's points of view. And the use of reason for refutals. Not the force. (The concept of meritocracy, per example, is the voice of Force, not reason).

And thus, that is why I take the time to write up detailed information that illuminates my path of thought.