Responding to Richard

[IAHM-COL]

Approaching scenery from the point of view of a data modelling exercise it's obvious to me that a database should be used for 2,3,4 in your list above. OSM data should be handled separately as it's an external source; and the database should only contain our own creations.

Not quite.

Fundamentally, the GIS database that comprises FG scenery is not the "scenery database" found here: https://scenery.flightgear.org/
This database found in the website is a collective of Shared and Static models, and its associated GIS information. Accepted.
But that database IS NOT what FlightGear Software reads. I think that's clear.

The GIS database that flightgear reads/uses is contained in a format, namely STG.

STG format (a GIS database on its own right) contains several fields: including object qualifier, name (absolute location in hard drive space), location (absolute lat, lon, elevation location), and turn in degrees (true heading) and pitch (rotation angle).
Every line corresponding to 1 object.

Lets' see an example:
https://raw.githubusercontent.com/FGMEMBERS-TERRAGIT/w010n50/Objects/w004n55/2892922.stg

A GIS database in that format IS understood by FG. Not another one.

Now.

a database should be used for 2,3,4 in your list above.

Currently,
2. Scenery is the configuration files for airports, (xml files) are not specified in the STG database.
3. Artificial Intelligence traffic files: are not specified in the STG database
4. 3d model objects populating the world ARE specified in the STG database. (both shared and static)
5. OSM2City objects: ARE specified in the STG database. (Stuart with the moderator hat on, actually just recently commited changes that allow the FG STG reader to interpret additional "verbs", in addition to OBJECT, OBJECT_BASE, OBJECT_SHARED, OBJECT_SIGN, and OBJECT_STATIC)

***
6. Terrain Tiles ARE specified in the STG database (OBJECT)
7. Airports ARE specified in the STG database (OBJECT_BASE)
8. Signs (Like taxisigns) ARE specified in the STG database (OBJECT_SIGN)

(Example file:
https://raw.githubusercontent.com/FGMEM ... 892922.stg )

So, in reality, beyond what may seem obvious in the first place, the actual database interpreted by FG includes lots of specification for many structures within scenery (including OSM2City[1]).
This means, OSM is not (nor intended) stored is a separated external source.

And more importantly, xml configuration files for airports are not incorporated in this QGis database.
(instead the are maintained in xml files in Airports/I/C/A/ICAO_file.xml ==> more about this later)

---Footnotes
[1]
See again
https://github.com/FGMEMBERS-TERRAGIT/w ... 892922.stg
lines 184 (currently), to the end

[IAHM-COL]

/Terrain seems to me to be something that doesn't belong either in a database or in a repository - maybe the source files; but the output can just remain on disk and be merged with the rest.

It actually belongs, thou.

So, what exactly is terrain (for FG)?


Terrain contains both the 3d mesh of the terrain (think of it as the horizon line, or the terrain ripples as it crumbles and deeps in your horizon). The earth surface if we may.
It is actually a 3d object, that you can appreciate if you lauch the ufo and look at the mesh, not the object. You'll see such 3d structure is composed of a rather large collection of triangulations.
In addition, It contains specification for the textures applied to different faces of such triangulation. These are the landclasses (that we take mostly either from Corine or V0).
In addition, It contains airports specification, and signs (taxisigns) etc if appropriate.

Now, is this information contained (stored and catalogued inside the GIS STG database of the scenery, or as you say above, this doesnt' belong to such database?

Well... lets look at a random example. Like Randomly, Edinburgh.
Lets see the corresponding directory.

https://github.com/FGMEMBERS-TERRAGIT/w ... in/w004n55

Right there.
The Objects are in the form of gunzipped btg files (a binary file, .bgt.gz).

A number corresponds to an World-wide unique identifier of the terrain tile numeric subdivision. Like, lets say, tile # 2892864.btg.gz
But for FG understand where to place such object, that object MUST exist and belong to the GIS databse. So we have a database entry # 2892864.stg

https://github.com/FGMEMBERS-TERRAGIT/w ... 892864.stg

That file, just says: Do include the base (terrain) tile number such (The number, since it correspond to a World Wide grid, implies identification of location).

So, yes. The object was indeed specified in the GIS STG database.

Then, you can find Airports. Usually named by ICAO.btg.gz
(like EGPH.btg.gz).

That object is then incorporated within the scenery by being tracked by the STG database. Where to locate it in the world grid specified in the proper tiled stg file
https://github.com/FGMEMBERS-TERRAGIT/w ... 892922.stg

It reads

Code: Select all

OBJECT_BASE 2892922.btg #Do load the tile
OBJECT EGPH.btg #Do Load the airport Object
OBJECT_SHARED Models/Airport/windsock_lit.xml -3.385236 55.943415 33.3 0.00 #Do load next and next and next objects


That way the STG database file self contains and catalogs all of the information of FG scenery

This database IS the database that Flightgear Do Understand.

[IAHM-COL]

In terms of the models that are in the database - I can understand why the current approach is taken it's effectively a GIS that can be converted to use as scenery. It keeps the identity of each model and ensures a granularity that allows the GIS data to be processed. A large mesh that happens to have a group of buildings loses the definition of what the mesh is and it becomes purely a graphics asset. I'd rather have the GIS. For the runtime performance shared models are preferable, as are shared textures. Importantly the GIS approach means that someone could write a program to combine the .ac models into a much larger model that is a single mesh with a single texture; or maybe even something that built an OpenFlight format database. Something that I've been looking at on and off for a while is using hardware instancing to improve rendering performance (for trees). My expertise isn't in 3d rendering but it seems like it could work and be a big improvement - but it would only work where there are identical models in the visual database that can be instanced.

This is all interesting, but it is deeply rooted in the misconception that this:
https://scenery.flightgear.org/
Is the actual database. Which is not.

The database you see in that url is another database that attempts at cataloging the 3d Objects (Shared and Static) of the FG content distributed via terrasync.

Now, the content of the database is used to rewrite the STG database on the /Objects directory. Overwriting any content on its way.

This is a bad approach for scenery managament.
You say, you can't see how Git can be used for this task? but do you really think that such database in the page above "is" the mechanism?

At this point, I am limited to bow to your knowledge in computational science and Engineer, but it seems from my experience within FG, that it was an sloppy approach all from the beginning.

In reality, the database in the url above does not precede scenery.
In reality, it was created "a posteriori" with one major intent in mind: Present a gateway for authors of 3d models to populate the FG world.

And the rules created, whether lumping objects, or not, and whether to tile and share texture or not. Are all, added on top of the rules of the gateway of content "method of curation". Not because they were necesarily the most computational optimum, nor because it helped the developer. Some of those rules, are mainly for cataloguing purposes (like the need of a thumbnail). and as such, those are really stubborn rules.

The rules should be in benefit of FG scenery: Better rendering, faster (thus object grouping) and more sharp and clear texture. Lower amount of hard-disk required for content storage, smaller textures (thus tiling and reusing).
Those rules are contrary to those of the database of the url. Again, the rules of the url-database main concern is to serve the database. NOT to serve the scenery.

That's why I call for a whole revaluation of the role an usage of that database. Because we need to re=structure priorities!.

[IAHM-COL]

Lumping or not lumping/ How much when/ why: "That is the question"

Those are technical issues needing major treatment, and right now is not the time. But a time must come to define these, and it is going to be an interesting conversation.

you are correct in saying that lumping too much changes the Geospecification of the database into mere graphical assets.
But not lumping enough leads to performance hit and overedundancy.

It is a perfect balance of the whole situation where we find a good gain/effort ratio.
Not lumping at all, is insanity.
OSM2City could not exist if lumping wouldn't be used. The graphic cards would be saturated of drawables. End of story.
Trees, and other identical objects, your idea of howto treat them is correct. But it is not how is done now.

In reality, object lumping IS unarguably the correct way to do it.
In fact, stuart, the moderator with a hat on, did write a code that merges stg objects in larger groups (stgmerge) and has independently *to me* tested the idea of object grouping with identical success.

Fighting object grouping is beyond any logical reason at this point. And the gateway of the scenery database, outright Forbids building and submitting objects that better fit a heavily/densely build airport with groups of objects within logic and convenience.

Again, proof of concept, Edinburgh

Example. look at the stg line 81 (in the meantime)
https://github.com/FGMEMBERS-TERRAGIT/w ... 892922.stg

Code: Select all

OBJECT_STATIC cargo.xml -3.35173457 55.95092667 33.1870 0 0

That object contains ALL of the cargo apron. The alternative to that would be using 5 Hangars, 7 or so buildings and 1 tower. To construct the same content in a computational inefective manner, and also in a way that would have forced me to do 15 times the same work of creating separating geolocating rotating texturizing and lightmaping objects

It is not appropriate for the computer
It is not appropriate for the developer.

And all and all, so finally eventually someone runs stgmerge onto it, and computes what you could have since the beginning at a cost of the effort.

So. You can argue about how that rule is convenient. I say:
No way. It just, currently, benefit the Scenery Website Database, nothing and no-oneelse

And when I talk about priorities my point is:

Such website database should serve Flightgear, its content creators for a purpose. Not the other way around (that is Flightgear and the content creators serving the database)

[IAHM-COL]

I've been thinking about how git fits into this and the simple answer is that I don't know. It seems wrong (from a software engineering perspective) to use git for controlling large amounts of binary data.

This is full of misconceptions. I'll try approaching them:

1.The Flightgear scenery is mostly, plain file text content. Not binaries!!
And as such. Git is very VERY VERY efficient managing its development.

Lets see

We got

1. Terrain and airports (in /Terrain). Binaries. Truly.
But ask yourself this question:

How frequently do this change?

Rarely never!!
Neither the terrain mesh changes, nor the landclasses specification changes at all, if ever.

Only the apt binary could And must change frequently.
So yes. Git would have to manage the binaries of these.

Lets see how big of a binary for a large airport like edinburgh
https://github.com/FGMEMBERS-TERRAGIT/w ... GPH.btg.gz
1.39 MB

A binary of 1 MB that changes once every couple of years, would not have the posibity of choking git. Simply.
and the reality is that many of those airports have not receive any change for 10+ years. and others get a major revamp and are left alone for long periods of time.
So the Terrain and airports binaries ARE OF NO CONCERN to git

Texture files for 3d objects. PNG binaries. Truly

Example:
https://github.com/FGMEMBERS-TERRAGIT/w ... hangar.png

Usually, if well made should be small binaries.
And do you know how frequently they change, causing git to track versions?

Rarely never.
At most more get added due to new objects requiring textures. But if done right, and texture sharing is used then you have a good solution to that little issue.

EXAMPLE:

OSM2city uses a rather large texture file.
In terraGIT i decided to share this texture (which in reality most OSM2City buildings do) by placing it on
/Models/OSM/tex/ directory of terraGIT, and allowing all City.ac object to use that same texture by redirection.

Reuse of the texture effectively makes git the perfect tool to manipulate and locate a binary resource that in general terms is rather unchangeable.

https://github.com/FGMEMBERS-TERRAGIT/t ... ls/OSM/tex

The GIS STG database itself. NOT A BINARY

It is a collection of STG files with objects per line.
GIT is a fantastic tool to SCM content.
(in addition to clearly being distributed, which is a great advantage over subversion in terrasync, to facilitate cooperative development)

The Configuration XML files. NOT BINARY

Again, you wouldnt find a better SCM system to collaborate in developing ALL of those XML files and track them during development with a minimal repository overload. (since they are text files)

https://github.com/FGMEMBERS-TERRAGIT/t ... r/Airports

https://github.com/FGMEMBERS-TERRAGIT/t ... orts/E/G/P

https://github.com/FGMEMBERS-TERRAGIT/t ... undnet.xml

The 3d models. In format AC. .... .... NOT A BINARY

The ac 3d specification is a plain text specification.
Therefore, actually, chaging 3d objects is smoothly tracked by git!!!
(Wonderfully as we all know)

https://github.com/FGMEMBERS-TERRAGIT/w ... 5/cargo.ac

There you are, the well exemplified cargo apron, in Edinburgh.

[IAHM-COL]

As you see above, flightgear scenery does consist mostly of plain text files, with a very limited exception in Terrain

(just what Flightgear has been unable to modify for years)

And furthermore, something that could change in Scenery v3. For the best.

[IAHM-COL]

It is only solving one problem - which is the difficulty in getting scenery submitted;

Do you think "just that one problem?

THAT IS THE PROBLEM.

That is the central issue central of the chasm.

Facilitating, and making scenery changes and contributions accesible to everyone willing to give it a go, this should be a central premise on the decision on how FlightGear should be moving forward.

that is why I created FGMEMBERS.
and why I created TerraGIT.

This facilitating seems to have fallen in odds to other priorities of the core community. But we are always in time to turn the ship in a better direction.

Reducing the difficulty of submit scenery changes (at all levels: Airport/terrain/config files/3d models) is the goal and the objective for the next step.
This is way more important that changing how the landclasses are going to be drapped over the terrain mesh (shader or landclasses) and whether the airport is cut within the terrain mesh, or whetter we drapped on top.

those are secondary points.

How easy is to improve our content assets to make them better, and how easy is to contribute to the content is, in my opinion, what should drive decisions at the design level forward. Seconded, really close, by improved performance and increased realisms on such content.

[IAHM-COL]

but I'm not convinced that large meshes are an improvement over individual buildings. Individual buildings seem to me to have the advantage of being neutral to the underlying terrain height, or at least more so than a large mesh over the same area which will need to better fit to the terrain.

I talk a bit about this above.
Read "to lump or not to lump, that is the question, post

We can discuss this further later, but it is not the central problem right now.

[IAHM-COL]

However to follow this through to the logical conclusion and taking what we know about the V3 format

- it will have levels of details
- new terrain meshes which may have different heights
- airports that are probably draped over the terrain rather than being cut into it
- landclasses implemented via a shader

I know it's hard to say, but how do you forsee merging this into all of the work that will be in terragit by the time that the V3 is available? That to me is the question that needs to be answered before deciding what the future of scenery could be.

Sounds great!
All four things above, which I've seen P. Sadro working on seems like an interesting direction, and I'm keen to see him succeed bringing us that to a reality.


Things that concern me personally are 2:

1. P. Sadro is doing a massive work almost alone. The coding of such re-structuration of our terrain, and restructuring terragear to construct Next Generation (or V3) scenery is a massive undertake. And it seems to me P. Sadro is alone against the code there. He could use more hands. Problem is, only he, that I know of, has real clarity of the little details of terragear. A terribly complicated FG scenery compiler. Even building it and satisfying all dependencies and getting a working toolchain is a real odyssey, that very few do succeed. Let alone change that software. Let alone change it in a way that is doing something dramatically different. So. Damn. the guy is really alone doing this. But! Oh! we do want him to succeed. Because it could facilitate users to bring airport and terrain changes into FG scenery. And that is a worth goal.
2. The second concern of mine is Backward/Forward compatibility. It seems the changes he is making are so massive that we'll found ourselves in a point in time on which FG 2017.4 (lets say) can only use the V3 scenery, but it is mostly unable to understand V2. Or more likely to happen, older FG versions are going to be blind into what/how to interpret the new scenery. The disconnect in scenery compatibility will bring an interesting split point for FG, and I fear it a little bit.

But your fear, I do not comprehend.

Once FG releases FG Scenery V3 (the terrain) we simply update TerraGIT Terrain branches (or possibly even better, split the content in 2 repositories one for terrain other for Objects).
TerraGIT is not married to V2 and unable to adapt and move forward with FG development.
So yeah! when V3 comes accross, terraGIT will ship it.

Simple part A.

About the Objects.
If an object only exist in terraGIT? what is the problem? I dont see it.

If FG scenery will still understand and use the 10000+ objects in ac/xml format present in the database, + the newly imported OSM files...
Why would it fail to use the 1000+ additional objects only present in terraGIT?

yeah! clearly it will not have any issue at all. The object specification is not changing quite right now. Nor the way FG reads it.

A change that could have this size of an impact, would deprecate all the objects now existing in the database, and not only affect those only existing in terraGIT.

So, in conclusion, in P. Sadro on its loneliness succeeds to bring us Scenery V3, then TerraGIT will move all together to the new specification with FlightGear.
But if his loneliness plays against him, there is no need for us to sit an wait with a 3year/old scenery when we can enjoy new airports in the here and in the now.

And if we can coordinate to bring terrasync as a synced method downstream of terraGIT, then the community at large would benefit. Not only the fortunate enough to know terraGIT exist, and are technical enough to be able to use it ,or develop on it.

That is where I think the community needs us together.... not appart.

[Richard]

misconception that ... https://scenery.flightgear.org/ is the actual database. Which is not.

Sorry for the slow reply; this has taken me days to write.

What I think of as the scenery database is the sources that are used to build the run time scenery database which I think of (and might refer to) as the visual database.

So we have two databases; the source data that contributors create is really what we need to take care with for the future; the run time database can (and probably will) be regenerated from the sources.

At this point, I am limited to bow to your knowledge in computational science and Engineer, but it seems from my experience within FG, that it was an sloppy approach all from the beginning.

Thanks, but feel free to challenge anything I say as I'm not always right. In this instance separating the source from the target data is a good thing in my opinion because it adds a layer between the authored content and the run time - and in this layer any manipulations that are needed could be performed. Equally it becomes possible to write something to generate a completely different (e.g. OpenFlight) database from the source scenery database.

Looking at the XML these are my questions:

1. Is the numeric ID in the STG guaranteed never to change?
2. What is your opinion of the bulk upload facility; is it something that could be used with modifications or is it not suitable. A brief glance through seems it's only for shared objects, but there are quite a lot of them in 2892922.stg; and maybe some of the static objects could be shared? (I have no idea what the implications of this would be)

From your list
>2. Scenery is the configuration files for airports, (xml files) are not specified in the STG database.
>3. Artificial Intelligence traffic files: are not specified in the STG database

It seems to me that these really should be stored in a database; but I'm guessing that (2) above is probably a result of other items that could be in a database or built as a result of some process, and (3) above is probably quite complex and might not be that easy to store.

The terrain mesh can remain as binary files that are computed. I believe the original plan (for scenery.flightgear) was to somehow include the terrain mesh so that the XPlane database changes could be picked up on demand; but I'm not sure I'm right about this. The fact that the terrain mesh hasn't changed in the last few years isn't really a good guide as to what might happen in the future as this all depends on what happens with the cutting out of airports. If this had been automated for a few years the git size could have grown a lot. I don't know, and actually nobody knows this until it happens in the future - but we need to consider the future when thinking about how things should be done. So in many ways what's been happening with the mesh and other binary assets is a bad guide to what might be needed in the future; but bear in mind that I'm not against using git where it works well - but at the same point git repositories do get slower over time, both to download and for certain operations (such as blame).

Merging STG files and lumping objects together at run time is a good thing; I just don't think that merged scenery should be submitted to the source scenery database as it then becomes is impossible (or really hard) to undo a merge; whereas it is easy to perform a merge.

I thought the bulk upload was to allow more things to be submitted at once.

It is only solving one problem - which is the difficulty in getting scenery submitted;

THAT IS THE PROBLEM.

When I say "one problem" I mean only one problem because everything else is working fine. So if too much infrastructure is changed without proper consideration then other things that are working well may no longer work; and maybe not immediately in an obvious way.

But your fear, I do not comprehend.

Once FG releases FG Scenery V3 (the terrain) we simply update TerraGIT Terrain branches (or possibly even better, split the content in 2 repositories one for terrain other for Objects).

My fear is just based on experience that it is better to control the source rather than the generated or built runtimes.

For example; I think the current scheme has survived the V1 to V2 transition so it should survive going to V3; whereas I'm genuinely not sure about what's going to happen with git when it comes to a merge with a newly generated V3 database. Obviously I'd hope it went well and git managed to sort out any conflicts or merges; but until it's been done we just don't know what will happen. The best case scenario with V3 is that it all just works; and the worst case is that it is entirely incompatible and TG becomes locked with V2. The likely case is that some things will pull through and others will result in conflicts of some sort.

For me the most important thing is stability and making sure that things continue to work in the next 20 years as FlightGear continues to evolve. Currently the codebase supports V1 and V2 scenery and this will probably continue so that V3 is also supported (I suspect there'd have to be a good reason to remove either of these). Unless someone backports V3 then older versions of FG will definitely not be able to use the new scenery. Backporting may be a non trivial task because of dependency changes that are necessary.

Hand crafted, detailed and often beautiful improvements to the scenery are fantastic to see and need to be encouraged; but in a way that will be compatible with the future. I'm not saying TG isn't compatible, as we can't know this, just that compatibility is an important consideration.

Pete is doing a making good progress; and it might just be one of those jobs that often are best done by a single mind. There is always a concern that if he gives up then V3 will not happen any time soon.

I'm still not well enough informed to be able to make a valued judgement about the future. I can see that there are possibilities with TG, but I can also see the risks in the longer term with compatibility and can't help wondering if what we should be looking for is a way to make changes to scenery in a compatible way so that we can write a script to go through git, figure out the differences and prepare these for submission.

It looks to me as though XPlane uses a similar source -> target database for submissions to the WED - and then builds out the apt.dat at timely intervals for their releases which definitely makes me think that the scenery database is probably a good idea that needs improvement rather than replacement.

Finally everything that is in the scenery should as close as possible to the real world and nothing fictional; this is manually reviewed for submissions to the scenery database so maybe there should be some way of change approval like in XPlane.