Dom2 file formats
 

Hi

Anyone know which Dom2 data files (map, player-data) are readable and where I can find a file format documentation? I gather the player files are encrypted with the player password, but if I got the password (to my own files) will I be able to read these and use the data?

I'm hoping to make a short of "quick-bar" to help get a better overview of my armies etc., though I still wish Illwinter would simply put better main-map filters in (like Show/Hide: Scouts/Commanders/Priest/Garrisoned Troops (troops not in army)).

Has anyone else done this or is it hopeless?

Re: Dom2 file formats
 

Hmm, yes it was indeed the specific game data files I was interrested in. So for a running game, is the following correct? Otherwise please elaborate:

jotun.2h - contains Jotun orders for current turn.
Is this the file sent to host in PBEM games?
So at start of game, this contains God data?

jotun.trn - contains Jotun player turn file describing situation at start of current turn.
Is this the file returned from host to the Jotun player in PBEM games?
In a running game does this file contains God data?

ftherlnd = history file
Province data? Anyting else?
When is this file updated in a PBEM game? When the player opens his new .trn file?

What I'm looking for of course is a way to open and decode all these files (provided I have the appropriate password, ie. my own), and present chosen data in another manner.

Btw. I read that something called Torvak's savegame utility can read the turn numer, so obviously someone has been able to figure out the data bytes http://forum.shrapnelgames.com/images/icons/icon7.gif

Oh yea, and ordered the full game yesterday so soon no more demo for me http://forum.shrapnelgames.com/images/icons/icon10.gif

Edit: ... after taking a look at the files with a hex-editor, it seems that byte 15 is the "turn number" byte - this goes for all three files: ftherlnd, .trn and .2h. So now I only need the rest http://forum.shrapnelgames.com/images/icons/icon10.gif

[ May 12, 2004, 16:08: Message edited by: Innocence ]

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">That's right, Chief.

<font size="2" face="sans-serif, arial, verdana">Right again.

<font size="2" face="sans-serif, arial, verdana">Only in the sense of the gods as units. I'm not sure if any other pertinent god data is included.

<font size="2" face="sans-serif, arial, verdana">ftherlnd is the "world state" file. It contains all of the information about the world, which is needed to host a game. It's updated at every hosting. In a PBEM game, players should not be sent this file unless you wish them to be able to do oracular hosting, or you wish to change who is handling the hosting.

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">I think its ansi-C
I didnt write it (and cant remember who did)
I also dont know if it works
Its here if anyone wants to look at it but remember it was just the start of a project, and it was for Dom1 not for Dom2
http://www.dom2minions.com/~dominion/dumpgod.c

[ May 12, 2004, 19:16: Message edited by: Gandalf Parker ]

Re: Dom2 file formats
 

Well the map image is readable (.TGA paint program format)
the map commands are readable (.map file, ascii text)
the MODs are readable (.dm file, ascii text, plus .tga's in the mods dubdirectory)

the games scoreboard is readable if you use the --scoredump switch when running the game. (score.html, web format)

I think thats about it. I have a bit of code someone did way back in Dom1 which is suppposed to be able to read god files (.2h) but I dont know how well it does Dom2 files. And I dont know if anyone has ever tried modifiing it to read the later .2h files in an ongoing game

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">In which language is the Code ? Where did you got the information about the .2h formats ? Can you send me the code at liga(AT)treemme.org ?

thkx

Liga

Re: Dom2 file formats
 

There's a pro and a con to generally known file formats:

pro:
Some folks could start working on "external", nation-specific AIs.
(I still wonder if you could write one in F77 ...)


con:
Some folks would start working on cheats and exploits to ruin multiplayer games ...

I would like to know the file formats very much - sadly the other game, which would have been much more suited for such a task, Stars!, died without the devs giving the specs to the public.
Security by obscurity isn't the way to go, but that's only IMHO.

Maybe our devs here could drop a line how they feel about this issue ??

[ May 13, 2004, 01:43: Message edited by: Arralen ]

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">It would seem to be the case that the data is encrypted and compressed to prevent exactly that. I've pulled open .trns and .2hses just to see if anything particularly edifying was revealed, but not only are they at least partially encrypted and compressed, as no plaintext strings such as the name was found anywhere, but they are also checksummed against modification.

Re: Dom2 file formats
 

dumpgod doesn't seem to work on dom2 files. under linux, it dies with "g2 seems to be misaligned".

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">Ok thanks - so it's a host-only file and in a normal multiplayer game the regular players wont even have this file http://forum.shrapnelgames.com/images/icons/icon7.gif
<font size="2" face="sans-serif, arial, verdana">Ah yes Stars! I wouldn't call it dead, though the sequel sadly is.

It's likely that Illwinter has enrypted the files, but if they wont release the file-specs and encryption metod (and noone else has been able to figure it out) I say we stand almost zero chance of any progress. Too bad.

Alternatively, maybe they could implement an export feature like Stars! has, dumping ALL information from the players PoV into semi-colon separated text-files, readable with anything from Excel to custom made programs. All without comprimising the file-security.

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">The problem with releasing info on the save files is that the 'con' part is very very much easier to do than the 'pro' part. Doing an external AI for Dominions 2 would be a humongous task. That said I believe the most common use for this info would be cheating.

Re: Dom2 file formats
 

[quote]Originally posted by Johan K:

Quote:

The problem with releasing info on the save files is that the 'con' part is very very much easier to do than the 'pro' part. Doing an external AI for Dominions 2 would be a humongous task. That said I believe the most common use for this info would be cheating.

<font size="2" face="sans-serif, arial, verdana">Ok, I fully understand. I just only want to know if this informations are somwhere aviabile in the .trn and .2h and if possible to extract them (it will be really nice to have a small utility who could do that, a precompiled one, so no secrets could be obtained). Just something to run

info abysia.2h
info abysia.trn

and get something like

turn 4 on map Aran with mods A,B and C

I know that could be tedious for devs to produce this kind of utils (that the reasons why we are trying to have the files formats) but it will be really usefull becouse playing a lot of game by email sometimes is easy to make confusion with the .2h files recived and the .trn I have to send ...

and also it will be easy to use the utils to move the files and send in an automatic way ...

thank you anyway for this great game!
Liga

Re: Dom2 file formats
 

there is still another possibility, which is to give the file format to some trusted individuals (a beta tester willing to code an utility for example).

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">Yes, that is in fact very likely. Plus, Illwinter might feel obligated to keep track of every single future change to the format, which isn't very flexible.

So in the end I guess that any utility reading the files directly will either have to be very simple (like extracting the turn number), or written by Illwinter, which is fair I guess.

However, that still leaves the possibility of an ASCII export feature. If Illwinter would implement this we'd have all the data we need and the file-security wouldn't need to be compomised.

Do Illwinter read these forums or should I rather mail them directly?

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">However, that still leaves the possibility of an ASCII export feature. If Illwinter would implement this we'd have all the data we need and the file-security wouldn't need to be compomised.

Do Illwinter read these forums or should I rather mail them directly? </font><hr /></blockquote><font size="2" face="sans-serif, arial, verdana">That was Illwinter you were answering. The programmer side in fact (which reads all, but Posts rarely).

The game does do ascii dumps of some info. There is the stats dump...
Statistics for game 'Cradle_and_RandMod' turn 1
Abysia played this turn
Man played this turn
Ulm played this turn
Arcoscephale played this turn
Marignon played this turn
Pangaea played this turn
Jotunheim played this turn

And there is the scoredump, which is an html ascii file. At the top is...
Dominions II Scores, Cradle_and_RandMod turn 2
and then the numeric scores of the players as far as armies, forts, dominion, etc

As you can see, both files have the turn number in them. And both files exist because we asked. Its true that Illwinter is unlikely to offer a full dump but if we can show a real (safe) use for some info then we can probably get it added to one file or the other.

As I see it, the stats file is meant as info for the host and game management. The scoredump is more the place for player info about the game. So as far as I can tell it might work to request that the stat info also include map name and mod names.

Anything else? Something with a reason, short of dump it all and let me play with it. http://forum.shrapnelgames.com/images/icons/icon7.gif Im a hacker and would love it but I understand not getting it.

[ May 13, 2004, 13:31: Message edited by: Gandalf Parker ]

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">I know about that files and the options to get it ... but my problem is: I got 2 pythium.2h from the same player who is playing 2 different game by PBEM with me ... if I don't pay too much attention the 2 files go in the same directory (one will be called pythium.1.2h) and after taht I have no way to understnd which one is for which game ... so, I was asking if there is a way to understand each .2h file from which game comes and for which turn ...

anyway I know that there are more important problems/tasks for devs ...

bye bye
Liga

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">Oh, hehe! http://forum.shrapnelgames.com/images/icons/icon7.gif
<font size="2" face="sans-serif, arial, verdana">Ditto, and if you know Stars! you know what kind of info I'm looking for - no more or less info than what the current player knows.

Also, dumping should be a player action, not an automatic job (or it could be done via a command line option).

ASCII dumping should be done to sepearate filesnames, like:

'gamename'-world-'nation'-'turn#'.txt
=all world data from players PoV (shrines, provinces, fortresses, garrisons etc.)
Example: genesis-world-jotun-5.txt

'gamename'-units-'nation'-'turn#'.txt
=all unit and army data, army composition, unit afflictions etc.
Example: genesis-units-jotun-5.txt

'gamename'-orders-'nation'-'turn#'.txt
=current orders given to which units, build orders etc.
Example: genesis-orders-jotun-5.txt

All in all, every piece of info available to the player in-game, dumped in ASCII format.

(Afterthough: Actually, if the regular savefiles were named uniquely like above there'd be no problem idetifying which jotun.2h file belonged to which game. 8.3 length filenames have long been made obsolete by any modern OS)

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">The "right" way to do things is to have the game files (.trn, .2h, ftherlnd) signed, rather than encrypted, to make it possible to read them, but not change them.

As in, have most of the files be in a public knowledge format, with a fingerprint appended. Then have the dom2 program refuse to use files that don't have that fingerprint, or that have the wrong fingerprint. That way, one can examine the contents of files if needed, but only the game (or programs that know the fingerprinting validation mechanism, which means programs that Illwinter makes) can create/modify these files in such a way that they can be used.

That would also remove the incentive for people to figure out how .2h files are encoded (which, if they manage to do it, will let them cheat, even if it's not the primary motivation for trying to figure it out). From looking at the god files (yeah, I tried to figure the format myself, but didn't try very hard), I don't think the files are encrypted, just not in a publicly described format (it's pretty easy to figure out which byte describes the nation, etc). This, IMHO, is the worst solution.

Edit: formating

[ May 14, 2004, 12:46: Message edited by: PhilD ]

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">But then anyone could read another players data without needing his password. This would be cheating too.

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">The problem with releasing info on the save files is that the 'con' part is very very much easier to do than the 'pro' part. Doing an external AI for Dominions 2 would be a humongous task. That said I believe the most common use for this info would be cheating. [/QB]</font><hr /></blockquote><font size="2" face="sans-serif, arial, verdana">I'm not sure how the information exchange is organized, but in principle, tampering with the 2h and trn files should not be a problem. trn essentially contains the projection of the world state for the specific nation and 2h contains player's instructions on what to do. As long as the server verifies logical validity of 2h file, the scheme is well protected.

If for example, the cheater modifies trn file to increase his earth gem treasury to 500, so that the client will allow him to summon 20 iron dragons, the server will check the command and discover that somebody is casting 20 iron dragons when he only has 10 earth gems (which is known to the server independently from trn and 2h files). The same thing would happen if the cheater has modified his 2h file.

So the only problem here is somebody getting information from other player's turns, but this can be prevented by encrypting the files with some standard method using the password as a key, so that only the person with the correct password can get the information.

Of course, this mechanism is not necessarily easy to implement http://forum.shrapnelgames.com/images/icons/icon12.gif

Re: Dom2 file formats
 

So it seems that...
1) the files seem secure from cheating
2) if the formats were public neat tools could be made
2a) if the formats were public there would probably be cheating
3) with some work the formats might be able to be made public and secure from cheating

[ May 15, 2004, 23:55: Message edited by: Gandalf Parker ]

Re: Dom2 file formats
 

<font size="2" face="sans-serif, arial, verdana">But then anyone could read another players data without needing his password. This would be cheating too. </font><hr /></blockquote><font size="2" face="sans-serif, arial, verdana">Oh right. Makes sense.

No good solution, then...