Modders - Need a Data Format Checker?

[Wenin]

JonMacLeod, I noticed that you have over 41,000 lines of data in your components file for your Imperium Verus Mod (Great Mod BTW). How did you make sure that all that data was in the proper format, that everything was properly referenced? I've found that the Component Editor .90 chokes on a Components file that is misformatted.

To all Modders

Do you find your data to be improperly formatted at times?

Do you find that you've said that a component has 3 Tech Reqs, but in fact it has 2 or 4?

Have you found that you set a Tech Level Req higher what is possible?


I've only began getting into the modding of SEIV and was wondering if this is common place in the community. I'm building a Data Integrity checker that will analyze the Components, Abilities, and TechArea file.


Does the Abilities.txt file actually change? I guess abilities can be removed from the list, but certainly nothing can be added. Right?

Thanks

[Captain Kwok]

The abilities.txt file is just a listing of the available abilities in the game (although not all of them work).

Fyron's modding tutorial has a list of all of the abilties and which files they work in:
http://se4modding.spaceempires.net/ModdingTutorial.html

The formatting itself is not really a problem (because it's copy/paste or automated) and the error reporting by SE:IV when loading a mod catches missing ability fields, but not extra ones. It also does not pick up on unused levels etc. - although you can usually find these by using the internal tech tree report. But a data file check would be cool.

DavidG made a data file editor for SE:IV that works fairly well with most of the files:
http://www.captainkwok.net/files/se4modder.zip

[Fyron]

Abilities.txt is just a reference file.

How are you writing your program? I've got a program written in C++ that is designed to create various forms of display files based on a mod's data, such as html files, CSV files, and MySQL database schema files. Perhaps we could combine our work? I've been meaning to rewrite the file classes to use a more intelligent inheritance model. Perhaps I should get on that!

Edit:
That modder by DavidG isn't necessarily what Wenin was thinking of, though. It doesn't necessarily act as a constraints checker of a manually edited file. Does it properly handle malformated files?

[Suicide Junkie]

Abilities.txt is merely informational, like the readme.txt

The most common error in datafiles is the double blank-line effect, which acts much like an end-of-file. Useful when testing, though.

I wrote a quick little Components repair tool which helps to fix erroneous newlines, particularily in multi-megabyte tech gridded mods.

SE4 will tell you if you have not enough tech or ability entries to match the number you gave. Having too many means the last ones will be ignored.
Typically quite rare, from what I've seen. Only a couple of times during a mod's development.

[Wenin]

Thanks for the info on the Abilities.txt file. I won't be checking that file then.

Fyron, I found that the Launch Drone ability isn't listed at http://se4modding.spaceempires.net/M...Tutorial.html. Nor is it in the Abilities.txt file.

I'm writing it in Visual Basic 6 (Yeah out of date, but I don't do this for a living so I can't afford .NET) =) I'd be open to sharing what I'm working on with you Fyron.

SE4 will tell you if you have not enough tech or ability entries to match the number you gave. Having too many means the last ones will be ignored.

Are you sure? I was playing a mod that had tech Req that couldn't be met, and I got no error messages on that. I ran across the Components when I was making modifications to have it more suit my desired play.

It also loaded a components file where X number of Techs were required, but X+1 were listed. SE4 gave no errors.

It was when I started using the Component Editor .90 that I started finding the issues. Component Editor has no data integrity check. When it would rewrite the Components.txt file it would destroy data caused by malformed data.


What I'm planning is for it to go a few steps further than your Component Repair Tool.

It will check all the following
- Proper format of the field names. Example: Name - Correct, name - Incorrect (I don't know if this is a problem in the components file, but I've found capitalization is important in the AI files.)

- Techs you indicate are listed in the TechArea.txt
- Tech Level Req can be achieved
- # of indicated Tech Req and Abilities is listed
- Abilities listed exist and are spelled properly.
- Fields that have specific input requirements

For any error that is encountered it will give the Line the error exists upon and what the program was expecting. Then the program ends, so that you can fix the error and rerun the application.

I wasn't thinking of having it check if a Tech Level goes unused. Would this be helpful? If so, should it just check if the highest value Tech is used, or if each Tech Level is used? Again I've only played two mods now so I don't know what's all out there.

[Wenin]

Got a question about the general rules of formatting.

Are Tabs within the Components.txt file a problem. SE4 doesn't give any errors, but the Component Editor .90 eats them for breakfast.

What got me going on this was my experience with the Component Editor, and how it reacted to malformed data. So not only am I looking to make the files SE4 friendly, but also Component Editor .90 friendly. This will result in a more cleanly data files.

[Fyron]

Nobody needs .NET anyways. It makes for huge, bloated programs! And besides, there are plenty of free IDEs out there. Eclipse + CDT plugin using the GCC compiler via cygwin, while sounding complicated to set up, works rather well. I only use MS VS 2003 cause I got it free from school.

Do you know much C++? VB and C++ can't really be integrated very well.

Also, note that my program has no GUI, since there isn't necessarily a need for one when there is no run-time interactivity.

Currently writing up generic classes so adding all the files can be done much more easily. Hopefully I can remember this stuff!

=0=

RE: Tech Reqs

He was refering to entries in the data file, not necessarily levels of those reqs.

"It also loaded a components file where X number of Techs were required, but X+1 were listed. SE4 gave no errors."

That's exactly what he said would happen. The X+1 Tech Req is ignored.

Also, note that modders frequently put high level tech reqs to temporarily remove items, so such issues should be handled as warnings, rather than errors.

[Wenin]

Looking at the Abilities section of http://se4modding.spaceempires.net/ModdingTutorial.html it will take to long to check so I'll just ask.

With the "Launch Drone" ability not being listed in the Abilities.txt file, are there any other such undocumented abilities?

Thanks

[Fyron]

Tabs are nasty creatures and should be replaced with spaces in any repair tool. SE4 can handle them well, since they are just white space. They make for really ugly text files, however, since text editors rarely agree on how to display them.

Also, it might be good to standardize column spacing automatically when the tool is run (or have an option for it).

[Wenin]

I don't know any C++.

My App will have a simple GUI to allow people to create a Family output file which will list all of the components by family so that modders can manually confirm that data.

It may have a few other similiar options.

Thanks for the pointers on why unatainable Components would be made.

Apologies on my poor reading comprehension, Suicide Junkie.