Modders - Need a Data Format Checker?
 

JonMacLeod, I noticed that you have over 41,000 lines of data in your components file for your Imperium Verus Mod (Great Mod BTW). How did you make sure that all that data was in the proper format, that everything was properly referenced? I've found that the Component Editor .90 chokes on a Components file that is misformatted.

To all Modders

Do you find your data to be improperly formatted at times?

Do you find that you've said that a component has 3 Tech Reqs, but in fact it has 2 or 4?

Have you found that you set a Tech Level Req higher what is possible?


I've only began getting into the modding of SEIV and was wondering if this is common place in the community. I'm building a Data Integrity checker that will analyze the Components, Abilities, and TechArea file.


Does the Abilities.txt file actually change? I guess abilities can be removed from the list, but certainly nothing can be added. Right?

Thanks

Re: Modders - Need a Data Format Checker?
 

The abilities.txt file is just a listing of the available abilities in the game (although not all of them work).

Fyron's modding tutorial has a list of all of the abilties and which files they work in:
http://se4modding.spaceempires.net/ModdingTutorial.html

The formatting itself is not really a problem (because it's copy/paste or automated) and the error reporting by SE:IV when loading a mod catches missing ability fields, but not extra ones. It also does not pick up on unused levels etc. - although you can usually find these by using the internal tech tree report. But a data file check would be cool.

DavidG made a data file editor for SE:IV that works fairly well with most of the files:
http://www.captainkwok.net/files/se4modder.zip

Re: Modders - Need a Data Format Checker?
 

Abilities.txt is just a reference file.

How are you writing your program? I've got a program written in C++ that is designed to create various forms of display files based on a mod's data, such as html files, CSV files, and MySQL database schema files. Perhaps we could combine our work? I've been meaning to rewrite the file classes to use a more intelligent inheritance model. Perhaps I should get on that!

Edit:
That modder by DavidG isn't necessarily what Wenin was thinking of, though. It doesn't necessarily act as a constraints checker of a manually edited file. Does it properly handle malformated files?

Re: Modders - Need a Data Format Checker?
 

Abilities.txt is merely informational, like the readme.txt

The most common error in datafiles is the double blank-line effect, which acts much like an end-of-file. Useful when testing, though.

I wrote a quick little Components repair tool which helps to fix erroneous newlines, particularily in multi-megabyte tech gridded mods.

SE4 will tell you if you have not enough tech or ability entries to match the number you gave. Having too many means the last ones will be ignored.
Typically quite rare, from what I've seen. Only a couple of times during a mod's development.

Re: Modders - Need a Data Format Checker?
 

Thanks for the info on the Abilities.txt file. I won't be checking that file then.

Fyron, I found that the Launch Drone ability isn't listed at http://se4modding.spaceempires.net/M...Tutorial.html. Nor is it in the Abilities.txt file.

I'm writing it in Visual Basic 6 (Yeah out of date, but I don't do this for a living so I can't afford .NET) =) I'd be open to sharing what I'm working on with you Fyron.

Are you sure? I was playing a mod that had tech Req that couldn't be met, and I got no error messages on that. I ran across the Components when I was making modifications to have it more suit my desired play.

It also loaded a components file where X number of Techs were required, but X+1 were listed. SE4 gave no errors.

It was when I started using the Component Editor .90 that I started finding the issues. Component Editor has no data integrity check. When it would rewrite the Components.txt file it would destroy data caused by malformed data.


What I'm planning is for it to go a few steps further than your Component Repair Tool.

It will check all the following
- Proper format of the field names. Example: Name - Correct, name - Incorrect (I don't know if this is a problem in the components file, but I've found capitalization is important in the AI files.)

- Techs you indicate are listed in the TechArea.txt
- Tech Level Req can be achieved
- # of indicated Tech Req and Abilities is listed
- Abilities listed exist and are spelled properly.
- Fields that have specific input requirements

For any error that is encountered it will give the Line the error exists upon and what the program was expecting. Then the program ends, so that you can fix the error and rerun the application.

I wasn't thinking of having it check if a Tech Level goes unused. Would this be helpful? If so, should it just check if the highest value Tech is used, or if each Tech Level is used? Again I've only played two mods now so I don't know what's all out there.

Re: Modders - Need a Data Format Checker?
 

Got a question about the general rules of formatting.

Are Tabs within the Components.txt file a problem. SE4 doesn't give any errors, but the Component Editor .90 eats them for breakfast.

What got me going on this was my experience with the Component Editor, and how it reacted to malformed data. So not only am I looking to make the files SE4 friendly, but also Component Editor .90 friendly. This will result in a more cleanly data files.

Re: Modders - Need a Data Format Checker?
 

Nobody needs .NET anyways. It makes for huge, bloated programs! http://forum.shrapnelgames.com/image...ies/stupid.gif And besides, there are plenty of free IDEs out there. Eclipse + CDT plugin using the GCC compiler via cygwin, while sounding complicated to set up, works rather well. I only use MS VS 2003 cause I got it free from school.

Do you know much C++? VB and C++ can't really be integrated very well.

Also, note that my program has no GUI, since there isn't necessarily a need for one when there is no run-time interactivity.

Currently writing up generic classes so adding all the files can be done much more easily. Hopefully I can remember this stuff!

=0=

RE: Tech Reqs

He was refering to entries in the data file, not necessarily levels of those reqs.

"It also loaded a components file where X number of Techs were required, but X+1 were listed. SE4 gave no errors."

That's exactly what he said would happen. The X+1 Tech Req is ignored.

Also, note that modders frequently put high level tech reqs to temporarily remove items, so such issues should be handled as warnings, rather than errors.

Re: Modders - Need a Data Format Checker?
 

Looking at the Abilities section of http://se4modding.spaceempires.net/ModdingTutorial.html it will take to long to check so I'll just ask.

With the "Launch Drone" ability not being listed in the Abilities.txt file, are there any other such undocumented abilities?

Thanks

Re: Modders - Need a Data Format Checker?
 

Tabs are nasty creatures and should be replaced with spaces in any repair tool. http://forum.shrapnelgames.com/images/smilies/wink.gif SE4 can handle them well, since they are just white space. They make for really ugly text files, however, since text editors rarely agree on how to display them.

Also, it might be good to standardize column spacing automatically when the tool is run (or have an option for it).

Re: Modders - Need a Data Format Checker?
 

I don't know any C++.

My App will have a simple GUI to allow people to create a Family output file which will list all of the components by family so that modders can manually confirm that data.

It may have a few other similiar options.

Thanks for the pointers on why unatainable Components would be made.

Apologies on my poor reading comprehension, Suicide Junkie.

Re: Modders - Need a Data Format Checker?
 

What would be a good standard for spacing? I found 25 to be standard, until I ran into the Optional fields being used in the Imperium Verus Mod. Or would it be fine to allow the User to specify the standard?

I guess if most editors just care about the white space between the field name and := and the space between := and the data, then having it customizable would work.

Again this is my inexperience with working with the various mods and tools made available for this game.

Re: Modders - Need a Data Format Checker?
 

The amount of spaces does not matter to SE4 (so long as it is at least 1). We could standardize all files to 1 more than the longest field name, making every file use the same spacing. I never understood why Aaron didn't do this in the first place.

Re: Modders - Need a Data Format Checker?
 

Hello, well, you can always download the express version of VB, C# and others .NET languages from microsoft web site. I haven't tested them yet, (need a service pack).

Re: Modders - Need a Data Format Checker?
 

VS 2005 is rubbish. VS 2003 is vastly superior. Stay away from 2005 at all costs!

Re: Modders - Need a Data Format Checker?
 

VS? Visual Scripting?

Re: Modders - Need a Data Format Checker?
 

Visual Studio.

Re: Modders - Need a Data Format Checker?
 

Posted the first version of the utility in the Mod forum.

Re: Modders - Need a Data Format Checker?
 

I am pretty meticulous about my work and have not found any errors other than things not working “just” the way I wanted. That is a side effect of abilities, game balance, etc though.

I am using DavidG’s SE4Modder v1.7.6 which is ripe with bugs and errors. I have learned to be very careful about how I use it to avoid another crash. I have been working on a long list of bugs but at some point grew board (play on words) keeping the list up.

Point is I would still suggest it for error avoidance. Once you get used to the bugs it is fairly quick and accurate. Most importantly with a few clicks you can easily double check your work. Although sometimes a few clicks include having to close then open the program.

Re: Modders - Need a Data Format Checker?
 

*El Linko* (click me)

Since people are lazy http://forum.shrapnelgames.com/images/smilies/wink.gif

Re: Modders - Need a Data Format Checker?
 

Wenin that almost sounds like my mod. I had built in a few components with x number of abilities and in the process would drop one but not delete the line for the ability. Is that part of what you are describing? If it is my old StarFire mod you may want to send me a private message.

Re: Modders - Need a Data Format Checker?
 

Yes that would be something my Data Checker would find, but I've never downloaded a StarFire mod. My data checker is far more picky than SE IV is actually.

I've actually only downloaded three mods. ExMod, Imperium Verus, and Trek_Bab5

After attempting to make my own modifications I ran into problems, and through searching found the causes. Digging deeper I found other errors that though didn't cause problems were still improperly formatted.

Anyhow the problems I ran into caused all kinds of headaches and wasted time so I created this data checker. =) Most if not all of them were caused by using the Component Editor .90 on malformed data.

Now I'll begin working on my own mod. =)

Re: Modders - Need a Data Format Checker?
 

If it had been my old StarFire mod I would have been ready to aid you. Good luck in your endeavor.

Re: Modders - Need a Data Format Checker?
 

Released 1.05 today

Re: Modders - Need a Data Format Checker?
 

I did it manually in notepad for Imperium Verus 1.02. The component editor I tried screwed up my mod (badly, http://forum.shrapnelgames.com/images/smilies/evil.gif) so I went back to doing it manually. I have someone helping me check Imperium Verus 1.03 for errors by playtesting it.

Re: Modders - Need a Data Format Checker?
 

I also prefer to use notepad myself. However, Wenin's data checker was a useful program to find those little bugs like unused tech levels of additional requirements that were not being used. Unfortunately, the last version had a bug IIRC that reduced functionality.

Re: Modders - Need a Data Format Checker?
 

41,000 lines of data done manually...Fyron, I think you have a competitor. http://forum.shrapnelgames.com/images/smilies/laugh.gif

Re: Modders - Need a Data Format Checker?
 

The B5 Mod has a 9.5 MB, 270976 line components file. http://forum.shrapnelgames.com/images/smilies/eek.gif Although, it was generated by a spreadsheet, not by hand.

Re: Modders - Need a Data Format Checker?
 

Who did that one, and was it done by hand?

(Grumbler, methinks?)

Re: Modders - Need a Data Format Checker?
 

It was mostly started by Val, with some changes done by others (including Grumbler) down the road. It was generated by a spreadsheet, not all by hand.

Re: Modders - Need a Data Format Checker?
 

Still, even with a *-1 penalty, I think it wins the 'done by hand contest.

* I'm turning into a Hero System geek. Make that 'divided by half'.

Re: Modders - Need a Data Format Checker?
 

I use the SE4Modder by DavidG for all my modding needs nowadays. You can use it to mod anything, with a GUI, as well as numerous other small but handy features. The only thing is that you have to edit everything manually, but I like the control http://forum.shrapnelgames.com/images/smilies/happy.gif

Re: Modders - Need a Data Format Checker?
 

I use both DavidG's modder and Wordpad (better Find and Replace function than Notepad).

Re: Modders - Need a Data Format Checker?
 

I like the find/replace function in EditPlus because it allows regular expressions, including newline characters (which a lot of other advanced tech editors deny the existence of). Makes modding much faster once you get the hang of them. http://forum.shrapnelgames.com/images/smilies/happy.gif

Re: Modders - Need a Data Format Checker?
 

How do you generate a components.txt file from a spreadsheet? Do you use Mail Merge? I never could figure that function out.

Any tutorials out there?

Re: Modders - Need a Data Format Checker?
 

No idea. I would probably export it as a CSV file and write a c++ program or a python script to do it.

Re: Modders - Need a Data Format Checker?
 

Considering that someone used Microsoft Excel to make pacman...