Author's Note: This article was updated at the request of ElminsterAU to add additional users to the team section. (11/01/2019 23:15 GMT)
For those in our community who want to know a little about the man behind the ElminsterAU handle, could you tell us a little about yourself?
I grew up in mid-west Germany, along the Rhine valley, in the '80s and early '90s, moved to Australia a bit over 16 years ago, and am now living in south-east Queensland, north of Brisbane, with my wife and our 6-year-old twins. I got my first computer (an Atari 520ST) when I was about 8 and was instantly fascinated with it. Not just playing games, but also figuring out how to make the computer do things I wanted it to do. I taught myself programming using Omikron-Basic by looking at the examples. Still have fond memories of that time.
A year later we got a "family PC" (a 286 at first, a bit later a 386) which I was using on and off, but the Atari was my mainstay for quite a few more years. I dabbled with GW-BASIC on the PC but found it lacking, and so, shortly after Turbo Pascal 4.0 came out, I started learning Pascal by looking at examples and a pocket-sized "Turbo Pascal 4.0 - The Reference" book. I've learned plenty of other programming languages over the years after that (Everything from Assembler to Smalltalk, and more recently Java, C# and many more), but Pascal (now in the form of Delphi) has been my language of choice ever since.
When I was 15, while still finishing school, I started my own registered trade (which required getting special permission from a family court as a minor), building custom PCs and doing custom development work for a number of customers.
After 10 years of school ("Realschule" in Germany), I started an industry apprenticeship as "Kommunikationselektroniker Fachrichtung Informationstechnik" (“Communication engineer specialized in information technology”) and finished it 1 year early with a top grade.
I was still doing custom development work during all that time and directly started full-time work for one of my major customers, Microtech where I quickly went from "new guy" to "in charge of designing and developing the application framework for our next generation software". After 4 eventful years at Microtech, during which time I also provided a completely new design for FlashFiler (the database engine I was using at Microtech) to TurboPower (the make for FlashFiler) for them to implement, I moved to Australia and was supported to start working on COSMOS which was using the new release of FlashFiler that was based on my design.
Shortly after I arrived in Australia, TurboPower unexpectedly got bought and exited the Delphi component market (the buyer just wanted the developers for their in-house projects), leaving a lot of FlashFiler users without any real way forward. So instead of working on COSMOS as planned, over the next 8 months, I developed a new database engine from scratch, NexusDB, which I've been working on, maintaining and improving, for the last 16 years.
Outside of computers, I've been an avid table-top RPG player and DM since my childhood, starting with 2nd edition DSA (Das Schwarze Auge, The Dark Eye, a German RPG) and later AD&D2 and following editions.AD&D got me started on reading fantasy novels. I have very fond memories of trying to read the original Dragonlance trilogy in English for the first time. This started me down the road as a ravenous reader of fantasy and science fictions books that continues until today.
Can you remember how you first got into gaming?
My earliest memories of "gaming" are of visiting a schoolmate and playing a whole bunch of games on his C64. The only title that has stuck in my mind from then is Giana Sisters, but there were many more.
When I got the Atari, I initially only had the high-resolution monochrome display for it, which means I couldn't play any of the commercial games on it. I have fond memories of my Dad driving me to the next larger city on weekends to visit a shop that had various freeware and shareware games and you could buy floppy disks with them for a small copy fee. One of my favourite games of these early days (I was 8 at the time) was Ballerburg.
When I finally got a colour monitor connected to the Atari my first game was Dungeon Master which cemented my love of RPGs. I also fondly remember spending countless hours with Airball. There are way too many games to remember them all by now.I've long lost count of the number of games I've played over the years. I've dabbled in pretty much any genre, but overall found the most enjoyment in role-playing, tactical, and strategy games. Ultima 7 is still very much on my mind as the first computer RPG where I really felt that the world is “alive”.
Do you still find the time to play any games?
I try to take the time to play, but it has gotten much rarer than I like. I try out a pretty decent number of games every year, but hardly find the time to actually finish them. It's one of the reasons why I enjoy working on xEdit so much, as it keeps me closer to my gaming roots.
For someone who is just dipping their toe into the modding scene and has never heard of xEdit, what does it do and why would they need it, how would you explain it?
I'll be explaining some of the history of xEdit as well, so please excuse me for maybe being a bit long-winded.
When I started playing Oblivion, it was the first time I came in contact with a game that had such an extensive modding scene. I downloaded a whole bunch of mods (mainly from tessource.net which has become NexusMods nowadays), threw them all together semi-randomly and tried to play... to find that many things were suddenly broken and didn't work right. Looking around, I found that there weren't really any tools available to help me understand why that is.
Not having yet encountered a computer-related problem that I couldn't solve by throwing a custom program at it, I started working on TES4View, which today has grown into xEdit.But its roots are in "conflict detection". Loading your complete setup of mods in the same way that the game does, and providing direct visual information about how the different modules conflict and interact with each other. Everyone who uses any mods in a game that xEdit supports should be using xEdit to look for conflicts in their setup, to make informed decisions about load order and necessary patches. There is really no alternative to this. Things like LOOT or information from the author on the mod pages can give you some general information, but at the end of the day, every modded setup is pretty much unique, and it is up to the user to understand how their installed mods interact with each other and to make sure any conflicts are correctly resolved. There are no shortcuts for this. Mods are not “DLC from a third party”. Making them work together will require a certain level of willingness and dedication to learning how they work and interact with each other.
After I initially developed TES4View, I found out that a large number of conflicts between mods were caused by what I then termed "Identical to Master" or ITM records. Basically, every mod available at that time contained anywhere between some and countless records that had nothing to do with the function of the mod and which were identical to the same record in the official game master. By removing these unnecessary records, many mods that were incompatible before could then be used together. As it proved difficult to actually remove these records using the Construction Set (the official editor for Oblivion, which was later called GECK for FO3/FNV, and is called CK (= Creation Kit) for TES5/SSE and FO4), I then started expanding TES4View, which could only be used to view module files, into TES4Edit, to allow directly modifying module files to fix problems.
This brings us to the 2nd major use of xEdit, the process that is now termed "cleaning". The goal of which is to find and remove unnecessary records from a mod to decrease the chance that it conflicts with other mods. This is a process that normally should be done by the mod author before the release of the mod.
Unfortunately, not all mod authors do this, and newer versions of xEdit might get better at detecting ITMs, which, if the mod author doesn't update their mod anymore, means that mod users might want to perform this operation if conflict detection shows that ITMs are causing unnecessary conflicts.
This is not a process that should be done ritually or habitually for every mod. It is no longer the early days of Oblivion modding when there were no tools for it and mod authors weren't aware of it, so every uploaded mod required cleaning. Nowadays, mod users should generally only perform it for mods that conflict detection has shown are causing unnecessary conflicts because of ITMs.TES4Edit quickly grow from just being able to remove ITMs to being able to fully edit all data in existing modules and even creating new modules from scratch.
The initial purpose of this was "conflict resolution" by creating patches, new modules that contained only conflicting records from two other mods, with their contents merged such that both original mods can work together. This is still one of the cornerstones of xEdit functionality for mod users today. While there may be pre-built patches for certain mod combinations available, almost every modded setup is unique, and there is a high chance that it contains combinations of conflicting mods for which no pre-build patches are available. Being able to create your own patches, based on the results of conflict detection, is an important skill that any mod user should have in their toolbox. Finally, once TES4Edit could reliably edit all information in module files, it became possible to "create new mods" from scratch, or make extensive changes to existing mods, using only TES4Edit. This really just builds on the skills developed for creating patches. Many mods on NexusMods today have been completely created in xEdit.Later I used TES4Edit as the basis for creating TES4LODGen, which was the first tool for Oblivion that could generate Tree/Object LOD information for your complete load order.
The software has been maintained by a team of people over its lifetime, could you tell us who the key developers are and how they contribute to the project?
The software that is xEdit today was initially closed source and I was the only developer working on it during the Oblivion, Fallout 3, and early Fallout New Vegas timeframe. If I remember correctly that should be everything up to Version 3.0.19.
After that, when it became clear to me that, because of work commitments and the birth of our twins, I would not have time to adequately maintain xEdit, I decided to open source the program.
Thankfully a group of dedicated people took the job while I went on hiatus, foremost Hlp, Zilav, Sharlikran.
- Sharlikran took over a lot of the organizational work and has been working on decoding record definitions.
- Zilav added the scripting engine and localization support that was necessary to support Skyrim.
- Hlp did a large amount of work to enable save game viewing (which still a work in progress).
- fireundubh contributed greatly to the Fallout 4 record definitions, hardcoded records, and various other additions and fixes.
- Jonathan Ostrus made significant contributions to the Fallout 76 record definitions.
- There are various other minor contributors that can be found on GitHub
All of them have been instrumental in creating the record definitions for Skyrim, Fallout 4, and Skyrim Special Edition.
Sheson joined the project to improve LOD generation in xLODGen and work on DynDOLOD (the program part of which is a fork of xEdit with additional functionality). He has also been instrumental in updating xEdit to enable compilation for 64bit.
During these years I would (very) occasionally find a week or two of time to work on issues and features that required a deeper understanding of the xEdit core than the other project members possessed.
Around mid-2018 I found some spare time (thanks to the twins having started school by then) to come back to xEdit and was planning to do what I did before, spend a week or two on some difficult issues or features and then move on.
But what was planned as some quick fixes and new features has turned into a months-long project, for which I took time off from my usual work, with a huge number of improvements and fixes to xEdit that culminated in the release of xEdit 4 in mid-December.
All development work on xEdit since mid-2018 has been done by me, but Sharlikran has continued to provide valuable organizational work and updating the online documentation that is now reachable through the Help button in the top right corner of the xEdit main form. Zilav and Sheson have provided valuable feedback.
You’ve recently released a significant milestone release with xEdit (v4.0), what would you say are the headline improvements in the update?
The most visible improvement is probably support for UI Themes, both light and dark ones, as well as theme colour awareness of the conflict colours (a dark theme with lighter text on darker background will make xEdit automatically use a darker background and a lighter font colour to show the different conflict statuses).
A very important addition is full ESL support. This means that xEdit now supports loading modules in exactly the same way as the game engine, with ESL flagged ones being mapped into the FExxx space. Without this improvement, valid SSE and FO4 setups that had over 255 modules in total couldn't be loaded into xEdit. Now, if it's valid for the game, it's valid for xEdit.To go hand in hand with the addition of ESL support, the complete load order handling in xEdit has been overhauled and will now behave identically to the game engine for all supported games.
The module selection dialog, used both during program start and at various times while using xEdit has been completely reworked, it's much more informative, has support for saving and loading presets, and filtering.
Language and codepage support in xEdit has been completely reworked. Before, xEdit simply used the system ANSI code page for most strings. Now, xEdit is very explicit about what code page is used for which strings and there are various ways to configure this.
Previous xEdit versions were slow to start as they used to build the information required for the "Referenced By" tabs on every launch. They also didn't build reference information for the game master file automatically, which had to be done explicitly after the program was running if desired. Now, the second and subsequent starts after updating to a new version of xEdit will be much faster, as the required information is read in from a persistent cache on disk instead of having to be built on each launch. On the first launch after an update, this reference information is built once and then saved to disk in a cache. While this first launch is slower, it is still faster than launching the previous version and then building the reference information for the game master, as this building process will now run using a multi-threaded approach.
The ModGroups functionality has been completely reworked. This is a centrally important feature of xEdit, which has been present for over a decade, about which, hardly anyone was aware. It completely changes how you go about looking for conflicts with xEdit. This feature makes conflict detection and resolution even with gigantic load orders a breeze. ModGroups are, at their most basic, a way to tell xEdit, "if this set of modules is loaded in this order, then any conflict between them has been resolved and you need to consider only the winning override for each record". In previous versions, it was necessary to use a text editor to create .modgroups files outside of xEdit, which almost nobody did. In this version, ModGroup support is fully implemented inside xEdit, with ways to create, edit, and delete ModGroups right in xEdit, as well as the ability to reload a different selection of ModGroups at any time.
Aligning alignable array elements is one of these features that hardly anyone notices when it's added because it just feels so natural and right, but if you remove it after someone had just a few minutes to get used to it, it's instantly and painfully noticeable. There are a number of arrays where the order is important, one of the primary ones is lists of conditions.
This is best explained in a short example:
Without this feature, a record with 2 overrides looked like this:
A B B
B C A
C
With alignment active, it will now look like this:
A
B B B
A
C C
The ability to filter in the View tab and to collapse nodes in the view tab hugely improves working with large records.
Two other new features go hand in hand with each other. One is the ability to create a delta patch. Given two versions of a module, xEdit can create a patch module that contains only exactly the records that are different between these two modules. The other is the ability to take a module (like such a delta patch, but also a plugin created in CK by editing records that already exist in a master) and merge this module into its master. Given the original version of a module and a delta patch, you can now re-create the 2nd version of that file by merging in the patch. This function can also replace the “merge into master” functionality of CK which normally requires the use of VC (version control).In addition to these big-ticket items above, there are hundreds of other smaller improvements and bug fixes all over the place.
On top of all this, which affects every existing game-specific version of xEdit, there is now full support for Fallout 76 in the form of FO76Edit.
When Bethesda releases a new game, what is the process you go through in order to make xEdit compatible with the new plugins?
This is a rather involved multi-step process which is helped along by a number of normally hidden functions in the xEdit core to analyse the structure of module files. It is usually possible to use the record definitions of an existing game as the basis.
The first step is to analyse the gross structure of the new game master, what record signatures and groups are contained in the file in what order?
Once that has been established and the basic record definitions updated or created, each record type (there are hundreds in the most current games) is analysed on its own to understand what subrecords it can contain, and how these subrecords are structured. This can get rather complicated, with arrays and structs of subrecords, sometimes nested multiple levels deep.
The next step is to analyse the contents of each sub record. Is there only a single value? Or a complex structure? Most locations where FormID, strings, or floats are stored can be found with the right analysis tools. This doesn't tell you what these values are used for or named. It only helps to identify what type of value is stored where. It is especially important to find any and all places that can contain FormIDs in this step. Without knowing the location of every single FormID, it is impossible for xEdit to add, remove, or sort masters for a file.
Up to this point, the availability of an official tool like CK is irrelevant. The process described above was followed to generate the record definitions for FO76Edit (starting from the record definitions for FO4 and modifying/extending them where analysis found discrepancies) and took about 150 man hours in total.
What follows after this depends on the availability of an official editor for the module files.
If no such tool is available, like currently in the case of FO76, it is still possible to try and make informed guesses about what the likely labels for some of these unknown new or changed values are, but there is a lot of guesswork involved.
If an official editor is available, it's possible to use that to create test plugins by putting unique values into each field visible in the editor for some record type, then load that test plugin into xEdit (or a number of other internal tools) and try to match up the information in the official editor with what has been written into the file. This is a very involved and repetitive process that can take hundreds of hours to further refine the record definitions.
Perhaps the only noticeable gap in xEdit’s library of supported games is The Elder Scrolls III: Morrowind - is there a technical reason why it hasn’t been supported?
There is indeed a technical reason for that. xEdit was originally developed for TES4: Oblivion. Oblivion was the first game for which Bethesda used FormIDs to identify individual records, and the fundamental design of xEdit is based on the assumption that records have FormIDs that identify them. This has been the case for any game that followed Oblivion so far.
Morrowind does not use FormIDs and has a much less "generalized" overall structure to its module files compared to later games. This makes it much harder to support Morrowind within the existing framework xEdit has established to support all later games.
I have recently been working on finding ways to support Morrowind in xEdit. It's very much a case of trying to get a square peg through a round hole. But I've been making progress by following an approach where I synthesize virtual FormIDs for records on load.
This will require a lot more work and time, but I believe TES3Edit is an achievable goal.
Have you ever wanted to make a tool like this for a different game engine?
There are a number of other modable games for which I would love to have a tool like xEdit, and I've looked into the data format for these games. Unfortunately, none of them had a data format that would make adapting xEdit for them something that could be achieved with reasonable effort, if at all.
I do have to congratulate Bethesda, the basic structure of their module files is much more suited for supporting setups with multiple mods than anything I've found in other games.
Given unlimited resources, what features would you like to bring to xEdit?
Beyond Morrowind support, which I'm currently working on, there are a number of other big items I would like to address in the future.
The first one is to modularize the main form. That's what you see when you start xEdit, with the navigation tree view on the left and the tabs on the right. All of this is currently one giant strongly interconnected whole which has grown over the last decade. The goal here is to separate it all into distinct, isolated modules (the navigation tree view, the view tab, the spreadsheet tabs, the different functions that can be executed). This involves a large amount of work, and will not, initially, result in a significant change from the user's point of view. But it's really the required foundation for a lot of future work.
Once the main form is modularized, it's easier to build on that and support multiple navigation tree views (each one can be differently filtered) and views (showing different records concurrently).
This will also enable quality of life improvements like a scratch pad (an area where you can drag in anything from any record, be it a single value or a whole struct or array, then later use this as a drag source to copy into any assignment compatible place in other records).
The next big feature would be to support assets at the same level as modules are currently supported. The navigation tree view would list the .bsa/.ba2 files, as well as loose files folders in the correct order. You can open them up to explore the contained folders and files. When a file is selected, the view tab will show the different file versions, with information extracted appropriately based on file type, side by side, the same way that records are shown side by side. This should also include the ability to preview textures, possible models and other file types.
After that would be graphical visualization of path grids, navmeshes, and general CELL contents.
Outside of these really big items, there are countless more minor improvements that are possible, among them various quality of life improvements and improved error checking for navmeshes and other complex records.
Obviously, whenever Bethesda releases any new game based on a variation of the current module format, implementing support for that would have the priority.
I’m sure a large number of modders who’ve used xEdit would like to know, what is the best way to support the development of xEdit?
The primary means of support would be to pledge on my Patreon as this gives a somewhat predictable monthly income which would allow me to make long term plans in regards to how much time I can spend on xEdit development in the future.
Another possibility is direct donations via PayPal, either using the donation function here on NexusMods (which will forward you to my PayPal, Nexus Mods is not involved in the processing of it) or directly. Both of these are currently easily reachable via buttons in the top right corner of the xEdit main form.
The last possibility is Ko-fi, which right now is simply another way to be forwarded to PayPal for a one-off donation so it doesn’t currently provide any benefit over making a direct PayPal donation. Please keep in mind that no matter what way you use to support xEdit development, the lower the amount, the higher the relative cut the payment processor takes. This is especially noticeable for very small amounts. If you donate $1, only about half of it will make it to me.
The huge amount of progress that has been made in xEdit 4 has only been possible by investing many hundreds of hours of development work, which, for me, came at a significant cost of lost income from my usual work. If I had purely worked on it in my spare time, xEdit 4 would still be 2-3 years in the future.
I obviously can't sustain ongoing xEdit development at a significant financial loss permanently, so the speed of progress for future xEdit development depends directly on the level of support I can achieve from the modding community.
At the time I'm writing this, current pledges on Patreon and average direct donations on PayPal together can finance roughly 1/8th of a month of full-time work on xEdit.
Is there anything else you’d like to say to the Nexus Mods community?
I've greatly enjoyed my work on xEdit and interactions with many members of the modding community. I'll try to answer any questions that come up in the comments of this interview. I am usually available on the xEdit Discord server (also reachable via the Discord button in the top right corner in the xEdit main form).
A big thank you to ElminsterAU for answering our questions. If there's an author or mod project you'd like to know more about, send your suggestions to BigBizkit or Pickysaurus.
37 comments
Comments locked
A moderator has closed this comment topic for the time beingThank you for everything Elm, Zilav & team! Very much appreciate all the coding lessons, all the time, work and patience. It's been awesome to get to know you guys :-D
-KK
This editor supports my relaxation away from my own children at times to re-energize before I head back into the fray! :)