wmlscope is sooooo slow
Moderator: Forum Moderators
- Atreides
- Posts: 1079
- Joined: March 30th, 2019, 10:38 pm
- Location: On the 2nd story of the centre village of Merwuerdigliebe turning the lights on and off
wmlscope is sooooo slow
OK, cheesy title aside has anyone noticed that wmlscope is much slower since 1.14?
My 1.12 wmlscope run from the GUI (includes core as well as whatever you choose) generally takes about 30 seconds with no extras picked.
1.14/16 it takes 10 times as long! 5 minutes.
Is it just me? I have to confess that it might be, just possibly. 1.12 is a debian installed package while 1.14+ is all flatpak (since 1.12.6 is the very newest as far as debian 9 (yes I know there's 10, I'll risk upgrading when I'm forced to) is concerned hehe) but surely the directoryies they are in is the only real difference for the core files... I really doubt that could be it unless wmlscope has some kind of hang up with the length of the pathname (the flatpaks tend to be in a longer path)...
Anyways, just airing my discontent and curious if anyone else has can relate...
My 1.12 wmlscope run from the GUI (includes core as well as whatever you choose) generally takes about 30 seconds with no extras picked.
1.14/16 it takes 10 times as long! 5 minutes.
Is it just me? I have to confess that it might be, just possibly. 1.12 is a debian installed package while 1.14+ is all flatpak (since 1.12.6 is the very newest as far as debian 9 (yes I know there's 10, I'll risk upgrading when I'm forced to) is concerned hehe) but surely the directoryies they are in is the only real difference for the core files... I really doubt that could be it unless wmlscope has some kind of hang up with the length of the pathname (the flatpaks tend to be in a longer path)...
Anyways, just airing my discontent and curious if anyone else has can relate...
- Elvish_Hunter
- Posts: 1580
- Joined: September 4th, 2009, 2:39 pm
- Location: Lintanir Forest...
Re: wmlscope is sooooo slow
Moved into the Coder's Corner.
There are a lot of possible reasons for this slowness. I must admit that I haven't been running the 1.12 version for a long time, so I don't have a reference about this.
First of all, one of the reasons is that we have more content in mainline: more macros, more images and more units mean more things that wmlscope needs to keep track of, which makes it slower. There isn't much we can do, except cutting content from mainline - but nobody wants to do that, right?
The second reason is directly depending on the porting to Python 3: one of the things that went lost was Python's
The third reason depends on the square braces notation that we use now to define animations: when things were simpler, each image path could point only to a specific file. This isn't the case any more: a path can now point to several sprites and I had to actually write in Python the square braces expansion code, because the checks for missing files had become useless.
And then I also had to add other features, like support for optional macro arguments (back then we didn't have them!), detection of deprecated macros (which I'm thinking to use to output a warning when a call to them is found)...
So, you get the idea. wmlscope back then was faster because it didn't had to handle several features that are part of current WML and also because it wasn't very much maintained. Can it be made faster? Probably yes, but it isn't easy to do so.
There are a lot of possible reasons for this slowness. I must admit that I haven't been running the 1.12 version for a long time, so I don't have a reference about this.
First of all, one of the reasons is that we have more content in mainline: more macros, more images and more units mean more things that wmlscope needs to keep track of, which makes it slower. There isn't much we can do, except cutting content from mainline - but nobody wants to do that, right?
The second reason is directly depending on the porting to Python 3: one of the things that went lost was Python's
cmp()
function, and to have a certain part of the code correctly sorting items, I had to use the total_ordering
module, which is slower than writing all the six comparison functions by hand (although it didn't seem really slower to me).The third reason depends on the square braces notation that we use now to define animations: when things were simpler, each image path could point only to a specific file. This isn't the case any more: a path can now point to several sprites and I had to actually write in Python the square braces expansion code, because the checks for missing files had become useless.
And then I also had to add other features, like support for optional macro arguments (back then we didn't have them!), detection of deprecated macros (which I'm thinking to use to output a warning when a call to them is found)...
So, you get the idea. wmlscope back then was faster because it didn't had to handle several features that are part of current WML and also because it wasn't very much maintained. Can it be made faster? Probably yes, but it isn't easy to do so.
Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
Re: wmlscope is sooooo slow
There's also Debian 11 now too.
I think just upgrading to Debian 10 or 11 would probably improve performance. Debian 9 has a version of Python 3 that is several years old (Python 3.5) and, as a general rule, early versions of Python 3 were incredibly slow. This has improved considerably in recent releases.
Here is a test I ran on a machine with several different versions of Python 3 installed (this is a Windows machine so the results might be considerably different on Linux, but I think it gives an idea how much Python 3 performance has improved):
Code: Select all
C:\Users\U\AppData\Local\Programs\Python\Python35\python.exe --version
Python 3.5.4
C:\Users\U\AppData\Local\Programs\Python\Python35\python.exe data/tools/wmlscope data/core
...
#Cross-reference time: 97 seconds
C:\Users\U\AppData\Local\Programs\Python\Python37\python.exe --version
Python 3.7.9
C:\Users\U\AppData\Local\Programs\Python\Python37\python.exe data/tools/wmlscope data/core
...
#Cross-reference time: 74 seconds
C:\Users\U\AppData\Local\Programs\Python\Python39\python.exe --version
Python 3.9.10
C:\Users\U\AppData\Local\Programs\Python\Python39\python.exe data/tools/wmlscope data/core
...
#Cross-reference time: 67 seconds
- Atreides
- Posts: 1079
- Joined: March 30th, 2019, 10:38 pm
- Location: On the 2nd story of the centre village of Merwuerdigliebe turning the lights on and off
Re: wmlscope is sooooo slow
Thanks for the replies. Interesting stuff! I suspect that the rewrite for 1.14 is the likely culprit. Amazing to hear how much had to be changed. I'd like to move to Debian 10 (and will be "forced" to when they stop updating 9 this summer (I think)) but it sounded very daunting when I read about it.
One thing I do that helps a lot is to run wmlscope from the command line just on the addon dir (not including the core, that I also avoid with wmllint as it too is super slow when the core gets checked).
One thing I do that helps a lot is to run wmlscope from the command line just on the addon dir (not including the core, that I also avoid with wmllint as it too is super slow when the core gets checked).
Re: wmlscope is sooooo slow
There's a thing in Python where you make use of multiprocessing/multithreading. wmlunits uses this that's why it's fast. But wmlunits will not run on windows,
not sure if it might be tricky to make multiprocessing/multithreading be cross-platform?
Also, it can be tricky to write this code to run on all devices, for example you would need to detect the number of cores/threads on a computer so that the script knows how many it can use.
not sure if it might be tricky to make multiprocessing/multithreading be cross-platform?
Also, it can be tricky to write this code to run on all devices, for example you would need to detect the number of cores/threads on a computer so that the script knows how many it can use.
- Atreides
- Posts: 1079
- Joined: March 30th, 2019, 10:38 pm
- Location: On the 2nd story of the centre village of Merwuerdigliebe turning the lights on and off
Re: wmlscope is sooooo slow
Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds. Not a very accurate comparison, nor sure I used same parameters or add-on but the difference is large enough to make that moot. Man... feels good to be back at the command line.
Re: wmlscope is sooooo slow
Check if the passed arguments to wmlscope when you ran it in the command line are the same as the arguments that were passed to wmlscope when you ran it in the GUI version.Atreides wrote: ↑February 13th, 2022, 4:17 pm Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds. Not a very accurate comparison, nor sure I used same parameters or add-on but the difference is large enough to make that moot. Man... feels good to be back at the command line.
- Atreides
- Posts: 1079
- Joined: March 30th, 2019, 10:38 pm
- Location: On the 2nd story of the centre village of Merwuerdigliebe turning the lights on and off
Re: wmlscope is sooooo slow
Mmm yes I'd thought of that, but didn't think about it in this context, obviously they could account for a difference. Thanks for pointing it out.max_torch wrote: ↑February 26th, 2022, 6:24 pmCheck if the passed arguments to wmlscope when you ran it in the command line are the same as the arguments that were passed to wmlscope when you ran it in the GUI version.Atreides wrote: ↑February 13th, 2022, 4:17 pm Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds. Not a very accurate comparison, nor sure I used same parameters or add-on but the difference is large enough to make that moot. Man... feels good to be back at the command line.
- Elvish_Hunter
- Posts: 1580
- Joined: September 4th, 2009, 2:39 pm
- Location: Lintanir Forest...
Re: wmlscope is sooooo slow
However, multiprocessing can't be really used in this case because wmlscope relies on the files being processed in a certain order (and so does wmllint). For multithreading the situation is even worse because of the GIL (global interpreter lock), which makes sure that all the Python code runs only one thread at any given time; since our Python scripts already seem to take one whole core for their computations, there wouldn't be any benefit.
It's surprising and yet it isn't at the same time. I mean, if you think about it, when using Tkinter Python needs to call an interpreter of another language, which is Tcl/Tk. If you need to run it a lot of times, using the command line and excluding all the GUI stuff is a good idea.Atreides wrote: ↑February 13th, 2022, 4:17 pm Oh, I should mention that I tried the command line (finally learned about those weird symbolic links - we never had those in MS-DOS) version of wmlscope and while the gui version takes 300 seconds to scan core plus a addon the command line set to scan the same takes about 80 seconds.
BTW, since commits 21bd75c (master) and 0da1ea4 (1.16) wmlscope is capable of reporting usages of deprecated macros, for example like
{SPECIAL_NOTES}
and {FOREACH} {NEXT}
.Current maintainer of these add-ons, all on 1.16:
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)
The Sojournings of Grog, Children of Dragons, A Rough Life, Wesnoth Lua Pack, The White Troll (co-author)