So I have finally finished a proof-of concept after several frustrating hours:
Turns out the documentation is wrong for the range color 1 index is:
160~191
Color2:
192~223
And added fun is that your preview has to have color 1 & 2 ranges SWAPPED! ugh.
Current photoshop+wally method I have:
1.) get your dm_base image and isolate the 2 areas of color you want changable
2.) copy the 2 selection and paste them into 2 different documents with the exact same dimentions as your original (using paste special > paste in place). Also save the selections in each document as an alpha channel. Make a background layer of black or the darkest color of your image.
3.) In your original, load the selections for the 2 areas and fill them with black
4.) Then on your original set to indexed colors but limit the amount of colors to 160. This will build the first 160 colors. Save a BMP copy out named "base"
5.) In your copied color 1 document set it to indexed but have 32 colors max save a copy out as "color1"
6.) Do the same for color 2
7.) Now open wally (because photoshop cannot save simple text-editable pallete format) and open the 3. Select each image and go to edit pallete and go to save pallete. Do it for all 3 so now you have a base.pal, color1.pal, color2.pal
8.) Now the tricky part... Open all 3 .pal files in notepad++ (I like it because it has lines numbers).
9.) Go to color1.pal and select all the numbers in line 4 to 35 and copy.
10.) Go to base.pal and paste these into lines line 164 to 195.
11.) Go to color2.pal and select all the numbers in line 4 to 35 and copy.
12.) Go to base.pal and paste these into lines line 196 to 227.
13.) check the end of the document, there shoudl be a bunch of lines that say "0 0 0" and the last "0 0 0" should be at 259. There should be a blank space on line 260. If there is no line 260 or if the document ends at 261 delete the line. This is important or else the pallete file will corrupt. Save the pallete as "fixed.pal"
14)Open base.bmp in wally and go to load pallete and load "fixed.pal" Then go to save the image as "fixed.bmp"
15) Go back to photoshop and go open "fixed.bmp". Go to both "color1.bmp" and "color2.bmp" which you should still have open in photoshop, and select the alpha channel. Copy paste special > paste in place "fixed.bmp" for both color1 & color parts.
16) Final step is go to image size then go uncheck "resample" and change the resolution from 96 to 72dpi (for some stupid reason wally saves as 96 which breaks all our hard work). Then save as "dm_base.bmp". Now compile your model.
Thumbnail image method:
1.) Open fixed.pal, then swap the contents of lines 164 to 195 with 196 to 227. You may need to copy out to another text document to do this. Save as "fixedthumb.pal"
2.) Open dm_base.bmp in wally and load "fixedthumb.pal" Save as "thumbnail.bmp", then save out the pallette as "fixedthumb.ACT"
3.) Open thumbnail.bmp and set mode to RBG color (this is for resizeing purposes.) Resize to 160 X 200 @ 72dpi. Then go to mode > indexed > custom pallete and load "fixedthumb.act". Save your image as the same name as your model in the model folder of your playermodel.
Holy shit that took a while, I will have to make a video because this is confusing as shit. Not as fast as I wanted but it works and takes advantage of 32 colors on each range properly with minimal waste. I still cannot find a reliable pallete swapping program just yet so wally+notepad has to do for now.
EDIT: Ive found a better method using tilesetpal than text editing and I can cut wally out of the equation and add another 32 colors to the base image. Will do a rewrite tomorrow or later.