RCT Discussion / Download ALL nedesigns RCT2 objects with this Python code

Hey everyone so I'm kind of a completionist and I wanted to have all custom RCT2 objects that are uploaded to this website but there's no "download all" button which is really annoying, you have to download them all one by one. So I got together with my friend who's a *little* more fluent with Python than I and we wrote the following code which goes through every page and downloads every downloadable object to a folder. The Python code is written to the text file I attached, just download it, OPEN IT AND READ THE COMMENTS and put in your info, then run it. It will take a while!!! I suggest disabling your computer's sleep timer and leaving it on overnight. It will download something between 10,000 and 12,000 files, totaling somewhere in the range of 500MB - 1GB (but it has to load the web page and locate the file every time).

Also I strongly recommend using OpenRCT2 for this!! It seems that some of the objects can't be loaded in vanilla RCT2 (even with expansions) and may cause the launcher to crash.

For those out there that aren't familiar with Python, you will need to download it here: https://www.python.org/downloads/. Then I recommend getting an editor, I like this one: https://pyzo.org. You will need to install the packages mechanicalsoup, requests, and re (use the command "pip install re" etc in the command line within the editor). Then copy-paste the text to the editor, make the necessary adjustments, save it as something with a .py extension and run it, and that's it! If it doesn't work, you can try this: https://www.pythoncentral.io/add-python-to-path-python-is-not-recognized-as-an-internal-or-external-command/. If you encounter any further problems please try to troubleshoot them on your own, I don't want to write a whole getting-started-with-python tutorial here.

Hope that helps! Enjoy your boatload of new RCT2 objects!

This is a great idea and thank you for sharing the code. one problem however (no fault of your own) is that there are several duplicate .dat names. i'm unsure if its even possible to work around that

Even if the names are identical, the contents aren't. They're likely neglectable though. Just, the system creates MD5 hashes that work like fingerprints for all objects, and creates new entries for what it doesn't have yet, even if the object name is the same.

^

The RCT objects has 193 pages with a total of 12,338 objects. The other pages list the original dat's that came with the game and expansion packs. Hope this helps.

I'm trying to reconfigure this file for Mac (for use with OpenRCT2). Thankfully, when I initially ran the script, it failed immediately because it couldn't find 'mechanicalsoup.' On second glance, I realized everything was arranged for Windows only.

Here's the GitHub repository for MechanicalSoup.

I've gotten mechanicalsoup, now, but it still doesn't recognize it as a module in Pyzo, but it works in IDLE.

Also, I upgraded it a little bit by adding a feature to eliminate duplicates by preserving only the most recently uploaded downloadable version of an object under each filename. Will post soon once I finish it.

All finished! I have ported the code for use in macOS (Mojave, version 10.14.6) and run it successfully using IDLE!

Some important things to note:

• Pyzo won't work since it doesn't support virtual environments ('venv'), so it can't detect MechanicalSoup.

• Executing the code still requires that you have the MechanicalSoup module installed. See the installation instructions on its GitHub repository if you're unsure how to do this. (Both Python3 and MechanicalSoup can be installed from Terminal or Command Line using HomeBrew.)

• I estimate a download speed of roughly one minute per page, so don't sit around waiting for it to finish! This process will demand a lot from your CPU (I've got an Intel Core i9 with 32GB of RAM 2.4 GHz, and it still ran hot), so don't attempt any of this if you're not confident in its stability.

• Changing the '.txt' to '.py' is no longer necessary– unless you want to see the coloration and dissect the code. Don't change any of the coding if you don't know what you're doing! (I also don't recommend running this from Windows unless you're somewhat familiar with Python.)

If you want to attempt this but don't have Python, follow this procedure to obtain and use it for this task:
Step 1. Install Python (v3 or later) from Python's Downloads directory. IDLE automatically installs with Python.

Step 2. Launch IDLE, then in the menu bar, select: File > Open

Step 3. Navigate to OpenRCT2's 'object' folder* and locate rct2objscrapermac.txt** (see attached file).

Step 4. In the menu bar, select: Run > Run Module

*You should already know where to find it if you're willing to download ~355 pages of downloadable objects.

**You should save this file either in the 'object' folder, or in a newly created folder within the 'object' folder. The objects will save to the same location as this document is located when you run it, and any existing objects will be overwritten if they share the same name as one downloaded through this process.

THANK YOU FOR MAKING THIS PROCESS SO MUCH FASTER FOR US!! I couldn't have done it without your help (or your friend's contributions, or the help team's assistance from the Python Discord server)!

If this ever gets outdated, reach out to me however you can!

Wow, glad this is still helping people almost 3 years later! Thanks OPStellar for your troubleshooting notes for anyone else who has having difficulty with it. I mentioned that you need to "use the command 'pip install re' etc" and by "etc" I meant that you also need to run "pip install requests" and "pip install mechanicalsoup," that should be the easiest way to install those packages. If you're using Anaconda, then I suppose you would want to try running those 3 commands but with "conda" instead of "pip." Also, looks like you had to change the directory based on where Mac installs RCT, but you got that handled.

I haven't played RCT in a while nor used mechanicalsoup so I'm grateful you're willing to help whatever other crazy people out there want to download every single RCT mod ever made since I probably wouldn't be able to figure out what I originally wrote or how it was supposed to work.

I mentioned that you need to "use the command 'pip install re' etc" and by "etc" I meant that you also need to run "pip install requests" and "pip install mechanicalsoup," that should be the easiest way to install those packages. If you're using Anaconda, then I suppose you would want to try running those 3 commands but with "conda" instead of "pip."

I'm glad you dipped back in to view the achievement! Again, thank you for making this possible! My next project will be porting the Game Utilities I've gathered… eh, maybe after I finish this semester in graduate school. XD

The RCT objects has 193 pages with a total of 12,338 objects. The other pages list the original dat's that came with the game and expansion packs. Hope this helps.

I have now completed the download. 917.9 MB, 13,304 items (items #2434-27696; several get skipped in the middle, clearly), 355 pages (432 are displayed, but the first 33 pages are native objects, as are all of the objects past page 388), 3:21-8:36PM EST (about 5 hours and 15 minutes). Largest object is 5.3 MB, smallest is 214 bytes.

Why would you go through the trouble to do this? What is the goal of downloading them all?

 

Why would you go through the trouble to do this? What is the goal of downloading them all?

 

To never see this again when selecting a scenery group:

https://pasteboard.co/3z8eMRTnWETx.png

Unfortunately, it still happens.

Also, there's apparently so many parts that they won't all show up in the small scenery selection list (it cuts off in the middle of the D's, alphabetically)! Not only that, but not all objects will even load, either! There's a cap to how many custom objects can be imported, it seems; scenery objects I know I've seen before are not showing up even when I search, namely those that started with 'zz'– so they just exceed the maximum that OpenRCT2 can handle!

Also, I have a statistical update:

534.4 MB (3,182 objects) of the downloads were 404 error pages  database of missing data.txt (41.82KB)
downloads: 285

Therefore, only ~383.5 MB (10,122 .DAT objects) are valid objects. I'm getting mixed results since I combined it all with existing data… so at this time, you will probably want to make sure that the final page in the range is 388 and not 432. I think I made the change already (I started the process before making the change, initially), but I'm saying it anyway just in case.

I believe that notification is a result of the limit on scenery objects within the game, 252 small scenery, 128 large, 128 walls etc...  Not a lack of having all the .dats under that tab.  But maybe I'm mistaking here somewhere.

I believe that notification is a result of the limit on scenery objects within the game, 252 small scenery, 128 large, 128 walls etc...  Not a lack of having all the .dats under that tab.  But maybe I'm mistaking here somewhere.

That's correct, it also happens in Vanilla+Expansions where some of the expansion pack scenery groups were made too large. 

"…limit on scenery objects within the game, 252 small scenery, 128 large, 128 walls etc...  Not a lack of having all the .dats under that tab."

That's correct, it also happens in Vanilla+Expansions where some of the expansion pack scenery groups were made too large. 

It was happening for every pack, even with all other objects deselected.

Anyway, I just went to the extent of getting this thing upgraded threefold!

The next renditions resolve several shortcomings of the old code:

1. Objects with duplicate filenames are placed in a separate folder

2. All other objects are placed in one folder

3. STANDALONE EXECUTABLE! No more need for Python savvy!

4. MACOS + PC VERSIONS! Plus source code for Linux!

I'll make a new post later once I get it all properly uploaded to GitHub.

Okay, so I finally got it working again. This time, it renames everything slightly. OpenRCT2 is still able to recognize it, regardless.

Current stats:

21,865 files, 837.1 MB

That's not counting the collection of ~168 KB error pages, most of which get sent to the 'Dupes' folder. Still gotta figure out some convenience features. I'm still not going to release it until I've successfully made it a standalone, because it now requires a few more Python extensions.

However, for those who are Python-savvy (and ONLY those who are Python-savvy enough to know what extensions to install and how, or how to find out that info without instruction), here's the script:

 NEDDb_NSF.txt (5.09KB)
downloads: 170

Be sure to change the end to '.py' before running it (idk how it will behave if you don't).

Note: I used Python-3.8, so that version or later should work. I managed to download up to 50 at a time. For now, will look like nothing is happening for a while, so run it in the background and don't put your machine to sleep while this process runs.