Arty - Digital Art Assisstant by Team The Ryuto 6

Arty is a productivity tool specifically tailored for digital 2D art illustrators. It's a platform that allows artists to automate frequent, repetitive, and time-consuming tasks like reference searching, idea organisation and visualisation, while ensuring the creative process remains entirely in artist’s control.

Link to our website hosted on UQ-Zone (UQ credentials needed): https://arty.uqcloud.net/

*Generative AI and Image searching may not work as our cloud GPU isn't always online.

Main features

Get a better idea of how our platform works from our explainer video here!

• Cross-Platform Compatibility
• Integrated Reference Searching
• Generative AI for Visualisation
• Cross-Device History Syncing
• Side-by-Side Panel

Contributors

Table of Contents

• Arty - Digital Art Assisstant by Team The Ryuto 6
  • Main features
  • Contributors
• Table of Contents
  • Release History
  • Local Installation and Running
    • Front-end
      • Installation
      • Running
    • Flask Server (Back-end)
      • Installation
      • Running
    • Stable Diffusion Backbone (Back-end)
      • Installation
      • Running
  • Screenshots and Usage Example
  • References
  • Prompts

Release History

• 1.0.0 Final Codebase Realse << Current Version

  • Revamped readme structure and content
  • Improved security measures
  • Improved front-end visual design and feedback

• 0.4.0 Interim Submission V4

  • Added bookmarking for searched/generated images
  • Added support for loading more images
  • Fixed incorrect API calls

• 0.3.0 Interim Submission V3

  • Improved file structure
  • Improved content in readme
  • Implemented Flask back-end server with database access
  • Added about page for displaying user info and history
  • Added inpaint canvas in generator page
  • Added inpaint with controlnet support in stable diffusion backbone

• 0.2.0 Interim Submission V2

  • Added Arty icon
  • Added stable diffusion backbone code
  • Added image searching support for search engine
  • Improved front-end page design

• 0.1.0 Interim Submission V1

  • Added ControlNet test code
  • Initial version of seach engine
  • Initial version of front-end

• 0.0.0 Repository Creation

  • Initial readme written

Local Installation and Running

Front-end

Installation

1. Download and install Node.js (a JavaScript runtime environment) from their website.
2. Start a new terminal instance from the project root directory, run cd frontend to navigate to frontend's folder.
3. Automatically install the dependencies needed by running npm install.

Running

1. Start a new terminal instance from the project root directory, run cd frontend to navigate to frontend's folder.
2. Run npm start, the front-end server will be created at http://localhost:3000.

*For more details, please refer to the front-end README.

Flask Server (Back-end)

Installation

Python environment configuration:

Note: The list of dependencies used can be found in backend/requirements.txt.

1. Download and install Conda (a Python package/environment management system) from their website.

2. Create a conda virtual environment and install the dependencies by running the following command in your terminal:

   conda create --name arty_backend python=3.12
   

3. Activate the conda virtual environment by running conda activate arty_backend.

4. Install Pytorch from their website.

5. Install dependencies by running:

   pip install -r backend/requirements.txt
   

Environment variable setup:

1. Navigate to /backend folder, create a file named .env.

2. Open the .env file with a text editor, insert:

   DB_LINK = <private link to our database>
   

Running

1. Make an SSH tunnel to UQ-Zone by entering the command:

   ssh -L 3306:localhost:3306 -J <your_uq_account>@remote.labs.eait.uq.edu.au <your_uq_account>@arty.zones.eait.uq.edu.au
   

   Make sure to keep this terminal open. This connects to the mysql database run on UQ-Zone as it can be ea pain to set up locally. If you need to test the database and don't have access, please contact Jason as he has been granted access and should be able to add other users.

   *If this does not work you may have to set your ssh key to the EAIT Remote Desktop first!

2. Create a new terminal at the root directory, activate the conda virtual environment by running conda activate arty_backend.

3. From the same terminal, run:

   cd backend
   python app.py
   

   and keep this terminal open.

4. Now you are set up to run tests for the codebase. Feel free to tweek around the codes here and there.

*For more details, please refer to the back-end README.

Stable Diffusion Backbone (Back-end)

Installation

Download models here:

• Stable Diffusion 1.5 (v1-5-pruned.ckpt)
• SD1.5 ControlNet models (all of them)

Installing stable-diffusion-webui:

Note: The controlnet folder contains a copy of stable-diffusion-webui with the ControlNet extension pre-installed. If you encounter any issues during installation following the steps below, please refer to the original installation guide for stable-diffusion-webui and sd-webui-controlnet.

1. Run webui.bat under /backend/controlnet folder.

   *The first initialisation will fail as it doesn't have access to the model weights.

2. Copy v1-5-pruned.ckpt to /backend/controlnet/models/Stable-diffusion, copy the ControlNet model weights to /backend/controlnet/models/ControlNet.

3. Run webui.bat again, and the stable diffusion backbone server will start.

Running

Run webui.bat under /backend/controlnet folder to start the stable diffusion backbone server.

*For more details, please refer to the controlnet README.

Screenshots and Usage Example

The product is to be used as a side assisstant for artists to search for artwork references.

Screenshots

Side-by-Side Usage Example

References

1. The BLIP Model:
   • Hugging Face: BLIP
2. ControlNet Extension for Stable Diffusion Web UI:
   • Mikubill ControlNet GitHub Repository
3. Flask Documentation:
   • Flask Official Documentation
4. Flask-SQLAlchemy Documentation:
   • Flask-SQLAlchemy Official Documentation
5. Printerest:
   • Pinterest Website
6. React 18.3.1:
   • React Documentation
7. Stable Diffusion v1.5:
   • Hugging Face: Stable Diffusion v1.5
8. Stable Diffusion Web UI:
   • AUTOMATIC1111 GitHub Repository
9. Button:
   • Animation Bro
10. Icon:
    • FlatIcon

Prompts

1. Inpainting Canvas:
   • Help me create a inpainting canvas.
2. Comments in CSS files:
   • Help me add comments in these CSS files.
3. Team member Profile:
   • Can you turn these profile images into oil painting effect?