Exploring the Potential of Novel Image-to-Text Generators as Prompt Engineers for CivitAI Models

Individuals looking to utilize AI image generation technologies to aid in creative content generation, such as anime or webtoon artists, may struggle to use new state-of-the-art text-to-image generators due to the lack of familiarity of prompt engineering. This paper explores the possibility of using models that have inherent image-to-text capacities like CLIP (ViT-L), CLIP (ViT-H), DeepDanbooru, GPT-4, and Gemini 1.5 pro, to automatically generate prompts that produce high-quality single-character images, which will help to streamline the creative content process for character image production during the character ideation phase. We employed image evaluation metrics like CLIP image-to-image (CLIPI-I), CLIP text-to-image (CLIPT-I), Contrastive Character Image Pretraining (CCIP), Bilingual Evaluation Understudy Score (BLEU), and ImageReward to compute quantitative measures to compare images representing CivitAI models to images produced by prompts that were automatically generated by different image-to-text generators. We found that the image-to-text generators' CLIPI-I scores were not statistically significant from one another, which means that the images were visually similar to each other. However, from the Bleu scores we found that the textual prompts were dissimilar between image-to-text generators. This means that visually similar images can be generated by different, but semantically similar tokens. We also found that most of the existing image evaluation metrics are not satisfactory to reflect the perceived preference of humans in their subjective ratings for images.