The prompt as map and compass
Prompt engineering is a vital skill for anyone making AI-generated art and text. When working with models like Stable Diffusion, a well-crafted prompt becomes your map and compass, directing the AI to create images that match your creative vision.
This tutorial demystifies prompt crafting with a structured template format that builds from foundational concepts to advanced techniques.
Whether you are a seasoned digital artist or a newcomer to AI assistance, this guide will equip you with the knowledge and insight to create compelling art that speaks with your unique voice.
Choosing your subject
The subject of your artwork is its heart. It’s what the viewer’s eye is drawn to, the center around which all other elements revolve. Selecting a subject decides the focal point of your creation.
People can be characters, figures, or self-portraits, each with their own stories and emotions. Objects can be everyday items, symbolic artifacts, or futuristic inventions that can carry meaning or significance. Scenes can be landscapes, cityscapes, or interiors, establishing setting and context. Abstract concepts are ideas, emotions, or themes that require a more metaphorical approach to visualization.
Consider the subject's distinctive features as well: physical description, historical era, or mythical attributes. Subjects can have implied qualities that suggest motion or emotion, such as “weary traveler” or “battle-worn knight.”
Start with a single, simple subject to ensure clarity of focus. Gradually introduce complexity by adding adjectives or additional elements. Decide on the scale of your subject matter. Is it a solitary figure, a duo, or a crowd? Reflect on how the multiplicity changes the narrative or the dynamic of the scene.
- Base: “tree”
- Specific Type: “oak tree”
- Detailed Description: “ancient oak tree with gnarled branches”
- Emotional Context: “ancient oak tree with gnarled branches, standing as the last sentinel of the forgotten forest”
Choosing your subject is a journey from the general to the particular. Each layer of detail adds depth, guiding the AI’s creative process toward generating artwork that embodies the essence and nuance of your vision.
Determining the verb
Verbs are the engines of our sentences; they propel our subjects into action and frame their state of being. In AI-generated art, verbs inject life into static images, turning them into stories in a single frame.
Action verbs indicate physical or mental actions a subject performs, such as running or thinking. State of being verbs describe conditions or situations a subject is in, such as exists or seems. Transitive verbs require an object to receive the action, such as holding a flower. Intransitive verbs do not require an object, such as laughing.
Present tense suggests an action happening now. Past tense indicates a completed action or a historical scene. Future tense offers a glimpse of what may come or a speculative scene.
Use adverbs to clarify how an action is performed. Consider the sensory experience: what does the action sound like, feel like, or how is it executed? Verbs can relate the subject to their surroundings, suggest interactivity with other subjects, set up an implied backstory, or predict future events.
- Simple Action: “cat sleeping”
- Action with Adverb: “cat sleeping peacefully”
- Interactive Action: “cat sleeping peacefully in a sunbeam”
- Narrative Action: “elderly cat sleeping peacefully in a sunbeam, recalling a lifetime of adventures”
Through versatile verb usage, you can direct the Stable Diffusion model to depict anything from a tranquil moment to a dynamic sequence of events. By selecting the right verb and pairing it with strategic details, you ensure the final image resonates with the dynamic essence you envision.
Deciding on color and style
Colors communicate mood, atmosphere, and emotion. The right color palette can turn a piece from a flat depiction into a visceral experience.
Reds are intense and can convey passion, energy, or danger. Blues evoke calmness, stability, or sorrow. Yellows are bright and can express cheerfulness, energy, or caution. Greens are associated with nature, healing, and tranquility. Purples suggest luxury, creativity, or mystery. Oranges are vibrant and can reflect enthusiasm, creativity, or warmth.
Warm colors, such as reds, oranges, and yellows, are often used to create a sense of warmth, excitement, or vibrancy. Cool colors, such as blues, greens, and purples, can produce a sense of calm, distance, or cold.
When crafting your prompt, name the colors and describe their application for more nuanced control. Instead of stating “warm colors,” try “a warm, golden sunset casting long shadows,” which guides the AI within the warm color range and suggests a time of day.
Choosing an art style is akin to selecting the lens through which the viewer will interpret the subject matter of your creation. Each style changes the feel, interpretation, and emotional response to the image.
Basic styles include realistic, stylized, and abstract. Historical art movements include impressionism, expressionism, surrealism, cubism, and pop art. Digital and contemporary styles include pixel art, low poly, and vaporwave. Genre-specific styles include fantasy, sci-fi, and horror. Technique-oriented styles include watercolor, ink drawing, and charcoal.
Constructing your prompt
The art of building prompts is much like crafting a recipe; you start with the basic ingredients and then add flavors and spices to taste. In prompt engineering, you begin with the core concept and then layer in details that refine and direct the AI’s output.
- Version 1: “cat”
- Version 2: “orange cat”
- Version 3: “orange cat sleeping”
- Version 4: “orange cat sleeping on a sunny window ledge”
- Version 5: “orange cat sleeping on a sunny window ledge in a watercolor style”
- Version 6: “peaceful orange cat sleeping on a sunny window ledge in a watercolor style”
- Version 7: “peaceful orange cat:2 sleeping on a sunny window ledge in a watercolor style”
This minimal prompt gives the AI the broadest possible interpretation for generating an image of a cat. Each added detail narrows the result: the subject color, the action, the setting, the art style, the mood, and finally the weighted focal point.
Fine-tuning with weights
Weights in Stable Diffusion give you the ability to fine-tune your prompt by controlling the influence of individual components within your generated art or text. This advanced feature allows for a higher level of customization and specificity in the outcome.
Weights are numerical values assigned to specific terms or concepts in your prompt, which tell the model how much emphasis to place on that element during the generation process. To assign a weight to an element in your prompt, follow the element with a colon and the desired weight value. Generally, the higher the weight, the more prominence that feature will receive in the generated result.
If you want to ensure that a plant plays a significant role in your image, you might use Plant:2 or Plant:3 to increase its prominence relative to other elements in the scene.
When using weights, it’s important to balance them carefully. Overweighting an element can overshadow other details, while underweighting may cause it to be barely noticeable. A default weight is usually considered to be 1. If you don’t specify a weight for an element, the model will assume a default weight of 1 for it.
In this prompt, the plants have been assigned a weight of 3, signaling the model to highlight the vegetation more than the pond or the bench.
Experimentation is Key: Weights can vary significantly in their visual impact depending on the complexity of the prompt and the intricacies of the generation model. It’s recommended to start with modest weight adjustments and progressively experiment to see the resulting changes in your generated artwork or text.
By following these steps and understanding the components, anyone can engineer prompts that produce bespoke, high-quality images or narratives using Stable Diffusion. Skillful prompt engineering pairs the content with weights that steer the outcome.
Related walls
First published on Medium. Rewritten for this site.