Original subtitles

A few weeks ago, I created an episode on transcribing with artificial intelligence, and it was a

really neat application because it essentially used a Python script which ran as a service,

and with that, we were able to make requests from a Rails application over to this Python

service to transcribe audio and video files.

However, there were several pieces of that implementation that I really did not like,

and so in this episode, we're going to have a revisit to this and how we can make it not

only more stable but also thread safe.

And we're also going to be looking at creating a text to image with stable diffusion, and

again, all of this is hosted on our own local machine, and we're not using any APIs to a

third-party image generation service.

So we'll just use this prompt, tiger walking through a forest, and I'll create the image.

It will go from pending to processing, and then processing, it'll then get completed,

automatically showing us the image.

And just for fun, so we can create multiple of these, we're going to also have a regenerate

button which will just create a new record with the same prompt, which we'll go through

and then recreate that image.

And so you can see we'll get a different kind of image each time, all based off of that

prompt.

And so I have it set up here, you'll see on the bottom right hand side, this is a program

called MVTOP, and it gives us an overview of the GPU.

So currently you see we have about four gigabytes of VRAM being used, because we do have one

instance of our model being loaded.

And so on the left hand side, I have four instances where we can then regenerate the

image.

And on the top right, we have the Python script running as a microservice.

So I'm going to go through and quickly click on the generate on each one of these, and

you'll see that it is generating multiple lines at the same time.

So it is then going to be able to complete all four images concurrently.

And so here you see that we have four different images generated each time.

And so this does work pretty well.

Again, I'll just regenerate the images.

And you'll see it goes from a pending to processing.

And now we're not limited to just one thread on our sidekick worker to handle these.

And we're not even going to use sidekick in this case, it's just going to use the normal

active job to queue them all up and then process them.

And the nice thing about this approach is that it's going to be even easier than the

previous episode because of our approach that we're taking with the Python script.

So to get started, I am SSH into my development machine that has an Nvidia GPU.

I do already have Python installed on here, and I'm going to create two different files.

I'm going to create a web dot Python.

And then I'm also going to create a main dot Python.

And we are actually going to need one more file.

And that's going to be the requirements dot txt, because on a new machine, we want to

be able to quickly install all of the requirements for this particular application.

So just to go through what we're going to need, we're going to need flask.

And this is basically a Sinatra like web service that we're going to be able to use.

And it's going to allow us to create an endpoint.

We'll then need the diffusers and also the transformers.

And this is going to be the hugging face libraries that we use to pull down and interact with

the stable diffusion model.

And then we're also going to be using torch.

So once you have these, we can go into our terminal and then we can do a pip install

dash are and the requirements dot txt.

And it'll go through and make sure that all of those are installed.

Again, I am using a SDF to host the Python interpreter.

And the Python version that I'm using in this episode is three dot 10 dot 11.

And so let's start out with the Python script that's actually going to be doing the image

generation.

So we do need to bring in from the library diffusers.

We need to import in the stable diffusion pipeline.

We also need to import in torch.

And then we also need to import in IO because we're going to be generating an image and

we're not going to be saving it on this machine.

Instead, we're just going to return the contents of that image.

We then set a model underscore ID and this is just a variable.

And we then need to point to a hugging face model.

And there is a company stability AI, which has created the stable dash diffusion model.

We're going to be using the version two point one.

We can then set a pipe that's going to be equal to the stable diffusion pipeline.

And then we need to load in that model.

So we can do a from pre trained and then we can pass in that model ID and then we can

call pipe dot two.

We can pass in CUDA saying that we want to push this to our Nvidia GPU.

So this model is going to be loaded into our Vram.

We then create a method and we're just going to call this the generate underscore image.

And then we're going to take in some keywords and we're also going to take in a number of

steps, which is basically just going to be the accuracy of the image that we're going

to be generating.

We can set an image is equal to we can then call our model with the pipe and then pass

in the keywords.

We then set the number of the inferred steps and then we'll set it to the number of steps

that we're passing into this function.

We then call the images and then we'll grab the first item in that array.

We can set another variable image bytes and we'll set that equal to the I.O. bytes I.O.

and then we can call our image dot save and we're going to save it to this image bytes.

So we're not saving it to our hard disk or anything.

Instead it's just going to be in memory for a moment.

We can also set this to a format of PNG.

We can then rewind this image bytes with a seek to zero and then we can call the torch

dot kuda dot empty cache.

And this is just going to free up any memory that may have been used or reserved when generating

this image.

And then finally we can return the image underscore bytes.

And so essentially what we're going to do is have this main Python function and this

is all we need to write, just the 15 lines of code.

But for this function we then need to interface with it and that's where this web dot Python

file is going to come into play.

So essentially we're going to create a endpoint and with that endpoint we're then going to

be able to make requests to it.

It'll then call that generate image function.

So to do that we need to import in the OS and from flask we also need to import in a

few different things.

We need flask the request and then also send underscore file from main.

We need to import in the generate image and this generate image and the main is actually

coming from this main dot pi and within there we have that generate image function.

So we can pull this in so we can then interact with it.

We'll set app is equal to flask and then we can create a route.

So we'll have the app dot route and let's just create this route to the generate underscore

image and it does not have to be the same name as our function here.

I just happen to have it as the same name and we're going to have another parameter

methods is equal to and then an array and we want to allow a post request to this endpoint.

We can then have a function and we're just going to call this a handle underscore request.

It really doesn't have to be named this.

You can have it called anything but really that generate image and so with this handle

request we're going to have a prompt is equal to the request which is what's coming into

our application and it is a form request that we're going to take in and then we're going

to have a parameter and let's just call this the keywords and if the form was posted with

no keywords value we can check that if not prompt we can then return and we'll just return

the hash and we'll just say that there was an error and we'll say that the keywords is

required and we'll give it a status of 400.

However if we did receive a prompt then we can set the image underscore bytes and we'll

set that equal to the generate image function.

We'll pass in our prompt and then we need to pass in the number of inference steps and

just so this happens quickly I'll say 10.

If you want it to be a lot more accurate you can set it to 50 but the higher number you

use it is going to take longer and longer to generate.

So once we have our image bytes we can return the send underscore file we want to return

that image underscore bytes and we'll set the mime type is equal to an image for slash

PNG and lastly we just had to call the app dot run we can set what host we want to bind

this on because this is not going to be happening on our local host we can listen on any interface

and then let's give this a port is equal to maybe something like 8000 and so that's all

the python we have to write we have our web interface which is importing in the function

generate image from our main file and when we accept a post request to this endpoint

that has a form parameter keywords with some value then it's going to call that generate

image with that prompt that generate image function is just going to pass into our stable

diffusion model generating the image temporarily saving it to memory and returning it and so

we can go ahead and start up our web server with a python and the web dot pi and now load

it up into memory you will get a warning that this is a development server don't use it

in a production deployment and so if you did want to deploy this I would recommend using

something like g unicorn so you could just run that instead saying that we basically

just want one worker but if you are going to do that they want to disable this app run

so that way we are solely just going to be using the g unicorn it'll load it up into

the vram as we would expect and then once it's up and running we can make our request

to the ip address of this machine and port 8000 and so I'm going to use paul here to

make this request just so we can test and make sure it's working so I can do the HTTP

colon forward slash forward slash I can put in the ip address of this machine and then

I can specify the port 8000 I made this a bit wider so we can see it all and then we

want to go to the endpoint forward slash generate underscore image because it is a post request

we do need to pass in into the body the form url encoded and we need to specify the keywords

and for the keywords let's just call the planet earth as an example oh and I do have

a few little mistypes here so I do need to fix this spelling on the inference steps and

then also this is supposed to be image bytes once we fix those issues we can then restart

our application we can make our request again we'll see that it's generating the image and

then we get our image back and the nice thing about using something like paw or the rapid

API now is that we can then just keep making this same request over and over and we should

then see a new image being generated each time and so I would say that this works and

now we can handle the Ruby on Rails side where we can then make this request to this endpoint

as well as looking at some of the gotchas around it so because this is up and running and working

I'm going to just leave it here I will copy the G unicorn and then just paste that in

here just so we have reference to it in the show notes but for the most part we have under

30 lines of Python code and it's going to do some really cool stuff and so now we can

start with a fresh real 7 application I'll generate a scaffold we'll just call this the

images we'll have a prompt which is a text and then we're also going to have a status

which is an integer we'll go ahead and generate that and then we'll also call the rails active

underscore storage colon install because we are going to be using active storage to store

these images before we run the migrations I am going to go into the images migration

and I just want to set a default value for the status it will set it equal to zero because

we are going to be using the enums for the status to just keep track of it and then in

the image model we can do the has one attached and then let's just call this our file we'll

have the enum for this status and again we're going to have just like we did in the previous

episode a pending a processing and then a completed and similar we're going to have

a broadcast so as we're getting updates from the background job it's then going to broadcast

it to our end user and then in the form I'm just going to remove the status because that's

going to be automatically generated and we don't need to take in anything except the

prompt in the controllers in the images controller we don't want to take in the status so we'll

just have the prompt there and then let's go into the show page and within the show

page we're rendering the image which means that we need to look into this image partial

and then we see the prompt and the status and so that's good and so let's also display

out that image so I'll just create a new paragraph and within there we can call the image underscore

tag and then the URL underscore four and then the image dot file and we only want to do

this if the image dot file is attached we still need to listen for the action cable

channel so I'm just going to put that under the show page and we can do that with an ERB

tag for the turbo stream from and we just want to stream from the image but let's also

go ahead and add in that regenerate image button so just under the status I'm just going

to create a button to and if you're unfamiliar with the button to that's basically going

to create a form and around that form we can then have a URL passing some parameters and

it's going to make a post request so I'm just going to have this called the regenerate and

we can send this to the images underscore path and then we can also pass something into

here and so just to keep it in the same format that our images controller is going to expect

where we have the prams we need to have an image and then within that image hash we can

have the prompt and we just want to set that equal to the image dot prompt and of course

we would only want to do this if that image is completed so that's going to look at the

enum to see if it's completed and if it is then we can regenerate the image and so in

the images controller whenever we are creating an image let's call on a job let's call the

process image job will perform underscore later and let's just pass in the image so

typically I'll pass in the image ID I do like doing that but I'm just going to pass in the

image here because active job does support passing in more complex objects like an active

record object and it will use global ID to return the record but if you're uncomfortable

with doing that or if you're using a background processor that doesn't support it then you're

going to be able to pass in the image ID and then look up that record within the background

job so we need to generate that job with a rails generate job and the process underscore

image that'll give us that process image job and within this job we need to take in the

image we could always set up some guard clauses return unless the image is there we could

also return if the image is completed we can then set the image to processing which by

setting it to processing that's going to broadcast out and update our view we can get a prompt

is equal to the image dot prompt and let's just make sure that it is present if it is

present then we can return the image dot prompt and if it's not present then let's just give

it a default prompt in this case I'm just going to call children book our cover monkey

hanging in a tree we can give it the same URL that we used within the rapid API and

that is to a local IP address port eight thousand and generate image and of course you probably

don't want to hard code an IP address in here I would probably extract that out as well

as support to an environment variable but in this case I'm just going to leave it in

here and so we're then going to expect some kind of image data back and I'm going to make

a separate class for this because I don't want to put in all this logic within here

so we're just going to have a class we'll call it HTTP we'll make a post request and

let's set up this post class method to take in a few different things we need the URL

and then we also need some kind of keywords and in this case I'm just going to pass in

the prompt and so before we go any further let's go ahead and create this HTTP class

in this post method under our models I have the HTTP dot RB we're just going to use the

standard net HTTP it's going to be a class of HTTP we'll have our method for the class

level post taking in the URL and also the form data we can create a new instance of

this class with the form data and our URL and then we can call the instance method post

so we can initialize this with our URL and the form data and then we can set instance

methods for our URL and the form data is equal to those local variables and then we create

our post method and within this post method let's just call our post underscore request

which is a method that we'll have to create and then we're going to expect an attribute

body to be called on in return so we can create this post underscore request we can set our

URI is equal to the URI dot parse and we want to parse our URL we then set the HTTP is equal

to the net colon colon HTTP we'll create a new instance with our URI dot host and the

URI dot port we can set the use SSL is equal to true if the URI dot scheme is equal to

HTTPS otherwise they'll use HTTP and depending on your application and the server that you're

hosting this on you may need to set the open timeout and I'm going to set it to 10 seconds

and then you also need to set the read timeout because if you do have a lot of traffic coming

into the server and if it's going to take a while to generate the images then you may

need a bit of higher number here we then make our request is equal to the net HTTP and this

is a post we'll create a new instance to the URI dot request underscore URI and then we

can set the form data for this and we'll just set that equal to our form data we'll finally

then make our request with the HTTP dot request and then we'll pass in the request and so

the nice thing about this is that now we're going to be able to reuse this multiple times

for basically any kind of HTTP post that simple enough where we just have a URL and the form

data so back in our image processing job we're sending our image data to that HTTP post if

we are getting some image data back then we can call the image dot file dot attach and

then we want to attach within IO with the string IO creating a new instance of that

image underscore data we do need to give it a file name which let's just call it the image

dot PNG we also need to set the content type is an image for slash PNG once that attaches

we can call the image dot completed which should then broadcast a new change however

there are a couple of thoughts that you may want to take into consideration one if you

want to throttle the number of requests coming into the server to generate the images then

you can create a separate queue and with that separate queue you have a separate service

maybe only one two items within that queue or two workers handling requests for that

queue and there's also a situation where if the python service is not running then it's

going to error out and so we could do a retry on there and we can retry on that error where

it's basically a connection refused because we don't want this to keep hammering that

service we could just wait five seconds let's say we're going to give it an unlimited number

of attempts we could also retry on if there is a situation where our timeout wasn't long

enough and again we're just going to wait five seconds on there and so now we can test

this out we can give it a prompt or we can leave the prompt blank and then it'll show

us a monkey hanging from the tree we see that it went to processing and then completed and

then we got our image we could try to regenerate this it'll create a new record and again we

got the processing and completed and so this all works and so i'm going to kill the server

and i'm going to change the inference because i want this to take a little bit longer so

i'll set it to 50 and it should take about 14 seconds to generate an image based on this

graphics card i can then set this up to run again so i'll just use a g unicorn it'll then

load it up into memory and i can get in the top running again just so we can see this

and then i'll have two browsers open where we generate one let's just test it with one

first we can see the cpu spiking on the gpu and then it's going through and then it's

taking a lot longer now to generate the image but that is going to make it a bit more detailed

and accurate to what we are looking for and so now let's test it with two simultaneously

where we're generating two images at a time so here you can see that it's going to generate

one image and then once it generates that first image it's then going to work on the

second one and so that's kind of one of the drawbacks with the g unicorn is that it's

only handling one request at a time but we are able to sort of fix that so i'll kill

the server we'll then just change the number of workers and we'll set it equal to two but

you do want to be careful doing that because if you notice that the memory it is going

to spike a lot higher because now we have two separate processes running so again i'll

click generate on both of these real quick and you'll see that it's going through and

it's generating them and it is kind of jumping back and forth because it is running two separate

processes to generate them but one thing you will notice is that it is going to take longer

to generate both of those images simply because now it is having to do two calculations at

the same time so we got the one image and then the second one will appear shortly after

and i really enjoy doing this image generation and the transcribing and i'm just coming up

with all new kinds of ideas for this i don't think i'll really make any more episodes around

the ai unless if there is something a bit more breakthrough but i do just want to kind

of reiterate the approach that we took on the python script because i do think that

this makes it very usable and i do think that this is a great case for a micro service because

essentially what we have is a very small script that's doing one thing and it does that one

thing very well and we also have one simple script that is our main entry point into this

ai image generation you could also dockerize this if you want but the one thing that i

would recommend if you are going to dockerize it is that i would copy out this model in

the pipe into a preload script so then you can bundle in that model that you're going

to be using into that docker image because i have found if you try to start the geonuclearn

process within docker the health checks will fail and then you're just going to have some

quirks there but if you preload that image in then it's going to deploy and run very

fast but it is going to result in a very large docker image and i have created a website

where i've kind of gone through all of the different ai things that i've been creating

and playing around with and so you can go to the ai.railsapp.dev to check some of these

out you can do different things like text to speech where you can generate audio clips

that are spoken you could also play around with the text to image which this text to

image is going to have a bunch of the different community generated images it works in a very

similar process where we have a pinning it'll then go into processing and then complete

it once it's finished and once it completes it'll then hide the progress bar and it'll

take a moment but then it'll load in the image you could also use this with closed captions

so you can upload a file and then it'll generate the closed captions for you because this is

just an example not really meant to be used as a utility because my resources are limited

i do have the first 1000 characters basically being displayed from that transcription and

closed caption generation and similar for the transcriptions where it is just doing

the transcription of it you can see where it's just displaying out the first 1000 characters

and this really just so you can get a taste on how it works well that's all for this episode

thanks for watching