To do: Work on breaking my process_transit_request_function to process one item of the queue at a time, read more about threads
What I did: I worked on doing a small sample of Celery~ness. I did an example celery process that was very helpful. My baby example had me pip install Celery & RabbitMQ and create three files:
- client.py – it creates an instance of Celery and configures it
- celeryconfig.py – the configuration details that client.py uses
- task.py – the actual function for celery to do
To get this baby going, I have to have three terminals open (this is the interface of running programs and talking to the “hardware” part of the computer) and run different parts of my application:
- Terminal #1 – activates my rabbit server, this is where my celery can do its processes
- Terminal #2 – activate my celery worker, so they can receive the task
- Terminal #3 – run my python task script, so I can give my worker’s their task
Great! Now to my project….
So I moved this around and I added to my server.py the information from client.py from my example so I can create an instance of celery. Almost everything that was in celeryconfig.py is need (expect for the args, I don’t need to pass any arguments), so I am my own file in my directory. Then pulled from my server.py the process_transit_request function and placed it into its own task.py so I can run him independently.
To get this connected, Kai (an instructor) let me explain to her what I was doing and the steps I need to do to get it started. She was so helpful and was able to the questions and help me figure out that I put put part of my configuration instruction in my server.py when it should be in my celeryconfig.py. We were able to get an error from celery! YEAH! I never thought that I would be so happy to see an error. This means that I am able to go with celery into my task and get him going. Its currently not able to process the request to get a list of transit request that need to be process, but I can figure that one, because now that i just the python file getting access to my database. Thank you so much Kai for being supportive!
To do Tomorrow: Fine tune my celery processes and add “processes_line_and_bound_selects_closest_vehicle” function to task so my user doesn’t have to wait for it to be processed
My Thoughts/Process: These past few days working on understanding celery, rabbit, threads, locks, deadlocks, etc. was challenging – there were many times that I felt I was starting to understand and then it would slip. It really wasn’t until I did the baby example did things start solidifying. Then using the white board to organize the set up and work flow gave me a map and then it was a matter of adding it into my code. Once I was there, it was a manner of having someone else (Kai) to explain what I was doing which supported my understanding and logic. The moment I got a celery error I was so excited! I told Kai that I never thought that I would have ever been so happy to see an error — the error meant it connected! Today was very satisfying and I am so happy that I didn’t give up on learning this new technology.
chatasweetie 