Day 14 – Project Transit Notifier

To do: Work on breaking my process_transit_request_function to process one item of the queue at a time, read more about threads

What I did: I worked on doing a small sample of Celery~ness. I did an example celery process that was very helpful. My baby example had me pip install Celery & RabbitMQ and create three files:

The example celery code
The example celery code
  • client.py – it creates an instance of Celery and configures it
  • celeryconfig.py – the configuration details that client.py uses
  • task.py – the actual function for celery to do

To get this baby going, I have to have three terminals open (this is the interface of running programs and talking to the “hardware” part of the computer) and run different parts of my application:

The example celery works!
The example celery works!
  • Terminal #1 – activates my rabbit server, this is where my celery can do its processes
  • Terminal #2 – activate my celery worker, so they can receive the task
  • Terminal #3 – run my python task script, so I can give my worker’s their task

Great! Now to my project….

My celery's task.py -- this is where I have the function (task) I want celery to do
My celery’s task.py — this is where I have the function (task) I want celery to do

So I moved this around and I added to my server.py the information from client.py from my example so I can create an instance of celery. Almost everything that was in celeryconfig.py is need (expect for the args, I don’t need to pass any arguments), so I am my own file in my directory. Then pulled from my server.py the process_transit_request function and placed it into its own task.py so I can run him independently.

My celery is connected and is processing my queue!
My celery is connected and is processing my queue!

To get this connected, Kai (an instructor) let me explain to her what I was doing and the steps I need to do to get it started. She was so helpful and was able to the questions and help me figure out that I put put part of my configuration instruction in my server.py when it should be in my celeryconfig.py. We were able to get an error from celery! YEAH! I never thought that I would be so happy to see an error. This means that I am able to go with celery into my task and get him going. Its currently not able to process the request to get a list of transit request that need to be process, but I can figure that one, because now that i just the python file getting access to my database. Thank you so much Kai for being supportive!

To do Tomorrow: Fine tune my celery processes and add “processes_line_and_bound_selects_closest_vehicle” function to task so my user doesn’t have to wait for it to be processed


My Thoughts/Process: These past few days working on understanding celery, rabbit, threads, locks, deadlocks, etc. was challenging – there were many times that I felt I was starting to understand and then it would slip. It really wasn’t until I did the baby example did things start solidifying. Then using the white board to organize the set up and work flow gave me a map and then it was a matter of adding it into my code. Once I was there, it was a manner of having someone else (Kai) to explain what I was doing which supported my understanding and logic. The moment I got a celery error I was so excited! I told Kai that I never thought that I would have ever been so happy to see an error — the error meant it connected! Today was very satisfying and I am so happy that I didn’t give up on learning this new technology.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s