Building an automated simulated Stock Trading System with Django, Scrapy, and Celery
Note: I am not judging nor proclaiming the (simulated) profitability of this system, I am merely building what the customer requested.
The customer’s request:
Simulate automated stock trading based on daily market movements.
Step 1: Find the worst 30 performers from yesterday
Step 2: Check if any stocks prices increases between when the market opens at 9:00am and 9:05am.
Step 3: At 9:06am, buy minimum amount of stock from step 2.
Step 4: Throughout the day, if any stocks go up or down by 3% of the purchase price, sell them.
Track and display all of this, to see if it’d be a profitable system.
Use python.
Phase 1:
I decided to use Django as the backend, Scrapy for web scraping, Celery for task scheduling, and Redis for managing background tasks.
In an empty repository, I initialized a Django project. Django’s ORM made it easy to structure my database, and Django’s admin panel let me interact with stored data, though phpMyAdmin is way better.
Next, I integrated Scrapy to scrape stock data from Yahoo Finance.
Here’s a spoiler to the final basic structure:
.
├── README.md
├── manage.py
├── requirements.txt
├── check_schedule.py
├── rebound_hunter #Project
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ ├── wsgi.py
│ ├── celery_app.py
├── hunter #App
│ ├── __init__.py
│ ├── admin.py
│ ├── apps.py
│ ├── models.py #Three Models
│ │ => WorstStock #Closing and 9:05am price
│ │ => OwnedShares #Bought and Sold shares
│ │ => ScrapingHistory #Every thirty minutes
│ ├── tasks.py #Four Celery Tasks
│ │ => scrape_worst_stocks_task() #8:30am
│ │ => scrape_changes_task(): #9:05am
│ │ => buy_minimum_shares_task(): #9:06am
│ │ => periodic_scrape_changes_task(): #Every 30 minutes
│ │ ├──>saves changes history to a database
│ │ helper() checks database for ±3% to trigger sale
│ ├── urls.py
│ ├── views.py
│ ├── templates
│ │ ├── base.html
│ │ ├── hunter
│ │ │ ├── recent_stored_data.html
│ │ │ ├── buy_sell_history.html
│ │ │ ├── owned_stocks.html
│ ├── stock_scraper
│ │ ├── scrapy.cfg
│ │ ├── stock_scraper
│ │ │ ├── output.json
│ │ │ ├── __init__.py
│ │ │ ├── settings.py
│ │ │ ├── spiders #Three Scrapy Spiders
│ │ │ │ ├── worst_thirty_spider.py #8:30am (& store in db)
│ │ │ │ ├── stock_price_change_spider.py #9:05am (& store in db)
│ │ │ │ ├── periodic_stock_price_spider.py #Every 30 minutes
├── accounts #App
│ ├── __init__.py
│ ├── models.py
│ ├── views.py
│ ├── urls.pyOnce Scrapy was working, I needed a way to schedule recurring tasks. Celery is perfect for this.
• Running Scrapy spiders at scheduled times
• Checking stock prices every 30 minutes
• Executing buy and sell decisions automatically
To support Celery, I installed Redis as the message broker. This setup allowed me to queue and execute tasks asynchronously in the background.
Automating Stock Buying and Selling
With the main infrastructure in place, I started implementing the trading logic:
1. Scrape “Worst 30” Stocks (8:30 AM) — My system pulls the bottom 30 performing stocks at the end of each trading day.
2. Fetch Their Prices at 9:05 AM — The following morning, I retrieve their updated prices.
3. Buy Stocks If Price Increases (9:06 AM) — If a stock’s price increased from its closing value, the system checks minimum units (usually 100) and buys at 9:06 price.
4. Monitor Every 30 Minutes — The system checks stock prices throughout the day. If a stock moves ±3%, it is automatically sold.
This is an overly simple strategy for identifying potential rebounds and locking in small gains.
Challenges
1. Scraping
Making sure the CSS selectors were just right, as well as dynamic URLs.
2. Managing Celery Tasks on a Production Server
Running Celery locally was easy, but deploying it (on two domains, a test and production) on my Plesk-managed server required extra configuration. I had to:
Set up Celery workers and Beats as separate background processes with logs each while ssh-ng inside their respective Python virtual environments.
ps aux | grep celery
pkill -9 -f "celery" # kill any living Celery
nohup celery -A rebound_hunter worker --loglevel=info
--queues=stock-queue --hostname=production.stock.com >
~/celery_production_worker.log 2>&1 & disown
nohup celery -A rebound_hunter beat --loglevel=info
--scheduler django_celery_beat.schedulers:DatabaseScheduler
--pidfile=/tmp/celery_beat_production_stock.pid & disown
nohup celery -A rebound_hunter worker --loglevel=info
--queues=stg-stock-queue --hostname=test.stock.com >
~/celery_test_worker.log 2>&1 & disown
nohup celery -A rebound_hunter beat --loglevel=info
--scheduler django_celery_beat.schedulers:DatabaseScheduler
--pidfile=/tmp/celery_beat_test_stock.pid & disownDjango’s settings.py needed to be quite specific for this to work:
# Stock Production Settings
ALLOWED_HOSTS = [
"production.stock.com",
"www.production.stock.com",
"127.0.0.1",
"localhost"
]
CELERY_DEFAULT_QUEUE = "stock-queue"
CELERY_ROUTES = {"hunter.tasks.scrape_worst_stocks_task":
{"queue": "stock-queue"}}
#Redis as broker with SPECIFIC database (0-15)
CELERY_BROKER_URL = "redis://localhost:6379/2" #This took me an entire day
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
STATIC_ROOT = "/var/www/vhosts/m-dev.work/production.stock.com/static/"
#Of course, change all of the above accordingly for the "test.stock.com"Configure Celery Beat for periodic tasks
Ensure Redis was running properly
Luckily, I made a check_schedule.py which can be run in the Django shell, plus a pretty comprehensive Readme that I made has some tips and copy-paste-able commands in it.
Check db schedule, too:
python manage.py shell
from django_celery_beat.models import PeriodicTask
for task in PeriodicTask.objects.all():
print(task.name, task.queue, task.crontab)
exit()3. Avoiding Anti-Scraping Measures
Yahoo Finance might(?) have rate limits and bot detection mechanisms. To avoid being blocked, I had to:
• Randomize headers and user-agents
• Implement delays between requests
• Limit the number of stocks scraped at a time
4. Handling Diverging Git Branches
There were times when my local branch diverged from the remote repository, causing push conflicts. Since I was working (99%) solo, I force-pushed a few times after very-temporarily unlocking my repo, but I also had to carefully resolve merge conflicts in Django views and templates.
5. Calculating Profit and Loss in the UI
I wanted my buy/sell history page to display not only total earnings but also percentage gains/losses per trade. This required fetching investment data dynamically and performing calculations in Django views before passing them to the template. There is also a template which, when accessed, shows the just-now scraped prices of owned stocks and their gain/loss with a manual “Sell now” button, so a user could do that.
And here are some of the other ones:
Phase 2:
Fetch significantly more data*, feed it to an AI at 9:06am and ask if it’s a good idea to buy right now.
*more data: A prompt with all of this more data could actually yield potentially valuable results from an AI model’s insights.
Opening price
Volume
Trading value
Price range limit
Market capitalization
Number of shares issued
PER (company forecast)
PBR (actual)
EPS (company forecast)
BPS (actual)
ROE (actual)
Equity ratio (actual)
Minimum purchase price
Number of shares per unit
Highest price since the beginning of the year
Lowest price since the beginning of the year
Bulletin board "Everyone's evaluation"Volatility (10 days)
RSI (Relative Strength Index)
MACD (Moving Average Convergence Divergence)
News sentiment
Sector performance
Fear and greed index
