PROJECT 04 / ML
Networked ML and RL Algorithm
Five ways for devices to set their transmit probability on one shared channel, compared against the slotted-ALOHA limit.
- Python
- NumPy
- Matplotlib
- Q-learning
- UCB bandit
- Type
- ML
- Source
- Public on GitHub
- Role
- My own project; I am the only author of the repo.
- Built with
- Python, NumPy, Matplotlib
01 / Overview
Overview
Simulate a shared channel
N devices contend for one channel, one time slot at a time.
Give each device a strategy
Each device sets its own transmit probability p using one of five strategies.
Plot against the 1/e line
Matplotlib plots compare the strategies with the slotted-ALOHA reference.
02 / Context and problem
Context and problem
A final project for my wireless networks course: a congestion study where many devices send packets to one receiver, and the goal is high throughput with low packet loss (project report). The repo README is out of date: it covers two strategies. This page is code-backed and describes the five strategies the code implements. There are no saved results, so none are quoted.
03 / My role
My role
My own project; I am the only author of the repo.
04 / How it works
How it works
Each slot, each device transmits with probability p
Exactly one transmitter is a success, two or more is a collision, none is idle.
Baseline
p goes up a little on success and down on collision, with a small amount of noise.
Q-learning
The state is the last result (idle, success or collision) and the action is to decrease, hold or increase p.
Adaptive step size
A step-size controller class: the step grows on success or idle and shrinks on collision, within limits.
Hybrid
Q-learning picks decrease, hold or increase, and the adaptive step sets the size of the change.
UCB bandit
Three actions chosen with an upper-confidence bonus, using the same decrease, hold or increase moves.
Reward and fairness
A success earns a reward, and a collision penalty is shared across the transmitters. A fairness term nudges devices on long success streaks to back off.
05 / Key decisions
Key decisions
Small noise on p
It stops the devices from synchronising.
Fairness term
It discourages one device from always transmitting.
1/e line as the benchmark
It is the standard slotted-ALOHA reference for throughput.
06 / Results and numbers
Results and numbers
5
strategies implemented in the code
10
devices in the simulation
1000 steps, 2 runs
per comparison plot
07 / Limitations
Limitations
- No measured results are published; the code has to be re-run to quote any.
- Only 2 runs of 1000 steps, with no fixed random seeds and no statistics.
- The reward uses the total number of transmitters, so the devices are not fully decentralised.
- No automated tests, and the README is out of date.
08 / What I learned
What I learned
- Reward shaping matters.
- Noise prevents synchronisation.
- Learning rate, exploration rate and the starting p all need tuning; keeping the simulation manageable without oversimplifying took a lot of trial and error (project report).
- The report's conclusion is qualitative: the RL agent adapted better than the baseline when network load changed quickly. No measured numbers are published.
09 / What's next
What's next
- Try more devices, multiple channels, or real network data (project report).
- Compare against AI agents connected to APIs (project report).
10 / Media gallery
Media gallery
11 / Next project