You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note that we have to comprehensively integrate the inner model of CellReplay. Though I haven't read its code, if it mimics the behavior of cellular channels, it will emulate bi-directional paths. While emulating both paths inside a single shell in mahimahi is a desired design, we may have to have the ability to break it into two cells, as our client-to-server and server-to-client paths may go through different cells.
Hi @BobAnkh, here's the CellReplay plan I mentioned in the email. it's only the replay part (not their recording tools) and there's no code yet, I wanted to agree on scope first.
What CellReplay adds on top of mahimahi: the base delay changes over time instead of being fixed, and the PDOs depend on the workload. a short burst gets "light" PDOs (measured with small packet trains) and only a long backlog gets the "heavy" saturator PDOs. each direction has 3 files: *-delay-light-pdo (one line about every 100 ms, time_ms one_way_delay_ms pdo0 pdo1 ..., the delay is RTT/2), *-heavy-pdo (mahimahi style ms timestamps) and latency-offset-* (a small delay adjustment per packet size, 100..1400 B). for tmobile/driving the down light trace has 18216 lines with 50 PDOs each (10 for up), and the base delay goes from 16 ms min to 24 median to 428 max.
The replay model (sec 4.3) is basically 2 states. when a packet comes and the link is inactive, it takes the base delay at that moment and keeps it for the whole active period, and the schedule is the light PDOs and then the heavy trace. when the queue is empty for F ms it goes back to inactive.
In mm-cellular, cellularshell.cc makes one PacketShell<CellularQueue> and calls start_uplink and start_downlink separately, so up and down are already 2 independent queues, each with its own files, its own F (--long-to-short-timer-up/down) and its own clock. heavy PDOs are 1400 B each.
Some things where the code and the paper dont match, so we should pick one:
F defaults to 0 and the README example never sets it, but Table 2 uses F = 5 or 7
--loss-rate creates drop_dist_ but nothing ever samples it, so it does nothing
"queue empty for F ms" is checked as time + base_delay > last_release + F (cellular_queue.cc:408), so its an approximation
tmobile/driving has 50 down light PDOs per train but Table 2 says D = 75 for that setup
My idea for Rattan: I don't think DelayReplay + BwReplay chained can do this, because the frozen delay, the light PDOs starting at activation and the switch back after F all depend on each other, and cells dont share state. so one new single direction cell, config something like light_pdo, heavy_pdo, latency_offset (optional), inactive_timer_ms (F, no hidden 0 default), bytes_per_pdo (default 1400) and a queue. Then one cell per direction, like you said in the comment above:
[links]
left = "cr_up"cr_up = "right"right = "cr_down"cr_down = "left"
and both start from TRACE_START_INSTANT like you said in the email. this also makes it easy to put a Loss cell on one side since CellReplay doesnt replay random loss anyway.
Edge cases I noticed:
netem-trace's mahimahi loader counts 1500 B per opportunity, so CellReplay heavy traces through BwReplay would give about 7% too much bandwidth
mm-cellular sets the MTU to 1392. in Rattan a packet could just use several PDOs byte by byte like mahimahi's link queue
the down heavy trace for tmobile/driving is about 41.5M lines, so I'd store per ms counts instead of a timestamp per line
the delay trace and heavy trace have different lengths so they wrap around at different times
For testing, mm-cellular can log every packet (arrival, release, delay, light or not), so I'd feed the same arrivals to both and compare release times, and after that do end to end tests like in the paper (packet train completion time, downloads).
Plan, as small PRs:
trace parsers + config types with unit tests on tiny files
the replay model as plain sync code (on_arrival(t, size) -> release_time), tested by hand and against mm-cellular logs
wrapping it as a cell + toml example + netns test. this one might take me longer since im still learning async rust
I assign this issue to you, and for other questions, we will discuss with @Centaurus99 and @Lethe10137 next week, they will also follow up on this issue. Briefly, we can follow the released code and any trace-like definition should be added to netem-trace, instead of define it only in Rattan. For the real test traces, we only need to include one or several pieces, which is enough. We will do a deep discussion next week on this thing (we are busy with camera-ready this week).
I agree that the trace replay for CellReplay deserves a new kind of cell.
I suggest that, the clearest way to implement such cell is just let the new cell to directly express "per-ms PDO".
Given that a mahimahi bandwidth trace is just a heavy-DTO trace, with no light-dto sending opportunities, have this new cell been implemented, we shall use it to replay mahimahi trace, other than our current method ( transform mahimahi trace into bw replay traces, then align the timestamp to the next ms at some ingress).
Thanks @Lethe10137, that makes sense to me and it's simpler than what I had. If the cell just takes "how many PDOs (bytes) in each ms", then:
mahimahi trace = heavy PDOs only, no light part, no inactive timer
CellReplay = light PDOs at the start of each active period, then the heavy ones, then back to inactive after F ms of empty queue
So the same cell covers both, and the CellReplay part is just a small state machine on top. It also fits the "per ms counts" idea from my note, since the tmobile/driving down heavy trace is ~41.5M lines.
A few things I'm not sure about:
Bytes per PDO. mahimahi traces in rattan count 1500 B per opportunity now, CellReplay uses 1400 B. Should it be a config field with 1500 as default?
If mahimahi traces move to this cell, should the current BwReplay path stay as it is (so old configs don't change) and the new cell just be another option for now?
The trace type itself would go in netem-trace like BobAnkh said. Maybe something like a per-ms PDO trace that both mahimahi files and the CellReplay heavy files can load into?
I'll wait for the discussion this week before writing code, and I'm opening the #149 PR first.
Thanks @Lethe10137, that makes sense to me and it's simpler than what I had. If the cell just takes "how many PDOs (bytes) in each ms", then:
* mahimahi trace = heavy PDOs only, no light part, no inactive timer
* CellReplay = light PDOs at the start of each active period, then the heavy ones, then back to inactive after F ms of empty queue
So the same cell covers both, and the CellReplay part is just a small state machine on top. It also fits the "per ms counts" idea from my note, since the tmobile/driving down heavy trace is ~41.5M lines.
A few things I'm not sure about:
1. Bytes per PDO. mahimahi traces in rattan count 1500 B per opportunity now, CellReplay uses 1400 B. Should it be a config field with 1500 as default?
2. If mahimahi traces move to this cell, should the current BwReplay path stay as it is (so old configs don't change) and the new cell just be another option for now?
3. The trace type itself would go in netem-trace like BobAnkh said. Maybe something like a per-ms PDO trace that both mahimahi files and the CellReplay heavy files can load into?
I'll wait for the discussion this week before writing code, and I'm opening the #149 PR first.
I think that the basic idea is that, there's no concept of bandwidth here in the CellReplay. Instead, we are handling PDO, or on a per-packet level.
I think this is not what the CellReplay should care. It should count by packets. Typically 1400B MTU is common in Cellular network, and 1500B MTU is common in ethernet. This, the MTU configuration, should be global for rattan, so that it could be configured when the veth pairs are created.
Yes, just remain the BwReplay as-is. Actually the BwReplay's behaviour is quite different from "per-ms PDO". We can not mimic the behaviour of mahimahi's bandwidth trace, which is based on per-ms PDO, with the current bandwidth cells that handles bandwidth.
Yes, we do need trace types in netem-trace that directly express the concept of PDO, or packet delivery opportunity. But I'm not sure if it has to be per-ms. Perhaps it could be expressed just as a list of (interval since last PDO, number of packets to be sent, delay for those packets) ?
Consider this example:
For a 18 Mbps static bandwidth trace, packets (supposed to be sufficient from upstream) would be sent at:
0.667 ms 1pkt
1.333 ms 1pkt
2.000 ms 1pkt
2.667 ms 1pkt
3.333 ms 1pkt
4.000 ms 1pkt
...
For a per-ms PDO trace that also gives a 18Mbps average bandwidth, it would be
Ok got it, no bandwidth at all, just packets and when they can go. Thats actually easier to think about.
MTU being global makes sense, I'll leave it out of the cell.
Ok, BwReplay stays as is.
A list of (interval, packets, delay) sounds good, and I agree it doesn't need to be per-ms. Your 18 Mbps example makes it pretty clear why the bw cell can't really do the 1,2,1,2 pattern.
Some questions before I start:
For CellReplay the delay isn't per PDO. The base delay is taken when the link goes from inactive to active and then it's kept for the whole active period. So should the trace still give a delay and the cell just uses the one from the first PDO of each active period? Or should the cell handle that part itself?
Same with the light PDOs. They start from the moment the link becomes active, so they depend on when packets arrive, and I don't think they can be one list made ahead of time. I was thinking the cell reads 2 traces (light and heavy) and switches between them. Does that sound ok or is there a cleaner way?
For small packets like ACKs, is it 1 packet per PDO no matter the size? I think in mahimahi a PDO is 1504 bytes and a few small packets can go out in one, so this could change things a bit compared to mm-link. not sure if we care about that
For mahimahi files, lines with the same ms would just become one entry with more packets, and delay 0 (base delay stays in a separate delay cell)?
If that's fine I'd start with the netem-trace side first (the trace type + a mahimahi loader + tests) as its own PR there, and then the cell in rattan after that.
Ok got it, no bandwidth at all, just packets and when they can go. Thats actually easier to think about.
1. MTU being global makes sense, I'll leave it out of the cell.
2. Ok, BwReplay stays as is.
3. A list of (interval, packets, delay) sounds good, and I agree it doesn't need to be per-ms. Your 18 Mbps example makes it pretty clear why the bw cell can't really do the 1,2,1,2 pattern.
Some questions before I start:
* For CellReplay the delay isn't per PDO. The base delay is taken when the link goes from inactive to active and then it's kept for the whole active period. So should the trace still give a delay and the cell just uses the one from the first PDO of each active period? Or should the cell handle that part itself?
* Same with the light PDOs. They start from the moment the link becomes active, so they depend on when packets arrive, and I don't think they can be one list made ahead of time. I was thinking the cell reads 2 traces (light and heavy) and switches between them. Does that sound ok or is there a cleaner way?
* For small packets like ACKs, is it 1 packet per PDO no matter the size? I think in mahimahi a PDO is 1504 bytes and a few small packets can go out in one, so this could change things a bit compared to mm-link. not sure if we care about that
* For mahimahi files, lines with the same ms would just become one entry with more packets, and delay 0 (base delay stays in a separate delay cell)?
If that's fine I'd start with the netem-trace side first (the trace type + a mahimahi loader + tests) as its own PR there, and then the cell in rattan after that.
I read the source code of CellReplay and Mahimahi again, and you are right. The suggestion I made does not reflect what CellReplay or Mahimahi actually does.
Trace format
Actually we should have 2 kinds of traces:
For the light-pdo, it just yields what a light-pdo trace line of CellReplay contains. i.e.
timestamp, base_delay, Vec<pdo_opportunity>
If we are simulating Mahimahi's behaviour, this can just be empty -- imposing 0ms delay on the packets.
For the heavy-pdo, or Mahimahi trace, it should be
interval since last PDO, bytes to send
I noticed that both CellReplay and Mahimahi's PDO is in bytes -- I thought it was in packet, my fault. Thus, we have to make the whole thing to be complicated, and count bytes. And packets needs to consume enough bytes to go out.
Suggestion for implementation
For each Cell in Rattan, it has Ingress and Egress part. They have a tokio::sync::mpsc in between, let's call that ingress queue.
In the Egress, like the current BwCell, we have another queue ( which can be Infinite / DropTail / DropHead / etc.), let's call that egress queue.
In the new Cell to be implemented:
In the ingress queue, passing (bool, impl Packet) instead of Packet. And the bool is for whether we can by pass the egress queue, if the egress-queue is empty. ( in other words, the egress-queue will ignore the bool, if it is not empty, and thus we do not need to care about the state of egress-queue when we set the bit.)
In the Ingress, which takes the light-pdo trace,
If the trace is missing (e.g., for Mahimahi). Do nothing with the packet's logical timestamp, and just sent (false, packet) into the ingress queue without waiting.
If the trace is not missing, ( for CellReplay). Calculate the "release time" (base delay, packet-size offset, light-pto), and if it is eligible for bypassing the heavy-pto. Adjust the logical timestamp for the packet according to the "release time". And send (bool, packet) to the ingress queuewithout waiting.
In the Egress, which takes the heavy-pdo trace.
It waits (asynchronously) on 3 kind of things:
new (bool, packet) from the ingress queue.
new PTO ( with the PTO's timestamp)
When getting a new (bool, packet) from the ingress queue,
If the egress queue is empty, and the bit come together with the bool is true: wait until the packet's timestamp (set at the ingress), and release that.
Otherwise, put the packet into the egress queue
When getting a new PTO:
Accumulate that to a "reserved PTO bytes".
If the egress queue is empty, clear the "reserved PTO bytes",
otherwise, check the head of egress queue
If the logical timestamp of that is later than the new PTO's, do nothing except clear the "reserved PTO bytes",
else if the "reserved PTO bytes" is enough to send it out, deduct the "reserved PTO bytes", adjust the packet's logical timestamp to the timestamp to the current PTO, and release the packet immediately. (notice that Mahimahi do the accounting on the packet side, i.e. packet_in_transit_bytes_left_ . I think the method I mentioned above is equivalent to that).
Notice that, we should never sleep in the Ingress of a Cell, but we can sleep in the Egress part.
Also, according to the CellReplay implementation, the release time of a packet should be non-decreasing. Make sure that is also the case for the packet in the ingress queue -- my suggested implementation was based on that.
Thanks, this is really detailed! Counting bytes makes sense, and it kind of answers my ACK question too since a few small packets can go out on one PDO now.
Just want to check I got it right before I start:
In the Egress you wrote it waits on 3 kinds of things but I only see 2 (new packet and new PDO). Is the 3rd one the timer for the bypass packet's release time? because while it waits for that packet it still needs to take PDOs.
After a PDO sends out the head packet, if there are reserved bytes left, it keeps going with the next packets in the queue until they run out, right?
For the first netem-trace PR should I also add a loader for CellReplay's light-pdo files, or just the two trace types + the mahimahi loader for now?
So the plan for netem-trace is the light-pdo trace (timestamp, base_delay, Vec<pdo_opportunity>), the heavy-pdo trace (interval since last PDO, bytes), a mahimahi loader where each line is 1504 bytes and lines with the same ms get merged into one, and tests. Then the cell here after that. I'll make sure the Ingress never sleeps and the release times stay non-decreasing.
Thanks, this is really detailed! Counting bytes makes sense, and it kind of answers my ACK question too since a few small packets can go out on one PDO now.
Just want to check I got it right before I start:
* In the Egress you wrote it waits on 3 kinds of things but I only see 2 (new packet and new PDO). Is the 3rd one the timer for the bypass packet's release time? because while it waits for that packet it still needs to take PDOs.
* After a PDO sends out the head packet, if there are reserved bytes left, it keeps going with the next packets in the queue until they run out, right?
* For the first netem-trace PR should I also add a loader for CellReplay's light-pdo files, or just the two trace types + the mahimahi loader for now?
So the plan for netem-trace is the light-pdo trace (timestamp, base_delay, Vec<pdo_opportunity>), the heavy-pdo trace (interval since last PDO, bytes), a mahimahi loader where each line is 1504 bytes and lines with the same ms get merged into one, and tests. Then the cell here after that. I'll make sure the Ingress never sleeps and the release times stay non-decreasing.
For the three kinds of things. It was a typo and I used to mean for waiting for packet's release time. But it's actually 2: Logically we need to handle the PDOs while waiting for the release time. However, we have this inequality while waiting for some release time： timestamp for the PDO that we are processing <= current time < the packet's release time <= any packet's release time that is still in the ingress queue <= the time that egress queue would be non-empty ---- which means that if we were to handle the PDO's at its timestamp, the only thing we would do is just drop it and clear the "reserved PTO bytes", given that when we start to waiting for some bypass packet's release time, there's nothing in the egress queue. Thus, we can actually safely sleep until the by pass packet's sleep time. And when we wake up, we can just drop any missed PTO during the sleep. Also, as the release time is non-decreasing, so it's OK to not check the ingress queue while we are waiting for some packet's release time. ---- Thus, if the above analysis holds, just simply sleep until the bypass packet's release time.
Yes. but be careful that in current design of Rattan, we can dequeue at most 1 packet from the dequeue() function call. You may expect that another dequeue() call will be made just after you returned a dequeue() if possible -- so make that the happy path of the code!
Thanks @Lethe10137, that makes sense! So while I'm waiting for a bypass packet's release time the egress queue is empty, and any PDO in that window would just get dropped anyway. So I'll just sleep until the release time and drop the PDOs I missed when it wakes up.
And ok, one packet per dequeue(). I'll keep the leftover PDO bytes in the cell, so the next dequeue() call can send the next packet right away. I'll make that the fast path.
The light-pdo loader is already in stack-rs/netem-trace#29 (load_cellreplay_light_pdo_trace, it parses time base_delay pdo0 pdo1 ... and checks that the times don't go down). Could you take a look at it when you have time? I'll start on the cell here next.
Activity
BobAnkh commentedon Jan 14, 2025
Note that we have to comprehensively integrate the inner model of
CellReplay. Though I haven't read its code, if it mimics the behavior of cellular channels, it will emulate bi-directional paths. While emulating both paths inside a singleshellin mahimahi is a desired design, we may have to have the ability to break it into two cells, as our client-to-server and server-to-client paths may go through different cells.limonuser commentedon Oct 7, 2026
Hi @BobAnkh, here's the CellReplay plan I mentioned in the email. it's only the replay part (not their recording tools) and there's no code yet, I wanted to agree on scope first.
What CellReplay adds on top of mahimahi: the base delay changes over time instead of being fixed, and the PDOs depend on the workload. a short burst gets "light" PDOs (measured with small packet trains) and only a long backlog gets the "heavy" saturator PDOs. each direction has 3 files:
*-delay-light-pdo(one line about every 100 ms,time_ms one_way_delay_ms pdo0 pdo1 ..., the delay is RTT/2),*-heavy-pdo(mahimahi style ms timestamps) andlatency-offset-*(a small delay adjustment per packet size, 100..1400 B). for tmobile/driving the down light trace has 18216 lines with 50 PDOs each (10 for up), and the base delay goes from 16 ms min to 24 median to 428 max.The replay model (sec 4.3) is basically 2 states. when a packet comes and the link is inactive, it takes the base delay at that moment and keeps it for the whole active period, and the schedule is the light PDOs and then the heavy trace. when the queue is empty for F ms it goes back to inactive.
In mm-cellular,
cellularshell.ccmakes onePacketShell<CellularQueue>and callsstart_uplinkandstart_downlinkseparately, so up and down are already 2 independent queues, each with its own files, its own F (--long-to-short-timer-up/down) and its own clock. heavy PDOs are 1400 B each.Some things where the code and the paper dont match, so we should pick one:
--loss-ratecreatesdrop_dist_but nothing ever samples it, so it does nothingtime + base_delay > last_release + F(cellular_queue.cc:408), so its an approximationMy idea for Rattan: I don't think DelayReplay + BwReplay chained can do this, because the frozen delay, the light PDOs starting at activation and the switch back after F all depend on each other, and cells dont share state. so one new single direction cell, config something like
light_pdo,heavy_pdo,latency_offset(optional),inactive_timer_ms(F, no hidden 0 default),bytes_per_pdo(default 1400) and aqueue. Then one cell per direction, like you said in the comment above:and both start from
TRACE_START_INSTANTlike you said in the email. this also makes it easy to put a Loss cell on one side since CellReplay doesnt replay random loss anyway.Edge cases I noticed:
BwReplayCellto panic #194For testing, mm-cellular can log every packet (arrival, release, delay, light or not), so I'd feed the same arrivals to both and compare release times, and after that do end to end tests like in the paper (packet train completion time, downloads).
Plan, as small PRs:
on_arrival(t, size) -> release_time), tested by hand and against mm-cellular logsI'll finish the #149 PR before starting this.
Questions:
thanks!
BobAnkh commentedon Oct 7, 2026
I assign this issue to you, and for other questions, we will discuss with @Centaurus99 and @Lethe10137 next week, they will also follow up on this issue. Briefly, we can follow the released code and any trace-like definition should be added to netem-trace, instead of define it only in Rattan. For the real test traces, we only need to include one or several pieces, which is enough. We will do a deep discussion next week on this thing (we are busy with camera-ready this week).
Lethe10137 commentedon Oct 8, 2026
I agree that the trace replay for
CellReplaydeserves a new kind of cell.I suggest that, the clearest way to implement such cell is just let the new cell to directly express "per-ms PDO".
Given that a mahimahi bandwidth trace is just a
heavy-DTOtrace, with no light-dto sending opportunities, have this new cell been implemented, we shall use it to replay mahimahi trace, other than our current method ( transform mahimahi trace into bw replay traces, then align the timestamp to the next ms at some ingress).limonuser commentedon Oct 11, 2026
Thanks @Lethe10137, that makes sense to me and it's simpler than what I had. If the cell just takes "how many PDOs (bytes) in each ms", then:
So the same cell covers both, and the CellReplay part is just a small state machine on top. It also fits the "per ms counts" idea from my note, since the tmobile/driving down heavy trace is ~41.5M lines.
A few things I'm not sure about:
I'll wait for the discussion this week before writing code, and I'm opening the #149 PR first.
Lethe10137 commentedon Oct 13, 2026
I think that the basic idea is that, there's no concept of
bandwidthhere in theCellReplay. Instead, we are handlingPDO, or on a per-packet level.CellReplayshould care. It should count by packets. Typically 1400B MTU is common in Cellular network, and 1500B MTU is common in ethernet. This, the MTU configuration, should be global for rattan, so that it could be configured when thevethpairs are created.mahimahi's bandwidth trace, which is based on per-ms PDO, with the current bandwidth cells that handles bandwidth.netem-tracethat directly express the concept ofPDO, or packet delivery opportunity. But I'm not sure if it has to beper-ms. Perhaps it could be expressed just as a list of (interval since last PDO,number of packets to be sent,delay for those packets) ?Consider this example:
For a 18 Mbps static bandwidth trace, packets (supposed to be sufficient from upstream) would be sent at:
For a
per-ms PDOtrace that also gives a 18Mbps average bandwidth, it would beWe used to minic the
per-ms PDOby bandwidth trace. It's indirect and error-prone. Let's just:PDO-based trace innetem-trace, which yields (interval since last PDO,number of packets to be sent,delay for those packets)When doing so, I suggest to just forget the concept of "bandwidth".
limonuser commentedon Oct 13, 2026
Ok got it, no bandwidth at all, just packets and when they can go. Thats actually easier to think about.
Some questions before I start:
If that's fine I'd start with the netem-trace side first (the trace type + a mahimahi loader + tests) as its own PR there, and then the cell in rattan after that.
Lethe10137 commentedon Oct 16, 2026
I read the source code of
CellReplayandMahimahiagain, and you are right. The suggestion I made does not reflect whatCellReplayorMahimahiactually does.Trace format
Actually we should have 2 kinds of traces:
If we are simulating Mahimahi's behaviour, this can just be empty -- imposing 0ms delay on the packets.
I noticed that both CellReplay and Mahimahi's
PDOis in bytes -- I thought it was in packet, my fault. Thus, we have to make the whole thing to be complicated, and count bytes. And packets needs to consume enough bytes to go out.Suggestion for implementation
For each Cell in Rattan, it has
IngressandEgresspart. They have atokio::sync::mpscin between, let's call thatingress queue.In the
Egress, like the currentBwCell, we have another queue ( which can be Infinite / DropTail / DropHead / etc.), let's call thategress queue.In the new Cell to be implemented:
In the
ingress queue, passing(bool, impl Packet)instead ofPacket. And the bool is for whether we can by pass theegress queue, if theegress-queueis empty. ( in other words, theegress-queuewill ignore the bool, if it is not empty, and thus we do not need to care about the state ofegress-queuewhen we set the bit.)In the
Ingress, which takes thelight-pdotrace,ingress queuewithout waiting.ingress queuewithout waiting.In the
Egress, which takes theheavy-pdotrace.It waits (asynchronously) on 3 kind of things:
(bool, packet)from theingress queue.When getting a new
(bool, packet)from theingress queue,egress queueis empty, and the bit come together with the bool istrue: wait until the packet's timestamp (set at the ingress), and release that.egress queueWhen getting a new PTO:
egress queueelse ifthe "reserved PTO bytes" is enough to send it out, deduct the "reserved PTO bytes", adjust the packet's logical timestamp to the timestamp to the current PTO, and release the packet immediately. (notice that Mahimahi do the accounting on the packet side, i.e.packet_in_transit_bytes_left_. I think the method I mentioned above is equivalent to that).Lethe10137 commentedon Oct 16, 2026
Notice that, we should never sleep in the Ingress of a Cell, but we can sleep in the Egress part.
Also, according to the CellReplay implementation, the release time of a packet should be non-decreasing. Make sure that is also the case for the packet in the
ingress queue-- my suggested implementation was based on that.limonuser commentedon Oct 16, 2026
Thanks, this is really detailed! Counting bytes makes sense, and it kind of answers my ACK question too since a few small packets can go out on one PDO now.
Just want to check I got it right before I start:
So the plan for netem-trace is the light-pdo trace (timestamp, base_delay, Vec<pdo_opportunity>), the heavy-pdo trace (interval since last PDO, bytes), a mahimahi loader where each line is 1504 bytes and lines with the same ms get merged into one, and tests. Then the cell here after that. I'll make sure the Ingress never sleeps and the release times stay non-decreasing.
Lethe10137 commentedon Oct 19, 2026
For the three kinds of things. It was a typo and I used to mean for waiting for packet's release time. But it's actually 2: Logically we need to handle the PDOs while waiting for the release time. However, we have this inequality while waiting for some release time：
timestamp for the PDO that we are processing<=current time<the packet's release time<=any packet's release time that is still in the ingress queue<=the time that egress queue would be non-empty---- which means that if we were to handle the PDO's at its timestamp, the only thing we would do is just drop it and clear the "reserved PTO bytes", given that when we start to waiting for some bypass packet's release time, there's nothing in the egress queue. Thus, we can actually safely sleep until the by pass packet's sleep time. And when we wake up, we can just drop any missed PTO during the sleep. Also, as the release time is non-decreasing, so it's OK to not check the ingress queue while we are waiting for some packet's release time. ---- Thus, if the above analysis holds, just simply sleep until the bypass packet's release time.Yes. but be careful that in current design of Rattan, we can dequeue at most 1 packet from the
dequeue()function call. You may expect that anotherdequeue()call will be made just after you returned adequeue()if possible -- so make that the happy path of the code!Please add the light-pdo files loader in that PR.
limonuser commentedon Oct 19, 2026
Thanks @Lethe10137, that makes sense! So while I'm waiting for a bypass packet's release time the egress queue is empty, and any PDO in that window would just get dropped anyway. So I'll just sleep until the release time and drop the PDOs I missed when it wakes up.
And ok, one packet per
dequeue(). I'll keep the leftover PDO bytes in the cell, so the nextdequeue()call can send the next packet right away. I'll make that the fast path.The light-pdo loader is already in stack-rs/netem-trace#29 (
load_cellreplay_light_pdo_trace, it parsestime base_delay pdo0 pdo1 ...and checks that the times don't go down). Could you take a look at it when you have time? I'll start on the cell here next.feat(model): add heavy and light PDO traces (#29)