Document that trip_updates are not required to occur in feed in block-order

[barbeau]

In the past, GTFS-realtime producers and consumers have asked me if TripUpdates need to be listed in block-order in the feed. For example, if there are trips with trip_ids 1, 2, and 3 that all belong to one block, and the vehicle travels trip 1, then trip 2, and then trip 3, do the trip_update entities in the feed need to be added in the order of 1, 2, 3?

Currently the GTFS-realtime spec is silent on this issue (it does say that within a trip, stop_time_updates must be provided in stop sequence-order).

This proposal clarifies this by recommending, but not requiring, that TripUpdates should be added in block-order in the feed.

EDIT After discussion, this proposal now simply states the current practice - that producers aren't required to add trips to a feed in block-order.

Announced on the GTFS-realtime Google Group at https://groups.google.com/forum/#!topic/gtfs-realtime/YYJmj91n0TM.

[skinkie]

Could you elaborate why this is important or beneficial for processing as consumer?

[barbeau]

Generally, adding this language to the spec prevents a consumer from implementing software that incorrectly assumes that trip_updates appear in block order in a feed.

In terms of benefits to an implementation of having trip_updates in block-order, it assists in processing if you're storing all the predictions for an entire block together in the same data structure (as opposed to storing each trip separately).

[mpaine-act]

Can a trip be assigned to multiple blocks? If so, what is the correct order for this case?

[skinkie]

We could make it ourselves a lot easier if we would mandate a departure time ascending sort.

[barbeau]

Can a trip be assigned to multiple blocks? If so, what is the correct order for this case?

It would be the active block order

We could make it ourselves a lot easier if we would mandate a departure time ascending sort.

This would actually be my preference (and where I started when I was writing this proposal).

What do producers think about this?

[skinkie]

@barbeau you might want to check two things: departure time (planned) sorting, or first departure (real time) sorted.

[abyrd]

My understanding is that GTFS-RT messages are unordered (that their order has no meaning). Rather than affirming that order has no meaning, the proposed text makes a new suggestion that feed producers should in fact be ordering these messages.

This could lead producers to put effort into implementing (potentially error-prone, moderately complex) code just to make output have certain characteristics, but consumers cannot rely upon those characteristics (because they are only recommended, not required). This means sorting / matching code would be needlessly duplicated on both ends of the chain.

The "GTFS philosophy" generally favors placing complexity on the consumer side, to encourage more widespread production of feeds. In this case, that would mean not placing any new order constraints on feeds.

[barbeau]

@abyrd You're right, the proposal as currently written is a compromise intended to clarify what is currently required, but push producers towards ordering feeds. As I mentioned above writing primarily consumer code I'm biased towards requiring producers to order messages, but I realize that this could mean extra work for producers.

My primary motivation with this pull request is to clarify what the spec requires. To that end, perhaps the least controversial change would make sense, which simply says that ordering isn't required - something along the lines of:

When multiple trips in one block are included in one feed, the respective TripUpdate entities are not required to be added to the feed in the same order that they are scheduled in the block (for more information about trips and blocks, please refer to GTFS trips.txt). For example, if there are trips with trip_ids 1, 2, and 3 that all belong to one block, and the vehicle travels trip 1, then trip 2, and then trip 3, the trip_update entities may appear in any order - trip 2, then trip 1, and then trip 3 is allowed.

[barbeau]

I've removed the recommendation to add trips in block-order to a feed in this proposal. Instead, this proposal now simply states the current practice - that producers aren't required to add trips to a feed in block-order. For now, I'd like to focus on the least controversial language just to get this documented in the spec.

This pull request has been open for more than one week, so per the Official Process I'm calling for a vote.

Vote will be closed on Wednesday August 14th at 23:59:59 UTC.

[gcamp]

I don't think anybody could be against that +1

[paulswartz]

+1

[anatolip-rt]

+1

[abyrd]

I am not against this in principle. But I am a little concerned that the wording will incite people to view feeds as inherently ordered. Someone who sees this text will learn that order does not matter in this particular case, but may then assume that order matters everywhere else, or at least begin wondering whether order matters elsewhere.

I would prefer a blanket statement that feeds are not ordered, higher up in the document hierarchy outside any particular topic like blocks of trips. Or if people find it confusing to say that the data structure is unordered (if they see files and streams as inherently ordered), then just a statement that the order does not have any meaning.

This way we can make a small number of exceptions in places where order does matter, rather than making exceptions in every topic saying order does not matter.

[barbeau]

@abyrd I agree that it would be useful to state somewhere at the top that entities aren't assumed to be ordered unless otherwise stated. However, I do think it's useful to call this out specifically for trip updates. The reason for the question here is that stop_time_updates ARE required to be ordered within a trip_update. Because trips have an order in a block in GTFS, it's reasonable to ask if the ordering of stop_time_updates in GTFS-realtime extends beyond the trip_update to the entire block (i.e., multiple trip_updates).

Given the vote for this passed, I'll go ahead and merge and then work on another proposal for this.

[barbeau]

The vote on this issue is closed with the following results:

• Yes - 3
• No - 0
• Abstain - 0

So it passes! Thanks all!